Image display method suitable for video conferencing system with wide-angle network camera

Abstract

The present invention provides an image display method, which is applicable to a video conferencing system with a wide-angle network camera, wherein the wide-angle network camera is used to capture a panoramic image, wherein the aspect ratio of the panoramic image is greater than or equal to 2:1, and the method comprises: based on the panoramic image, framing a plurality of interest regions, wherein each interest region has at least one of a plurality of attributes; based on whether a portion of the panoramic image is filled in and the attributes, positions and quantities of the plurality of interest regions, selecting one of a plurality of preset screen layouts as an output screen layout; and, based on the attributes of the plurality of interest regions, filling the portion of the panoramic image and at least one of the plurality of interest regions into a corresponding window of the output screen layout to form a combined picture frame.

Description

Technical Field

[0001] The invention discloses an image display method, and in particular relates to an image display method suitable for a video conference system with a wide-angle network camera. Background Art

[0002] Owl Labs has disclosed a method for human detection in the patent document of US Patent Certificate No. US 10,636154B2, and Ricoh has disclosed a method for sound detection in the patent document of Japanese Patent Certificate No. JP 4908543B2. Because existing video conferencing software (such as ZOOM) can only display 4:3 or 16:9 images, wide-angle network cameras with a horizontal angular field of view (HFOV) greater than or equal to 180 degrees cannot transmit the entire elongated panoramic image to the video conferencing software. Even if there is an image transmitted, the characters in the picture displayed by the video conferencing software will become very small and difficult to identify. In order to solve the above problems, the present invention is proposed. Summary of the invention

[0003] In view of this, how to alleviate or eliminate the deficiencies in the above-mentioned related fields is indeed a problem to be solved.

[0004] According to an embodiment of the present invention, there is provided an image display method, which is applicable to a video conferencing system with a wide-angle network camera, wherein the wide-angle network camera is used to capture a panoramic image, wherein the aspect ratio of the panoramic image is greater than or equal to 2:1, and the method comprises: based on the panoramic image, framing a plurality of regions of interest, wherein each region of interest has at least one of a plurality of attributes; based on whether a portion of the panoramic image is filled in and the attributes, positions and quantities of the regions of interest, selecting one of a plurality of preset screen layouts as an output screen layout; and, based on the attributes of the regions of interest, filling a portion of the panoramic image and at least one of the regions of interest into a corresponding window of the output screen layout to form a combined frame.

[0005] One of the advantages of the above embodiment is that it selects sub-scene data or regions of interest (ROI) from the panoramic image with a horizontal viewing angle greater than or equal to 180 degrees for combination according to user preferences, whether there is a speaker, and the number and distribution of attendees, just like a director discarding unimportant image fragments, so that the combined image frame finally seen by the attendees is like attending a meeting in person, without missing any key points, and the key ROI can be emphasized at any time according to the user's preferences. In addition, when the aspect ratio of the panoramic image obtained from different network cameras is different from the aspect ratio of the screen presented by general video software, it can only simply fill the black edges at the top and bottom of the screen, crop the left and right sides, or deform the screen. In comparison, the image display method of the present invention not only significantly improves the utilization rate of the screen, but also greatly improves the aesthetics of the screen.

[0006] Other advantages of the present invention will be explained in more detail with reference to the following description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0008] Figure 1 FIG. 4 is a schematic diagram showing a video conferencing system according to an embodiment of the present invention.

[0009] FIG. 2A to FIG. 2B A method for displaying an image is shown according to an embodiment of the present invention.

[0010] FIG. 3A to FIG. 3F The figure shows the process of selecting all ROIs from the panoramic image to the combined frame according to an embodiment of the present invention.

[0011] FIG. 4A to FIG. 4B An example of displaying a difference image and clustering multiple difference regions is shown.

[0012] Figure 5 The conditions and methods for merging ROIs of the present invention are shown.

[0013] FIG. 6A to FIG. 6G To display different preset screen layouts of the present invention.

[0014] 7A to 7E A schematic diagram showing how to fill ROIs and panoramic images with different aspect ratios into corresponding windows with different aspect ratios in an output screen layout according to the present invention.

[0015] FIG. 8A to FIG. 8C According to the present invention, when the output screen layout changes, different transition effects are displayed.

[0016] Figure Number:

[0017] 100: Video conferencing system

[0018] 110a, 110b: Wide-angle camera

[0019] 120a, 120b: broadcasting device

[0020] 130a, 130b: Communication device

[0021] 301: ROI positioning based on human shape / face detection

[0022] 302: Manually selected ROI

[0023] 303: Tracking ROI

[0024] 304: ROI in Speech

[0025] 305: Identified ROI

[0026] 306: Merged ROI

[0027] 310: Panoramic image

[0028] 320: Combination frame DETAILED DESCRIPTION

[0029] The following description is a preferred implementation of the invention, which is intended to describe the basic spirit of the invention, but is not intended to limit the invention. The actual content of the invention must refer to the scope of the following claims.

[0030] It must be understood that the words "comprise", "include", etc. used in this specification are used to indicate the existence of specific technical features, values, method steps, operation processing, elements and / or components, but do not exclude the addition of more technical features, values, method steps, operation processing, elements, components, or any combination of the above.

[0031] The words “first”, “second”, “third”, etc. used in the claims are used to modify the elements in the claims. They are not used to indicate a priority order, a preceding relationship, or that one element precedes another element, or a temporal sequence in performing method steps. They are only used to distinguish elements with the same name.

[0032] It should be understood that when an element is described as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, and there may be intervening elements. Conversely, when an element is described as being "directly connected" or "directly coupled" to another element, there are no intervening elements. Other words used to describe the relationship between elements may be interpreted in a similar manner, such as "between" versus "directly between", or "adjacent" versus "directly adjacent", etc.

[0033] Figure 1 The schematic diagram of a video conferencing system is shown according to an embodiment of the present invention. The video conferencing system 100 includes two wide-angle network cameras 110a and 110b, two director devices 120a and 120b, and at least two communication devices 130a-130b. The wide-angle network camera 110a installed in the conference room A shoots all the participants, generates a panoramic image A, and transmits it to the director device 120a through the network or USB video device class (UVC), while the wide-angle network camera 110b installed in the conference room B shoots all the participants, generates a panoramic image B, and transmits it to the director device 120b through the network or UVC. Among them, the wide-angle network cameras 110a-110b can be wide-angle cameras with a horizontal viewing angle greater than or equal to 180 degrees, and can generate panoramic images A-B with a horizontal viewing angle greater than or equal to 180 degrees. However, the panoramic image defined in the present invention is not limited to a horizontal viewing angle greater than or equal to 180 degrees, but includes all images with an aspect ratio greater than or equal to 2:1. It should be noted that the aspect ratio refers to the relationship between the width and height of an image, and does not represent the physical size of the image or the size in pixels. Specifically, the aspect ratio of the panoramic image defined in the present invention mainly includes common formats such as 360X180, 180X60, 360X60, 180X90, etc., but is not limited thereto.

[0034] In one embodiment, each broadcasting device is implemented using an independent processor and a storage medium (not shown). Figure 1 The broadcast director 120b is independent of the communication device 130b and has an independent processor and a storage medium (not shown). In another embodiment, each broadcast director is implemented using the processor and storage medium (not shown) of the corresponding communication device, for example Figure 1 The communication device 130a includes the broadcast director 120a. The storage medium of each of the broadcast director devices stores a plurality of instructions for execution by the corresponding processor: FIG. 2A to FIG. 2B All the steps in the method. FIG. 2A to FIG. 2BThe method will be described later. The communication devices 130a-130b include, but are not limited to, personal computers, tablet computers, smart phones, etc. The communication device 130a transmits the combination frame A output by the director device 120a to the communication device 130b by executing a first video software (please refer to the relevant description of step S220), and receives and displays the combination frame B sent by the communication device 130b through the network; similarly, the communication device 130b transmits the combination frame B output by the director device 120b to the communication device 130a by executing a second video software, and receives and displays the combination frame A sent by the communication device 130a through the network. Figure 1 As shown, the communication device 130b is set in conference room B, and the communication device 130a is set in conference room A, so that people in the two conference rooms can see the participants in the other conference room through the screens of the communication devices 130a-130b.

[0035] FIG. 2A to FIG. 2B An image display method according to an embodiment of the present invention is illustrated. FIG. 3A to FIG. 3F The drawing shows the process of selecting all ROIs of the combined frame after the panoramic image transmitted from the wide-angle network cameras 110a and 110b to the broadcasting devices 120a and 120b is processed according to an embodiment of the present invention. Figure 1 , Figure 2A , Figure 2B , FIG. 3A to FIG. 3F , the image display method of the present invention is described in detail.

[0036] Step S202: Receive a panoramic image 310 from a wide-angle network camera, wherein the horizontal viewing angle of the panoramic image 310 is greater than or equal to 180 degrees. Figure 3A An example of a panoramic image 310 captured by wide-angle network cameras (110a, 110b). In one embodiment, each wide-angle network camera (110a, 110b) includes at least two lenses and a microphone array.

[0037] Step S204: Detect or select a region of interest (ROI) in the panoramic image 310 by using one or a combination of human shape detection, face detection, and user manual selection. Figure 3BAs shown, according to the present invention, in order to locate the position of a person in a panoramic image 310, human shape detection or face detection is used to locate ROI 301. Alternatively, human shape detection can be used to locate the approximate position of the person first, and then face detection can be used to accurately locate ROI 301. Human shape detection and face detection include, but are not limited to, the following two methods: (1) Deep learning, which is used to perform artificial intelligence detection (AI detection), such as using the programming language Python with the computer vision library OpenCV, the open source neural network library Keras, and the open source database TensorFlow, using OpenCV to capture image information for image processing, using the neural network module provided by Keras with TensorFlow as the backend for model training; (2) Machine learning, which can be implemented using Hidden Markov Models (HMM) and Support Vector Machines (SVM). Basically, traditional machine learning requires developers to decide what features to use (for example, turning a picture of a person into a gradient array), and then pass it to a learning module (such as SVM or HMM) for training and testing. Deep learning, on the other hand, allows the module itself to determine the feature points (or it can be said that the extraction of feature points is already included in the module), so the image can be directly passed to a deep learning module (such as a multilayer perceptron (MLP), a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), etc.) for training and testing. The techniques of deep learning and machine learning are well known to those skilled in the art, so they will not be elaborated here. Please also note that the above-mentioned deep learning and machine learning are only examples, not limitations of the present invention, and other existing or future developed human figure / face detection methods may also be applicable to the concepts of the present invention.

[0038] The definitions of the terms used throughout this specification and in the subsequent claims are as follows, unless otherwise specified in this specification. The term "selecting an ROI" means using a selection box to define the effective range of an ROI (i.e., a scene) in a panoramic image. Please note that FIG. 3B to FIG. 3EThe rectangular selection box in is only an example and not a limitation of the present invention. Selection boxes of other shapes are also applicable to the concept of the present invention. The term "priority" refers to the order in which a ROI is filled into a corresponding window of an output screen layout. For example, the higher the priority, the earlier the ROI is filled into the corresponding window. For example, the priority ranges from 1 to 5, where 1 represents the lowest priority and 5 represents the highest priority; in another example, the priority ranges from A to C, where C represents the lowest priority and A represents the highest priority.

[0039] like Figure 3B As shown, the ROI 302 of the leftmost whiteboard cannot be automatically detected by the aforementioned human figure detection or face detection method, but can be manually selected by the user. After any ROI 301 and ROI 302 are selected, the selection box can be dragged, enlarged, and reduced. The number of selection boxes can also be increased or decreased after ROI 301 and ROI 302 are selected, and the ROIs 301-302 can even be adjusted to the desired position and range. Please note that in addition to being manually selected by the user, the ROI 302 of the leftmost whiteboard can also be detected by any known method to detect the ROI of the whiteboard, screen, or TV. In addition, the properties of the ROI 301-302 selected in this step are all defined as "static", that is, the size and position of the ROI 301-302 will not change with the movement of the person in the box.

[0040] Step S206: Select the ROI for object tracking according to the user's needs. The user mainly selects the ROI for object tracking from the static ROI selected in step S204. FIG. 3B to FIG. 3C For example, the user selects one of the ROIs 301 to ROI 302 selected in step S204 as the ROI 303 for object tracking. Please note that this step defines the attribute of the ROI 303 as "dynamic" (or "tracking"), that is, the position of the ROI 303 moves with the movement of the person in the frame.

[0041] This step uses the following four tracking methods to perform object tracking (such as tracking Figure 3C(a person in ROI 303): (1) Feature extraction method. Feature extraction: input ROI 303 => corner detection => output multiple corner points. Tracking: input feature points => use "optical flow / mean shift / motion estimation" to calculate the motion vector of each point => determine the motion vector of the group => determine the motion vector of ROI 303. (2) Human pose estimation method: use any existing and future developed human pose estimation method. For example: define the human body structure, mainly the main joints and their connections of the human body; for example, the Part Intensity Field (PIF) can be used to find each human body key point (main joint) and the Part Association Field (PAF) can be used to connect each key point into a complete human body model => put the human body structure into ROI 303 => align the human body structure with the human posture in ROI 303. (3) Foreground detection method (using the foreground to move the ROI 303): Smoothing a frame => calculating a difference image between the current smoothed image and the previous smoothed image (e.g. Figure 4A ), since the object moves gradually, the only obvious difference is at the "edge of the object", and the difference in the middle is not obvious because of the same material color, which makes the moving object look like a contour composed of "lines"; this difference image is grayscale and has different depths, and there is no clear connection between points => In order to more clearly obtain the range of the object, the difference image is binarized => Grouping multiple difference areas, such as dilation and erosion processing on the binary difference image (the result is shown in Figure 4B )=>If the clustered difference regions overlap with ROI 303, move ROI 303 to cover the difference regions. (4) Use scale-invariant feature transform (SIFT) or histogram of oriented gradient (HOG) to perform feature comparison and tracking. Since the above four detection and tracking methods are well known to those skilled in the art, they will not be described in detail here. Please also note that the above four detection and tracking methods are only examples and not limitations of the present invention. Other detection and tracking methods that exist or will be developed in the future may also be applicable to the concepts of the present invention.

[0042] Step S208: Detect the ROI in the speaker. This step uses the following two methods to detect whether someone is speaking from the three ROIs 301 to 303: (1) Using a microphone array to calculate beamforming to estimate the direction of arrival (DOA) of the sound source, and then generate a possible sound source direction. The sound source direction estimation method includes, but is not limited to, the maximum likelihood (ML) method, the time delay estimation (TDE) method, and the eigenstructure method, etc. (2) Using a microphone array to select the larger sound source. Taking a microphone array including four microphones (the sound source directions are 0 degrees, 90 degrees, 180 degrees and 270 degrees respectively) as an example, assuming that the energy received at the sound source directions of 90 degrees and 180 degrees is relatively large, 50 decibels and 60 decibels respectively, the two microphones with the largest energy are selected for interpolation to determine the angle. After converting decibels into energy, it is equal to that the sound energy of 180 degrees is 10 times that of 90 degrees, so the possible sound source direction interpolated will be (180*10+90*1) / 11≈172 degrees. In actual implementation, in order to facilitate the subsequent comparison of horizontal viewing angle and sound source direction, the lens and microphone array of the wide-angle network camera (110a, 110b) must be properly set up so that the horizontal viewing angle of the generated panoramic image 0 to 360 degrees just matches the sound source direction of the microphone 0 to 360 degrees, for example, the horizontal viewing angle of the panoramic image 0 degrees matches the sound source direction of the microphone 0 degrees, the horizontal viewing angle of the panoramic image 180 degrees matches the sound source direction of the microphone 180 degrees, and so on.

[0043] In this step, when detecting whether there are participants speaking in the conference room, it is assumed that the above three ROIs 301-303 have been selected, so the horizontal viewing angles of ROIs 301-303 are all known. When the microphone array detects someone speaking and calculates a possible sound source direction, if the horizontal viewing angle θ of any ROI 301-303 is subtracted from the possible sound source direction θ S The angle difference (θ-θ S ) is less than a critical value θ T1 When the angle difference is less than the critical value θ T1 The ROI of is regarded as the ROI in the current speech, such as Figure 3D Assume that two ROIs 301-303 are both close to the possible sound source direction and the angle difference is less than the critical value θ T1 When the angle difference is the smallest (ie, the closest) ROI, the ROI is selected as the currently speaking ROI 304. Please note that the attribute of the ROI 304 selected in this step is defined as "speaking".

[0044] Step S210: Perform face recognition from the ROI 301 of human shape detection and / or face detection, and automatically mark the name. The face recognition method used in the present invention includes but is not limited to using scale-invariant feature transform (SIFT) image feature points, accelerated robust features (SURF) image feature points, local binary pattern histogram (LBPH) as image feature points for recognition, and Eigenface based on principal component analysis (PCA) data analysis and Fisherface using linear discriminant analysis (LDA) to allow the system to learn from known samples and find classification methods according to different theories and methods to perform face recognition. Alternatively, deep learning modules (such as multi-layer perceptron (MLP), deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), etc.) can also be used for face recognition. This step can be to use a pre-trained model to perform face recognition on known samples, or it can be used with web crawler technology to perform face recognition on celebrities simultaneously. Figure 3E Please note that the attribute of ROI 305 selected in this step is defined as "identified". Figure 2A Shown in dashed lines.

[0045] Step S212: When the given conditions are met, merge the ROIs. When the image display method of the present invention reaches this step, the following data has been generated: (1) panoramic image 310; (2) ROIs of different attributes: for example, "static" ROIs 301, 302, "dynamic" ROI 303, "speaking" ROI 304, and "recognized" ROI 305. Figure 5 The conditions and method for merging ROIs of the present invention are shown. To avoid the appearance of some repeated characters / objects when the above ROIs are placed in each window of the output screen layout, if there are at least two ROIs that meet any of the following two conditions, this step must be merged: (i) ROIs with similar central point viewing angles: e.g. Figure 5 The horizontal viewing angle difference (θ1-θ2) between the center points of any two regions of interest ROI-1 and ROI-2 (i.e., the above ROIs 301-305) is less than a preset angle θ T2 When , ROI-1 and ROI-2 are merged, for example, θT2 =10 O (ii) ROIs with overlapping areas / pixels: Any two ROI-1 and ROI-2 (i.e., the above ROIs 301 to 305) that have overlapping areas or pixels are merged. As for the merging method, the union areas of all the ROIs to be merged are taken from top to bottom, left to right, and vice versa. For example, Figure 5 The dashed rectangular frame ROI 306 in FIG. 3 merges ROI-1 and ROI-2. It should be particularly noted that each ROI may correspond to more than one attribute, and in the subsequent steps, one of the attributes of the merged ROI 306 may be set to "merged", "having multiple persons", or "number of persons" (e.g., 2 persons). In addition, please note that, whether it is the merged ROI 306 or the ROI-1 and ROI-2 before merging, in the subsequent step S216, they will be considered whether to fill in a corresponding window of an output screen layout to generate a combined frame 320.

[0046] Step S214: Select one of a plurality of preset layouts as an output layout according to whether to fill in a portion of the panoramic image and the attributes, positions and quantities of the ROIs (301-306). The preset layouts of the present invention include, but are not limited to, layout AB (such as FIG. 6A to FIG. 6B ), focus layout (such as Figure 6C ), grid layout (such as Fig.6D ), picture-in-picture layout (such as Fig. 6E ), speaker layout (such as Fig. 6F ) and top-down layout (such as Figure 6G ).

[0047] 6A to 6F Display different preset screen layouts of the present invention. ROIs with different attributes and quantities are suitable for different preset screen layouts, and different preset screen layouts have different presentation methods to suit different meeting numbers, personnel distribution or scenes. Please note that each preset screen layout contains one or more windows of different or same sizes and aspect ratios. For example: Fig. 6A The layout A contains three windows of different sizes and aspect ratios, and Figure 6G A top-down layout consists of two windows of the same size and aspect ratio.

[0048] FIG. 6A to FIG. 6BTwo types of layouts AB are displayed, which are suitable for situations where there are many ROIs and more than one key person. The layouts AB present a panoramic image in a banner above the frame (all participants can be seen), and use more than two windows below the frame to present multiple key persons, and the size and aspect ratio of each window may not be the same; it should be specially noted that the layout AB is not limited to the complete presentation of the panoramic image above the frame, but can only present a part of the panoramic image, as long as all participants can be roughly seen. In one embodiment, the user can also freely adjust the part to be presented above the layout AB frame. In one embodiment, up to 8 key persons can be presented in multiple split windows below the frame. Figure 6C The display focus layout is suitable for multiple ROIs that are close to each other and only occupy a part of the entire 360 ​​degrees. For example, this layout can frame all the ROIs that are close to each other into a large window, or frame all the ROIs that speak within a specific period of time into a large window. Fig.6D The grid layout is suitable for scenes with a large number of important (highest priority and / or second highest priority) ROIs that are scattered, or scenes with a small number of unimportant ROIs that do not need to be presented with a panoramic image. This layout presentation method presents the grouped ROIs in a neat multi-grid window, not necessarily four grids, nor an even number of grids. Fig.6D In the example, four windows display four participants. When a new speaker joins, he / she will directly replace the participant who has not spoken for the longest time.

[0049] Fig. 6E The picture-in-picture layout is suitable for a scene with a relatively large ROI 302 (such as a whiteboard, a screen, or a TV) and participants. The layout presentation method is to place the ROI of the whiteboard, the screen, the TV, or the speaker in the large window, and the ROI of the lecturer or the questioner in the small window. Fig. 6F The display speaker layout is suitable for a scene with many participants and only one main speaker. This layout presentation method is to place a panoramic image at the bottom of the picture frame and place the speaker's ROI 304 at the top of the picture frame. If someone asks a question and the speaker may not speak for a while, the questioner's ROI will be placed in the large window above the picture frame. Figure 6GThe top-down layout is suitable for situations where no ROI needs to be specially displayed. This layout mainly allows the wide-angle panoramic image to be filled into the upper and lower windows of the combined picture frame 320 in a half-upper and half-lower manner. It is worth noting that in order to make the combined picture frame 320 meet the aspect ratio required by the video software when displaying the screen, part of the upper and lower edges of the panoramic image may be cropped during the filling process. For example, assuming that the aspect ratio of the wide-angle panoramic image is 6:1 and the aspect ratio required by the video software when displaying the screen is 16:9 (ratio approximately equal to 1.78), if the upper and lower windows of the combined picture frame each display half of the panoramic image, the aspect ratio will be equal to 3:2 (ratio approximately equal to 1.5). Under the premise that the width of the half panoramic images in the upper and lower windows does not change, and the upper and lower edges are evenly cropped to fill the upper and lower windows, the upper and lower edges of the upper and lower halves of the panoramic image will each be cut off by about 7.8%.

[0050] In this step, one of the preset screen layouts is selected as an output screen layout according to whether to fill in a part of the panoramic image and at least one of the attributes, positions and quantities of the above-mentioned ROIs (301-306). For example, if a part of the panoramic image is to be filled in, the layout AB, the speaker layout or the top-down layout can be selected; if a large ROI (whiteboard or screen) is needed for explanation (as an explanation screen), the picture-in-picture layout can be selected; if there is only one speaking ROI (i.e., a single speaker), the speaker layout can be selected; if there are multiple speaking ROIs (i.e., multiple people discussing), the layout AB, the grid layout or the focus layout can be selected; if the positions of multiple ROIs in the panoramic image are relatively close, the focus layout can be selected; if the positions of multiple ROIs in the panoramic image are relatively scattered, the grid layout can be selected.

[0051] Step S216: According to the properties of the ROIs (301-306), the ROIs or / and at least a portion of the panoramic image are filled into the corresponding windows of the output screen layout to generate a combined frame 320. It should be noted that the aspect ratio of each ROI (301-306) and the panoramic image may not be equal to the aspect ratio of its corresponding window, so the aspect ratio of the ROI and the at least a portion of the panoramic image must be adjusted first. In one embodiment, except that the layout AB, the speaker layout and the top-down layout need to place the panoramic image into the preset corresponding window first, each ROI is filled into the corresponding window of the output screen layout according to its own properties. For example, if each ROI corresponds to only one attribute, among all the attributes of the ROI (301-306), the priority of the "speaking" attribute may be set to be the highest, the priority of the "merging", "having multiple persons", or "the number of persons is more than 1" attributes may be set to be the second, the priority of the "dynamic" attribute may be set to be the third, and the priority of the "static" and "recognized" attributes may be set to be the lowest. Considering that the space of the frame and the number of windows are limited, in principle, each output screen layout will give priority to displaying the "speaking" ROI with the highest priority, and if there are remaining windows, then display the "merging", "having multiple persons", or "the number of persons is more than 1" ROI with the second highest priority, and if there are remaining windows, then display the "dynamic" ROI with the second highest priority, and finally, if there are remaining windows, then display the "static" and "recognized" ROIs with the lowest priority, and conversely, if there are no extra windows / space, first abandon the "static" and "recognized" ROIs. In another embodiment, each ROI corresponds to more than one attribute and different attributes have different priorities. For example, the "pinning" attribute has priority A, the "merging", "having multiple persons", or "more than 1 person" attributes have priority B, and the remaining attributes have priority C. However, among the attributes with the same priority C, different priorities can be further subdivided. For example, the "speaking" attribute has the highest priority, which is set to C3, the "dynamic" attribute has the second highest priority, which is set to C2; the "static" and "identified" attributes have the lowest priority, which is set to C1; in other words, the ROI including the "pinning" attribute, regardless of whether it contains other attributes, must have priority A. For the ROI that does not have the "pinning" attribute but includes the "merging", "having multiple persons", or "more than 1 person" attributes, it must have priority B. As for the ROI that only includes the remaining attributes, it has priority C.

[0052] According to the present invention, the "pin" attribute is directly given the highest priority. In other words, any ROI with the "pin" attribute will be selected and filled into the corresponding window of each output screen layout. Moreover, the "pin" attribute is an attribute manually selected by the user, and the user can decide to pin or unpin. It should be noted that the number of ROIs with the "pin" attribute will be different depending on the nature of each output screen layout (e.g., a panoramic image needs to be placed, a narration screen needs to be included, a single speaker or a multi-person discussion). For example, if only four ROIs with the "pin" attribute can be filled into an output screen layout, when the user pins the fifth ROI, the first (oldest) pinned ROI will be immediately unpinned. On the other hand, each output screen layout can choose to fill / display four ROIs with priorities of 1A1B2C (i.e., including one ROI with priority A, one ROI with priority B, and two ROIs with priority C) according to its nature and user preference. Since the number of ROIs with priority A or B is necessarily small, there is no need to further subdivide ROIs with priority A, and the same applies to ROIs with priority B. As for ROIs with priority C, since there are more types and numbers, they must be filled in the corresponding windows of each output screen layout in order according to the priority level.

[0053] In one embodiment, each ROI corresponds to more than one attribute and different attributes have different priorities. Each output screen layout sorts the ROIs according to the priority level to generate a sorted region of interest string, and then fills the corresponding window in order. In the sorting process of comparing the priorities of multiple ROIs, the highest priority of all the attributes of each ROI is taken for comparison. If the highest priority between two or more ROIs is the same, the second highest priority is compared, and so on, until the priorities of all ROIs are sorted out (hereinafter referred to as the one-by-one comparison method). For example: ROI-1 has four attributes, and their priorities are (5, 4, 2, 1) respectively, and ROI-2 has three attributes, and their priorities are (5, 4, 3) respectively. Because the first two priorities of ROI-1 and ROI-2 are the same, but the third priority is different, ROI-2 takes precedence over ROI-1. For example, ROI-3 has four attributes with priorities (5, 4, 2, 1), and ROI-4 has three attributes with priorities (5, 4, 2). Since the first three priorities of ROI-3 and ROI-4 are the same, when we continue to compare, ROI-3 has a fourth priority but ROI-4 does not, so ROI-3 takes precedence over ROI-4. This method of taking the highest priority also applies to the case where a ROI contains multiple ROIs.

[0054] In another embodiment, similarly, each ROI corresponds to more than one attribute and different attributes have different priorities, but each ROI needs to first calculate the sum of all attribute priorities of itself, that is, the summed priority, and then sort the ROIs according to the summed priority of each ROI. In the process of comparing the summed priorities of multiple ROIs, if there are two or more ROIs with the same summed priority, the one-by-one comparison method is used again, starting with the highest priority, and then comparing the second highest priority, and so on, until the priorities of all ROIs are separated to generate a sorted region of interest string (hereinafter referred to as the summed and one-by-one comparison method). Finally, the ROIs are filled into the corresponding window in order according to the priority.

[0055] In another embodiment, each ROI corresponds to more than one attribute and all attributes are divided into multiple levels, and the attributes of the same level are further divided into multiple categories, for example, all attributes are divided into three levels of X, Y, and Z, and the level priorities are 10, 5, and 1 respectively. In addition, the level of attribute Y is further divided into two categories Y1 and Y2, and the priorities are 6 and 7 respectively. Each output screen layout first sorts the ROIs according to the level priority. In the sorting process of comparing the level priorities of multiple ROIs, the highest level priority of all attributes of each ROI is taken for comparison. If the highest level priority between two or more ROIs is the same, the category priorities of the attributes of the same level are compared (for example, Y1 and Y2 are compared) by the one-by-one comparison method or the summing and one-by-one comparison method. If the category priorities of the attributes of the same level are also the same, the level priorities of the attributes of the next level are compared, and so on, until the priorities of all ROIs are sorted to generate a sorted region of interest string (hereinafter referred to as the level-category comparison method). Finally, the ROIs are filled into the corresponding window in order according to the level and category priorities.

[0056] In addition, according to the present invention, when any ROI is filled into a specific window, it will not affect the aesthetics of the output image presentation, and the ROI can be defined as having the priority S for the specific window. For example 1: when the size (scale) of the ROI is within a certain range (for example, between 0.5 and 1.5 times) of the size of the corresponding window and the image clarity will not be affected by enlarging or reducing the ROI too much, the ROI can be defined as having the priority S for the corresponding window; for example 2: when the aspect ratio of the ROI is different from the aspect ratio of the corresponding window and it is impossible to achieve the same aspect ratio as the corresponding window by expanding the selection box range of the ROI (that is, the edge of the panoramic image will be touched during the process of expanding the selection box range of the ROI, such as Figure 7C), the ROI does not have the priority S. Otherwise, the ROI can be defined to have the priority S for the corresponding window. In one embodiment, when all windows of an output screen layout select ROIs with priority S, a ROI with a most suitable size attribute and / or a most suitable aspect ratio attribute can be preferentially selected to fill in the corresponding window, so as to achieve the effect of uniform resolution or uniform size of the participants displayed in the screen, thereby beautifying the output screen layout. In another embodiment, assuming that one of the windows of an output screen layout needs to select a ROI with priority S, first, any of the above three methods (one-by-one comparison method, summing and one-by-one comparison method, and class comparison method) must be used to compare ROI-a with priority A and ROI-b with priority B. At this time, if the difference in priority between the two ROIs is less than a critical value TH, the ROI with priority S will be filled in the window before the ROI without priority S. Accordingly, the flexibility of the critical value TH can be retained. In other words, the larger the critical value TH is, the more important the user thinks the priority S is.

[0057] 7A to 7E A schematic diagram is shown for illustrating how to fill ROIs and panoramic images of different aspect ratios into corresponding windows of different aspect ratios in the output screen layout according to the present invention. There are four methods for filling ROIs and panoramic images of different aspect ratios into corresponding windows of different aspect ratios in the output screen layout as follows. Method 1: On the panoramic image, if the selection box range of the ROI can be expanded, then the selection box range is expanded left, right, up, and down so that its aspect ratio matches the ratio of the corresponding window, for example Fig. 7A , the original ROI-a has a ratio of 3:4. After the selection box is expanded left and right, it conforms to the aspect ratio of the corresponding window of 16:9; for example Figure 7B , the original ROI-b ratio is 6:1. After expanding the selection box up and down, it meets the aspect ratio of the corresponding window 6:3. Method 2: On the panoramic image, expand the selection box range if it can be expanded. If it cannot be expanded at all, then cut out the redundant image in the ROI according to the aspect ratio of the corresponding window, for example Fig.7D Method 3: When the selection box of the target area of ​​interest hits the edge of the panoramic image and cannot be expanded any further, the following two methods are adopted: Method (a): Push the selection box back from the edge of the panoramic image to match the aspect ratio of the corresponding window; Method (b): Fill one side of the selection box with a black border to match the aspect ratio of the corresponding window. For example Figure 7CFor example, the original ROI-c has a ratio of 6:1. After expanding the selection box up and down, it can only be expanded to an aspect ratio of 6:3, which is still some distance from the target value of 6:4. At this time, method (a) can be used to push the selection box back from the edge of the panoramic image to the bottom so that the selection box range conforms to the aspect ratio of 6:4 of the corresponding window. As a result, the original ROI is located slightly above the corresponding window (as shown in the lower right figure), but the content of the panoramic image can be fully adopted, which is more natural. Another option is to use method (b) to add a black border above the selection box (6:3) that can no longer be expanded to conform to the aspect ratio of 6:4 of the corresponding window. As a result, the original ROI can be located in the middle of the corresponding window (as shown in the lower left figure). Method 4: On the panoramic image, if it is impossible to expand the selection box range and crop the excess image in the ROI, the image in the ROI can be directly deformed according to the aspect ratio of the corresponding window, such as Fig. 7E The circle in the picture changes into an ellipse.

[0058] Step S220: Send the combined image frame 320 to a video software for display. Figure 1 In the embodiment, the communication device 130a transmits the combined picture frame A output by the broadcasting director 120a to the communication device 130b by executing a video software A, and receives and displays the combined picture frame B transmitted by the communication device 130b through the network. The combined picture frame 320 transmitted to the video software conforms to the aspect ratio or / and pixel number specified / commonly used by the existing general video software. The video software includes various video conferencing software and various webcam application software, and the various video conferencing software includes but is not limited to ZOOMcloud meeting, Skype, Tencent Meeting, Cisco Webex Meeting, Line, V-cube Meeting, Google Meet, etc. After the above-mentioned combined picture frame 320 is output to the application software of various webcams, it can be used for live broadcast, making meeting records, editing and sharing, etc.

[0059] Please note that for ease of explanation, the above Figures 6A-6G The communication devices 130a-130b display the same screen layout. In actual implementation, the image display method of the present invention executed on the director devices 120a-120b can select different output screen layouts according to the properties and quantities of different ROIs selected / detected on each panoramic image A-B to generate two different combined image frames 320a-b, and then transmit them to the communication devices 130a-130b through the network for display. For example, Figure 1 In FIG. 1 , the communication device 130 b displays the editing layout A, while the communication device 130 a displays the presenter layout.

[0060] In addition, please note that the entire process of the meeting is dynamic, the panoramic image will continue to change, the properties and number of the selected ROI will also change with the content of the panoramic image, and then the output screen layout will change accordingly. For example, from the meeting room being empty to the meeting room being full (at this time, the top-down layout may be selected), to the start of the meeting, to someone giving a speech (at this time, the speaker layout may be selected), to someone asking a question (at this time, the picture-in-picture layout may be selected), to someone moving, to someone leaving, to someone joining (at this time, the grid layout may be selected), to a few people gathering for discussion (the focus layout may be selected), to the end of the meeting.

[0061] FIG. 8A to FIG. 8C According to the present invention, when the output screen layout changes, different transition effects are displayed. When the output screen layout changes from A to B, some transition effects are required, such as Fig. 8A As shown in the figure, the output screen layout A originally displays ROI-X. After the transition, the output screen layout B wants to display ROI-Y. The thick dashed frame indicates the display range of the original ROI-X in the output screen layout A. If the display range is enlarged by 1.2 times (equal to 10% on both sides) (thin dashed frame), it will hit the target ROI-Y. You can directly move the lens (pan) from left to right on the panoramic image to the target ROI-Y. Otherwise, it means that the transition distance is too far, and you can directly switch between the two ROI-X and ROI-Y to make the transition, such as Figure 8B As shown. If the transition distance is too far, if the pan speed is fixed, it may take too long. If the transition time is fixed, the pan speed may be too fast. These are unacceptable to consumers, so ROI-X and ROI-Y are directly switched for transition. Taking the grid layout to the presenter layout as an example, slowly enlarge the proportion of one of the grid layout windows to the size of the presenter window of the presenter layout, such as Figure 8C shown.

[0062] In the output screen layout, you can choose different projection methods to present the screen according to the different viewing areas of the ROI / selection box. The preset projection methods include, but are not limited to, equirectangular projection, cylinder projection, perspective projection, Panini projection, and fisheye projection.

[0063] The above are only preferred embodiments of the present invention and are not intended to limit the scope of the present invention; any other equivalent changes or modifications that do not deviate from the spirit disclosed by the present invention should be included in the scope of the following patent applications.

Claims

1. An image display method, characterized in that: Applicable to a video conferencing system having a wide-angle webcam, the wide-angle webcam being used to capture a panoramic image, the aspect ratio of the panoramic image being greater than or equal to 2:1, the method comprising: Frame selection step: Frame multiple regions of interest according to the panoramic image, wherein each region of interest has at least one of multiple attributes; A selection step: selecting one of a plurality of preset screen layouts as an output screen layout according to whether a portion of the panoramic image and at least one of the attributes, positions and quantities of the plurality of regions of interest are filled in; as well as Filling step: filling the portion of the panoramic image and at least one of the plurality of regions of interest into at least one corresponding window of the output picture layout according to the attributes of the plurality of regions of interest to form a combined picture frame; Wherein, the filling step comprises: Adjusting step: adjusting the aspect ratio of a portion of the panoramic image and the plurality of regions of interest to match the aspect ratio of a corresponding window; Wherein, the adjustment step comprises: In the panoramic image, expanding a selection frame range of a target region of interest; When the range of the selection frame of the target region of interest cannot be expanded, cutting out redundant images in the target region of interest according to the aspect ratio of the corresponding window; and When it is impossible to expand the selection frame range of the target region of interest and it is impossible to cut out the redundant image in the target region of interest, the image in the target region of interest is deformed according to the aspect ratio of the corresponding window.

2. The image display method according to claim 1, characterized in that: Also includes: transmitting the combined picture frame to a video software so that the combined picture frame is displayed on the video software; The aspect ratio of the combined picture frame corresponds to the screen aspect ratio required by the video software for display.

3. The image display method according to claim 1, characterized in that: The attributes include: static, dynamic, recognized, speaking, merged, with multiple people, number of people, pinned, size, and aspect ratio.

4. The image display method according to claim 3, characterized in that: The frame selection step further includes: Performing sound source direction estimation on the multiple regions of interest to obtain a possible sound source direction; comparing the horizontal viewing angles of the multiple regions of interest and the angle differences between the possible sound source directions; as well as The region of interest with the smallest angle difference is defined as being in speech; The wide-angle network camera includes at least two lenses and a microphone array, and the lenses and the microphone array are properly set up so that the horizontal viewing angle of the panoramic image matches the sound source direction of the microphone array.

5. The image display method according to claim 3, characterized in that: The frame selection step includes: Selecting the plurality of regions of interest using one or a combination of human shape detection, face detection and user manual selection; and The properties of the plurality of regions of interest are defined as static.

6. The image display method according to claim 5, characterized in that: The frame selection step further includes: Selecting at least one region of interest from the static regions of interest for object tracking; and The attribute of the at least one region of interest for object tracking is defined as dynamic.

7. The image display method according to claim 5, characterized in that: The frame selection step further includes: Performing face recognition on the plurality of regions of interest selected by human shape detection and face detection to mark the names of the persons; and Attributes of the plurality of regions of interest labeled with person names are defined as identified.

8. The image display method according to claim 3, characterized in that: Also includes: After the frame selection step and before the selection step, when the horizontal viewing angle difference between the center points of any two regions of interest is less than a preset angle, merging the upper, lower, left, and right joint regions of the two regions of interest; as well as The attributes of the two merged regions of interest are defined as at least one of merging, having a plurality of persons, and a number of persons.

9. The image display method according to claim 3, characterized in that: Also includes: After the frame selection step and before the selection step, when any two regions of interest overlap, merge the upper, lower, left, and right joint regions of the two regions of interest; as well as The attributes of the two merged regions of interest are defined as at least one of merging, having a plurality of persons, and a number of persons.

10. The image display method according to claim 1, characterized in that: The adjusting step further comprises: When the selection frame of the target region of interest hits the edge of the panoramic image and cannot be expanded any further during the process of expanding the selection frame, the selection frame is pushed back from the edge of the panoramic image to meet the aspect ratio of the corresponding window.

11. The image display method according to claim 1, characterized in that: The adjusting step further comprises: When the selection frame of the target region of interest hits the edge of the panoramic image and cannot be expanded any further during the process of expanding the selection frame, a black border is added to the corresponding edge of the expanded selection frame to meet the aspect ratio of the corresponding window.

12. The image display method according to claim 1, characterized in that: Also includes: When a current output screen layout is not equal to a previous output screen layout, enlarging a display range of a current region of interest to a preset multiple; If the enlarged display range of the current region of interest overlaps with a target region of interest, switching the current region of interest and the target region of interest by moving the lens; as well as If the enlarged display range of the current region of interest does not overlap with the target region of interest, the current region of interest and the target region of interest are directly switched on the current output screen layout.

13. The image display method according to claim 1, characterized in that: The filling step comprises: Determining whether to fill a portion of the panoramic image into a corresponding window of the output screen layout; Comparison step: comparing the priorities of the regions of interest to generate a sorted region of interest string; as well as The sorted ROI strings are filled into corresponding windows of the output screen layout.

14. The image display method according to claim 13, characterized in that: The comparing step further comprises: If the priority difference between two adjacent regions of interest in the sorted region of interest string is less than a critical value and the region of interest with a smaller priority among the two adjacent regions of interest contains a specific attribute, the positions of the two adjacent regions of interest in the sorted region of interest string are exchanged, wherein the specific attribute indicates that the region of interest with a smaller priority will not affect the aesthetics of the output screen presentation when filling the corresponding window.

15. The image display method according to claim 13, characterized in that: The comparison step comprises: (a1) respectively setting the attributes of the plurality of regions of interest to have different priorities; (a2) setting the representative priority of each region of interest to be equal to the highest priority among all its attributes; (a3) sorting the multiple regions of interest according to their representative priorities; (a4) when the representative priorities of at least two regions of interest are the same, setting the representative priorities of the at least two regions of interest to be equal to the next highest priority; as well as (a5) Repeat steps (a3) ​​to (a4) until the sorted region of interest string is generated.

16. The image display method according to claim 13, characterized in that: The comparison step comprises: (b1) respectively setting the attributes of the plurality of regions of interest to have different priorities; and (b2) summing up the priorities of all attributes of each region of interest to obtain a summed priority; as well as (b3) Sort the multiple regions of interest according to the total priority of the multiple regions of interest.

17. The image display method according to claim 16, characterized in that: The comparing step further comprises: (b4) when the total priority of at least two regions of interest are the same, the representative priority of each region of interest is set equal to the highest priority among all its attributes; (b5) sorting the at least two regions of interest according to representative priorities of the at least two regions of interest; (b6) when the representative priorities of the at least two regions of interest are the same, setting the representative priority of each region of interest equal to the next highest priority; as well as (b7) Repeat steps (b5) to (b6) until the sorted region of interest string is generated.

18. The image display method according to claim 13, characterized in that: The comparison step comprises: (c1) classifying the attributes of the plurality of regions of interest into a plurality of levels; (c2) dividing the attributes at the same level into multiple categories, wherein each level has a different level priority and each category has a different category priority; (c3) setting the priority level of each region of interest to be equal to the highest priority level among all its attributes; as well as (c4) Sort the multiple regions of interest according to their level priorities.

19. The image display method according to claim 18, characterized in that: The comparing step further comprises: (c5) when the level priorities of at least two regions of interest are the same, respectively setting the category priorities of the at least two regions of interest to be equal to the highest category priority among all attributes of the corresponding level; (c6) sorting the at least two regions of interest according to the category priorities of the at least two regions of interest; (c7) when the category priorities of the at least two regions of interest are the same, setting the level priority of each of the at least two regions of interest to be equal to the next highest level priority; and (c8) Repeat steps (c4) to (c7) until the sorted region of interest string is generated.

20. The image display method according to claim 18, characterized in that: The comparing step further comprises: (d1) when the level priorities of at least two regions of interest are the same, summing up the category priorities of all attributes in the corresponding levels of the at least two regions of interest to obtain a summed category priority; (d2) sorting the at least two regions of interest according to the summed category priorities of the at least two regions of interest; (d3) when the total category priorities of the at least two regions of interest are the same, setting the level priority of each of the at least two regions of interest to be equal to the next highest level priority; as well as (d4) Repeat steps (c4) and (d1) to (d3) until the sorted region of interest string is generated.

Images