Screen splitting method for video conference picture, and related device
By detecting and tracking target objects in the video conference screen, dynamically dividing the picture and adjusting the layout, the problem of difficult to identify speakers and poor interactivity in video conferences is solved, improving the user experience and real-time split screen and virtual background are realized.
Patent Information
- Application Number
- PCT/CN2024/095004
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-26
- Filing Date
- 2024-05-23
- Publication Date
- 2024-12-05
AI Technical Summary
In video conferencing, it is difficult to distinguish and identify the current speaker, and it is difficult to realize real-time split-screen adjustments in the prior art to improve interactivity.
By detecting the target object in the video conference screen, the screen is automatically divided into multiple sub-screens, and the split-screen layout is dynamically adjusted according to the number of target objects and the speaker detection results, and the target object is displayed in the corresponding sub-screen, using target detection and Tracking technology combines image processing and semantic segmentation to realize real-time split screen and virtual background functions.
Improve the interaction between participants in conference room in video conferences and online participants, enhance the user experience, solve the problem of difficult to identify speakers and distinguish current speakers, and realize real-time split-screen adjustments and virtual backgrounds filling.
Smart Images

Figure CN2024095004_05122024_PF_FP_ABST
Abstract
Description
Video conference screen splitting method and related equipment
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to Chinese invention patent application number 202310611376.5, entitled “Screen splitting method and related equipment for video conference screen” and filed on May 26, 2023, and the entire application is incorporated herein by reference. Technical Field
[0003] The present disclosure relates to the field of computer technology, and in particular to a method for splitting a video conference screen and related equipment. Background Art
[0004] Face-to-face interaction is crucial in video conferencing. However, when multiple people gather in a conference room, most cameras capture only a single video feed, making it difficult to distinguish the participants and their number, especially identifying the current speaker.
[0005] Summary of the Invention
[0006] The present disclosure proposes a method for splitting a video conference screen and related equipment to solve or partially solve the above problems.
[0007] In a first aspect of the present disclosure, a method for splitting a video conference screen is provided, comprising:
[0008] Acquire a target image acquired by an acquisition unit;
[0009] detecting a target object in the target image;
[0010] Dividing the video conference screen into at least two sub-screens according to the target object;
[0011] The target object in the target image is displayed correspondingly in the at least two sub-pictures.
[0012] In a second aspect of the present disclosure, a device for splitting a video conference screen is provided, comprising:
[0013] An acquisition module is configured to: acquire the target image acquired by the acquisition unit;
[0014] A detection module is configured to: detect a target object in the target image;
[0015] A division module is configured to: divide the video conference screen into at least two sub-screens according to the target object;
[0016] The display module is configured to display the target object in the target image in the at least two sub-pictures accordingly.
[0017] In a third aspect of the present disclosure, a computer device is provided, comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the method according to the first aspect.
[0018] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium containing a computer program is provided. When the computer program is executed by one or more processors, the processors are caused to execute the method described in the first aspect.
[0019] In a fifth aspect of the present disclosure, a computer program product is provided, comprising computer program instructions, which, when executed on a computer, cause the computer to execute the method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0021] FIG1A shows a schematic diagram of an exemplary system provided by an embodiment of the present disclosure.
[0022] FIG1B is a schematic diagram showing a video conference screen captured in the scene shown in FIG1A .
[0023] FIG2 shows a schematic diagram of an exemplary process according to an embodiment of the present disclosure.
[0024] FIG3A shows a schematic diagram of an exemplary target image according to an embodiment of the present disclosure.
[0025] FIG3B shows a schematic diagram of displaying a detection frame in a target image according to an embodiment of the present disclosure.
[0026] FIG3C shows a schematic diagram of an exemplary video conference screen according to an embodiment of the present disclosure.
[0027] FIG3D shows a schematic diagram of another exemplary video conference screen according to an embodiment of the present disclosure.
[0028] FIG3E shows a schematic diagram of a split-screen mode according to an embodiment of the present disclosure.
[0029] FIG3F shows a schematic diagram of another split-screen mode according to an embodiment of the present disclosure.
[0030] FIG3G shows a schematic diagram of another exemplary video conference screen according to an embodiment of the present disclosure.
[0031] FIG3H shows a schematic diagram of facial key point detection.
[0032] FIG3I shows a schematic diagram of a rotating human face.
[0033] FIG4 shows a schematic diagram of an exemplary method provided by an embodiment of the present disclosure.
[0034] FIG5 shows a schematic diagram of the hardware structure of an exemplary computer device provided by an embodiment of the present disclosure.
[0035] FIG6 shows a schematic diagram of an exemplary device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0036] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.
[0037] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the usual meanings understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative position relationships. When the absolute position of the described object changes, the relative position relationship may also change accordingly.
[0038] FIG1A shows a schematic diagram of an exemplary system 100 provided by an embodiment of the present disclosure.
[0039] As shown in Figure 1A, the system 100 may include at least one terminal device (e.g., terminal devices 102 and 104), a server 106, and a database server 108. A medium providing a communication link may be provided between the terminal devices 102 and 104 and the server 106 and the database server 108, such as a network. The network may include various connection types, such as wired or wireless communication links or fiber optic cables.
[0040] Users 110A-110C can use terminal device 102 to interact with server 106 over the network to receive or send messages, etc. Similarly, user 112 can use terminal device 104 to interact with server 106 over the network to receive or send messages, etc. Various applications (APPs) can be installed on terminal devices 102 and 104, such as video conferencing applications, reading applications, video applications, social applications, payment applications, web browsers, and instant messaging tools. In some embodiments, users 110A~110C and user 112 can respectively use video conferencing applications installed on terminal devices 102 and 104 to use video conferencing services provided by server 106, and terminal devices 102 and 104 can capture images 1022 and 1042 through cameras (for example, cameras set on terminal devices 102 and 104) and can capture live audio through microphones (for example, microphones set on terminal devices 102 and 104) and upload it to server 106, so that users 110A~110C and user 112 can respectively view each other's images and hear each other's voices through video conferencing applications on terminal devices 102 and 104.
[0041] The terminal devices 102 and 104 herein can be either hardware or software. When the terminal devices 102 and 104 are hardware, they can be various electronic devices with display screens, including but not limited to smartphones, tablet computers, e-book readers, MP3 players, laptop computers, and desktop computers (PCs). When the terminal devices 102 and 104 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules (for example, to provide distributed services), or as a single software or software module. No specific limitations are given here.
[0042] Server 106 may be a server that provides various services, such as a backend server that supports various applications displayed on terminal devices 102 and 104. Database server 108 may also be a database server that provides various services. It will be understood that if server 106 can implement the relevant functions of database server 108, database server 108 may not be provided in system 100.
[0043] The server 106 and database server 108 herein can also be hardware or software. When they are hardware, they can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When they are software, they can be implemented as multiple software programs or software modules (for example, to provide distributed services), or as a single software program or software module. No specific limitations are imposed herein.
[0044] It should be noted that the screen splitting method of the video conference screen provided in the embodiment of the present application is generally executed by terminal devices 102 and 104.
[0045] It should be understood that the number of terminal devices, users, servers and database servers in Figure 1A is merely illustrative and any number of terminal devices, users, servers and database servers may be provided as required.
[0046] FIG1B is a schematic diagram showing a video conference screen 120 captured in the scene shown in FIG1A .
[0047] As shown in Figure 1B, in the scenario shown in Figure 1A, the video conference screen 120 can display the images captured by the two cameras in two sub-screens 1202 and 1204 respectively, where the camera on the side of the terminal device 102 captures the image of the entire conference room. Therefore, the image of the target object (for example, facial image) corresponding to users 110A to 110C will be displayed accordingly in the sub-screen 1202.
[0048] As can be seen, since the camera in the conference room is fixed in position, the size and orientation of the target objects in sub-screen 1202 will vary depending on the relative position of users 110A-110C to the camera. Furthermore, since conference room cameras are typically far from the seats, it can be difficult to clearly see each participant in sub-screen 1202. Therefore, a solution for automatically splitting the image captured by a single camera is of great practical significance for improving interaction between the conference room and online participants during video conferencing.
[0049] In view of this, an embodiment of the present disclosure provides a method for splitting a video conference screen, by detecting a target object in a target image captured by an acquisition unit, and then dividing the video conference screen into at least two sub-screens according to the target object, and displaying the target object in the target image in the sub-screen accordingly. In this way, when the captured target image contains multiple participants, the video conference screen can be automatically split, thereby helping to enhance the sense of interaction between the participants in the conference room and other online participants during the video conference, thereby improving the user experience.
[0050] FIG2 illustrates a flow chart of an exemplary method 200 provided in an embodiment of the present disclosure. This method 200 can be used to automatically split a video conference screen. Optionally, this method 200 can be implemented by the terminal devices 102 and 104 of FIG1A , or by the server 106 of FIG1A . The following description uses the server 106 as an example to illustrate this method 200.
[0051] As shown in FIG. 2 , the method 200 may further include the following steps.
[0052] At step 202, server 106 may obtain a target image captured by a capture unit. Taking FIG1A as an example, the capture unit may be a camera provided in terminal devices 102 or 104, and the target image may be images 1022 and 1042 captured by the camera. After the camera captures the image, terminal devices 102 or 104 may upload the captured image to server 106 for processing.
[0053] FIG3A shows a schematic diagram of an exemplary target image 300 according to an embodiment of the present disclosure.
[0054] The target image 300 may be an image captured by any terminal device participating in the video conference in the system 100 . As shown in FIG3A , the target image 300 may include target objects 302A to 302C of multiple participants.
[0055] Next, at step 204, the server 106 may detect the target object in the target image 300. As an optional embodiment, an object detection technology may be used to detect the target object in the target image 300. Alternatively, a pre-trained object detection model may be used to detect the target objects 302A-302C in the target image 300, and detection boxes 304A-304C corresponding to the target objects 302A-302C may be obtained, as shown in FIG3B .
[0056] Furthermore, considering that the position of the participants in the conference room is constantly changing, if the position of the detection frame is fixed after obtaining the detection frame, it may not be possible to follow the position of the face in time when the participant moves. Therefore, in some embodiments, target tracking technology can be used to track the position changes of the target object in real time. Accordingly, the detection frame will also change accordingly. In this way, even if the participant moves during the video conference, the target object can be tracked. Optionally, a pre-trained target tracking model can be used to track the target objects 302A to 302C in the target image 300. The target tracking model can be a real-time face frame and face key point detection and tracking model based on deep learning. The model structure includes but is not limited to various forms of convolutional neural networks and various forms of transformer networks. In this way, each sub-screen of the split screen achieved through image face detection and tracking can follow the face on the screen in real time.
[0057] In some embodiments, after detecting the target object in the target image 300 , the process may proceed to step 206 , where the server 106 may directly determine a split-screen layout based on the target object, and then divide the video conference screen based on the split-screen layout.
[0058] FIG3C shows a schematic diagram of an exemplary video conference screen 310 according to an embodiment of the present disclosure. As shown in FIG3C , the video conference screen 310 is divided into multiple sub-screens. Specifically, when dividing the sub-screens, in some embodiments, considering that the video conference scene requires multi-party interaction, the video conference screen can be divided according to the total number of target objects in the target images captured by each terminal device currently participating in the video conference. For example, taking the scene shown in FIG1A as an example, the number of sub-screens can be determined based on the total number of target objects in the images captured by each terminal device 102 and 104, which number is 4 in this example. After determining the number of sub-screens, the split-screen layout can be determined. There are many types of split-screen layouts. For example, in order to maintain the basic split-screen layout of an existing video conference, the video conference screen 310 can be first divided into n sub-screens based on the number of terminal devices n, and then each terminal device can be associated with the n sub-screens. Next, the sub-screens corresponding to each terminal device can be further divided based on the number of target objects in the images captured by each terminal device, thus obtaining a split-screen layout. As shown in Figure 3C, the left and right sub-screens correspond to the images captured by the two terminal devices 102 and 104, respectively. The left screen further includes three sub-screens, corresponding to target objects 302A to 302C. The right sub-screen displays the image captured by terminal device 104 and may include target object 312. This constitutes a complete video conference screen 310. Furthermore, because the sub-screens corresponding to different terminal devices in screen 310 are divided (e.g., into equal-sized groups), the user can know the specific number of terminals participating in the video conference through this split-screen layout.
[0059] It can be understood that in addition to adopting the split-screen layout of the aforementioned embodiment, there can also be other split-screen layout methods, for example, directly dividing the sub-screen according to the number of all target objects. Figure 3D shows a schematic diagram of another exemplary video conference screen 320 according to an embodiment of the present disclosure. As shown in Figure 3D, the video conference screen 320 is divided into 4 sub-screens of equal size according to the total number of target objects, corresponding to target objects 302A to 302C and target object 312 respectively. In this way, all participants, whether they are multiple participants in the conference room or participants attending alone, can occupy a sub-screen of equal size with others, so that each participant can interact with each participant very clearly.
[0060] In some embodiments, when performing a split-screen layout, different layout options may be available depending on the number of target objects. For example, the screen may be divided into an equilateral array of sub-screens (e.g., N×N sub-screens) or an unequal array of sub-screens (e.g., N×M sub-screens) based on the number of target objects. For example, as shown in FIG3D , when the number is 4, 2×2 sub-screens may be obtained. For another example, when the number is 3, the screen 320 may be divided into two rows, with one sub-screen in the first row and two sub-screens in the second row, as shown in the left screen of FIG3C .
[0061] It is understood that when the number of target objects is greater, the aforementioned method can also be used for split-screen layout. For example, when the number of target objects is 7, 3×3 sub-screens can be used to correspond to the target objects, wherein two sub-screens can be left blank, as shown in FIG3E .
[0062] In some embodiments, considering that the ratio of a typical display screen is not a square but a rectangle with a length greater than a width (for example, 16:9), it is possible to consider increasing the number of sub-screens in the length direction when performing a split-screen layout. For example, the maximum number of people supported by a video conference screen is 12. When N<5, the screen can be divided into 1 row and N columns. When N≤8, the screen can be divided into 2 rows, the first row is N / 2 columns (which can be rounded up or down), and the second row is NN / 2 columns (which can be rounded down or up). When N≤12, the screen can be divided into 3 rows, the first two rows are divided into N / 3 columns (which can be rounded up or down), and the last row is NN / 3×2 columns (which can be rounded down or up). The width of each sub-screen in the last row is consistent with the width of the sub-screen in the previous row, and can be arranged in the center. Figure 3F shows a schematic diagram of another exemplary video conference screen according to an embodiment of the present disclosure. As shown in Figure 3F, when the number of target objects is 7, the above method can be used to split the screen into a 4+3 layout.
[0063] It can be seen that by detecting the target objects in the target image, the video conference screen is divided into at least two sub-screens according to the number of target objects, and the corresponding target objects are displayed in the sub-screens. In this way, when the captured target image contains multiple participants, the video conference screen can be automatically split, which helps to enhance the sense of interaction between the participants in the conference room and other online participants during the video conference, thereby improving the user experience.
[0064] In some embodiments, when performing a split-screen layout, in addition to considering the number of target objects, it is also possible to detect whether a participant is speaking, and then perform corresponding processing based on the sub-screen corresponding to the detected speaking participant. Therefore, as shown in Figure 2, method 200 further includes step 208, performing speaker detection on the target object. Optionally, this step can be performed in parallel with target object detection to increase processing speed.
[0065] Optionally, when it is determined that the target object in the target image is speaking (i.e., the participant corresponding to the target object is speaking), an indicator, such as an icon indicating that the participant is speaking, may be displayed in the sub-image corresponding to the speaking participant. As shown in FIG3D , a microphone-shaped icon may be displayed in the sub-image corresponding to the speaking participant to alert others that the participant is speaking.
[0066] As an optional embodiment, the split-screen layout may also be changed based on whether a participant who is speaking is detected.
[0067] Optionally, if the detection result determines that the target object in the target image is speaking, the video conference screen may be divided into at least two sub-screens according to a first split-screen mode; if the detection result determines that all participants in the video conference are silent, the video conference screen may be divided into at least two sub-screens according to a second split-screen mode. The second split-screen mode may be any of the split-screen layouts in the aforementioned embodiments.
[0068] Furthermore, the video conference screen is divided into at least two sub-screens according to the first split-screen mode, including: enlarging and displaying the first sub-screen of the at least two sub-screens, and displaying the other sub-screens of the at least two sub-screens side by side on at least one side of the first sub-screen, and the first sub-screen can be used to display the target object who is speaking.
[0069] Figure 3G shows a schematic diagram of another exemplary video conference screen 330 according to an embodiment of the present disclosure. As shown in Figure 3G, screen 330 includes four sub-screens, corresponding to target objects 302A-302C and target object 312, respectively. First sub-screen 3302 is enlarged and corresponds to target object 302A of participant 110A who is speaking. The other sub-screens are displayed side by side to one side of first sub-screen 3302. In this way, based on the speaker detection results, the sub-screen of the speaking participant is arranged in the center of the screen, occupying a larger portion of the screen, while the sub-screens of non-speakers are arranged to the sides, occupying a smaller portion of the screen, thereby enhancing interactivity.
[0070] Thus, when someone is speaking, speaker mode (the first split-screen mode) is used. In speaker mode, the person currently speaking is placed on the largest sub-screen, and the remaining participants are arranged side by side on at least one side of the largest sub-screen (or on two or more sides if there are a large number of participants). When no one is speaking, normal split-screen mode (the second split-screen mode) is used, with each sub-screen the same size. By selecting different split-screen modes based on the speaker detection results, the interactivity of the video conference can be increased and the user experience can be improved.
[0071] Generally, the video stream and audio stream of online participants are independent. In related technologies, video conferencing software can confirm whether someone is speaking in the corresponding video based on the audio stream. However, the participants in the conference room share the video stream and audio stream. The audio stream collected by the terminal device on one side of the conference room cannot determine whether the person is speaking, nor can it distinguish who is currently speaking, which reduces the interactivity of the meeting.
[0072] In view of this, in some embodiments, speaker detection is performed through image processing, which can avoid the problem of being unable to distinguish who is currently speaking through the audio stream.
[0073] As an optional embodiment, the server 106 may perform key point detection on each detected target object, and then determine whether the participant corresponding to the target object in the target image is speaking based on the key point detection result.
[0074] FIG3H shows a schematic diagram of facial key point detection.
[0075] As shown in Figure 3H, facial key point detection can use a 68-point detection method. These key points are distributed across various parts of the face. Points 0-16 correspond to the chin, points 16-21 correspond to the right eyebrow (in this case, the mirror image is the right eyebrow of the person in the image), points 22-26 correspond to the left eyebrow, points 27-35 correspond to the nose, points 36-41 correspond to the right eye, points 42-47 correspond to the left eye, and points 48-67 correspond to the lips. By detecting key points, faces can be recognized, and based on the changes in key points across multiple frames of the target object, it can be determined whether the corresponding participant is speaking.
[0076] It should be noted that the facial key point detection method with 68 key points is only an example. It is understandable that facial key point detection can also have other key point numbers, for example, 21 key points, 29 key points, and so on.
[0077] As an optional embodiment, 106 key points may be used to implement key point detection, thereby obtaining a more accurate detection result.
[0078] In some embodiments, after detecting the key points of the target object, the lip height and lip width can be determined based on the key points of the target object. The lip height-to-width ratio of the target object's lips is then determined based on the lip height and lip width. Based on the changes in the lip height-to-width ratio, whether the target object is speaking is determined. This enables speaker detection using image processing, avoiding the issue of being unable to discern the current speaker from an audio stream.
[0079] Furthermore, considering that the rotation of the face may cause the position of the lip key points to change, in some embodiments, when performing key point detection, the detected key points can be corrected based on the rotation angle of the face.
[0080] FIG3I shows a schematic diagram of a rotating human face.
[0081] As shown in FIG3I , for a human face, there are three rotation angles in three-dimensional space: yaw angle (Yaw), roll angle (Roll), and pitch angle (Pitch).
[0082] As an optional embodiment, the influence of the roll rotation can be offset by an affine transformation, and then the influence of the pitch and yaw rotation can be offset by the pitch and yaw information of the face detection.
[0083] Specifically, multiple key points (e.g., coordinates of 106 key points) can be obtained based on target object detection. The key points of the target object are then matched with the key points of a standard (average) human face (i.e., standard key points), thereby obtaining an affine transformation matrix (mapping relationship). Based on applying the affine transformation matrix to the multiple key points obtained based on target object detection, the multiple key points can be subjected to roll angle correction, thereby obtaining multiple corrected key points, i.e., the coordinates of the key points of the currently detected target object when Roll = 0.
[0084] Furthermore, multiple first key points and multiple second key points corresponding to the lip height and lip width, respectively, can be selected from the corrected multiple key points, and the multiple first key points can be corrected for pitch angles to obtain a corrected lip height, and the multiple second key points can be corrected for yaw angles to obtain a corrected lip width. In this way, the corrected lip height and the corrected lip width offset the effects of pitch and yaw rotations. As an optional embodiment, taking 106 key points as an example, the length of the line segment from 98 to 102 can be calculated to represent the lip height and divided by cos(pitch) to offset the effects of pitch, thereby obtaining the corrected lip height. Similarly, the length of the line segment from 96 to 100 can be calculated to represent the lip width and divided by cos(yaw) to offset the effects of yaw, thereby obtaining the corrected lip width. The angle information of pitch and yaw can be provided by the face detection module.
[0085] Then, based on the corrected lip height and the corrected lip width, the key point detection result (the detection result for the lip height and lip width) can be obtained, so that the degree of mouth opening can be expressed by the height-to-width ratio of the lips after correction.
[0086] Given that speaking is a dynamic process, relying solely on the current lip aspect ratio may not accurately determine whether the current participant is speaking. Therefore, in some embodiments, the changes in the lip aspect ratio over a period of time can be monitored, and the variance of the aspect ratio over this period can be used to determine whether the subject is currently speaking.
[0087] As an optional embodiment, the lip height-to-width ratio of the target object may be calculated based on the corrected lip height and the corrected lip width, and the lip height-to-width ratio may be stored.
[0088] Then, when determining whether the participant corresponding to the target object in the target image is speaking based on the key point detection results, the key point detection results can be combined to determine the change information corresponding to the lip aspect ratio of the target object in the target image, and based on the change information, determine whether the participant corresponding to the target object in the target image is speaking. For example, whether the participant is speaking can be determined based on whether the variance of the lip aspect ratio within a preset time period (e.g., within 1 second) is greater than a variance threshold. If so, it can be determined that the participant is speaking. In this way, the result of determining whether the participant is speaking can be more accurate.
[0089] Considering that speakers may pause intermittently while speaking, in order to make speech detection more stable, in some embodiments, a counter can be maintained to record the number of times the speaker has been determined to be speaking in a recent period of time. Whether the speaker is speaking is then determined based on the relationship between this number and a preset counting threshold. Optionally, determining whether the target subject is speaking based on information about changes in the lip aspect ratio includes: setting a preset time period (e.g., within 2 seconds); counting the number of changes in the lip aspect ratio within the preset time period; and determining that the target subject is speaking when the number of changes reaches a preset number. In this way, the timing information of key points is utilized to enhance the timing stability of speaker detection and reduce fluctuations in the detection state.
[0090] As an optional embodiment, in response to determining based on the change information that the participant corresponding to the target object in the target image is speaking (that is, the subject is determined to be speaking in the current frame), the count value is +1; in response to determining based on the change information that the participant corresponding to the target object in the target image is not speaking (that is, the subject is determined to be not speaking in the current frame), the count value is -1.
[0091] Then, based on the count value within a preset time period (e.g., within 2 seconds), it can be determined whether the participant corresponding to the target object in the target image is speaking. For example, when the counter value is greater than a preset count threshold (e.g., 2 times), an effect that the subject is speaking can be displayed (e.g., the sub-screen corresponding to the speaker is enlarged and / or a microphone icon is displayed); when the counter value is less than the preset count threshold, the effect that the subject is speaking can be canceled (e.g., the sub-screen corresponding to the speaker is restored to the same size as other sub-screens and / or the microphone icon is hidden).
[0092] In this way, the position of the lips is determined based on the key points of the face after the split screen, and the relative position of the lip key points is used to determine whether the current participant is speaking. To mitigate speech detection errors caused by facial movement and rotation, the key points of face detection are mapped to a standard, unrotated face before making a speech determination, reducing the impact of facial movement. Furthermore, the timing information of the key points is used to enhance the temporal stability of speaker detection and reduce fluctuations in detection status.
[0093] Considering the potential for error in speaker identification using image processing alone, in some embodiments, after determining that the number of changes has reached a preset number, audio data from the video conference can be obtained. Then, based on the key point detection results and the audio data from the video conference, it can be determined whether the target subject in the target image is speaking. This combination of key point detection results and the current video conference audio data can further improve the accuracy of speaker identification. As an optional embodiment, when the audio pickup is a dual-channel pickup, the speaker can be located based on the two sets of audio data collected by the two channels, further improving the accuracy of speaker identification.
[0094] Some video conferencing automatic screen splitting software uses body detection to determine the position of participants in the current frame, and then performs cropping and layout based on the position of the participants to achieve the automatic screen splitting function. However, it is difficult to achieve real-time screen splitting changes when participants enter or leave the meeting room.
[0095] Therefore, in order to increase or decrease the number of split screens in real time, after determining the split screen layout, as shown in FIG2 , step 210 can be entered to match the target object with the detection frame according to the detection frame position and the split screen layout.
[0096] According to the above embodiments, target detection or target tracking technology can be used to detect the target object in the target image and obtain a detection frame corresponding to the target object, for example, detection frames 304A to 304C in FIG3B . In this way, each detected target object can correspond to a detection frame, and then the position of each target object in the split-screen layout can be determined based on the split-screen layout, and then the detection frame of the target object can be matched with the position in the split-screen layout. In this way, by detecting the target object 300, the relative position relationship of different faces can be determined by the position of the detection frame of the target object, and the position of the split screen can be matched with the position of the face, and then the detection frame can be matched with the split-screen layout. The number of split screens can be increased or decreased according to the detection results, and the layout of the split screen can be changed, so that when someone joins or leaves the conference room, the relative position of the original participants can be kept unchanged, and the number of split screens can be increased or decreased in real time.
[0097] Optionally, the ROI (Region of Interest) corresponding to each participant in the split-screen layout can be determined based on the coordinates of each participant's detection frame in the original image (target image), thereby determining the one-to-one correspondence between the person, the detection frame, and the sub-screen based on the position of each person's detection frame and the split-screen layout.
[0098] Then, as shown in Figure 2, at step 212, the contents of the detection frame can be matched with the sub-screen. Optionally, the coordinates of the detection frame corresponding to the target object and the coordinates of the sub-screen corresponding to the target object can be first determined; then, based on the coordinates of the detection frame corresponding to the target object and the coordinates of the sub-screen corresponding to the target object, the image corresponding to the detection frame can be translated and / or scaled to the sub-screen corresponding to the target object. By scaling the target object, all participants can clearly see each other, enhancing the interactivity of the meeting.
[0099] As an optional embodiment, after each detection frame corresponds to a split-screen sub-picture, the detection frame can be expanded to both sides according to a certain ratio (for example, 20% for height and 40% for width). On this basis, the width and height of the detection frame can be expanded to both sides until its aspect ratio is the same as the aspect ratio of the corresponding sub-screen. If the expanded ROI exceeds the boundary of the picture, it will be translated to the range of the picture, so that the content of the detection frame can be matched with the sub-picture.
[0100] In some embodiments, when translating and scaling the image corresponding to the detection frame, the ROI of the sub-screen corresponding to each detection frame at the current moment can be calculated by linear interpolation, and a smooth movement and scaling effect from the current screen to the target face can be achieved, realizing the horizontal transformation-vertical transformation-scaling function similar to that of a surveillance camera when switching the split-screen effect.
[0101] Specifically, assuming that the original coordinates of a sub-image are represented by the coordinates of the upper left and lower right vertices of the rectangle as (x1_0, y1_0, x2_0, y2_0), the translation and zoom duration is T, and the coordinates of the ROI of the detection box of the target face are represented by the coordinates of the upper left and lower right vertices of the rectangle as (x1_T, y1_T, x2_T, y2_T), then:
[0102] First, a time interval Δt may be determined, and then the number of linear interpolations may be determined according to the pan-scaling duration T and the time interval Δt.
[0103] Then, the updated coordinates of each sub-picture corresponding to each interpolation are determined based on the number of linear interpolations and the original coordinates of the sub-picture and the coordinates of the ROI of the detection frame. For each interpolation, the updated coordinates can be transformed at equal intervals relative to the coordinates corresponding to the previous interpolation.
[0104] Then, in a manner of equal time intervals, the sub-picture is gradually subjected to horizontal transformation, vertical transformation, and scaling processing according to the updated coordinates of the sub-picture corresponding to each interpolation, until the duration reaches T.
[0105] In this way, there can be a transition time of time T from turning on the split-screen function to completing the translation and scaling of the sub-screen, and then from a visual effect, there can be a horizontal transformation-vertical transformation-zooming function similar to that of a surveillance camera, which improves the user experience and improves the problem of abrupt switching effects caused by direct switching or simple scaling of the cropped frame in related technologies.
[0106] In some embodiments, the clarity of the sub-picture processed in the aforementioned manner may be affected. Therefore, a super-resolution technology may be used to increase the resolution of the sub-picture, thereby improving the clarity.
[0107] In the related art, there is usually no virtual background function that supports sprites. The embodiments of the present disclosure provide a virtual background function to fill the gap.
[0108] Therefore, as shown in Figure 2, in step 214, it can be determined whether the virtual background function is turned on. If the virtual background function is turned on, proceed to step 216 to perform semantic segmentation of the current image based on the target object (which can be processed in parallel with face detection to improve processing efficiency). In this way, the virtual background function of each split screen is realized by using the semantic segmentation capability. Optionally, a pre-trained semantic segmentation model can be used to segment the target object from the background image. The semantic segmentation model can be a real-time portrait semantic segmentation model based on deep learning, and the model structure includes but is not limited to various forms of convolutional neural networks (Convolutional Neural Network) and various forms of Transformer networks.
[0109] In some embodiments, the portrait segmentation function can be applied to perform portrait segmentation on the entire current input image. After the split-screen function is turned on, the segmentation result of each sub-screen corresponds to the segmentation result of the full image segmentation result under the ROI corresponding to the sub-screen. Then, for each pixel point of each sub-screen, the pixel value of the virtual background result at that pixel point is calculated based on the value of the segmentation result at that pixel point (normalized to [0,1]), the value corresponding to that pixel point in the input image, and the pixel value of the new background to be replaced at that pixel point.
[0110] Specifically, for each pixel point, the first value of the segmentation result at the pixel point (normalized to [0,1]) can be multiplied by the value corresponding to the pixel point in the input image, and the second value of the segmentation result at the pixel point (1 minus the first value) can be added and multiplied by the pixel value of the new background to be replaced at the pixel point to obtain the pixel value of the virtual background result at the pixel point.
[0111] In this way, the process of replacing the real background with the virtual background is completed.
[0112] Next, the process proceeds to step 218 to render the video conference screen.
[0113] Optionally, each sub-screen can be rendered using the content of the original image's ROI area. If the virtual background function is enabled, background replacement is performed by combining the portrait segmentation results and the background to be replaced. Any special processing required for the detected speaker is also completed in this step. For example, the target object corresponding to the speaking participant can be displayed in the first sub-screen 3302, and the virtual background can be displayed in the second sub-screen 3304.
[0114] Then the current frame processing is completed and the next frame process is entered, and the next frame processing can begin.
[0115] As can be seen from the above embodiments, the embodiments of the present disclosure adopt an automatic split-screen system for video conferencing. The automatic split-screen function of video can enable people sitting in the same conference room to communicate "face to face" with colleagues participating remotely. In some embodiments, speaker detection can easily identify who is speaking in the current conference room and place the speaker's video stream in a prominent position to enhance the video conferencing experience. In some scenarios, such as outdoor live broadcasts, sometimes the proportion of the subject in the picture may be very small due to the venue and distance. The PTZ function implemented by software can realize the function of automatic tracking and focusing of the lens without any operation.
[0116] It should be noted that the aforementioned embodiment is described with the server 106 as the execution subject. In fact, the aforementioned processing steps do not need to limit the execution subject. For example, the terminal devices 102 and 104 can also implement these processing steps. Therefore, it is also possible to use the terminal devices 102 and 104 as the execution subjects of the aforementioned embodiments.
[0117] The present disclosure also provides a method for splitting a video conference screen. FIG4 illustrates a flowchart of an exemplary method 400 provided by the present disclosure. This method 400 can be applied to the server 106 of FIG1A or to the terminal devices 102 and 104 of FIG1A. As shown in FIG4 , this method 400 can further include the following steps.
[0118] In step 402, a target image acquired by an acquisition unit is obtained.
[0119] Taking FIG. 1A as an example, the acquisition unit may be a camera provided in the terminal devices 102 and 104 , and the target image may be images 1022 and 1042 acquired by the camera.
[0120] At step 404 , target objects (eg, target objects 302A to 302C in FIG. 3A ) in the target image (eg, image 300 in FIG. 3A ) are detected.
[0121] As an optional embodiment, object detection technology can be used to detect the target object in the target image 300. Alternatively, a pre-trained object detection model can be used to detect the target objects 302A-302C in the target image 300, and detection boxes 304A-304C corresponding to the target objects 302A-302C can be obtained, as shown in FIG3B .
[0122] In step 406 , the video conference screen is divided into at least two sub-screens according to the target object.
[0123] In step 408 , the target object in the target image is displayed in the at least two sub-pictures correspondingly, as shown in FIG. 3C to FIG. 3G .
[0124] The embodiment of the present disclosure provides a method for splitting the video conference screen. The method detects the target object in the target image captured by the acquisition unit, and then divides the video conference screen into at least two sub-screens according to the target object, and displays the corresponding target object in the sub-screen. In this way, when the captured target image contains multiple participants, the video conference screen can be automatically split, which helps to enhance the sense of interaction between the participants in the conference room and other online participants during the video conference, thereby improving the user experience.
[0125] In some embodiments, the video conference screen is divided into at least two sub-screens according to the number of the target objects, including: determining whether the target object in the target image is speaking; in response to determining that the target object in the target image is speaking, dividing the video conference screen into at least two sub-screens according to the first split-screen mode, as shown in Figure 3G. In this way, when someone is speaking, the speaker mode (first split-screen mode) is adopted. In the speaker mode, the person who is currently speaking is placed on the largest sub-screen, and the remaining participants are arranged side by side on at least one side of the largest sub-screen (when the number is large, they can be placed on two or more sides). When no one is speaking, the normal split-screen mode (second split-screen mode) is adopted, and each sub-screen is the same size. By selecting different split-screen modes based on the speaker detection results, the interactivity of the video conference can be increased and the user experience can be improved.
[0126] In some embodiments, the video conference screen is divided into at least two sub-screens according to a first split-screen mode, including: enlarging and displaying a first sub-screen (for example, sub-screen 3302 in Figure 3G) of the at least two sub-screens, and displaying other sub-screens of the at least two sub-screens side by side on at least one side of the first sub-screen, wherein the first sub-screen is used to display the target object who is speaking.
[0127] In this way, according to the speaker detection result, the sub-screen of the participant who is speaking is arranged in the middle of the screen and occupies a larger screen, and the sub-screen of the non-speaker is arranged on the side and occupies a smaller screen, thereby better improving interactivity.
[0128] In some embodiments, determining whether the target object in the target image is speaking includes: detecting key points of the target object; determining the lip height based on the key points of the target object; determining the lip width based on the key points of the target object; obtaining the lip height-to-width ratio of the target object according to the lip height and the lip width; and determining whether the target object is speaking based on change information of the lip height-to-width ratio, thereby performing speaker detection through image processing and avoiding the problem of not being able to distinguish who is currently speaking through the audio stream.
[0129] In some embodiments, determining whether the target object in the target image is speaking includes: performing key point detection on the target object; determining whether the target object in the target image is speaking based on the key point detection result, thereby performing speaker detection through image processing, avoiding the problem of being unable to distinguish who is currently speaking through the audio stream.
[0130] In some embodiments, performing key point detection on the target object includes:
[0131] Obtaining a plurality of key points according to the target object detection;
[0132] Performing roll angle correction on the plurality of key points according to the correspondence between the key points and the standard key points to obtain a plurality of corrected key points;
[0133] Selecting a plurality of first key points corresponding to the lip height from the corrected plurality of key points, and performing pitch angle correction on the plurality of first key points to obtain a corrected lip height;
[0134] selecting a plurality of second key points corresponding to the lip width from the corrected plurality of key points, and performing yaw angle correction on the plurality of second key points to obtain a corrected lip width;
[0135] The key point detection result is obtained according to the corrected lip height and the corrected lip width.
[0136] In order to improve the speech detection misjudgment caused by face movement, rotation and other actions, the key points of face detection will be mapped to a standard face that has not been rotated before speaking judgment is made to reduce the impact of face movement.
[0137] In some embodiments, the method further comprises: calculating the lip height-to-width ratio of the target object according to the corrected lip height and the corrected lip width, and storing the lip height-to-width ratio;
[0138] Determining whether a target object in the target image is speaking based on a key point detection result includes: determining, in combination with the key point detection result, change information corresponding to the height-to-width ratio of the lips of the target object in the target image, and determining whether the target object in the target image is speaking based on the change information.
[0139] In this way, the timing information of key points is used to enhance the temporal stability of speaker detection and reduce the fluctuation of detection status.
[0140] In some embodiments, determining whether a participant corresponding to a target object in the target image is speaking includes: setting a preset time period; counting the number of changes in the lip height-to-width ratio within the preset time period; and determining that the target object is speaking when the number of changes reaches a preset number. In this way, the timing information of key points is utilized to enhance the temporal stability of speaker detection and reduce fluctuations in detection status.
[0141] In some embodiments, when the number of changes reaches a preset number, determining that the target subject is speaking includes: determining that the number of changes reaches the preset number; obtaining audio data from the video conference; and determining that the target subject is speaking in combination with the audio data from the video conference. In this way, combining the key point detection results with the audio data from the current video conference can further improve the accuracy of speaker identification. As an optional embodiment, when the microphone collecting audio is a dual-channel microphone, the speaker can be located based on the two sets of audio data collected by the two channels, further improving the accuracy of speaker identification.
[0142] In some embodiments, detecting the target object in the target image includes: detecting the target object in the target image using target detection or target tracking technology to obtain a detection frame of the target object;
[0143] Displaying the target object in the target image in the at least two sub-screens includes: determining the coordinates of the detection frame corresponding to the target object; determining the coordinates of the sub-screen corresponding to the target object; and translating and / or scaling the image corresponding to the detection frame to the sub-screen corresponding to the target object based on the coordinates of the detection frame and the coordinates of the sub-screen.
[0144] By zooming in and out on the target object, all participants can clearly see each other, thus improving the interactivity of the meeting.
[0145] In some embodiments, displaying the target object in the target image in the at least two sub-screens further includes: in response to determining that the virtual background function in the sub-screen is turned on, using segmentation technology to segment the target object and the background to obtain a segmentation result; based on the segmentation result, displaying the virtual background in the sub-screen, filling the technical gap in the related technology of not displaying the virtual background in the sub-screen.
[0146] In some embodiments, the target object in the target image is displayed correspondingly in the at least two sub-screens, including: in response to determining that the virtual background function of the second sub-screen among the at least two sub-screens is turned on, displaying the virtual background in the second sub-screen (for example, sub-screen 3304 in Figure 3G), filling the technical gap in the related technology of not displaying the virtual background in the sub-screen.
[0147] In some embodiments, the method further comprises: segmenting the target object and the background in the target image using a semantic segmentation technique to obtain a segmentation result;
[0148] Displaying a virtual background in the second sub-picture includes: displaying a virtual background in the second sub-picture according to the segmentation result.
[0149] In this way, semantic segmentation technology is used to segment the target object from the actual background, thereby effectively replacing the virtual background.
[0150] In some embodiments, displaying the target object in the target image in the at least two sub-screens further includes: in response to determining that the target object is speaking, displaying an indicator icon in the sub-screen corresponding to the target object, thereby reminding others that the participant in the sub-screen corresponding to the icon is speaking, thereby improving interactivity.
[0151] It should be noted that the method of the embodiments of the present disclosure can be performed by a single device, such as a computer or server. The method of the embodiments of the present disclosure can also be applied in a distributed scenario, where multiple devices cooperate to perform the method. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the method of the embodiments of the present disclosure, and the multiple devices will interact with each other to complete the method.
[0152] It should be noted that the above description is limited to some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0153] The present disclosure also provides a computer device for implementing the aforementioned method 200 or 400. FIG5 illustrates a hardware structure diagram of an exemplary computer device 500 provided in the present disclosure. Computer device 500 can be used to implement the server 106 in FIG1A , or the terminal devices 102 and 104 in FIG1A . In some scenarios, computer device 500 can also be used to implement the database server 108 in FIG1A .
[0154] 5 , a computer device 500 may include a processor 502, a memory 504, a network module 506, a peripheral interface 508, and a bus 510. The processor 502, the memory 504, the network module 506, and the peripheral interface 508 are connected to each other in communication within the computer device 500 via the bus 510.
[0155] The processor 502 may be a central processing unit (CPU), an image processor, a neural network processor (NPU), a microcontroller (MCU), a programmable logic device (PLD), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or one or more integrated circuits. The processor 502 may be used to perform functions related to the technology described in this disclosure. In some embodiments, the processor 502 may also include multiple processors integrated into a single logical component. For example, as shown in FIG5 , the processor 502 may include multiple processors 502 a, 502 b, and 502 c.
[0156] The memory 504 can be configured to store data (e.g., instructions, computer codes, etc.). As shown in Figure 5, the data stored in the memory 504 can include program instructions (e.g., program instructions for implementing the method 200 or 400 of the embodiment of the present disclosure) and data to be processed (e.g., the memory can store configuration files of other modules, etc.). The processor 502 can also access the program instructions and data stored in the memory 504, and execute the program instructions to operate on the data to be processed. The memory 504 can include a volatile storage device or a non-volatile storage device. In some embodiments, the memory 504 can include a random access memory (RAM), a read-only memory (ROM), an optical disc, a magnetic disk, a hard disk, a solid-state drive (SSD), a flash memory, a memory stick, etc.
[0157] The network interface 506 can be configured to provide the computer device 500 with communication with other external devices via a network. The network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, near field communication (NFC)), a cellular network, the Internet, or a combination thereof. It will be understood that the type of network is not limited to the specific examples above.
[0158] The peripheral interface 508 can be configured to connect the computer device 500 to one or more peripheral devices to enable information input and output. For example, the peripheral devices can include input devices such as a keyboard, a mouse, a touchpad, a touch screen, a microphone, and various sensors, as well as output devices such as a display, a speaker, a vibrator, and an indicator light.
[0159] The bus 510 may be configured to transmit information between various components of the computer device 500 (e.g., the processor 502, the memory 504, the network interface 506, and the peripheral interface 508), such as an internal bus (e.g., a processor-memory bus), an external bus (USB port, PCI-E bus), etc.
[0160] It should be noted that although the architecture of the computer device 500 shown above only shows the processor 502, memory 504, network interface 506, peripheral interface 508, and bus 510, in a specific implementation, the architecture of the computer device 500 may also include other components necessary for normal operation. In addition, those skilled in the art will understand that the architecture of the computer device 500 may also include only the components necessary to implement the embodiments of the present disclosure, and does not necessarily include all the components shown in the figure.
[0161] The present disclosure also provides an interactive device. Figure 6 shows a schematic diagram of an exemplary device 600 provided by the present disclosure. As shown in Figure 6, the device 600 can be used to implement method 200 or 400, and can further include the following modules.
[0162] The acquisition module 602 is configured to acquire the target image acquired by the acquisition unit.
[0163] Taking FIG. 1A as an example, the acquisition unit may be a camera provided in the terminal devices 102 and 104 , and the target image may be images 1022 and 1042 acquired by the camera.
[0164] The detection module 604 is configured to detect a target object (eg, target objects 302A to 302C in FIG. 3A ) in the target image (eg, image 300 in FIG. 3A ).
[0165] As an optional embodiment, object detection technology can be used to detect the target object in the target image 300. Alternatively, a pre-trained object detection model can be used to detect the target objects 302A-302C in the target image 300, and detection boxes 304A-304C corresponding to the target objects 302A-302C can be obtained, as shown in FIG3B .
[0166] The division module 606 is configured to divide the video conference screen into at least two sub-screens according to the target object.
[0167] The display module 608 is configured to display the target object in the target image in the at least two sub-pictures accordingly.
[0168] The embodiment of the present disclosure provides a method for splitting the video conference screen. The method detects the target object in the target image captured by the acquisition unit, and then divides the video conference screen into at least two sub-screens according to the target object, and displays the corresponding target object in the sub-screen. In this way, when the captured target image contains multiple participants, the video conference screen can be automatically split, which helps to enhance the sense of interaction between the participants in the conference room and other online participants during the video conference, thereby improving the user experience.
[0169] In some embodiments, the division module 606 is configured to: determine whether the target object in the target image is speaking; in response to determining that the target object in the target image is speaking, divide the video conference screen into at least two sub-screens according to the first split-screen mode, as shown in Figure 3G. In this way, when someone is speaking, the speaker mode (first split-screen mode) is adopted. In the speaker mode, the person who is currently speaking is placed on the largest sub-screen, and the remaining participants are arranged side by side on at least one side of the largest sub-screen (when the number is large, they can be placed on two or more sides). When no one is speaking, the normal split-screen mode (second split-screen mode) is adopted, and each sub-screen is the same size. By selecting different split-screen modes based on the speaker detection results, the interactivity of the video conference can be increased and the user experience can be improved.
[0170] In some embodiments, the dividing module 606 is configured to: enlarge and display a first sub-screen (e.g., sub-screen 3302 in FIG. 3G ) of the at least two sub-screens, and display other sub-screens of the at least two sub-screens in parallel on at least one side of the first sub-screen, wherein the first sub-screen is used to display the target object who is speaking;
[0171] The display module 608 is configured to display the target object corresponding to the participant who is speaking in the first sub-screen.
[0172] In this way, according to the speaker detection result, the sub-screen of the participant who is speaking is arranged in the middle of the screen and occupies a larger screen, and the sub-screen of the non-speaker is arranged on the side and occupies a smaller screen, thereby better improving interactivity.
[0173] In some embodiments, the detection module 604 is configured to: detect the key points of the target object; determine the lip height based on the key points of the target object; determine the lip width based on the key points of the target object; obtain the lip height-to-width ratio of the target object according to the lip height and the lip width; determine whether the target object is speaking based on the change information of the lip height-to-width ratio, so that speaker detection can be performed through image processing, which can avoid the problem of not being able to distinguish who is currently speaking through the audio stream.
[0174] In some embodiments, the detection module 604 is configured to: perform key point detection on the target object; determine whether the target object in the target image is speaking based on the key point detection result, so that speaker detection can be performed through image processing, avoiding the problem of not being able to distinguish who is currently speaking through the audio stream.
[0175] In some embodiments, the detection module 604 is configured to:
[0176] Obtaining a plurality of key points according to the target object detection;
[0177] Performing roll angle correction on the plurality of key points according to the correspondence between the key points and the standard key points to obtain a plurality of corrected key points;
[0178] Selecting a plurality of first key points corresponding to the lip height from the corrected plurality of key points, and performing pitch angle correction on the plurality of first key points to obtain a corrected lip height;
[0179] selecting a plurality of second key points corresponding to the lip width from the corrected plurality of key points, and performing yaw angle correction on the plurality of second key points to obtain a corrected lip width;
[0180] The key point detection result is obtained according to the corrected lip height and the corrected lip width.
[0181] In order to improve the speech detection misjudgment caused by face movement, rotation and other actions, the key points of face detection will be mapped to a standard face that has not been rotated before speaking judgment is made to reduce the impact of face movement.
[0182] In some embodiments, the detection module 604 is configured to: calculate the lip height-to-width ratio of the target object according to the corrected lip height and the corrected lip width, and store the lip height-to-width ratio;
[0183] In combination with the key point detection result, change information corresponding to the lip aspect ratio of the target object in the target image is determined, and based on the change information, it is determined whether the participant corresponding to the target object in the target image is speaking.
[0184] In this way, the timing information of key points is used to enhance the temporal stability of speaker detection and reduce the fluctuation of detection status.
[0185] In some embodiments, the detection module 604 is configured to: set a preset time period; count the number of changes in the lip height-to-width ratio within the preset time period; and determine that the target object is speaking when the number of changes reaches a preset number.
[0186] In this way, the timing information of key points is used to enhance the temporal stability of speaker detection and reduce the fluctuation of detection status.
[0187] In some embodiments, detection module 604 is configured to determine that the target subject is speaking when the number of changes reaches a preset number, including: determining that the number of changes reaches the preset number; obtaining audio data from the video conference; and determining that the target subject is speaking in combination with the audio data from the video conference. In this way, combining the key point detection results with the audio data from the current video conference can further improve the accuracy of speaker identification. As an optional embodiment, when the microphone collecting audio is a dual-channel microphone, the speaker can be located based on the two sets of audio data collected by the two channels, further improving the accuracy of speaker identification.
[0188] In some embodiments, the detection module 604 is configured to: detect the target object in the target image using target detection or target tracking technology, and obtain a detection frame corresponding to the target object;
[0189] The display module 608 is configured to: determine the coordinates of the detection frame corresponding to the target object; determine the coordinates of the sub-screen corresponding to the target object; and translate and / or scale the image corresponding to the detection frame to the sub-screen corresponding to the target object based on the coordinates of the detection frame and the coordinates of the sub-screen.
[0190] By zooming in and out on the target object, all participants can clearly see each other, thus improving the interactivity of the meeting.
[0191] In some embodiments, the display module 608 is configured to: in response to determining that the virtual background function in the sub-screen is turned on, use segmentation technology to segment the target object and the background to obtain a segmentation result; based on the segmentation result, display the virtual background in the sub-screen, filling the technical gap in the related technology of not displaying the virtual background in the sub-screen.
[0192] In some embodiments, the display module 608 is configured to: in response to determining that the virtual background function of the second sub-screen among the at least two sub-screens is turned on, display the virtual background in the second sub-screen (for example, the sub-screen 3304 in Figure 3G), filling the technical gap in the related technology of not displaying the virtual background in the sub-screen.
[0193] In some embodiments, the display module 608 is configured to segment the target object and the background in the target image using semantic segmentation technology to obtain a segmentation result; and display the virtual background in the second sub-screen based on the segmentation result. In this way, the semantic segmentation technology is used to segment the target object and the actual background, thereby effectively achieving the replacement of the virtual background.
[0194] In some embodiments, the display module 608 is configured to: in response to determining that the target object is speaking, display an indicator icon in the sub-screen corresponding to the target object, thereby reminding others that the participant in the sub-screen corresponding to the icon is speaking, thereby improving interactivity.
[0195] For the convenience of description, the above devices are described as being functionally divided into various modules. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.
[0196] The apparatus of the above embodiment is used to implement the corresponding method 400 in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.
[0197] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute method 400 as described in any of the above embodiments.
[0198] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.
[0199] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute method 200 or 400 as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.
[0200] Based on the same inventive concept, corresponding to any of the above-described embodiments of method 200 or 400, the present disclosure further provides a computer program product comprising a computer program. In some embodiments, the computer program is executable by one or more processors to cause the processors to perform method 200 or 400. For each step in each embodiment of method 200 or 400, the processor that executes the step may be a member of the corresponding execution entity.
[0201] The computer program product of the above embodiment is used to enable a processor to execute the method 400 described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.
[0202] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Within the scope of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.
[0203] In addition, to simplify the description and discussion, and so as not to obscure the embodiments of the present disclosure, known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, devices may be shown in the form of block diagrams to avoid obscuring the embodiments of the present disclosure, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure are to be implemented (i.e., these details should be fully within the purview of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that the embodiments of the present disclosure may be implemented without these specific details or with variations in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.
[0204] Although the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.
[0205] The embodiments of the present disclosure are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.
Claims
1. A method for splitting a video conference screen, comprising: Acquire a target image acquired by an acquisition unit; Detecting a target object in the target image; Dividing the video conference screen into at least two sub-screens according to the target object; The target object in the target image is displayed in the at least two sub-pictures correspondingly.
2. The method of claim 1, wherein: Dividing the video conference screen into at least two sub-screens according to the target object further includes: determining whether a target object in the target image is speaking; In response to determining that the target object is speaking, the video conference screen is divided into at least two sub-screens according to a first split-screen mode.
3. The method of claim 2, wherein: Dividing the video conference screen into at least two sub-screens according to the first split-screen mode includes: Enlarging and displaying a first sub-picture of the at least two sub-pictures, wherein the first sub-picture is used to display a target object who is speaking; The other sub-pictures of the at least two sub-pictures are displayed in parallel on at least one side of the first sub-picture.
4. The method of claim 2, wherein: Determining whether a target object in the target image is speaking includes: Detecting key points of the target object; Determining lip height based on key points of the target object; Determining lip width based on key points of the target object; Obtaining a lip height-to-width ratio of the target object according to the lip height and the lip width; Based on the change information of the lip height-to-width ratio, it is determined whether the target object is speaking.
5. The method of claim 4, wherein: Determining whether the target object is speaking based on the change information of the lip height-to-width ratio includes: Set a preset time period; Counting the number of changes in the lip height-to-width ratio within the preset time period; When the number of changes reaches a preset number, it is determined that the target object is speaking.
6. The method of claim 5, wherein: When the number of changes reaches a preset number, determining that the target object is speaking includes: Determining that the number of changes reaches a preset number; Get audio data of the video conference; Combined with the audio data of the video conference, it is determined that the target object is speaking.
7. The method of claim 1, wherein: Detecting the target object in the target image includes: Detecting a target object in the target image using target detection or target tracking technology to obtain a detection frame of the target object; Displaying the target object in the target image in the at least two sub-pictures correspondingly includes: Determine the coordinates of the detection box corresponding to the target object; Determine the coordinates of the sub-picture corresponding to the target object; According to the coordinates of the detection frame and the coordinates of the sub-picture, the image corresponding to the detection frame is translated and / or scaled to the sub-picture corresponding to the target object.
8. The method of claim 1, wherein: Displaying the target object in the target image in the at least two sub-pictures correspondingly further includes: In response to determining that the virtual background function in the sub-picture is turned on, segmenting the target object from the background using a segmentation technology to obtain a segmentation result; According to the segmentation result, a virtual background is displayed in the sub-picture.
9. The method of claim 1, wherein: Displaying the target object in the target image in the at least two sub-pictures correspondingly further includes: In response to determining that the target object is speaking, an indication mark is displayed in the sub-picture corresponding to the target object.
10. A screen splitting device for a video conference, comprising: An acquisition module is configured to: acquire a target image acquired by an acquisition unit; A detection module is configured to: detect a target object in the target image; A division module is configured to: divide the video conference screen into at least two sub-screens according to the target object; The display module is configured to display the target object in the target image in the at least two sub-pictures accordingly.
11. A computer device comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the method according to any one of claims 1 to 9.
12. A non-volatile computer-readable storage medium containing a computer program, which, when executed by one or more processors, causes the processors to perform the method according to any one of claims 1 to 9.
13. A computer program product, comprising computer program instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 9.
Citation Information
Patent Citations
Method, system, server and terminal for displaying speaker in video conference
CN101080000A
Video conference device and video conference management method
CN108933915A
Image processing method, apparatus and system
CN109492506A
Method and system based on portrait segmentation precision improvement, and storage medium
CN113065534A
Picture presentation method, video conference system and readable storage medium
CN113676693A
Cited By
Financial service video post quality inspection method, device, equipment and medium
CN122290225A
A financial service video post-event quality inspection method, device, equipment and medium
CN122290225B
