Conference picture processing method, device, storage medium and system

Abstract

This application relates to a conference screen processing method, device, storage medium, and system. It involves acquiring multiple audio data streams using a data acquisition device, determining the sound source location information based on the first audio data stream where the human voice energy exceeds a first threshold, acquiring the image corresponding to the sound source location information, identifying the speaker in the image whose human voice energy meets preset conditions, and obtaining the facial deflection angle between a preset position representing the speaker's face and the data acquisition device. The speaker's image and facial deflection angle are then sent to a data processing device. The data processing device can display the image of the speaker with the smallest facial deflection angle using a display device. This eliminates the need for the data processing device to expend significant computing resources for image recognition, reducing its computational resource consumption and improving the efficiency of displaying the speaker's face at the conference.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to a method, apparatus, storage medium, and system for processing conference screens. Background Technology

[0002] With the development of technology, video conferencing is being used more and more widely, and participants have increasingly higher requirements for the video conferencing experience.

[0003] Currently, in order to make it easier for participants to observe the speaker's expressions, movements and other subtle gestures, cameras and other equipment are usually focused on the speaker during video conferences, which is known as speaker face tracking.

[0004] Existing speaker face tracking technology typically involves setting up multiple cameras at the meeting venue, and then using video processing equipment to analyze (e.g., merge, analyze) the videos captured by these cameras to determine the video that most closely resembles the speaker's face for display.

[0005] However, the above methods have high performance requirements for video processing equipment, and the video processing equipment needs to spend a lot of computing resources to analyze and process the video, resulting in low tracking efficiency. Summary of the Invention

[0006] This application provides a method, device, storage medium, and system for processing conference images, which can improve the efficiency of displaying the face of the conference speaker.

[0007] In a first aspect, embodiments of this application provide a conference screen processing method, applied to a acquisition device in a conference system. The conference system includes a display device, a data processing device, and multiple acquisition devices; the data processing device is connected to the display device and each acquisition device. The conference screen processing method includes:

[0008] Collect multiple audio data streams and determine the sound source location information of the first audio data stream; the first audio data stream is the audio data stream in which the human voice sound energy is greater than a first threshold.

[0009] Obtain the image corresponding to the sound source location information;

[0010] Identify the speaker in the image; the speaker is the person in the image whose voice meets the preset conditions.

[0011] Determine the face deflection angle; the face deflection angle is used to characterize the angle between the preset position of the main person's face in the image and the acquisition device;

[0012] The presenter's image and face deflection angle are sent to the data processing device. The face deflection angle is used by the data processing device to determine the presenter's image to be displayed and drive the display device to display the presenter's image to be displayed. The presenter's image to be displayed is the presenter's image with the smallest face deflection angle determined by the data processing device.

[0013] Optionally, the speaker in the aforementioned image includes:

[0014] Identify the speaker in the image;

[0015] If there is a speaker in the image, determine that speaker as the main speaker;

[0016] When there are at least two speakers in the image, determine the voice energy value of each speaker and the distance between each speaker and the acquisition device; determine the weight value of the corresponding speaker based on the determined distance; determine the weighted voice energy value of each speaker based on the determined weight value and voice energy value; determine the speaker whose weighted voice energy value meets the preset conditions as the main speaker; the weighted voice energy value is used to characterize the voice energy.

[0017] In this embodiment, by identifying the speaker in the image, if there is only one speaker in the image, that speaker is taken as the main speaker. If there are at least two speakers in the image, a weighted speech energy value is calculated based on the speaker's speech energy value and the distance between the speaker and the acquisition device. The main speaker is then determined based on the weighted speech energy value, thereby improving the accuracy of the main speaker identification.

[0018] Optionally, determining the face deflection angle as described above includes:

[0019] The image is input into a pre-trained face deflection angle acquisition model to determine the face deflection angle.

[0020] In this embodiment of the application, the face deflection angle is obtained by inputting the image into the pre-trained face deflection angle acquisition model.

[0021] Optionally, sending the face deflection angle to the data processing device includes:

[0022] Create a custom frame;

[0023] Write the face deflection angle into a custom frame;

[0024] Send a custom frame to the data processing device.

[0025] In this embodiment of the application, a custom frame is created and the facial deflection angle is written into the custom frame. The custom frame can be transmitted along with the speaker's image, which is more convenient and improves transmission efficiency.

[0026] Optionally, sending the speaker's image to the data processing device includes:

[0027] The obtained image is scaled and cropped to control the speaker's face image to be located in a preset area in order to obtain the speaker image;

[0028] Send the speaker's image to the data processing equipment.

[0029] In this embodiment of the application, by scaling and cropping the image, the speaker's face image is controlled to be located in a preset area of ​​the speaker's image screen. When the display device is driven to display the speaker's image, the speaker's face can be displayed better, improving the viewing experience.

[0030] Secondly, embodiments of this application provide a conference screen processing method applied to a data processing device in a conference system. The conference system includes a display device, a data processing device, and multiple acquisition devices; the data processing device is connected to the display device and each of the multiple acquisition devices; the conference screen processing method includes:

[0031] The speaker's image and face deflection angle are acquired from multiple acquisition devices; the face deflection angle is the angle between the preset position of the speaker's face determined by the corresponding acquisition device and the corresponding acquisition device; the speaker is determined by the corresponding acquisition device based on the acquired audio data and the image corresponding to the audio data.

[0032] From the received speaker images, determine the speaker image to be displayed. The speaker image to be displayed is the one with the smallest face deflection angle. The speaker image to be displayed is used for display on the display device.

[0033] Optionally, the meeting screen processing methods also include:

[0034] In response to the display device's call command to the data processing device, the image of the speaker to be displayed is sent to the display device.

[0035] In this embodiment of the application, the display device can obtain the image of the speaker to be displayed by calling the data processing device, thereby realizing the display of the image of the speaker to be displayed and reducing the amount of data transmission.

[0036] Optionally, determining the speaker image to be displayed from the received speaker image includes:

[0037] Obtain the face deflection angle of each acquisition device within a preset time period, determine the average face deflection angle of each acquisition device within the preset time period, and use the speaker image sent by the acquisition device with the smallest average value as the speaker image to be displayed.

[0038] or,

[0039] Obtain a first device identifier and a second device identifier. The first device identifier is used to identify the acquisition device with the smallest face deflection angle in the image frame to be displayed, and the second device identifier is used to identify the acquisition device of the currently displayed image frame. If the first device identifier and the second device identifier are different, obtain the face deflection angle in the currently displayed image frame. If the difference between the smallest face deflection angle in the image frame to be displayed and the face deflection angle in the currently displayed image frame is greater than a target threshold, use the speaker image acquired by the acquisition device corresponding to the first device identifier as the speaker image to be displayed. If the difference between the smallest face deflection angle in the image frame to be displayed and the face deflection angle in the currently displayed image frame is less than or equal to the target threshold, use the speaker image acquired by the acquisition device corresponding to the second device identifier as the speaker image to be displayed.

[0040] In this embodiment, the speaker image sent by the acquisition device with the smallest average face deflection angle within a preset time period is used as the speaker image to be displayed; or, the speaker image to be displayed is determined by combining the device identifier and the difference between the face deflection angle, which effectively avoids the problem of frequent switching of speaker images and improves the viewing experience.

[0041] Thirdly, embodiments of this application provide a conference screen processing method applied in a conference system. The conference system includes a display device, a data processing device, and multiple acquisition devices, wherein the data processing device is connected to the display device and each acquisition device. The corresponding conference screen processing method includes:

[0042] Each acquisition device acquires multiple audio data streams and determines the sound source location information of the first audio data stream; the first audio data stream is the audio data stream where the human voice energy is greater than a first threshold; the image corresponding to the sound source location information is acquired, and the speaker in the image is identified, where the speaker is the speaker whose human voice energy in the image meets the preset conditions; the face deflection angle is determined; the face deflection angle is used to characterize the angle between the preset position of the speaker's face in the image and the acquisition device; the speaker's image and face deflection angle are sent to the data processing device.

[0043] The data processing device receives the speaker image and face deflection angle sent by each acquisition device, and determines the speaker image to be displayed from the received speaker images. The speaker image to be displayed is the speaker image with the smallest face deflection angle.

[0044] The display device displays the image of the speaker to be displayed.

[0045] Fourthly, embodiments of this application provide a conference screen processing apparatus, applied to a acquisition device in a conference system. The conference system includes a display device, a data processing device, and multiple acquisition devices; the data processing device is connected to the display device and each acquisition device; the apparatus includes:

[0046] The sound source location information determination module is used to collect multiple audio data and determine the sound source location information of the first audio data; the first audio data is the audio data of human voice with a sound energy greater than a first threshold among the multiple audio data;

[0047] The image acquisition module is used to acquire images corresponding to the sound source location information;

[0048] The speaker determination module is used to determine the speaker in the image. The speaker is the person in the image whose voice meets the preset conditions.

[0049] The face deflection angle determination module is used to determine the face deflection angle; the face deflection angle is used to characterize the angle between the preset position of the main face in the image and the acquisition device;

[0050] The data transmission module is used to send the speaker's image and face deflection angle to the data processing device. The face deflection angle is used by the data processing device to determine the speaker's image to be displayed and drive the display device to display the speaker's image to be displayed. The speaker's image to be displayed is the speaker's image with the smallest face deflection angle determined by the data processing device.

[0051] Fifthly, embodiments of this application provide a conference screen processing apparatus, applied to a data processing device in a conference system. The conference system includes a display device, a data processing device, and multiple acquisition devices; the data processing device is connected to the display device and each of the multiple acquisition devices; the apparatus includes:

[0052] The data receiving module is used to acquire images of the speaker and facial deflection angles from multiple acquisition devices; the facial deflection angle is the angle between the preset position of the speaker's face determined by the corresponding acquisition device and the corresponding acquisition device; the speaker is determined by the corresponding acquisition device based on the acquired audio data and the image corresponding to the audio data;

[0053] The speaker image determination module is used to determine the speaker image to be displayed from the received images. The speaker image to be displayed is the speaker image with the smallest face deflection angle. The speaker image to be displayed is used for display on the display device.

[0054] Sixthly, embodiments of this application provide a data acquisition device, including a memory, a processor, and a computer program stored in the memory and executable by the processor;

[0055] When a processor executes a computer program, it implements the steps of the conference screen processing method described in any of the above.

[0056] In a seventh aspect, embodiments of this application provide a data processing apparatus, including a memory, a processor, and a computer program stored in the memory and executable by the processor;

[0057] When a processor executes a computer program, it implements the steps of the conference screen processing method described in any of the above.

[0058] Eighthly, embodiments of this application provide a conference system, including a display device, a data processing device, and multiple acquisition devices; the data processing device is connected to the display device and each acquisition device.

[0059] Each acquisition device is used to acquire multiple audio data streams and determine the sound source location information of the first audio data; the first audio data is the audio data in the multiple audio data streams where the human voice sound energy is greater than a first threshold; acquire the image corresponding to the sound source location information, determine the speaker in the image, the speaker is the speaker whose human voice sound energy in the image meets the preset conditions; determine the face deflection angle; the face deflection angle is used to characterize the angle between the preset position of the speaker's face in the image and the corresponding acquisition device; send the speaker image and face deflection angle to the data processing device;

[0060] The data processing device is used to receive the speaker image and face deflection angle sent by each acquisition device, and to determine the speaker image to be displayed from the received speaker images. The speaker image to be displayed is the speaker image with the smallest face deflection angle.

[0061] Display device used to display the image of the speaker to be displayed.

[0062] Ninthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the conference screen processing method as described in any of the preceding claims.

[0063] In this embodiment, the acquisition device acquires audio data and corresponding images, and determines the speaker's (the speaker whose voice meets preset conditions) facial deflection angle (the angle between the speaker's preset face position and the acquisition device) based on the audio data and images. It then sends the speaker's image and facial deflection angle to the data processing device. The data processing device then selects the speaker image with the smallest facial deflection angle from the received facial deflection angle and speaker image and drives the display device to display it. It can be seen that the data processing device does not need to expend significant computing resources for image recognition, thus reducing the computational resource usage of the data processing device and improving data processing efficiency.

[0064] To better understand and implement this application, the following detailed description is provided in conjunction with the accompanying drawings. Attached Figure Description

[0065] Figure 1 This is a schematic diagram illustrating an application scenario of the conference screen processing method in this application embodiment;

[0066] Figure 2 This is a flowchart of the meeting screen processing method in the embodiments of this application;

[0067] Figure 3 This is a schematic diagram of the face deflection angle in an embodiment of this application;

[0068] Figure 4 This is a flowchart of step S130 in an embodiment of this application;

[0069] Figure 5 This is a flowchart of step S150 in another embodiment of this application;

[0070] Figure 6 This is a flowchart of a conference screen processing method in another embodiment of this application;

[0071] Figure 7 This is a flowchart of a conference screen processing method in another embodiment of this application;

[0072] Figure 8 This is a schematic diagram of the structure of a conference screen processing device according to an embodiment of this application;

[0073] Figure 9 This is a schematic diagram of the structure of a conference screen processing device in another embodiment of this application;

[0074] Figure 10 This is a schematic diagram of the structure of a data acquisition device according to an embodiment of this application;

[0075] Figure 11 This is a schematic diagram of the structure of the data processing device in the embodiments of this application;

[0076] Figure 12 This is a schematic diagram of the structure of a conference system according to an embodiment of this application. Detailed Implementation

[0077] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0078] It should be understood that the described embodiments are merely some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of the embodiments of this application.

[0079] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the embodiments of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0080] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of devices and methods consistent with some aspects of this application as detailed in the appended claims. In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0081] Furthermore, in the description of this application, unless otherwise stated, "several" refers to two or more. "And / or" describes the correspondence between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0082] Existing speaker face tracking technology typically involves setting up multiple cameras at the meeting venue, and then using video processing equipment to analyze and process the videos captured by these cameras to determine the video that most closely resembles the speaker's face for display.

[0083] However, the above methods have high performance requirements for video processing equipment, and the video processing equipment needs to spend a lot of computing resources to analyze and process the video, resulting in low tracking efficiency.

[0084] Therefore, to address the aforementioned problems, this application provides a method for processing conference video feeds. This method involves processing multiple audio and image streams collected by a data acquisition device to determine the image and face angle (the angle between the preset position of the speaker's face and the data acquisition device) of the corresponding speaker (a speaker whose voice meets preset conditions). The method then sends the speaker's image and face angle to a data processing device. The data processing device can then select the speaker's image with the smallest face angle from the received face angle and image data and drive the display device to display it. Compared to existing solutions, this method eliminates the need for extensive processing by the data processing device, saving computational resources and improving data processing efficiency.

[0085] The conference screen processing method provided in this application is applied to a conference system. The conference system includes a display device, a data processing device, and multiple acquisition devices. The data processing device is connected to the display device and each acquisition device. The data processing device and each acquisition device can be directly or indirectly connected via wired or wireless communication, and this application does not impose any limitations on this connection. Similarly, the data processing device and the display device can be directly or indirectly connected via wired or wireless communication, and this application does not impose any limitations on this connection.

[0086] Optionally, the data processing device and the display device can be integrated together or they can be independently configured devices; this application embodiment does not specifically limit this. Similarly, the data processing device and the acquisition device can be integrated together or they can be independently configured devices; this application embodiment does not specifically limit this.

[0087] The display device can be a computer, a smart interactive whiteboard, a mobile phone, or any other device that can be used to display the speaker's image.

[0088] The data processing device is used to acquire the speaker image and face deflection angle sent by at least one acquisition device, determine the image with the smallest face deflection angle from the received speaker images, and drive the display device to display the image.

[0089] The data processing device can be a controller or device with the above functions, and this application embodiment does not specifically limit it.

[0090] The data acquisition device may include an audio acquisition device and an image acquisition device, or it may be an electronic device that integrates audio and image acquisition functions (such as a smart interactive whiteboard, smart speaker, etc.). The audio acquisition device is used to acquire multiple audio data streams. The audio acquisition device may be a microphone array or other devices used to acquire multiple audio data streams; this application does not specifically limit this. The microphone array is an array composed of multiple microphones with identical characteristics. The image acquisition device is used to acquire facial images of attendees. The image acquisition device may be a camera or other devices used to acquire images. The data acquisition device is used to acquire audio data and images, and to process the audio data and images.

[0091] Optionally, the above-mentioned conference system can be applied to conference scenarios to quickly acquire the frontal video data of the speaker in the conference and realize the tracking of the speaker's frontal face; it can also be applied to teaching scenarios to quickly acquire the frontal video data of the speaker (such as teacher or student) in the teaching process and realize the tracking of the speaker's frontal face; it can also be applied to other scenarios, which are not specifically limited in this application embodiment.

[0092] To facilitate understanding, the following explanation uses a meeting scenario as an example.

[0093] like Figure 1 As shown in (A), the meeting scenario involves five attendees. Four data acquisition devices (Acquisition Device 1-Acquisition Device 4) are deployed around the perimeter of the meeting room. The data processing device is positioned above or below the conference table, and the display device is mounted on the wall. Each acquisition device can acquire audio and image data from multiple speakers (the attendees). Based on the acquired audio and image data, it identifies the speaker as attendee A (the speaker whose voice meets preset conditions) and determines the facial deflection angle (the angle between attendee A's preset face position and the acquisition device). It then sends the speaker's image and facial deflection angle to the data processing device. The data processing device selects the speaker's image with the smallest facial deflection angle from all received facial deflection angles and images and drives the display device to display it.

[0094] For example, if all three acquisition devices (1-4) determine that the speaker is participant A, then each device will determine the angle between participant A and its respective location, and send the image of participant A and its facial deflection angle to the data processing device. If the facial deflection angle determined by acquisition device 4 is the smallest (which can be understood as the face of participant A captured by acquisition device 4 being closer to a frontal view), then the data processing device will use the image of participant A captured by acquisition device 4 as the image to be displayed and drive the display device to display it.

[0095] It is understood that, in addition to the examples above, the data acquisition device can also be a mobile device that is connected to a data processing device and has data acquisition capabilities, such as a tablet computer, mobile phone, or laptop computer.

[0096] For example, such as Figure 1 As shown in (B), this meeting scenario involves five attendees. Five data acquisition devices (Acquisition Device 1-Acquisition Device 5) are deployed around the perimeter of the meeting room, with the data processing device positioned above or below the conference table. Acquisition Device 1 and Acquisition Device 5 can also be equipped with display functions. Each acquisition device can acquire audio and image data from multiple speakers (the attendees). Based on the acquired audio and image data, it determines the image and facial deflection angle (the angle between the preset position of the face and the acquisition device) of the main speaker (the speaker whose voice meets preset conditions) and sends the main speaker's image and facial deflection angle to the data processing device. The data processing device selects the main speaker's image with the smallest facial deflection angle from all received facial deflection angles and images and drives the display device to display it. In this scenario, both Acquisition Device 1 and Acquisition Device 5 can display the data.

[0097] In combination with the above Figure 1 ,like Figure 2 As shown, this application provides a method for processing conference video feeds, applied to a capture device in a conference system. The method specifically includes:

[0098] S110: Collect multiple audio data streams and determine the sound source location information of the first audio data stream.

[0099] The first audio data is the audio data of human voice with a sound energy greater than a first threshold among multiple audio data. The first threshold can be set according to the actual application scenario.

[0100] Audio data is data converted from collected sound. Audio data can be categorized into different types based on the timbre of the sound, and also into different frequencies based on the pitch of the sound. In this embodiment, the multi-channel audio data refers to audio data distinguished according to the type of timbre; different channels of audio data have different timbre types.

[0101] Understandably, the acquisition device can determine the number of speakers based on different types of audio data, and can also accurately determine the distance between the speakers and the acquisition device based on audio data of different frequencies.

[0102] Specifically, after acquiring multiple audio data streams, the acquisition device determines the vocal energy of each audio stream and identifies audio data with a vocal energy greater than a first threshold as the first audio data. After determining the first audio data, the device then determines the sound source location information of that audio data.

[0103] Human voice acoustic energy refers to the sound energy emitted by a human voice. Based on human voice acoustic energy, the volume of the human voice in various audio data streams can be determined. Sound source location information can include information such as the direction and distance of the sound source.

[0104] Optionally, after acquiring multiple audio data, the acquisition device can also perform noise reduction processing on the multiple audio data to filter out ambient noise such as keyboard sounds and chair movement sounds, so that the multiple audio data can better preserve human voices and improve the accuracy of sound source localization.

[0105] Acquisition devices can determine the magnitude of human voice energy using voice energy values, which can be used to describe changes in sound signal intensity. When calculating voice energy values, audio data can be divided into several frames, and the voice energy value of each frame can be calculated based on the frame length and the signal value of each frame.

[0106] When determining the first audio data, the acquisition device may use audio data with a voice energy value greater than a first threshold as the first audio data.

[0107] When there are many audio data points with voice energy values ​​greater than the first threshold, the acquisition device can sort the audio data points from largest to smallest according to their voice energy values. The top N audio data points with the highest voice energy values ​​are then selected as the first audio data point for sound source localization, thereby reducing the amount of data to be calculated and improving the efficiency of sound source localization. Here, N is a natural number greater than 1, and N is less than the number of audio data points.

[0108] In this embodiment, N can be 3. The acquisition device selects the top 3 audio data points in terms of speech energy value as the first audio data. When locating the sound source from the first audio data, a direction of arrival (DOA) estimation algorithm can be used to obtain the sound source location information. The DOA estimation algorithm estimates the location of the sound source by measuring the time difference of the signal arriving at different microphones.

[0109] It should be noted that when the meeting venue is relatively quiet, resulting in only N or fewer audio data being picked up from multiple audio sources, the sound source localization is performed based on the specific N or fewer audio data points picked up.

[0110] Optionally, the acquisition device can perform sound source localization on the first audio data based on any sound source localization algorithm, thereby determining the sound source location information of the first audio data.

[0111] S120: Obtain the image corresponding to the sound source location information.

[0112] Specifically, the acquisition device can adjust the angle of the image acquisition device according to the sound source location information, thereby realizing the acquisition of the image corresponding to the sound source location information.

[0113] It is understandable that multiple images captured by the acquisition device can be combined to form a video stream.

[0114] S130: Identify the speaker in the image.

[0115] The speaker is the person whose voice energy in the image meets preset conditions. Optionally, the preset condition can be that the speaker with the highest voice energy in the image is the speaker.

[0116] In practical applications, the images captured by the acquisition device may include one or more people, and some or all of these people may be speaking (this can be determined by observing the user's lip movements in the image, which will not be elaborated here). To more accurately identify the speaker, when multiple people are in the image, the acquisition device can determine the speaker by combining the speaking status of the people in the image with the speaker's voice energy value; that is, by determining the vocal energy of each speaker in the image. When a speaker's vocal energy meets preset conditions, that speaker is identified as the speaker.

[0117] For example, the image captured by acquisition device 1 includes user A, user B, and user C. Since both user A and user B are speaking, acquisition device 1 determines the vocal power of user A and user B. If the vocal power of user A is greater than that of user B, then user A is identified as the main speaker.

[0118] S140: Determine the face deflection angle.

[0119] After the acquisition device identifies the speaker, it determines the angle between the preset position of the speaker's face and the acquisition device, that is, the face deflection angle.

[0120] The preset position of the speaker's face can be determined according to the display requirements of the speaker's face. Optionally, in this embodiment, the preset position of the face can be the midpoint of the line connecting the speaker's two eyes, the line of symmetry of the speaker's face, or the speaker's mouth. This embodiment does not specifically limit this.

[0121] For example, such as Figure 3 As shown, the default position of the speaker's face is the speaker's nose. For example... Figure 3 As shown in (a), the face is directly facing the data acquisition device, and the face deflection angle is 0°. Figure 3 As shown in (b), the speaker's face is deflected relative to the acquisition device, and this deflection angle can be 90°. Figure 3As shown in (c), the speaker's face is deflected in another direction relative to the acquisition device, and the face deflection angle can be -90°.

[0122] The position of the speaker's entire face in the image can be determined by using facial recognition technology to identify the positions of facial features such as eyes, lips, and ears.

[0123] S150: Send the speaker's image and face deflection angle to the data processing device.

[0124] After determining the face deflection angle, the acquisition device sends the face deflection angle and the speaker's image to the data processing device. The speaker's image can be an image acquired by the acquisition device, or it can be an image obtained by scaling and cropping the acquired image.

[0125] The presenter's image can be a single image frame from the video stream captured by the acquisition device.

[0126] The data processing device can determine the image sent by the acquisition device that meets the requirements (i.e., determine the image of the speaker to be displayed) based on the facial deflection angle, and drive the display device to display the image of the speaker to be displayed. For example, it can determine whether the face in the image is a frontal face based on the facial deflection angle, thereby driving the display device to display the image of the speaker with a frontal face, thus realizing the display of the speaker's frontal face.

[0127] In this embodiment, multiple audio data are collected using a data acquisition device. The sound source location information is determined based on the first audio data in the multiple audio data where the human voice energy is greater than a first threshold. The image corresponding to the sound source location information is obtained. By determining the speaker in the image whose human voice energy meets the preset conditions, the face deflection angle between the preset position representing the speaker's face and the data acquisition device is obtained. By sending the speaker image and the face deflection angle to the data processing device, the data processing device can select the speaker image with the smallest face deflection angle and drive the display device to display it. The data processing device does not need to spend a lot of computing resources to perform image recognition, which can reduce the computing resource occupation of the data processing device and improve the efficiency of the speaker's face display in the meeting.

[0128] like Figure 4 As shown, in one embodiment, determining the speaker in the image includes:

[0129] S131: Identify the speaker in the image.

[0130] When identifying the speaking state of a person in an image, the speaking state can be determined by recognizing features such as lip movements and facial expressions, and the speaker can be identified based on the speaking state.

[0131] S132: If there is a speaker in the image, determine that speaker is the main speaker.

[0132] S132: When there are at least two speakers in the image, determine the speech energy value of each speaker and the distance between each speaker and the acquisition device; determine the weight value of the corresponding speaker based on the determined distance; determine the weighted speech energy value of each speaker based on the determined weight value and speech energy value; determine the speaker whose weighted speech energy value meets the preset conditions as the main speaker; the weighted speech energy value is used to characterize the human voice energy.

[0133] When there are at least two speakers in the image, a weighted speech energy value can be calculated by combining the speaker's speech energy value and the distance between the speaker and the conference equipment. The speaker can then be identified based on the weighted speech energy value, thus improving the accuracy of speaker recognition.

[0134] The distance between the speaker and the acquisition device can be obtained using distance measuring devices or other devices that can be used to detect distance.

[0135] In this embodiment of the application, the acquisition device includes an image acquisition device, which can be a TOF (Time of Flight) camera or other cameras with distance detection function. The TOF camera calculates the distance by measuring the time it takes for light to travel from emission to reflection from an object. The distance between the speaker and the acquisition device can be obtained by the TOF camera.

[0136] The speaker's voice energy level can be detected by voice acquisition equipment set up at the conference venue.

[0137] In this embodiment, the greater the distance, the larger the corresponding weight value. The correspondence between distance and weight value can be set according to actual application.

[0138] The weighted speech energy value is the speech energy value obtained by weighting the speech energy value according to the weight values.

[0139] When determining the speaker whose weighted speech energy value meets the preset conditions, the speaker with the highest weighted speech energy value can be selected as the main speaker.

[0140] Suppose there are two speakers, A and B, in the image. Speaker A has a speech energy value of M1, speaker B has a speech energy value of N1, speaker A has a weight value of X, and speaker B has a weight value of Y.

[0141] Then the weighted speech energy value M2 = M1 * X corresponding to speaker A, and the weighted speech energy value N2 = N1 * X corresponding to speaker A.

[0142] When M2 > N2, speaker A is determined to be the main speaker; when M2 < N2, speaker B is determined to be the main speaker.

[0143] In this embodiment, by identifying the speaker in the image, if there is only one speaker in the image, that speaker is taken as the main speaker. If there are at least two speakers in the image, a weighted speech energy value is calculated based on the speaker's speech energy value and the distance between the speaker and the acquisition device. The main speaker is then determined based on the weighted speech energy value, thereby improving the accuracy of the main speaker identification.

[0144] In one embodiment, a pre-trained face deflection angle model can be used to obtain the face deflection angle. Specifically, determining the face deflection angle includes:

[0145] The image is input into a pre-trained face deflection angle acquisition model to determine the face deflection angle.

[0146] The face deflection angle acquisition model can be a model built based on existing deep learning algorithms such as neural network algorithms. This model can be used to analyze the face deflection angle of a face in an image, thereby obtaining the face deflection angle.

[0147] The face rotation angle acquisition model can be obtained by pre-training a large number of face images at different angles. The training samples can include face images from different angles, such as left side face, right side face, back face, and front face.

[0148] In this embodiment of the application, the face deflection angle is obtained by inputting the image into the pre-trained face deflection angle acquisition model.

[0149] In step S150, the acquisition device can send the face deflection angle and the acquired speaker image together to the data processing device.

[0150] like Figure 5 As shown, in one embodiment, sending a face deflection angle to a data processing device includes:

[0151] S151: Create a custom frame.

[0152] S152: Write the face deflection angle into a custom frame.

[0153] S153: Send a custom frame to the data processing device.

[0154] A custom frame can be a video frame defined by the user. The parameters of the custom frame can be the same as those of the video frame containing the speaker's image, which makes it convenient to transmit the custom frame and the speaker's image together and improves transmission efficiency.

[0155] Optionally, the custom frame may also include a facial image of the speaker. This facial image can be used by the data processing device to drive the display device to show the speaker's face in case of abnormal image transmission or other situations, thus improving the user experience.

[0156] In this embodiment of the application, a custom frame is created and the facial deflection angle is written into the custom frame. The custom frame can be transmitted along with the speaker's image, which is more convenient and improves transmission efficiency.

[0157] In one embodiment, sending a speaker image to a data processing device includes:

[0158] The acquired image is scaled and cropped to control the speaker's face image to be located in a preset area in order to obtain the speaker's image.

[0159] Send the speaker's image to the data processing equipment.

[0160] The preset area can be the center area of ​​the speaker's image.

[0161] The speaker's face is positioned in a preset area within the speaker's image. This allows for a better display of the speaker's face and improves the viewing experience.

[0162] In this embodiment of the application, by scaling and cropping the image, the speaker's face image is controlled to be located in a preset area of ​​the speaker's image screen. When the display device is driven to display the speaker's image, the speaker's face can be displayed better, improving the viewing experience.

[0163] Optionally, the conference screen processing method in this application embodiment can be executed periodically according to a set time period. By periodically sending the face deflection angle to the data processing device, the tracking of the speaker can be satisfied while reducing the data processing load of the acquisition device, improving the performance of the acquisition device, and avoiding frequent switching of conference screens from affecting the user experience.

[0164] Please see Figure 6 This is another embodiment of the conference screen processing method in this application. The method can be applied to a data processing device in a conference system. The conference system includes a display device, a data processing device, and multiple acquisition devices; the data processing device is connected to the display device and each acquisition device; the conference screen processing method includes:

[0165] S210: Acquire the speaker's image and face deflection angle from multiple acquisition devices; the face deflection angle is the angle between the preset position of the speaker's face determined by the corresponding acquisition device and the acquisition device; the speaker is determined by the corresponding acquisition device based on the acquired audio data and the image corresponding to the audio data.

[0166] The speaker's image and face rotation angle can be obtained by the acquisition device through the conference screen processing method described in the above embodiments. The acquisition method can be found in the above embodiments and will not be repeated here.

[0167] S220: Determine the speaker image to be displayed from the received speaker image; the speaker image to be displayed is the speaker image with the smallest face deflection angle, and the speaker image to be displayed is used for display on the display device.

[0168] The speaker image to be displayed is used to display on the display device. The speaker image to be displayed can be used to show the speaker's face at the conference.

[0169] In one embodiment, the data processing device is communicatively connected to a terminal device, and the method further includes the following steps:

[0170] In response to the display device's call command to the data processing device, the image of the speaker to be displayed is sent to the display device.

[0171] Data processing devices can communicate with display devices wirelessly via networks or other wireless methods, or they can communicate with display devices via wired methods such as USB cables. Compared to wireless connections, wired connections can better ensure the stability and speed of data transmission.

[0172] Optionally, the data processing device can also be used to control at least one acquisition device to perform an operation corresponding to the control command, based on the control command of the display device.

[0173] The correspondence between control commands and operations performed by the acquisition device can be determined by the human-machine interaction protocol pre-stored in the data processing device.

[0174] In this embodiment of the application, the human-computer interaction protocol can be the HID (Human Interface Device) protocol pre-input by the user. The HID protocol is used to realize communication and data exchange between two devices.

[0175] In this embodiment of the application, the display device can control the acquisition device based on a human-computer interaction protocol. For example, the sensitivity of the image acquisition device's screen switching can be adjusted.

[0176] In another embodiment, the data processing device may also transmit the image of the speaker to be displayed only after receiving the calling instruction from the display device. Specifically, the method further includes the following steps:

[0177] In response to a call command from the display device to the data processing device, the image of the speaker to be displayed is sent to the display device. Optionally, the data processing device may generate a virtual camera device identifier on the display device, thereby enabling the data processing device to invoke the data processing device by triggering the virtual camera device identifier.

[0178] A virtual camera device identifier can be a virtual camera device configured by a data processing device through software. This virtual camera device identifier can be invoked by a display device to display the target video.

[0179] When using some conferencing software, display devices typically need to select a camera for video display. This application generates a virtual camera device identifier on the display device, allowing users to trigger the virtual camera device identifier on the display device just like using any other camera, thereby calling the data processing device without requiring additional learning of how to call the data processing device, reducing the difficulty of use and improving the user experience.

[0180] In this embodiment of the application, the display device can obtain the image of the speaker to be displayed by calling the data processing device, thereby realizing the display of the image of the speaker to be displayed and reducing the amount of data transmission.

[0181] In this embodiment, the speaker image can be real-time transmitted data, and the facial rotation angle in the video data can change as the speaker moves. When determining the speaker image to be displayed, the speaker image with the smallest facial rotation angle can always be selected as the speaker image to be displayed, based on the facial rotation angle.

[0182] In practical applications, due to the complexity of the meeting environment, when multiple people compete to speak, the frequent changes in the main speaker can lead to frequent switching, which affects the viewing experience.

[0183] Therefore, in one embodiment, determining the speaker image to be displayed from the received speaker image includes:

[0184] Obtain the face deflection angle of each acquisition device within a preset time period, determine the average face deflection angle of each acquisition device within the preset time period, and use the speaker image sent by the acquisition device with the smallest average value as the speaker image to be displayed.

[0185] or,

[0186] Obtain a first device identifier and a second device identifier. The first device identifier is used to identify the acquisition device with the smallest face deflection angle in the image frame to be displayed, and the second device identifier is used to identify the acquisition device of the currently displayed image frame. If the first device identifier and the second device identifier are different, obtain the face deflection angle in the currently displayed image frame. If the difference between the smallest face deflection angle in the image frame to be displayed and the face deflection angle in the currently displayed image frame is greater than a target threshold, use the speaker image acquired by the acquisition device corresponding to the first device identifier as the speaker image to be displayed. If the difference between the smallest face deflection angle in the image frame to be displayed and the face deflection angle in the currently displayed image frame is less than or equal to the target threshold, use the speaker image acquired by the acquisition device corresponding to the second device identifier as the speaker image to be displayed.

[0187] In this embodiment, the speaker image to be displayed can be the speaker image with the smallest facial deviation value. The facial deflection angle can be an absolute value; the smaller the facial deflection angle, the closer the speaker's face is to a frontal view.

[0188] For example, suppose the current minimum frontal face deviation value is ang1, corresponding to acquisition device 1, and the previous minimum frontal face deviation value is ang2, corresponding to acquisition device 2. If the difference between ang1 and ang2 is greater than the target threshold, the speaker image from acquisition device 1 is selected. If the difference between ang1 and ang2 is less than or equal to the target threshold, the speaker image from conference device 2 is selected. This avoids the problem of poor viewing experience caused by frequent switching of speaker images.

[0189] In this embodiment, by obtaining the average face deflection angle of each acquisition device within a preset time period, the speaker image of the acquisition device with the smallest average value is selected as the speaker image to be displayed; or, the current minimum face deflection angle is compared with the previous minimum face deflection angle, and the speaker image of the acquisition device corresponding to the current minimum face deflection angle is selected as the speaker image to be displayed only when the difference between the current face deflection angle and the previous face deflection angle is greater than a target threshold, thereby avoiding the problem of frequent switching of speaker images and improving the viewing experience.

[0190] Please see Figure 7 This application also provides a conference screen processing method for use in a conference system. The conference system includes a display device, a data processing device, and multiple acquisition devices, wherein the data processing device is connected to the display device and each acquisition device. The conference screen processing method includes:

[0191] S310: Each acquisition device acquires multiple audio data streams and determines the sound source location information of the first audio data stream; the first audio data stream is the audio data stream where the human voice energy is greater than a first threshold; acquires the image corresponding to the sound source location information and determines the speaker in the image; determines the face deflection angle; the face deflection angle is used to characterize the angle between the preset position of the speaker's face in the image and the acquisition device; and sends the speaker image and face deflection angle to the data processing device.

[0192] The speaker is the person whose voice in the image meets the preset conditions.

[0193] The specific processing procedure of the data acquisition equipment can be referred to the above description, and will not be elaborated on here.

[0194] S320: The data processing device receives the speaker image and face deflection angle sent by each acquisition device, and determines the speaker image to be displayed from the received speaker image.

[0195] The image of the speaker to be displayed is the image of the speaker with the smallest facial deflection angle. The specific processing procedure of the data processing device can be found in the description above, and will not be elaborated upon here.

[0196] S330: The display device displays the image of the speaker to be displayed.

[0197] In this embodiment, the acquisition device processes the acquired multi-channel audio data and images to determine the image and face deflection angle (the angle between the preset position of the speaker's face and the acquisition device) of the corresponding speaker (a speaker whose voice meets preset conditions). The speaker's image and face deflection angle are then sent to the data processing device. In this way, the data processing device can select the speaker's image with the smallest face deflection angle from the received face deflection angles and images and drive the display device to display it. Compared to existing solutions, the data processing device does not need to perform a large number of processing operations, saving computing resources and improving data processing efficiency.

[0198] For technical details not disclosed in the embodiments of this application, please refer to the description of the above embodiments.

[0199] Please see Figure 8 This application also provides a conference screen processing device, which is applied to the acquisition device in a conference system. The conference system includes a display device, a data processing device, and multiple acquisition devices. The data processing device is connected to the display device and each acquisition device. The device includes a sound source location information determination module 210, an image acquisition module 220, a speaker determination module 230, a face deflection angle determination module 240, and a data transmission module 250.

[0200] The sound source location information determination module 210 is used to collect multiple audio data and determine the sound source location information of the first audio data; the first audio data is the audio data of human voice with a sound energy greater than a first threshold among the multiple audio data.

[0201] The image acquisition module 220 is used to acquire the image corresponding to the sound source location information.

[0202] The speaker determination module 230 is used to determine the speaker in the image. The speaker is the speaker in the image whose voice meets the preset conditions.

[0203] The face deflection angle determination module 240 is used to determine the face deflection angle; the face deflection angle is used to characterize the angle between the preset position of the main face in the image and the acquisition device.

[0204] The data transmission module 250 is used to send the speaker image and face deflection angle to the data processing device; the face deflection angle is used by the data processing device to determine the speaker image to be displayed and drive the display device to display the speaker image to be displayed; the speaker image to be displayed is the speaker image with the smallest face deflection angle determined by the data processing device.

[0205] In one embodiment, the speaker determination module 230 is specifically used for:

[0206] Identify the speaker in the image;

[0207] If there is a speaker in the image, determine that speaker as the main speaker;

[0208] When there are at least two speakers in the image, determine the speech energy value of each speaker and the distance between each speaker and the acquisition device; determine the weight value of the corresponding speaker based on the determined distance; determine the weighted speech energy value of each speaker based on the determined weight value and speech energy value; determine the speaker whose weighted speech energy value meets the preset conditions as the main speaker; the weighted speech energy value is used to characterize the sound energy of human voice.

[0209] In one embodiment, the face deflection angle determination module 240 is specifically used to input the image into a pre-trained face deflection angle acquisition model to determine the face deflection angle.

[0210] In one embodiment, the data sending module 250 is specifically used for:

[0211] Create a custom frame;

[0212] Write the face deflection angle into a custom frame;

[0213] Send a custom frame to the data processing device.

[0214] In one embodiment, the data sending module 250 is specifically used for:

[0215] The image acquired by the image acquisition module 220 is scaled and cropped to control the speaker's face image to be located in a preset area in order to obtain the speaker's image;

[0216] Send the speaker's image to the data processing equipment.

[0217] It should be noted that the conference screen processing device provided in the above embodiments is only illustrated by the division of the above functional modules when executing the conference screen processing method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the conference screen processing device provided in the above embodiments and the conference screen processing method in the above embodiments belong to the same concept, and its implementation process is detailed in the above embodiments, which will not be repeated here.

[0218] Please see Figure 9 This application provides a conference screen processing device, which is applied to a data processing device in a conference system. The conference system includes a display device, a data processing device, and multiple acquisition devices. The data processing device is connected to the display device and each acquisition device. The device includes a data receiving module 310 and a speaker image determination module 320.

[0219] The data receiving module 310 is used to acquire the speaker's image and face deflection angle from multiple acquisition devices; the face deflection angle is the angle between the preset position of the speaker's face determined by the corresponding acquisition device and the acquisition device; the speaker is determined by the corresponding acquisition device based on the acquired audio data and the image corresponding to the audio data.

[0220] The speaker image determination module 320 is used to determine the speaker image to be displayed from the received images. The speaker image to be displayed is the speaker image with the smallest face deflection angle. The speaker image to be displayed is used for display on the display device.

[0221] In one embodiment, the apparatus further includes:

[0222] The speaker image sending module is used to send the speaker image to be displayed to the display device in response to the display device's call command to the data processing device.

[0223] In one embodiment, the speaker image determination module 320 is specifically used for:

[0224] Take the face deflection angle of each acquisition device within a preset time period, determine the average face deflection angle of each acquisition device within the preset time period, and take the speaker image sent by the acquisition device with the smallest average value as the speaker image to be displayed.

[0225] or,

[0226] Obtain a first device identifier and a second device identifier. The first device identifier is used to identify the acquisition device with the smallest face deflection angle in the image frame to be displayed, and the second device identifier is used to identify the acquisition device of the currently displayed image frame. If the first device identifier and the second device identifier are different, obtain the face deflection angle in the currently displayed image frame. If the difference between the smallest face deflection angle in the image frame to be displayed and the face deflection angle in the currently displayed image frame is greater than a target threshold, use the speaker image acquired by the acquisition device corresponding to the first device identifier as the speaker image to be displayed. If the difference between the smallest face deflection angle in the image frame to be displayed and the face deflection angle in the currently displayed image frame is less than or equal to the target threshold, use the speaker image acquired by the acquisition device corresponding to the second device identifier as the speaker image to be displayed.

[0227] It should be noted that the conference screen processing device provided in the above embodiments is only illustrated by the division of the above functional modules when executing the conference screen processing method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the conference screen processing device provided in the above embodiments and the conference screen processing method in the above embodiments belong to the same concept, and its implementation process is detailed in the above embodiments, which will not be repeated here.

[0228] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the conference screen processing method as described in any of the above embodiments.

[0229] The embodiments of this application may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0230] like Figure 10 As shown, this application embodiment also provides a data acquisition device 400, including a memory 410, a processor 420, and a computer program stored in the memory 410 and executable by the processor 420;

[0231] When the processor 420 executes a computer program, it implements the steps of the conference screen processing method as described in any of the above.

[0232] The memory 410 includes a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0233] The processor 420 is the control unit of the acquisition device 400. It connects to various components of the acquisition device 400 via various interfaces and lines. By running or executing programs or modules stored in the memory 410, and by calling data stored in the memory 410, it performs various functions of the conference device 400 and processes data. For example, when the processor 420 executes the computer program stored in the memory 410, it implements all or part of the steps of the conference screen processing method in this embodiment; or it implements all or part of the functions of the conference screen processing device. The processor 420 can be composed of integrated circuits, such as a single packaged integrated circuit, or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips.

[0234] like Figure 11 As shown, this application embodiment also provides a data processing device 500, including a memory 510, a processor 520, and a computer program stored in the memory 510 and executable by the processor 520;

[0235] When the processor 520 executes a computer program, it implements the steps of the conference screen processing method as described in any of the above.

[0236] The memory 510 includes read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0237] The processor 520 is the control unit of the data processing device 500. It connects to various components of the data processing device 500 via various interfaces and lines. It executes programs or modules stored in the memory 510 and calls data stored in the memory 510 to perform various functions and process data. For example, when the processor 520 executes the computer program stored in the memory 510, it implements all or part of the steps of the conference screen processing method described in this embodiment; or it implements all or part of the functions of the conference screen processing device. The processor 520 can be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips.

[0238] like Figure 12 As shown in the figure, this application embodiment also provides a conference system 600, including a display device 630, a data processing device 620, and a plurality of acquisition devices 610; the data processing device 620 is connected to the display device 630 and each acquisition device 610;

[0239] Each acquisition device 610 acquires multiple audio data streams and determines the sound source location information of the first audio data stream; the first audio data stream is the audio data stream where the human voice energy is greater than a first threshold; it acquires the image corresponding to the sound source location information, determines the speaker in the image, and the speaker is the speaker whose human voice energy in the image meets the preset conditions; it determines the face deflection angle; the face deflection angle is used to characterize the angle between the preset position of the speaker's face in the image and the acquisition device; and it sends the speaker image and face deflection angle to the data processing device 620.

[0240] The data processing device 620 receives the speaker image and face deflection angle sent by each acquisition device 610, and determines the speaker image to be displayed from the received speaker images. The speaker image to be displayed is the speaker image with the smallest face deflection angle.

[0241] Display device 630 displays the image of the speaker to be displayed.

[0242] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0243] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for processing conference video, characterized in that, A data acquisition device is used in a conference system, the conference system including a display device, a data processing device, and multiple acquisition devices; the data processing device is connected to the display device and each of the acquisition devices; the conference screen processing method includes: Collect multiple audio data streams and determine the sound source location information of the first audio data stream; the first audio data stream is the audio data stream in which the human voice sound energy is greater than a first threshold. Obtain the image corresponding to the sound source location information; Identify the speaker in the image, wherein the speaker is the person whose voice in the image meets preset conditions; Determine the face deflection angle; the face deflection angle is used to characterize the angle between the preset position of the speaker's face in the image and the acquisition device; The presenter image and the face deflection angle are sent to the data processing device; the face deflection angle is used by the data processing device to determine the presenter image to be displayed and drive the display device to display the presenter image to be displayed, wherein the presenter image to be displayed is the presenter image with the smallest face deflection angle determined by the data processing device.

2. The conference screen processing method according to claim 1, characterized in that, Determining the speaker in the image includes: Identify the speaker in the image; If there is a speaker in the image, the speaker is identified as the main speaker; When there are at least two speakers in the image, determine the voice energy value of each speaker and the distance between each speaker and the acquisition device; determine the weight value of the corresponding speaker based on the determined distance; determine the weighted voice energy value of each speaker based on the determined weight value and the voice energy value; determine the speaker whose weighted voice energy value meets the preset condition as the main speaker; the weighted voice energy value is used to characterize the voice energy.

3. The conference screen processing method according to claim 1, characterized in that, Determining the face deflection angle includes: The image is input into a pre-trained face deflection angle acquisition model to determine the face deflection angle.

4. The conference screen processing method according to any one of claims 1-3, characterized in that, Sending the face deflection angle to the data processing device includes: Create a custom frame; Write the face deflection angle into the custom frame; The custom frame is sent to the data processing device.

5. The conference screen processing method according to any one of claims 1-3, characterized in that, Sending the speaker's image to the data processing device includes: The image is scaled and cropped to control the speaker's face image to be located in a preset area in order to obtain the speaker image; The speaker's image is sent to the data processing device.

6. A method for processing conference video, characterized in that, A data processing device for use in a conference system, the conference system including a display device, the data processing device, and multiple acquisition devices; the data processing device is connected to the display device and each of the multiple acquisition devices; The conference screen processing method includes: The speaker's image and face deflection angle are acquired from the multiple acquisition devices; the face deflection angle is the angle between the preset position of the speaker's face determined by the corresponding acquisition device and the corresponding acquisition device; the speaker is determined by the corresponding acquisition device based on the acquired audio data and the image corresponding to the audio data. From the received speaker images, a speaker image to be displayed is determined, wherein the speaker image to be displayed is the one with the smallest facial deflection angle; the speaker image to be displayed is used for display on the display device.

7. The conference screen processing method according to claim 6, characterized in that, The method further includes: In response to the display device's call instruction to the data processing device, the image of the speaker to be displayed is sent to the display device.

8. The conference screen processing method according to claim 6 or 7, characterized in that, The step of determining the speaker image to be displayed from the received speaker image includes: The face deflection angle of each acquisition device within a preset time period is obtained, the average face deflection angle of each acquisition device within the preset time period is determined, and the speaker image sent by the acquisition device with the smallest average value is taken as the speaker image to be displayed. or, Obtain a first device identifier and a second device identifier. The first device identifier is used to identify the acquisition device with the smallest face deflection angle in the image frame to be displayed, and the second device identifier is used to identify the acquisition device of the currently displayed image frame. If the first device identifier and the second device identifier are different, obtain the face deflection angle in the currently displayed image frame. If the difference between the smallest face deflection angle in the image frame to be displayed and the face deflection angle in the currently displayed image frame is greater than a target threshold, use the speaker image acquired by the acquisition device corresponding to the first device identifier as the speaker image to be displayed. If the difference between the smallest face deflection angle in the image frame to be displayed and the face deflection angle in the currently displayed image frame is less than or equal to the target threshold, use the speaker image acquired by the acquisition device corresponding to the second device identifier as the speaker image to be displayed.

9. A method for processing conference video, characterized in that, The system is used in a conference system, which includes a display device, a data processing device, and multiple acquisition devices. The data processing device is connected to the display device and each of the multiple acquisition devices. The conference screen processing method includes: Each acquisition device acquires multiple audio data streams and determines the sound source location information of the first audio data stream, wherein the first audio data stream is the audio data stream in which the human voice energy is greater than a first threshold; it acquires an image corresponding to the sound source location information and determines the speaker in the image, wherein the speaker is the speaker in the image whose human voice energy meets a preset condition. Determine the face deflection angle; the face deflection angle is used to characterize the angle between the preset position of the speaker's face in the image and the corresponding acquisition device; send the speaker image and the face deflection angle to the data processing device; The data processing device receives the speaker image and face deflection angle sent by each acquisition device, and determines the speaker image to be displayed from the received speaker images. The speaker image to be displayed is the speaker image with the smallest face deflection angle. The display device displays the image of the speaker to be displayed.

10. A conference screen processing device, characterized in that, A data acquisition device for use in a conference system, the conference system including a display device, a data processing device, and multiple acquisition devices; the data processing device is connected to the display device and each of the acquisition devices; the device includes: The sound source location information determination module is used to collect multiple audio data and determine the sound source location information of the first audio data; the first audio data is the audio data of human voice with a sound energy greater than a first threshold among the multiple audio data; The image acquisition module is used to acquire the image corresponding to the sound source location information; The speaker determination module is used to determine the speaker in the image, wherein the speaker is the speaker in the image whose voice meets preset conditions. A face deflection angle determination module is used to determine the face deflection angle; the face deflection angle is used to characterize the angle between the preset position of the speaker's face in the image and the acquisition device; The data transmission module is used to send the speaker image and the face deflection angle to the data processing device; the face deflection angle is used by the data processing device to determine the speaker image to be displayed and drive the display device to display the speaker image to be displayed, wherein the speaker image to be displayed is the speaker image with the smallest face deflection angle determined by the data processing device.

11. A conference screen processing device, characterized in that, A data processing device for use in a conference system, the conference system including a display device, the data processing device, and multiple acquisition devices; the data processing device is connected to the display device and each of the multiple acquisition devices; the device includes: The data receiving module is used to acquire the speaker image and face deflection angle from the multiple acquisition devices; the face deflection angle is the angle between the preset position of the speaker's face determined by the corresponding acquisition device and the corresponding acquisition device; the speaker is determined by the corresponding acquisition device based on the acquired audio data and the image corresponding to the audio data; The speaker image determination module is used to determine the speaker image to be displayed from the received images, wherein the speaker image to be displayed is the speaker image with the smallest face deflection angle; the speaker image to be displayed is used for display on the display device.

12. A data acquisition device, characterized in that, Includes a memory, a processor, and a computer program stored in the memory and executable by the processor; When the processor executes the computer program, it implements the steps of the conference screen processing method as described in any one of claims 1-5.

13. A data processing device, characterized in that, Includes a memory, a processor, and a computer program stored in the memory and executable by the processor; When the processor executes the computer program, it implements the steps of the conference screen processing method as described in any one of claims 6-8.

14. A conference system, characterized in that, It includes a display device, a data processing device, and multiple acquisition devices; the data processing device is connected to the display device and each of the multiple acquisition devices. Each of the acquisition devices is used to acquire multiple audio data streams and determine the sound source location information of the first audio data stream, wherein the first audio data stream is the audio data stream in which the human voice energy is greater than a first threshold; acquire the image corresponding to the sound source location information and determine the speaker in the image, wherein the speaker is the speaker in the image whose human voice energy meets the preset conditions. Determine the face deflection angle; the face deflection angle is used to characterize the angle between the preset position of the speaker's face in the image and the corresponding acquisition device; send the speaker image and the face deflection angle to the data processing device; The data processing device is used to receive the speaker image and face deflection angle sent by each acquisition device, and to determine the speaker image to be displayed from the received speaker images, wherein the speaker image to be displayed is the speaker image with the smallest face deflection angle. The display device is used to display the image of the speaker to be displayed.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When executed by a processor, the computer program implements the steps of the conference screen processing method as described in any one of claims 1-5; Alternatively, the steps of the conference screen processing method as described in any one of claims 6-8 may be implemented; Alternatively, the steps of the conference screen processing method as described in claim 9 can be implemented.

Images