Video conference speaker display method and device, equipment and storage medium

Abstract

The application provides a video conference speaker display method and device, equipment and a storage medium. The method comprises the following steps: acquiring a video image frame sequence of a video conference; acquiring a face image sequence corresponding to each face in the multiple faces according to the video image frame sequence; acquiring a speaking state parameter sequence corresponding to each face image sequence, and obtaining a speaker sequence according to the speaking state parameter sequence; and displaying a face image sequence corresponding to at least one speaker on a screen according to the speaker sequence. According to the technical scheme provided by the application, when there are multiple speakers, the multiple speakers can be displayed on the screen according to the corresponding speaker sequence, the frequency of switching the screen picture between different speakers is reduced, and the conference experience of the participants is improved.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a method, apparatus, device and storage medium for displaying speakers in a video conference. Background Technology

[0002] Video conferencing refers to meetings in which people in two or more locations conduct face-to-face conversations via communication devices and networks. Video conferencing systems require real-time location tracking of the person speaking in the meeting and displaying that person on the screen.

[0003] The relevant technology uses microphone array sound source localization to determine the speaker at the meeting. When displaying the video of the speaker, it can only locate the person who is currently speaking and display a close-up of the speaker. When multiple speakers are speaking at the same time, the screen will switch frequently between different speakers, which will cause visual fatigue and poor meeting experience for the participants. Summary of the Invention

[0004] In view of this, in order to solve the above-mentioned technical problems, this application provides a video conferencing speaker display method, apparatus, device and storage medium.

[0005] Specifically, this application is implemented through the following technical solution: According to a first aspect of the embodiments of this application, a method for displaying a speaker in a video conference is provided, the method comprising: Acquire a sequence of video image frames from a video conference, the sequence of video image frames including multiple faces; Based on the video image frame sequence, obtain the face image sequence corresponding to each face among the plurality of faces; Obtain the speech state parameter sequence corresponding to each face image sequence, and obtain the speaker sequence based on the speech state parameter sequence, wherein the speaker sequence includes the speech state parameter sequence corresponding to the speaker; Based on the speaker sequence, display a sequence of facial images corresponding to at least one speaker on the screen.

[0006] Optionally, obtaining the face image sequence corresponding to each face among the plurality of faces based on the video image frame sequence includes: The video image frame sequence is subjected to face detection processing. When a face image region is detected, the face corresponding to the face image region is tracked to obtain the face image sequence corresponding to the face.

[0007] Optionally, obtaining the speech state parameter sequence corresponding to each face image sequence, and obtaining the speaker sequence based on the speech state parameter sequence, includes: Facial landmark detection processing is performed on each of the face image sequences to obtain a face image sequence containing facial landmarks; Based on the facial key points, the speaking state corresponding to each frame of the face image is determined, and the speaking state parameter sequence corresponding to the face image sequence is obtained.

[0008] Optionally, the step of determining the speech state corresponding to each frame of the face image based on the facial key points to obtain the speech state parameter sequence corresponding to the face image sequence includes: Based on the facial key points, the mouth region of the face in each frame image is determined and the mouth region features are extracted; Based on the mouth region features corresponding to each frame of the image, a classification network is used to classify the speech state corresponding to the mouth region, thereby determining the speech state corresponding to each frame of the face image.

[0009] Optionally, the speaking state parameters include a first parameter and a second parameter, wherein the first parameter is used to indicate the state of speaking and the second parameter is used to indicate the state of not speaking; The step of obtaining the speech state parameter sequence corresponding to each face image sequence, and obtaining the speaker sequence based on the speech state parameter sequence, includes: If the speaking state parameter sequence includes at least one first parameter, the face corresponding to the speaking state parameter sequence is determined to be the speaker's face.

[0010] Optionally, the speaker sequence is obtained by sliding a time window over the speaking state parameter sequence corresponding to the plurality of faces, wherein the time window has a set time length and step size; The step of displaying at least one sequence of facial images corresponding to a speaker on the screen according to the speaker sequence includes: If the first time window is the first time window on the parameter sequence, or if there is no speaker sequence in the second time window, the speaker to be displayed on the screen is determined according to the speaker sequence corresponding to the first time window, and the second time window is the time window of the previous moment of the first time window. Based on the speaker sequence corresponding to the first time window and the speakers displayed on the screen in the second time window, determine the speakers displayed on the screen in the first time window.

[0011] Optionally, the screen includes a first display frame and a second display frame; The step of determining the speaker to be displayed on the screen based on the speaker sequence corresponding to the first time window includes: The first display frame displays a sequence of facial images of the main speaker, who is the first person to speak in the speaker sequence. In response to the speaker sequence including more than one speaker, a sequence of facial images of secondary speakers is displayed in the second display frame, the secondary speaker being the first person to speak in the speaker sequence other than the main speaker.

[0012] Optionally, determining the speaker displayed on the screen in the first time window based on the speaker sequence corresponding to the first time window and the speakers displayed on the screen in the second time window includes: If the main speaker displayed on the screen in the second time window exists in the speaker sequence of the first time window, the sequence of facial images of the main speaker is displayed in the first display box; If a secondary speaker is present in the speaker sequence of the first time window and is displayed on the screen in the second time window, the sequence of facial images of the secondary speaker is displayed in the second display box. If the secondary speaker displayed on the screen in the second time window does not exist in the speaker sequence of the first time window, the first person to speak in the speaker sequence other than the main speaker is identified as the secondary speaker, and the sequence of facial images of the secondary speaker is displayed in the second display frame.

[0013] Optionally, determining the speaker displayed on the screen in the first time window based on the speaker sequence corresponding to the first time window and the speakers displayed on the screen in the second time window further includes: If the main speaker displayed on the screen in the second time window does not exist in the speaker sequence of the first time window, determine whether the secondary speaker exists in the speaker sequence of the first time window; If the secondary speaker exists in the speaker sequence of the first time window, the secondary speaker is converted into the main speaker, and the converted main speaker's face image sequence is displayed in the first display box; If the secondary speaker does not exist in the speaker sequence of the first time window, the speaker to be displayed on the screen is determined according to the speaker sequence corresponding to the first time window.

[0014] Optionally, displaying the sequence of facial images of the secondary speaker in the second display frame further includes: If the second speaker is displayed for more than a set time in the second display box, the first person to speak in the speaker sequence other than the main speaker and the second speaker will be updated to the second speaker, and the updated second speaker's face image sequence will be displayed in the second display box.

[0015] Optionally, the method further includes: If a primary speaker is identified, the frequency of detecting the primary speaker's speaking status is reduced; if a secondary speaker is identified, the frequency of detecting the secondary speaker's speaking status is increased.

[0016] According to a second aspect of the embodiments of this application, a video conferencing speaker display device is provided, the device comprising: A video image acquisition module is used to acquire a sequence of video image frames from a video conference, wherein the sequence of video image frames includes multiple faces; A face image sequence acquisition module is used to acquire a face image sequence corresponding to each face among the plurality of faces based on the video image frame sequence; The speaker sequence determination module is used to obtain the speaking state parameter sequence corresponding to each face image sequence, and to obtain the speaker sequence based on the speaking state parameter sequence, wherein the speaker sequence includes the speaking state parameter sequence corresponding to the speaker; The speaker display module is used to display a sequence of facial images corresponding to at least one speaker on the screen according to the speaker sequence.

[0017] Optionally, the face image sequence acquisition module is specifically used for: The video image frame sequence is subjected to face detection processing. When a face image region is detected, the face corresponding to the face image region is tracked to obtain the face image sequence corresponding to the face.

[0018] Optionally, the speaker sequence determination module is specifically used for: The key point detection module is used to perform facial key point detection processing on each of the facial image sequences to obtain a facial image sequence containing facial key points; The speaking state determination module is used to determine the speaking state corresponding to each frame of the face image based on the facial key points, and to obtain the speaking state parameter sequence corresponding to the face image sequence.

[0019] Optionally, the speaking state determination module is specifically used for: Based on the facial key points, the mouth region of the face in each frame image is determined and the mouth region features are extracted; Based on the mouth region features corresponding to each frame of the image, a classification network is used to classify the speech state corresponding to the mouth region, thereby determining the speech state corresponding to each frame of the face image.

[0020] Optionally, the speaking state parameters include a first parameter and a second parameter, whereby the first parameter indicates a speaking state and the second parameter indicates a non-speaking state; the speaker sequence determination module is specifically used for: If the speaking state parameter sequence includes at least one first parameter, the face corresponding to the speaking state parameter sequence is determined to be the speaker's face.

[0021] Optionally, the speaker sequence is obtained by sliding a time window over the speaking state parameter sequence corresponding to the plurality of faces, wherein the time window has a set time length and step size; The speaker display module is specifically used for: The first display module is used to determine the speaker to be displayed on the screen based on the speaker sequence corresponding to the first time window when the first time window is the first time window on the parameter sequence or when there is no speaker sequence in the second time window. The second time window is the time window at the previous moment of the first time window. The second display module determines the speaker displayed on the screen in the first time window based on the speaker sequence corresponding to the first time window and the speaker displayed on the screen in the second time window.

[0022] Optionally, the screen includes a first display frame and a second display frame; The first display module is specifically used for: The first display frame displays a sequence of facial images of the main speaker, who is the first person to speak in the speaker sequence. In response to the speaker sequence including more than one speaker, a sequence of facial images of secondary speakers is displayed in the second display frame, the secondary speaker being the first person to speak in the speaker sequence other than the main speaker.

[0023] Optionally, the second display module is specifically used for: If the main speaker displayed on the screen in the second time window exists in the speaker sequence of the first time window, the sequence of facial images of the main speaker is displayed in the first display box; If a secondary speaker is present in the speaker sequence of the first time window and is displayed on the screen in the second time window, the sequence of facial images of the secondary speaker is displayed in the second display box. If the secondary speaker displayed on the screen in the second time window does not exist in the speaker sequence of the first time window, the first person to speak in the speaker sequence other than the main speaker is identified as the secondary speaker, and the sequence of facial images of the secondary speaker is displayed in the second display frame.

[0024] Optionally, the second display module is further configured to: If the main speaker displayed on the screen in the second time window does not exist in the speaker sequence of the first time window, determine whether the secondary speaker exists in the speaker sequence of the first time window; If the secondary speaker exists in the speaker sequence of the first time window, the secondary speaker is converted into the main speaker, and the converted main speaker's face image sequence is displayed in the first display box; If the secondary speaker does not exist in the speaker sequence of the first time window, the speaker to be displayed on the screen is determined according to the speaker sequence corresponding to the first time window.

[0025] Optionally, the device further includes: If the second speaker is displayed for more than a set time in the second display box, the first person to speak in the speaker sequence other than the main speaker and the second speaker will be updated to the second speaker, and the updated second speaker's face image sequence will be displayed in the second display box.

[0026] Optionally, the device further includes: If a primary speaker is identified, the frequency of detecting the primary speaker's speaking status is reduced; if a secondary speaker is identified, the frequency of detecting the secondary speaker's speaking status is increased.

[0027] According to a third aspect of the embodiments of this application, an electronic device is provided, the electronic device comprising: a memory and a processor; the memory being used to store a computer program; the processor being used to execute the above-described video conferencing speaker display method by invoking the computer program.

[0028] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the above-described video conferencing speaker display method.

[0029] The technical solutions provided in this application embodiment may include the following beneficial effects: In the technical solution provided in this application, by acquiring the video image frame sequence of the video conference, a sequence of each face image is obtained. Based on the speaking state parameter sequence corresponding to each face image sequence, a speaker sequence is determined. At least one speaker is displayed on the screen according to the speaker sequence. When there are multiple speakers, multiple speakers can be displayed on the screen according to the corresponding speaker sequence, which reduces the frequency of screen switching between different speakers and improves the meeting experience for participants.

[0030] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Furthermore, no embodiment in this application needs to achieve all the effects described above. Attached Figure Description

[0031] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0032] Figure 1 This is a flowchart illustrating a video conferencing speaker display method according to an exemplary embodiment of this application; Figure 2 This is a schematic diagram illustrating two speech state parameter sequences corresponding to a face image sequence, as shown in an exemplary embodiment of this application. Figure 3 This is a schematic diagram illustrating an exemplary embodiment of this application of obtaining a speaker sequence by setting a time window and a sequence of speaking state parameters; Figure 4 This application provides an exemplary embodiment of a method for displaying a speaker in a video conference scenario involving multiple participants in the same meeting room. Figure 5 This is a schematic diagram illustrating the structure of a video conferencing speaker display device according to an exemplary embodiment of this application; Figure 6 This is a hardware schematic diagram of an electronic device illustrated in an exemplary embodiment of this application. Detailed Implementation

[0033] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0034] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0035] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, a first classification threshold may also be referred to as a second classification threshold, and similarly, a second classification threshold may also be referred to as a first classification threshold. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."

[0036] In video conferencing systems, there is a need to locate the person speaking in the meeting in real time and display the person speaking on the screen.

[0037] The relevant technology uses microphone arrays to locate the sound source and determine the speaker at the meeting. However, using microphone arrays requires additional equipment, which is costly and increases the complexity of the system. For small meeting rooms or classrooms where space is limited, the cost of using microphone arrays is very high.

[0038] When displaying video feeds of conference speakers, only the currently speaking person can be located and a close-up of that speaker can be shown. When multiple speakers are speaking simultaneously, the screen will frequently switch between different speakers, which can easily cause visual fatigue and make it difficult for participants to focus their attention on one speaker. For example, if three people, A, B, and C, are discussing and speaking at a certain moment, the screen will display a close-up of one person and frequently switch between A, B, and C.

[0039] To address the aforementioned problems, this application provides a method for displaying a speaker in a video conference, see below. Figure 1 As shown, the method may include the following steps: S101, acquire a video image frame sequence from the video conference, the video image frame sequence including multiple faces; The video image frame sequence may include image frames corresponding to the video stream of the video conference. The video image frames may be acquired using a set frequency, for example, at a frame rate of 25 frames per second.

[0040] In related technologies, the resolution of cameras used for speaker tracking is typically 1080p or 720p. When the speaker's position is further located through video, it is impossible to accurately determine the speaker's state and location, and the displayed image becomes blurry when the speaker is zoomed in.

[0041] This application embodiment can acquire real-time video from a video conference using a high-resolution camera and convert it into a corresponding video image frame sequence to provide clear video images for subsequent face tracking and speech status assessment. For example, a 4K resolution camera can be used to acquire real-time video of the meeting.

[0042] In the video image frame sequence, the video images include the faces of multiple conference participants, and each video image frame may include timestamp information corresponding to that frame.

[0043] S102, based on the video image frame sequence, obtain the face image sequence corresponding to each face among the plurality of faces; The video images include the faces of multiple conference participants. Face detection and face tracking technologies can be used to process the video image frame sequence to obtain the face image sequence corresponding to each face.

[0044] Face detection is performed frame by frame on the video image frame sequence. When a face image region is detected in a video image frame, face tracking is used in subsequent frames to continue capturing the image region corresponding to the face. Based on the location information of the face image region in each frame, images containing faces are extracted from the video image frames to obtain the face image sequence corresponding to each conference participant.

[0045] For example, in a video frame, faces A, B, and C are detected for the first time, and the position information of the corresponding three face detection boxes is obtained. Face tracking technology can be used to capture the position information of the face detection boxes corresponding to faces A, B, and C in subsequent frames. Based on the position information of the face detection boxes, an image containing a face can be cropped from the video image frame sequence to obtain a face image sequence containing face A, a face image sequence containing face B, and a face image sequence containing face C.

[0046] In some embodiments, obtaining the face image sequence corresponding to each of the plurality of faces based on the video image frame sequence may include the following steps: performing face detection frame by frame on each video image in the video image frame sequence using a face detection network, and determining the first video image in which a face image region is detected at the beginning of the video image start frame as the first detection frame; for each image after the first detection frame, when the overlap area between the face image region in each image and the face image region at the corresponding position in the previous image exceeds a set threshold, determining that the face image region and the face image region in the previous frame indicate the same face, and obtaining the face image sequence corresponding to each face.

[0047] For example, if a face is detected for the first time in a certain frame of a video image frame sequence, and the image contains face A and face B, this frame is determined as the first detection frame. Based on the position information of the face image regions corresponding to the two faces, the video image is cropped to obtain the first image of the image sequence corresponding to face A and face B. For each frame after this frame, the face image region in the current detection frame is obtained, and the overlap area between the face image region in the current detection frame and the face image region in the previous frame is calculated. If the overlap rate of the two face image regions exceeds a set threshold, such as 80%, then it is determined that the two face image regions indicate the same face, such as face A. When it is determined that the face image region in the current detection frame indicates face A, the image containing face A in the current detection frame is cropped and added to the face image sequence corresponding to face A.

[0048] During face detection, for face images where a face is detected but facial landmarks cannot be obtained, the speech state parameters corresponding to the face image can be determined based on the speech state parameters of the previous frame. For example, if the speech state parameters of the previous frame indicate that someone is speaking, then the speech state parameters of the face image can be set to be the same as those of the previous frame.

[0049] In some embodiments, obtaining a face image sequence corresponding to each of the plurality of faces based on the video image frame sequence may also include: performing face detection processing on the video image frame sequence, and if a face image region is detected, tracking the face corresponding to the face image region in subsequent frames to obtain the face image sequence corresponding to the face.

[0050] In this application, the tracking of the face corresponding to the face image region in subsequent frames can be achieved using face tracking technology, which will not be elaborated here.

[0051] S103, obtain the speech state parameter sequence corresponding to each face image sequence, and obtain the speaker sequence based on the speech state parameter sequence, wherein the speaker sequence includes the speech state parameter sequence corresponding to the speaker; Each face image sequence is a sequence of images containing a face that is cropped from a continuous segment of image frames in the video image frame sequence. Each face image frame may include the same timestamp information as the corresponding video image.

[0052] For each face image sequence, the speaking state corresponding to the face in each frame is determined frame by frame. The speaking state can include a state of speaking and a state of not speaking, corresponding to two speaking state parameters. Based on the speaking state corresponding to each frame and the corresponding parameter indicating the speaking state, the speaking state parameter sequence corresponding to the face image sequence can be obtained.

[0053] The speaking state parameter sequence may include a parameter sequence that corresponds one-to-one with the speaking state of each frame in the face image sequence; alternatively, after obtaining the parameter sequence that corresponds one-to-one with the speaking state of each frame in the face image sequence, the corresponding speaking state parameter for each frame that does not contain the face is determined as an indication of no speech, according to the frame number of the video image frame sequence, and the speaking state parameter sequence is obtained according to the order of the video frame sequence.

[0054] See Figure 2 As shown, taking a video image frame sequence containing 30 frames as an example, the illustration includes two ways to represent the speaking state corresponding to the same face image sequence, corresponding to two speaking state parameter sequences. Face A appears in frames 10 to 30 of the video image frame sequence. The face image sequence is judged frame by frame to obtain the speaking state corresponding to each face image, and represented by the corresponding speaking state parameters.

[0055] exist Figure 2 In the example, "0" indicates no speaking state, and "1" indicates speaking. The first speaking state parameter sequence corresponds one-to-one with each frame of the face image sequence, and the parameters are determined based on the speaking state of each frame. The second speaking state parameter sequence includes the first speaking state parameter sequence and contains the speaking state parameters corresponding to each frame of the video image frame sequence that does not contain face A. The speaking state parameter for each frame that does not contain face A is set to 0 by default, which means no speaking state.

[0056] In one example, the acquired video image frame sequence is transmitted in real time. A sequence of face images within a set time period can be acquired, and the corresponding speech state parameter sequence within that time period can be determined to obtain the corresponding speaker sequence. The set time period can be a time period corresponding to a set number of frames. When acquiring video image frames corresponding to the set number of frames, the speaker sequence within that time period is determined. For example, if the set number of frames is 30, during the reception of the real-time transmitted video image frames, processing operations are performed on each frame simultaneously. These processing operations include face detection, adding face images within the face image region to the corresponding face image sequence, acquiring the speech state parameters corresponding to the face image, and adding them to the corresponding speech state parameter sequence. When 30 video images are received, the speaker sequence within the time period corresponding to the set number of frames is output based on the speech state parameter sequence corresponding to each face.

[0057] In some embodiments, the speaking state parameter includes a first parameter and a second parameter, wherein the first parameter indicates a state of speaking and the second parameter indicates a state of not speaking. For example, the first parameter can be 1 and the second parameter can be 0.

[0058] Based on the speaking state parameter sequence, it can be determined whether the face corresponding to the parameter sequence is in a speaking state. If at least one parameter in the speaking state parameter sequence indicates a "speaking" state, then the face is determined to be in a speaking state, the face can be identified as the speaker's face, and the face and the corresponding speaking state parameter sequence can be added to the speaker sequence.

[0059] See Figure 2 The speech state parameter sequence corresponding to face A is obtained. This sequence includes parameter 1, which indicates that the speaker is in a speaking state. Therefore, face A is identified as the speaker's face, and face A and the first or second speech state parameter sequence corresponding to face A are added to the speaker sequence.

[0060] S104, based on the speaker sequence, display at least one speaker's corresponding facial image sequence on the screen.

[0061] The speaker sequence includes at least one speaker and a corresponding speaking state parameter sequence for that speaker. When the speaker sequence contains one speaker, the facial image sequence corresponding to that speaker is displayed on the screen; when the speaker sequence contains more than one speaker, facial image sequences corresponding to two speakers in the speaker sequence can be displayed simultaneously on the screen.

[0062] In the technical solution provided in this application embodiment, a sequence of video image frames from a video conference is obtained to obtain a sequence of each face image. A speaker sequence is determined based on the speaking state parameter sequence corresponding to each face image sequence. At least one speaker is displayed on the screen according to the speaker sequence. When there are multiple speakers, multiple speakers can be displayed on the screen according to the corresponding speaker sequence, which reduces the frequency of screen switching between different speakers and improves the meeting experience for participants.

[0063] In addition, the embodiments of this application are based on video tracking and speaker location, which eliminates the need for additional equipment and reduces the cost of speaker location.

[0064] In some embodiments, obtaining the speech state parameter sequence corresponding to each face image sequence may include: performing facial landmark detection processing on each face image sequence to obtain a face image sequence containing facial landmarks; determining the speech state corresponding to each frame of face image based on the facial landmarks to obtain the speech state parameter sequence corresponding to the face image sequence.

[0065] Among them, with Figure 2 Taking the processing of the face image sequence corresponding to face A as an example, a face key point detection network can be used to detect key points frame by frame in the face image sequence. Based on the face key points in each frame, the speaking state corresponding to the face in each frame can be determined, that is, whether the face is speaking or not speaking. The speaking state parameters include a first parameter indicating the speaking state and a second parameter indicating the not speaking state. The speaking state of each frame is represented by the corresponding state parameters to obtain the corresponding speaking state parameter sequence.

[0066] In some embodiments, determining the speaking state corresponding to each frame of a face image based on the facial key points may include: determining the mouth region of the face in each frame of the image based on the facial key points and extracting mouth region features; classifying the speaking state corresponding to the mouth region using a classification network based on the mouth region features corresponding to each frame of the image, and determining the speaking state corresponding to each frame of the face image.

[0067] In this method, the mouth region of a face can be located based on the facial key points of each frame image, and the mouth region image can be cropped to obtain the mouth region image. The mouth region image is then used as the input image of a preset mouth state classification network to obtain the speaking state corresponding to the face. Alternatively, the mouth region can be located based on the facial key points and the mouth region features can be extracted. The mouth region features are then used as the input value of a classification network to obtain the speaking state corresponding to the face.

[0068] In this embodiment of the disclosure, the mouth area is located by facial key points, and the speech state corresponding to the face is obtained by using a classification network, so that the speech recognition result is more accurate and can adapt to different forms of speech of the person to be recognized.

[0069] In some embodiments, the speaker sequence can be obtained by sliding a time window over the speaking state parameter sequences corresponding to the plurality of faces, wherein the time window has a set time length and step size; displaying at least one speaker's corresponding face image sequence on the screen according to the speaker sequence may include: If the first time window is the first time window on the parameter sequence, or if there is no speaker sequence in the second time window, the speaker to be displayed on the screen is determined according to the speaker sequence corresponding to the first time window, and the second time window is the time window of the previous moment of the first time window. Based on the speaker sequence corresponding to the first time window and the speakers displayed on the screen in the second time window, determine the speakers displayed on the screen in the first time window.

[0070] In one example, the step size of the time window can be set to be the same as the length of the time window, so that there is no overlap between two adjacent time windows during the sliding window process; alternatively, the step size of the time window can be set to be less than the length of the time window, so that there is overlap between two adjacent time windows.

[0071] The speaker sequence is obtained by sliding time windows over the speaking state parameter sequences corresponding to the plurality of faces, and may include: determining each corresponding speaking state parameter sequence based on each face image sequence within each time window; and determining the face corresponding to the parameter sequence as the speaker's face when the speaking state parameter sequence includes at least one of the first parameters, thereby obtaining the speaker sequence for each time window.

[0072] The step of displaying a sequence of facial images corresponding to at least one speaker on the screen based on the speaker sequence may include the following steps: S201, determine the first time window and the corresponding speaker sequence for the current moment; S202, determine whether the first time window is the first time window in the speaking state parameter sequence; S203, if it is the first time window on the speaking state parameter sequence, then determine the speaker to be displayed on the screen according to the speaker sequence corresponding to the first time window; S204, if it is not the first time window in the speaking state parameter sequence, then obtain the speaker displayed on the screen in the time window of the previous moment of the first time window; S205, if there is no speaker displayed on the screen in the previous time window, then the speaker displayed on the screen is determined according to the speaker sequence corresponding to the first time window; otherwise, the speaker displayed on the screen in the first time window is determined according to the speaker displayed on the screen in the previous time window and the speaker sequence corresponding to the first time window.

[0073] like Figure 3 As shown, faces A, B, and C are detected in the video image, and face image sequences of faces A, B, and C are obtained through face detection and face tracking. Speech state parameter sequences corresponding to each face are obtained through facial landmark detection and a preset mouth state classification network. For the speech state parameter sequences corresponding to faces A, B, and C, a speaker sequence corresponding to each time window is obtained based on a set step size and a set time length.

[0074] Based on each speaking state parameter sequence, the speaker sequence corresponding to each time window is obtained. As shown in the figure, the speaker sequence corresponding to time window 1 includes face A and its corresponding speaking state parameter sequence, and the speaker sequence corresponding to time window 2 includes faces A, B, and C and their corresponding speaking state parameter sequences.

[0075] exist Figure 3 In the example, the time window slides sequentially from the starting frame of the video image frame sequence according to the frame number order. If the frame number corresponding to each face image sequence is within the time window, then the time window contains the speech state parameters corresponding to that face image sequence.

[0076] Time window 1 is the first time window in the speaking state parameter sequence. Taking the time window in the figure as an example, displaying at least one sequence of facial images corresponding to a speaker on the screen according to the speaker sequence may include the following steps: Get the current time window and determine if the current time window is time window 1; If the current time window is time window 1, then the speaker to be displayed on the screen is determined according to the speaker sequence corresponding to the current time window; for example, the speaker sequence corresponding to time window 1 includes face A and the corresponding speaking state parameter sequence. If the current time window is not time window 1, retrieve the speaker displayed on the screen in the time window before the current time window; if the current time window is time window 2 and the previous time window is time window 1, retrieve the speaker displayed on the screen in time window 1. In response to the previous time window corresponding to the current time window having a speaker displayed on the screen, the speaker displayed on the screen in the current time window is determined based on the speaker displayed on the screen and the speaker sequence corresponding to the current time window; for example, if the current time window is time window 2, and the speaker displayed on the screen in time window 1 is face A, the speaker displayed on the screen in time window 2 is determined based on the speaker displayed on the screen in time window 1 and the speaker sequence corresponding to time window 2.

[0077] In this embodiment of the disclosure, a time window method is used to determine the speaker sequence corresponding to each window, which can more timely locate the speaker at each moment and update the speakers displayed on the screen.

[0078] In some embodiments, the screen includes a first display frame and a second display frame; when the first time window is the first time window on the parameter sequence, or when there is no speaker sequence in the second time window, determining the speaker displayed on the screen according to the speaker sequence corresponding to the first time window may include the following steps: The first display frame displays a sequence of facial images of the main speaker, who is the first person to speak in the speaker sequence. In response to the speaker sequence including more than one speaker, a sequence of facial images of secondary speakers is displayed in the second display frame, the secondary speaker being the first person to speak in the speaker sequence other than the main speaker.

[0079] See Figure 3 When the first time window is time window 1, then the first time window is the first time window in the speaking state parameter sequence. The screen includes two display frames. The first display frame displays the face image sequence of face A. The speaker sequence corresponding to time window 1 includes face A and the corresponding speaking state parameter sequence, determining the participant corresponding to face A as the main speaker.

[0080] If the speaker sequence corresponding to time window 1 includes two or more speakers, then after the above steps, the first person to speak in the speaker sequence, excluding the main speaker corresponding to face A, is determined as the secondary speaker, and the secondary speaker is displayed in the second display box.

[0081] In this embodiment of the disclosure, by setting two display frames on the screen, displaying the main speaker in the first display frame and the secondary speaker in the second display frame, it is possible to display two speakers on the screen at the same time, thereby reducing the frequency of switching speakers frequently in a single display frame.

[0082] In some embodiments, where the main speaker is displayed in the first display frame and the secondary speaker is displayed in the second display frame, determining the speaker displayed on the screen in the first time window based on the speaker sequence corresponding to the first time window and the speakers displayed on the screen in the second time window may include: If the main speaker displayed on the screen in the second time window exists in the speaker sequence of the first time window, the sequence of facial images of the main speaker is displayed in the first display box; If a secondary speaker is present in the speaker sequence of the first time window and is displayed on the screen in the second time window, the sequence of facial images of the secondary speaker is displayed in the second display box. If the secondary speaker displayed on the screen in the second time window does not exist in the speaker sequence of the first time window, the first person to speak in the speaker sequence other than the main speaker is identified as the secondary speaker, and the sequence of facial images of the secondary speaker is displayed in the second display frame.

[0083] For example, if the speakers displayed on the screen in the second time window include main speaker 1 and secondary speaker 2, and if the speaker sequence corresponding to the first time window includes speaker 1 and speaker 2, then speaker 1 is determined to be the main speaker and displayed in the first display box, and speaker 2 is determined to be the secondary speaker and displayed in the second display box; if the speaker sequence corresponding to the first time window includes speaker 1, speaker 3, and speaker 4, then speaker 1 is determined to be the main speaker and displayed in the first display box, and the first speaker among speakers 3 and 4 is determined to be the secondary speaker and displayed in the second display box.

[0084] Combination Figure 3 When the first time window is time window 1, the main speaker is determined to be the meeting participant corresponding to face A. The face image sequence of face A is displayed in the first display box. There is no secondary speaker. The second display box does not display any information.

[0085] When the first time window is time window 2, the speaker sequence corresponding to time window 2 is compared with the speaker displayed on the screen in the previous time window 1. The speaker sequence corresponding to time window 2 includes speaker A, speaker B, and speaker C, which correspond to face A, face B, and face C, respectively. The main speaker displayed on the screen in time window 1 corresponds to face A. Therefore, the above-mentioned situation is satisfied that the main speaker displayed on the screen in the second time window exists in the speaker sequence of the first time window. The main speaker is determined to be speaker A, and the main speaker is displayed in the first display box. If the secondary speaker is not displayed on the screen in time window 1, and the speaker sequence corresponding to time window 2 includes speaker A, speaker B, and speaker C, satisfying the condition that the secondary speaker displayed on the screen in the second time window does not exist in the speaker sequence of the first time window, then the first person to speak between speaker B and speaker C is determined to be the secondary speaker. According to the speaking state parameter sequence corresponding to speaker B and speaker C in time window 2, if parameter 1 indicating the speaking state appears first in speaker B's parameter sequence, then speaker B is determined to be the secondary speaker, and the secondary speaker is displayed in the second display box.

[0086] In this embodiment of the disclosure, the speaker currently displayed on the screen is determined based on the speaker displayed on the screen when the speaker sequence is obtained and the obtained speaker sequence, thereby achieving stable display of continuously speaking people on the screen.

[0087] In some embodiments, determining the speaker displayed on the screen in the first time window based on the speaker sequence corresponding to the first time window and the speakers displayed on the screen in the second time window further includes: If the main speaker displayed on the screen in the second time window does not exist in the speaker sequence of the first time window, determine whether the secondary speaker exists in the speaker sequence of the first time window; If the secondary speaker exists in the speaker sequence of the first time window, the secondary speaker is converted into the main speaker, and the converted main speaker's face image sequence is displayed in the first display box; If the secondary speaker does not exist in the speaker sequence of the first time window, the speaker to be displayed on the screen is determined according to the speaker sequence corresponding to the first time window.

[0088] Specifically, when converting a secondary speaker to a primary speaker, the indicator information of the first and second display frames can be exchanged. This indicator information is used to distinguish the two display frames. For example, the first display frame corresponds to the number 1, and the second display frame corresponds to the number 2. If the secondary speaker is displayed in the second display frame, when converting the secondary speaker to the primary speaker, the number of the second display frame can be changed to 1, and the number of the first display frame can be changed to 2, thus making the original second display frame the new first display frame.

[0089] For example, if the speakers displayed on the screen in the second time window include main speaker 1 and secondary speaker 2, and if the speaker sequence corresponding to the first time window includes speaker 2 and speaker 3, then the above-mentioned situation is satisfied: the main speaker displayed on the screen in the second time window does not exist in the speaker sequence of the first time window, and the secondary speaker 2 displayed on the screen exists in the speaker sequence. The secondary speaker 2 is then converted from a secondary speaker to a main speaker and displayed in the first display box. If the speaker sequence corresponding to the first time window includes speaker 3, speaker 4, and speaker 5, then the situation described above, where neither the main speaker nor the secondary speaker displayed on the screen in the second time window exists in the speaker sequence, is satisfied. Determining the speakers displayed on the screen based on the speaker sequence corresponding to the first time window may include: determining the first speaker in the speaker sequence as the main speaker and displaying them in the first display box; or, if the speaker sequence includes three speakers, determining the first speaker among the speakers other than the main speaker as the secondary speaker and displaying them in the second display box.

[0090] In some embodiments, after determining the secondary speaker, displaying the sequence of facial images of the secondary speaker in the second display frame may further include: If the second speaker is displayed for more than a set time in the second display box, the first person to speak in the speaker sequence other than the main speaker and the second speaker will be updated to the second speaker, and the updated second speaker's face image sequence will be displayed in the second display box.

[0091] If the speaker sequence includes more than one speaker besides the main speaker, then in the second display box, based on the speaking order of each speaker, if the time of the currently displayed secondary speaker exceeds the set time, the speaker following the currently displayed secondary speaker should be identified as the new secondary speaker and displayed in the second display box.

[0092] For example, if the speaker sequence includes speaker 2 and speaker 3 in addition to the main speaker, after speaker 2 is determined to be the secondary speaker, if the time for displaying speaker 2 exceeds the set time, such as the set time of 500 milliseconds, then according to the speaking order, if speaker 3 is the first speaker after speaker 2, then speaker 3 is determined to be the new secondary speaker and is displayed in the second display box.

[0093] In this embodiment of the disclosure, for speakers other than the main speaker in the speaker sequence, each speaker is switched on the screen according to the speaking order indicated by the speaking state parameter sequence, and the display time of each speaker is set according to the set time, which reduces the frequency of frequent switching of speaking and improves the video conferencing experience of the participants.

[0094] In some embodiments, after determining the main speaker, the frequency of detecting the main speaker's speaking status can be reduced; after determining the secondary speaker, the frequency of detecting the secondary speaker's speaking status can be increased.

[0095] In this embodiment of the disclosure, the frequency of detecting the speaking status of the main speaker is reduced, while the frequency of detecting the speaking status of the secondary speakers is increased, so that the main speaker is displayed on the screen more stably and smoothly, and the display switching of multiple secondary speakers can be performed more accurately.

[0096] This application provides another method for displaying a speaker in a video conference. See also... Figure 4 As shown, before the video conference speaker display method is executed, the entire screen displays the scene of the entire conference, and no content is displayed in the screen containing the first display frame and the second display frame; after the speaker sequence is obtained, the main speaker determined according to the speaker sequence is displayed in the first display frame, and the secondary speaker determined according to the speaker sequence is displayed in the second display frame.

[0097] During the initial execution of the method, the speaker sequence within a set time period corresponding to the current moment is first determined based on the acquired video image frame sequence. Determining the speaker sequence within each set time period based on the acquired video image frame sequence may include the following steps: S301, Obtain the video image frame sequence corresponding to the real-time video containing the conference participants; S302, perform face detection on the video image frame sequence. If the face image region of the meeting participant is detected, track the face corresponding to the face image region in the subsequent frame image to obtain the face image sequence corresponding to each meeting participant. S303, obtain the speech state parameter sequence corresponding to each face image sequence within a set time period, obtain the speech state parameter sequence corresponding to each face image sequence, and determine the speaker sequence corresponding to the set time period based on the speech state parameter sequence; Next, determine whether the main speaker is displayed in the first display box of the screen at the current moment, and perform the following steps based on the determination result: If the main speaker is not displayed in the first display box, the first person to speak in the speaker sequence is determined to be the main speaker and displayed in the first display box; further, it is determined whether the number of speakers in the speaker sequence is more than one. If the number of speakers is more than one, the first person to speak besides the main speaker is determined to be the secondary speaker and displayed in the second display box; if the number of speakers is no more than one, the speaker sequence is obtained again. When the main speaker is displayed in the first display box, determine whether the main speaker exists in the speaker sequence corresponding to the current moment, and perform the following steps according to the determination result: If so, meaning the main speaker exists in the speaker sequence at the current moment, update the main speaker's display in the first display frame; further, determine whether the number of speakers in the speaker sequence is more than one. If the number of speakers is less than one, continue to acquire the speaker sequence. If the number of speakers is more than one, further determine whether the second display frame of the screen displays the secondary speaker at the current moment, and perform the following operations according to the determination result: If the second display box does not display the secondary speaker, determine the first person to speak in the speaker sequence other than the main speaker as the secondary speaker, and display the secondary speaker in the second display box. If a secondary speaker is displayed in the second display box, it is further determined whether the secondary speaker exists in the speaker sequence corresponding to the current time. If the secondary speaker is not in the speaker sequence corresponding to the current time, the first person to speak in the speaker sequence other than the main speaker is identified as the secondary speaker, and the secondary speaker is displayed in the second display box. Conversely, if the secondary speaker is in the speaker sequence corresponding to the current time, it is determined whether the display time of the secondary speaker in the second display box exceeds the set duration. If not, the display screen of the secondary speaker in the second display box is updated. If so, it is further determined whether the number of speakers in the speaker sequence is more than two. If the number of speakers is more than two, the first person to speak in the speaker sequence other than the main speaker and the secondary speaker is identified as the new secondary speaker, and the new secondary speaker is displayed in the second display box.

[0098] If not, meaning the main speaker is not in the speaker sequence at the current moment, further determine whether the second display box on the screen displays the secondary speaker at the current moment, and perform the following operations based on the determination result: If the second display box shows the second speaker, continue to determine whether the second speaker exists in the speaker sequence corresponding to the current moment, and perform the following operations based on the determination result: If so, that is, in the speaker sequence corresponding to the current moment, the secondary speaker is determined to be the new main speaker, and the indicator of the first display box and the second display box are swapped, so that the original second display box is converted into the new first display box, and the original first display box is converted into the new second display box; further, it is determined whether the number of speakers in the speaker sequence is more than one. If the number of speakers is more than one, the first person to speak in the speaker sequence other than the main speaker is determined to be the secondary speaker, and the secondary speaker is displayed in the second display box. If the number of speakers is no more than one, the speaker sequence is obtained again. If not, the secondary speaker is not in the speaker sequence corresponding to the current moment. The first person to speak in the speaker sequence is determined to be the main speaker and displayed in the first display box. Further, it is determined whether the number of speakers in the speaker sequence is more than one. If the number of speakers is more than one, the first person to speak besides the main speaker is determined to be the secondary speaker and displayed in the second display box. If the number of speakers is not more than one, the speaker sequence is obtained again. If the second display box does not display the secondary speaker, the first person to speak in the speaker sequence is determined to be the main speaker, and the main speaker is displayed in the first display box; further, it is determined whether the number of speakers in the speaker sequence is more than one. If the number of speakers is more than one, the first person to speak besides the main speaker is determined to be the secondary speaker, and the secondary speaker is displayed in the second display box. If the number of speakers is no more than one, the speaker sequence is obtained again.

[0099] In the technical solution provided in this application embodiment, a sequence of video image frames from a video conference is obtained to obtain a sequence of each face image. A speaker sequence is determined based on the speaking state parameter sequence corresponding to each face image sequence. At least one speaker is displayed on the screen according to the speaker sequence. When there are multiple speakers, multiple speakers can be displayed on the screen according to the corresponding speaker sequence, which reduces the frequency of screen switching between different speakers and improves the meeting experience for participants.

[0100] In addition, by setting two display frames on the screen, with the main speaker displayed in the first display frame and the secondary speaker displayed in the second display frame, it is possible to display two speakers on the screen at the same time, reducing the frequency of switching speakers frequently in a single display frame; and by keeping the main speaker displayed in the first display frame and the secondary speaker displayed in the second display frame, the display of speakers on the screen is more stable and smooth.

[0101] Corresponding to the aforementioned embodiment of the video conference speaker display method, see [link to example]. Figure 5 As shown, this application also provides an embodiment of a video conferencing speaker display device, the device including: The video image acquisition module 501 is used to acquire a video image frame sequence of a video conference, wherein the video image frame sequence includes multiple faces; The face image sequence acquisition module 502 is used to acquire the face image sequence corresponding to each face among the plurality of faces based on the video image frame sequence; The speaker sequence determination module 503 is used to obtain the speaking state parameter sequence corresponding to each face image sequence, and to obtain the speaker sequence based on the speaking state parameter sequence, wherein the speaker sequence includes the speaking state parameter sequence corresponding to the speaker; The speaker display module 504 is used to display a sequence of facial images corresponding to at least one speaker on the screen according to the speaker sequence.

[0102] Optionally, the face image sequence acquisition module is specifically used for: The video image frame sequence is subjected to face detection processing. When a face image region is detected, the face corresponding to the face image region is tracked to obtain the face image sequence corresponding to the face.

[0103] Optionally, the speaker sequence determination module is specifically used for: The key point detection module is used to perform facial key point detection processing on each of the facial image sequences to obtain a facial image sequence containing facial key points; The speaking state determination module is used to determine the speaking state corresponding to each frame of the face image based on the facial key points, and to obtain the speaking state parameter sequence corresponding to the face image sequence.

[0104] Optionally, the speaking state determination module is specifically used for: Based on the facial key points, the mouth region of the face in each frame image is determined and the mouth region features are extracted; Based on the mouth region features corresponding to each frame of the image, a classification network is used to classify the speech state corresponding to the mouth region, thereby determining the speech state corresponding to each frame of the face image.

[0105] Optionally, the speaking state parameters include a first parameter and a second parameter, whereby the first parameter indicates a speaking state and the second parameter indicates a non-speaking state; the speaker sequence determination module is specifically used for: If the speaking state parameter sequence includes at least one first parameter, the face corresponding to the speaking state parameter sequence is determined to be the speaker's face.

[0106] Optionally, the speaker sequence is obtained by sliding a time window over the speaking state parameter sequence corresponding to the plurality of faces, wherein the time window has a set time length and step size; The speaker display module is specifically used for: The first display module is used to determine the speaker to be displayed on the screen based on the speaker sequence corresponding to the first time window when the first time window is the first time window on the parameter sequence or when there is no speaker sequence in the second time window. The second time window is the time window at the previous moment of the first time window. The second display module determines the speaker displayed on the screen in the first time window based on the speaker sequence corresponding to the first time window and the speaker displayed on the screen in the second time window.

[0107] Optionally, the screen includes a first display frame and a second display frame; The first display module is specifically used for: The first display frame displays a sequence of facial images of the main speaker, who is the first person to speak in the speaker sequence. In response to the speaker sequence including more than one speaker, a sequence of facial images of secondary speakers is displayed in the second display frame, the secondary speaker being the first person to speak in the speaker sequence other than the main speaker.

[0108] Optionally, the second display module is specifically used for: If the main speaker displayed on the screen in the second time window exists in the speaker sequence of the first time window, the sequence of facial images of the main speaker is displayed in the first display box; If a secondary speaker is present in the speaker sequence of the first time window and is displayed on the screen in the second time window, the sequence of facial images of the secondary speaker is displayed in the second display box. If the secondary speaker displayed on the screen in the second time window does not exist in the speaker sequence of the first time window, the first person to speak in the speaker sequence other than the main speaker is identified as the secondary speaker, and the sequence of facial images of the secondary speaker is displayed in the second display frame.

[0109] Optionally, the second display module is further configured to: If the main speaker displayed on the screen in the second time window does not exist in the speaker sequence of the first time window, determine whether the secondary speaker exists in the speaker sequence of the first time window; If the secondary speaker exists in the speaker sequence of the first time window, the secondary speaker is converted into the main speaker, and the converted main speaker's face image sequence is displayed in the first display box; If the secondary speaker does not exist in the speaker sequence of the first time window, the speaker to be displayed on the screen is determined according to the speaker sequence corresponding to the first time window.

[0110] Optionally, the device further includes: If the second speaker is displayed for more than a set time in the second display box, the first person to speak in the speaker sequence other than the main speaker and the second speaker will be updated to the second speaker, and the updated second speaker's face image sequence will be displayed in the second display box.

[0111] Optionally, the device further includes: If a primary speaker is identified, the frequency of detecting the primary speaker's speaking status is reduced; if a secondary speaker is identified, the frequency of detecting the secondary speaker's speaking status is increased.

[0112] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0113] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0114] This application also provides an electronic device, the structural schematic diagram of which is shown below. Figure 6 As shown, the electronic device 600 includes at least one processor 601, a memory 602, and a bus 603. At least one processor 601 is electrically connected to the memory 602. The memory 602 is configured to store at least one computer-executable instruction, and the processor 601 is configured to execute the at least one computer-executable instruction to perform the steps of any video conferencing speaker display method provided in any embodiment or optional implementation of this application.

[0115] Furthermore, the processor 601 can be an FPGA (Field-Programmable Gate Array) or other devices with logic processing capabilities, such as an MCU (Microcontroller Unit) or a CPU (Central Processing Unit).

[0116] In the technical solution provided in this application embodiment, a sequence of video image frames from a video conference is obtained to obtain a sequence of each face image. A speaker sequence is determined based on the speaking state parameter sequence corresponding to each face image sequence. At least one speaker is displayed on the screen according to the speaker sequence. When there are multiple speakers, multiple speakers can be displayed on the screen according to the corresponding speaker sequence, which reduces the frequency of screen switching between different speakers and improves the meeting experience for participants.

[0117] This application also provides another readable storage medium storing a computer program that, when executed by a processor, implements the steps of any video conferencing speaker display method provided in any embodiment or optional implementation of this application.

[0118] The readable storage media provided in this application include, but are not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic cards, or optical cards. In other words, readable storage media include any medium by which a device (e.g., a computer) stores or transmits information in a readable form.

[0119] In the technical solution provided in this application embodiment, a sequence of video image frames from a video conference is obtained to obtain a sequence of each face image. A speaker sequence is determined based on the speaking state parameter sequence corresponding to each face image sequence. At least one speaker is displayed on the screen according to the speaker sequence. When there are multiple speakers, multiple speakers can be displayed on the screen according to the corresponding speaker sequence, which reduces the frequency of screen switching between different speakers and improves the meeting experience for participants.

[0120] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0121] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0122] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0123] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for displaying a speaker in a video conference, characterized in that, The method includes: Acquire a sequence of video image frames from a video conference, the sequence of video image frames including multiple faces; Based on the video image frame sequence, obtain the face image sequence corresponding to each face among the plurality of faces; A speech state parameter sequence corresponding to each face image sequence is obtained, and a speaker sequence is obtained by sliding a time window on the speech state parameter sequences corresponding to the multiple faces, wherein the speaker sequence includes the speech state parameter sequence corresponding to the speaker; the time window has a set time length and step size; If the first time window is the first time window on the parameter sequence, or if there is no speaker sequence in the second time window, the speaker to be displayed on the screen is determined according to the speaker sequence corresponding to the first time window, and the second time window is the time window of the previous moment of the first time window. Based on the speaker sequence corresponding to the first time window and the speakers displayed on the screen in the second time window, determine the speakers displayed on the screen in the first time window.

2. The method according to claim 1, characterized in that, The step of obtaining the face image sequence corresponding to each face among the plurality of faces based on the video image frame sequence includes: The video image frame sequence is subjected to face detection processing. When a face image region is detected, the face corresponding to the face image region is tracked to obtain the face image sequence corresponding to the face.

3. The method according to claim 1, characterized in that, The step of obtaining the speech state parameter sequence corresponding to each face image sequence includes: Facial landmark detection processing is performed on each of the face image sequences to obtain a face image sequence containing facial landmarks; Based on the facial key points, the speaking state corresponding to each frame of the face image is determined, and the speaking state parameter sequence corresponding to the face image sequence is obtained.

4. The method according to claim 3, characterized in that, The step of determining the speech state corresponding to each frame of the face image based on the facial key points, and obtaining the speech state parameter sequence corresponding to the face image sequence, includes: Based on the facial key points, the mouth region of the face in each frame image is determined and the mouth region features are extracted; Based on the mouth region features corresponding to each frame of the image, a classification network is used to classify the speech state corresponding to the mouth region, thereby determining the speech state corresponding to each frame of the face image.

5. The method according to claim 1, characterized in that, The speaking status parameters include a first parameter and a second parameter. The first parameter is used to indicate the state of speaking, and the second parameter is used to indicate the state of not speaking. The step of obtaining the speech state parameter sequence corresponding to each face image sequence, and obtaining the speaker sequence based on the speech state parameter sequence, includes: If the speaking state parameter sequence includes at least one first parameter, the face corresponding to the speaking state parameter sequence is determined to be the speaker's face.

6. The method according to claim 1, characterized in that, The screen includes a first display frame and a second display frame; The step of determining the speaker to be displayed on the screen based on the speaker sequence corresponding to the first time window includes: The first display frame displays a sequence of facial images of the main speaker, who is the first person to speak in the speaker sequence. In response to the speaker sequence including more than one speaker, a sequence of facial images of secondary speakers is displayed in the second display frame, the secondary speaker being the first person to speak in the speaker sequence other than the main speaker.

7. The method according to claim 6, characterized in that, The step of determining the speaker displayed on the screen in the first time window based on the speaker sequence corresponding to the first time window and the speakers displayed on the screen in the second time window includes: If the main speaker displayed on the screen in the second time window exists in the speaker sequence of the first time window, the sequence of facial images of the main speaker is displayed in the first display box; If a secondary speaker is present in the speaker sequence of the first time window and is displayed on the screen in the second time window, the sequence of facial images of the secondary speaker is displayed in the second display box. If the secondary speaker displayed on the screen in the second time window does not exist in the speaker sequence of the first time window, the first person to speak in the speaker sequence other than the main speaker is identified as the secondary speaker, and the sequence of facial images of the secondary speaker is displayed in the second display box.

8. The method according to claim 7, characterized in that, The method further includes: If the main speaker displayed on the screen in the second time window does not exist in the speaker sequence of the first time window, determine whether the secondary speaker exists in the speaker sequence of the first time window; If the secondary speaker exists in the speaker sequence of the first time window, the secondary speaker is converted into the main speaker, and the converted main speaker's face image sequence is displayed in the first display box; If the secondary speaker does not exist in the speaker sequence of the first time window, the speaker to be displayed on the screen is determined according to the speaker sequence corresponding to the first time window.

9. The method according to any one of claims 6 or 7, characterized in that, The method further includes: If the second speaker is displayed for more than a set time in the second display box, the first person to speak in the speaker sequence other than the main speaker and the second speaker will be updated to the second speaker, and the updated second speaker's face image sequence will be displayed in the second display box.

10. The method according to any one of claims 6-8, characterized in that, The method further includes: If a primary speaker is identified, the frequency of detecting the primary speaker's speaking status is reduced; if a secondary speaker is identified, the frequency of detecting the secondary speaker's speaking status is increased.

11. A video conferencing speaker display device, characterized in that, The device includes: A video image acquisition module is used to acquire a sequence of video image frames from a video conference, wherein the sequence of video image frames includes multiple faces; A face image sequence acquisition module is used to acquire a face image sequence corresponding to each face among the plurality of faces based on the video image frame sequence; The speaker sequence determination module is used to obtain the speaking state parameter sequence corresponding to each face image sequence, and to slide a time window on the speaking state parameter sequences corresponding to the multiple faces, wherein the speaker sequence includes the speaking state parameter sequence corresponding to the speaker; the time window has a set time length and step size; The speaker display module is used to determine the speaker to be displayed on the screen based on the speaker sequence corresponding to the first time window when the first time window is the first time window on the parameter sequence, or when there is no speaker sequence in the second time window. The second time window is the time window at the previous moment of the first time window. Based on the speaker sequence corresponding to the first time window and the speakers displayed on the screen in the second time window, determine the speakers displayed on the screen in the first time window.

12. An electronic device, characterized in that, include: Processor, memory; The memory is used to store computer programs; The processor is configured to execute the video conferencing speaker display method as described in any one of claims 1-10 by invoking the computer program.

13. A readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the video conferencing speaker display method as described in any one of claims 1-10.

Images