Speaker identification method and device, storage medium and terminal

Abstract

The invention discloses a spokesman identification method and device, a storage medium and a terminal. Sound data and video frame data in a speech scene of a video conference are collected; frequency domain features of the sound data are extracted, and face features of each user in the speech scene are extracted according to the video frame data; the frequency domain features and the face features corresponding to each user are combined and input into a speaking prediction model, the speaking probability output by the speaking prediction model to each user is obtained, and the user with the speaking probability meeting the preset condition in all the users is determined as the current spokesman. Frequency domain features are extracted from the sound data, face features of each user are extracted from the video frame data, the frequency domain features and the face features of each user are combined, and fused multi-modal features corresponding to each user are input into a speech prediction model; therefore, the speaking prediction model can predict the spokesman more accurately according to the associated fusion information between the audio and the video.

Description

Speaker identification methods, devices, storage media, and terminals Technical Field

[0001] This application relates to the field of computer technology, and in particular to a speaker identification method, apparatus, storage medium, and terminal. Background Technology

[0002] In large or multi-party video conferences, multiple participants often coexist within a single meeting setting. When someone speaks, automatically and accurately locating the speaker among the participants and continuously displaying a close-up of them on the conference screen would greatly benefit other remote users, allowing them to receive the speaker's message more accurately and making the meeting process smoother and more efficient. The foundation for tracking and displaying speakers lies in accurately identifying them; therefore, an automatic speaker identification method is needed to effectively locate the user currently speaking among multiple participants. Summary of the Invention

[0003] This application provides a speaker identification method, device, storage medium, and terminal, which can solve the technical problem of inaccurate speaker location in related technologies.

[0004] In a first aspect, embodiments of this application provide a speaker identification method, the method comprising:

[0005] Collect audio data and video frame data from the speaking scenes in video conferences;

[0006] Extract the frequency domain features of the above audio data, and extract the facial features of each user in the above speaking scenario based on the above video frame data;

[0007] The frequency domain features and the facial features corresponding to each user are combined and input into the speech prediction model to obtain the speaking probability output by the speech prediction model for each user. The user whose speaking probability meets the preset condition is determined as the current speaker.

[0008] The beneficial effects brought about by the technical solution in the first aspect mentioned above include at least the following: in a conference scenario, audio data and video frame data are collected, frequency domain features are extracted from the audio data and facial features of each user are extracted from the video frame data, the frequency domain features are combined with the facial features of each user to achieve feature-level fusion, and the fused multimodal features corresponding to each user are input into the speech prediction model, so that the speech prediction model can make more accurate predictions of the speaker based on the correlation and fusion information between audio and video, thereby locating the speaker among multiple users.

[0009] In some possible implementations, after determining the user whose speaking probability meets the preset condition among all users as the current speaker, the method further includes: performing camera tracking on the current speaker based on the tracking modeling of the current speaker, wherein the tracking modeling is constructed based on the video frame data of the current speaker in the speaking scenario; and displaying a close-up image of the current speaker in the conference screen of the video conference.

[0010] Through the technical solutions in the above possible implementation methods, after determining the current speaker, their close-up image can be captured and displayed in the video conference screen based on their tracking model, thereby realizing automatic switching of the speaker's close-up image.

[0011] In some possible implementations, after determining the user whose speaking probability meets the preset condition among all users as the current speaker, the method further includes: determining whether there is a close-up of the previous speaker in the video conference screen and determining whether the close-up duration of the previous speaker reaches a first preset duration; if there is a close-up of the previous speaker and the close-up duration does not reach the first preset duration, then the close-up of the previous speaker is still displayed; if there is no close-up of the previous speaker, or if there is a close-up of the previous speaker and the close-up duration reaches the first preset duration, then the step of performing camera tracking on the current speaker based on the tracking model of the current speaker is executed.

[0012] Using the technical solutions described above in the possible implementation methods, when the current speaker is determined, the meeting screen may still be displaying a close-up of the previous speaker. In order to avoid affecting the user's viewing experience by frequently switching close-up shots between different speakers, it can be determined whether the close-up shot of the previous speaker has been played for more than a first preset duration. If the playback duration has not reached the first preset duration, the close-up shot of the current speaker will not be switched temporarily. If there is no close-up shot of the previous speaker or the playback duration of the close-up shot of the previous speaker has reached the first preset duration, the close-up shot of the current speaker can be switched, so as to achieve a smooth switching of close-up shots between multiple speakers.

[0013] In some possible implementations, the method further includes: detecting the face position information and body position information of each user in each frame of the video frame data; combining the face position information and body position information of each user in different frames according to the temporal information of the video frame data to obtain a position association model corresponding to each user; extracting the face feature information of each user in each frame of the video frame data; matching the face feature information in different frames to obtain a face feature association model corresponding to each user; extracting the body feature information of each user in each frame of the video frame data; matching the body feature information in different frames to obtain a body feature association model corresponding to each user; and obtaining a tracking model corresponding to each user based on the above position association model, the above face feature association model, and the above body feature association model corresponding to each user.

[0014] Through the technical solutions described above, the facial and body positions of each user are identified in continuous images within the video frame data. Then, the facial and body positions of each user in different frame images are correlated to construct a positional correlation model for each user. Furthermore, the facial and body features of each user are identified in continuous images within the video frame data. Then, the facial and body features of each user in different frame images are correlated to construct a facial feature correlation model and a body feature correlation model for each user. Accurate tracking of users in the scene is supported through multimodal temporal correlation data, including body position, facial features, and body features. The continuous data sequence ensures that even if user information is inaccurate or incomplete at certain moments, the tracking effect remains unaffected.

[0015] In some possible implementations, the above-mentioned camera tracking of the current speaker based on the current speaker's tracking model includes: acquiring the current position information of each user in the current frame image, performing position intersection calculation by associating the current position information of each user with the position of the current speaker in the current frame image, and obtaining the position intersection score of each user in the current frame image; acquiring the current facial feature information of each user in the current frame image, performing face similarity matching by associating the current facial feature information of each user with the facial feature of the current speaker in the current frame image, and obtaining the face matching score of each user in the current frame image; acquiring the current body feature information of each user in the current frame image, and performing face similarity matching by associating the current facial feature information of each user with the facial feature of the current speaker in the current frame image. For each user, the current body feature information is associated with the current speaker's body feature to perform body similarity matching, resulting in a body matching score for each user in the current frame image. Based on the position intersection score, face matching score, and body matching score of each user in the current frame image, the similarity score between each user in the current frame image and the current speaker is determined. Users whose similarity scores meet the preset similarity conditions are identified as the current speaker in the current frame image, and the current position information, current face feature information, and current body feature information of the current speaker in the current frame image are updated to the tracking model corresponding to the current speaker.

[0016] Through the technical solutions in the above possible implementation methods, when continuously tracking the current speaker, the position, face, and body of each user are analyzed in the new current frame image. The information in the current frame image is matched with the corresponding information of the current speaker in previous frames. The tracking result of the current speaker in the current frame is obtained from the matching score of multiple data. This allows for accurate tracking even if the tracked person loses some feature information in a certain frame, through a highly robust tracking algorithm. Furthermore, after a new frame is successfully tracked, the new position information and feature information are updated in its tracking model to ensure the stability and reliability of subsequent tracking.

[0017] In some possible implementations, after displaying the close-up of the current speaker in the video conference screen, the method further includes: if it is detected that the duration of the current speaker leaving the speaking scene reaches a second preset duration, or the duration of all speaking in the current scene reaches a second preset duration, then the conference screen is switched to a panoramic view of the current scene.

[0018] Using the technical solutions in the above possible implementation methods, if the current speaker leaves the speaking scene when displaying a close-up view, the meeting screen can be switched back to a panoramic view after the speaker leaves for a second preset time or after all speaking in the scene has ended for a second preset time, thus avoiding lag in the meeting screen when there is no one speaking or no speaker in the screen.

[0019] In some possible implementations, determining the user whose speaking probability meets the preset condition among all users as the current speaker includes: determining the user whose speaking probability exceeds the preset probability threshold and is the highest probability value among all users as the current speaker; or, determining the candidate speaker whose speaking probability exceeds the preset probability threshold among all users and whose location matches the sound source location of the aforementioned sound data, and determining the candidate speaker with the highest speaking probability value among all candidate speakers as the current speaker.

[0020] Through the technical solutions in the above possible implementation methods, after the speech prediction model outputs the speaking probability of each user, it can directly select the user whose speaking probability exceeds the preset threshold and has the highest probability as the current speaker; it can also filter multiple users whose speaking probability exceeds the threshold by the sound source location of the sound data, and determine the user with the highest probability value that matches the sound source location as the current speaker.

[0021] In some possible implementations, the above-mentioned combination of the frequency domain features and the facial features corresponding to each user and input into the speech prediction model to obtain the speech probability output by the speech prediction model for each user includes: performing feature concatenation and integration on the frequency domain features and the facial features corresponding to each user to obtain the fused features corresponding to each user, and inputting the fused features of each user into the speech prediction model to obtain the speech probability output by the speech prediction model for each user.

[0022] Through the technical solutions in the above possible implementation methods, when predicting the probability of a user speaking, the frequency domain features and the facial features corresponding to each user can be concatenated and integrated first, and then the fused features of each user can be input into the speaking prediction model to obtain the speaking probability of each user. Directly fusing the features and using a model for prediction can reduce the amount of computation to a certain extent.

[0023] In some possible implementations, the above-mentioned combination of the frequency domain features and the facial features corresponding to each user and input into the speech prediction model to obtain the speech prediction model outputting the speech probability for each user includes: simultaneously inputting the frequency domain features and the facial features corresponding to each user into a first prediction model and a second prediction model to obtain a first speech probability output by the first prediction model for each user and a second speech probability output by the second prediction model for each user, wherein the first prediction model and the second prediction model are different; inputting the first speech probability and the second speech probability corresponding to the same user into a third speech model to obtain the final speech probability output by the third prediction model for each user.

[0024] Using the technical solutions described above, when predicting the probability of a user speaking, one approach is to simultaneously input the frequency domain features and the facial features corresponding to each user into different first and second prediction models to obtain the first and second speaking probabilities for each user. Then, the first and second speaking probabilities of the same user are input into a third speaking model to obtain the final speaking probability for each user. This simultaneous prediction of multimodal features using multiple different prediction models allows for a better combination of the performance and advantages of each model, resulting in more accurate output results.

[0025] In some possible implementations, the aforementioned audio data is acquired based on a microphone array device. After acquiring the audio data and video frame data from the speaking scenario of the video conference, the method further includes: calculating the sound source location of the audio data based on the position of each microphone device in the microphone array device and the time difference between receiving the audio data between every two microphone devices; determining whether there are multiple users at the sound source location based on the video frame data; if there is a single user at the sound source location, determining the user at the sound source location as the current speaker; if there are multiple users at the sound source location, performing the step of extracting the frequency domain features of the audio data.

[0026] Using the technical solutions described above, when collecting sound data, the number of users at the sound source location can be determined first. If only one user exists at that location, they can be directly identified as the current speaker. If multiple users exist, the speaker can be determined by combining video frame data. This preliminary determination based on the sound source location simplifies the speaker identification process and improves speaker location efficiency.

[0027] In some possible implementations, after calculating the sound source location of the sound data, the method further includes: determining whether the duration of the sound at the sound source location reaches a third preset duration; if it does, then determining the sound source location as a valid location, and performing the step of determining whether there are multiple users at the sound source location based on the video frame data.

[0028] When locating a speaker based on the sound source location using the technical solutions described above, it is also necessary to note that in order to avoid misjudgment caused by brief sounds of informal speeches, after locating the sound source location, it is possible to further determine whether there is continuous sound at the sound source location for a period of time. If so, it means that this is a valid location that can be used to locate the speaker; if the sound disappears quickly, it means that it may be some noise, and the sound source location can be considered invalid. Therefore, the sound data can be left unprocessed to avoid wasting unnecessary computing resources.

[0029] In some possible implementations, after collecting the audio data and video frame data from the speaking scenario of the video conference, the method further includes: encoding the audio data to obtain audio features; generating voiceprint features corresponding to the audio features based on a voiceprint recognition model; comparing the similarity of the voiceprint features with at least one voiceprint feature in an online voiceprint database to determine whether there is a target voiceprint feature in the database whose similarity to the voiceprint feature meets a preset matching condition; the online voiceprint database is used to store the voiceprint features corresponding to all historical speakers in the video conference; if the target voiceprint feature exists, the historical speaker corresponding to the target voiceprint feature is determined as the current speaker; if the target voiceprint feature does not exist, the step of extracting the frequency domain features of the audio data is performed.

[0030] By employing the technical solutions described above, the voiceprints of users who spoke during the meeting are stored in an online voiceprint database. When new voice data is detected, its corresponding voiceprint features are first compared with those in the online database. If a match is found, the speaker can be directly located, improving the efficiency of speaker identification. The online voiceprint database only stores voiceprint data that appeared during the current meeting, enhancing its applicability in meetings and avoiding unnecessary storage space usage due to the presence of voiceprints from individuals not present at the meeting.

[0031] In some possible implementations, after determining the user whose speaking probability meets the preset condition among all users as the current speaker, the method further includes: updating the above-mentioned voiceprint features and the correspondence between the above-mentioned voiceprint features and the current speaker to the above-mentioned online voiceprint database.

[0032] Through the technical solutions described above, even if the current speaker is not matched in the voiceprint database, the voiceprint data can be updated to the online voiceprint database after the speaker is identified, so as to facilitate rapid identification when the same speaker speaks multiple times in the future.

[0033] Secondly, embodiments of this application provide a speaker identification device, the device comprising:

[0034] The data acquisition module is used to collect audio data and video frame data in the speaking scenarios of video conferences;

[0035] The feature extraction module is used to extract the frequency domain features of the above sound data, and to extract the facial features of each user in the speaking scene based on the video frame data.

[0036] The multimodal prediction module is used to combine the frequency domain features and the facial features corresponding to each user and input them into the speech prediction model to obtain the speaking probability output by the speech prediction model for each user, and determine the user whose speaking probability meets the preset conditions among all users as the current speaker.

[0037] Thirdly, embodiments of this application provide a computer storage medium storing multiple instructions adapted for loading by a processor and executing the steps of the above-described method.

[0038] Fourthly, embodiments of this application provide a terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being adapted to be loaded by the processor and to execute the steps of the above-described method.

[0039] Fifthly, a computer program product is provided, comprising: computer program code, which, when run on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation thereof. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 is an exemplary system architecture diagram of a speaker identification method provided in an embodiment of this application;

[0042] Figure 2 is a flowchart illustrating a speaker identification method provided in an embodiment of this application;

[0043] Figure 3 is a flowchart of the logical steps of a speech prediction model provided in an embodiment of this application;

[0044] Figure 4 is a schematic diagram of a process for tracking attendees on screen according to an embodiment of this application;

[0045] Figure 5 is a flowchart illustrating a speaker identification method provided in an embodiment of this application;

[0046] Figure 6 is a flowchart illustrating a speaker identification method provided in an embodiment of this application;

[0047] Figure 7 is a logical step diagram of a speaker identification method provided in an embodiment of this application;

[0048] Figure 8 is a structural block diagram of a speaker identification device provided in an embodiment of this application;

[0049] Figure 9 is a schematic diagram of the structure of a terminal provided in an embodiment of this application. Detailed Implementation

[0050] To make the features and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. Furthermore, in the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; the word "and / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist, for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, in the description of the embodiments of this application, "multiple" refers to two or more.

[0052] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0053] When hosting large-scale or multi-party video conferences, remote participants often struggle to quickly and accurately capture the speaker's face and subtle expressions, potentially impacting the effective transmission and depth of information comprehension. To overcome this challenge, modern video conferencing technology has introduced speaker identification and tracking systems, enabling automatic speaker location and real-time close-up shots. This targeted approach to video conferencing not only significantly improves the accuracy and efficiency of information transmission but also lays the foundation for in-depth discussions and efficient decision-making.

[0054] Specifically, there are various technical approaches to automatically locating speakers, mainly divided into manual and automatic methods. Manual speaker location, while traditional, is flexible. It mainly relies on the meeting host or operator to manually switch the screen view to the current speaker according to the meeting's progress. Although this method requires human intervention, it can still play an important role in specific situations (such as when it is necessary to particularly emphasize a participant or in a specific scenario).

[0055] Automatic speaker location represents a trend towards intelligence in video conferencing, typically involving speaker identification based on audio and video information. Audio-based speaker location primarily utilizes audio signals for speech recognition and sound source localization to determine the speaker's position. This method is particularly suitable for tracking voice information, but it cannot consistently achieve accurate results when multiple people are speaking simultaneously or when audio quality is poor. Video-based speaker location uses facial recognition and behavior analysis to identify participants in the frame and predict and track the speaker's movements based on visual cues such as facial orientation and gestures. However, the accuracy of this method heavily relies on facial orientation; accuracy drops significantly when the speaker is speaking from the side, and when the face is not visible, such as when the speaker is facing away from the camera, mouth movements cannot be determined, making speaker identification impossible. Another approach combines audio and video, fusing audio-based and video-based speaker identification results for speaker location. However, this fusion method ignores the deep correlation between sound and video information, resulting in unsatisfactory final results.

[0056] Therefore, this application provides a speaker identification method to solve the technical problem of the inability to accurately locate the speaker at a meeting.

[0057] Please refer to Figure 1, which is an exemplary system architecture diagram of a speaker identification method provided in an embodiment of this application.

[0058] As shown in Figure 1, the system architecture may include a terminal 101, a network 102, and a server 103. The network 102 serves as the medium for providing a communication link between the terminal 101 and the server 103. The network 102 may include various types of wired or wireless communication links, such as wired communication links including fiber optic cables, twisted-pair cables, or coaxial cables, and wireless communication links including Bluetooth, Wireless-Fidelity (Wi-Fi), or microwave communication links.

[0059] Terminal 101 can interact with server 103 via network 102 to receive messages from or send messages to server 103. Alternatively, terminal 101 can interact with server 103 via network 102 to receive messages or data sent to server 103 by other users. Terminal 101 can be hardware or software. When terminal 101 is hardware, it can be various electronic devices, including but not limited to smartphones, laptops, and desktop computers. When terminal 101 is software, it can be installed in the aforementioned electronic devices and can be implemented as multiple software programs or software modules (e.g., to provide distributed services) or as a single software program or software module; no specific limitation is made here.

[0060] In this embodiment, terminal 101 first collects audio data and video frame data from the speaking scene of a video conference; for the audio data and video frame data, terminal 101 further extracts the frequency domain features of the audio data, and extracts the facial features of each user in the speaking scene according to the video frame data; after determining the features of the audio and video data, terminal 101 combines the frequency domain features and the facial features corresponding to each user and inputs them into the speaking prediction model to obtain the speaking probability output by the speaking prediction model for each user, and determines the user whose speaking probability meets the preset conditions among all users as the current speaker.

[0061] Server 103 can be a business server providing various services. It should be noted that server 103 can be either hardware or software. When server 103 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 103 is software, it can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module; no specific limitations are made here.

[0062] Alternatively, the system architecture may not include server 103. In other words, server 103 may be an optional device in the embodiments of this specification. That is, the method provided in the embodiments of this specification can be applied to a system structure that only includes terminal 101. The embodiments of this application do not limit this.

[0063] It should be understood that the number of terminals, networks, and servers in Figure 1 is only illustrative, and can be any number of terminals, networks, and servers depending on the implementation needs.

[0064] Please refer to Figure 2, which is a flowchart illustrating a speaker identification method provided in an embodiment of this application. The execution entity in this embodiment can be a terminal performing speaker identification, a processor within the terminal performing the speaker identification method, or a speaker identification service within the terminal performing the speaker identification method. For ease of description, the specific execution process of the speaker identification method will be described below using the processor within the terminal as an example.

[0065] As shown in Figure 2, the speaker identification method may include at least:

[0066] S202. Collect audio data and video frame data from the speaking scene of the video conference.

[0067] Optionally, identifying a speaker solely through audio data suffers from significant accuracy degradation in complex scenarios such as simultaneous speech by multiple speakers or noise interference. Furthermore, unstable audio quality directly impacts tracking performance. Conversely, identifying a speaker solely through video frame data relies heavily on visual recognition technologies like facial recognition or motion analysis. Frequent changes in speaker position, poor lighting conditions, or the presence of obstructions severely affect recognition accuracy. Therefore, considering that audio and video data can respectively express different aspects of a speaker's characteristics, this embodiment of the application utilizes both audio and video data simultaneously for a more accurate analysis of the speaker on-site.

[0068] In some possible embodiments, video conferences involve remote communication among multiple parties. For example, user A and user B may participate in the same meeting, but user A is in meeting room A and user B is in meeting room B. When a speaker speaks, their audio and video data are collected within their offline context. Therefore, when multiple offline contexts exist simultaneously in a meeting, it's possible to first determine the speaker's context (i.e., the speaking context) across all offline contexts of the current meeting, and then further acquire the audio and video data within that speaking context. This allows for more accurate analysis of the speaker in subsequent audio and video frame data. It's important to note that to ensure sufficient data for audio and visual analysis, both audio and video frame data are continuous over a period of time. That is, audio data is not a single moment or a few moments of audio but rather a sustained duration. Similarly, video frame data consists of multiple continuously acquired image frames over a period of time; in other words, video frame data contains multiple consecutive images.

[0069] S204. Extract the frequency domain features of the audio data, and extract the facial features of each user in the speaking scene based on the video frame data.

[0070] Alternatively, current solutions, even those combining audio and visual recognition technologies, often only combine the audio and visual recognition results without focusing on the deep correlation between them. Therefore, the accuracy of this speaker identification method is not ideal in complex scenarios. Generally, when a user makes a sound, there are accompanying changes in facial expressions, body language, and lip movements. For example, in a meeting scenario, a user stands in front of an interactive screen to present and explain their design document, while other users at the meeting table sit and think. In this situation, the facial expressions, features, and other visual information of the speaker and non-speakers will differ significantly. Therefore, it can be understood that audio data and video frame data have a deep correlation in their representation, not just in the results.

[0071] Furthermore, to reflect the deep correlation between audio and video frame data in terms of representation, feature-level fusion can be performed to obtain multimodal features. Subsequent speaker prediction can then be based on these multimodal features, allowing for more accurate speaker prediction by focusing on the correlation between audio and video. Specifically, frequency domain features are extracted from the audio data, and continuous facial features of each user in the speaking scene are extracted from each frame of the video frame data. This facilitates subsequent identification of each user's speaking probability based on their frequency domain and facial features.

[0072] S206. Combine the frequency domain features and the facial features corresponding to each user and input them into the speech prediction model to obtain the speaking probability output by the speech prediction model for each user, and determine the user whose speaking probability meets the preset conditions among all users as the current speaker.

[0073] Optionally, based on the audio data and video frame data, frequency domain features of speech and facial features of each user are extracted when a speech occurs in the scene. This allows the frequency domain features to represent the frequency changes of the speaker during speech, and the facial features to represent the facial changes of each user during speech. The frequency domain features and the continuous facial features corresponding to each user are then combined, and a speech prediction model is used to predict the combined multimodal features to obtain the speaking probability output by the speech prediction model for each user. The speaking probability is the probability that each user is the speaker. Therefore, users whose speaking probabilities meet preset conditions can be identified as the current speaker. These preset conditions can be set to exceeding a preset probability threshold, the highest value among all probability values, or a comprehensive condition that combines other information to judge the speaking probability. For example, the user with the highest probability of speaking among all users can be directly identified as the current speaker; alternatively, candidate speakers whose speaking probability exceeds the preset probability threshold and whose location matches the sound source location of the audio data can be identified, and the candidate speaker with the highest speaking probability among all candidate speakers can be identified as the current speaker. Using the sound source location as the speaker selection criterion ensures that the location of the final selected current speaker is reasonable, improving the accuracy of the final result. In general, preset conditions can be set in various ways, depending on the needs of the actual application scenario, and this application embodiment does not impose a unique limitation on this.

[0074] Optionally, in the process of detecting a speaker's specific flow using a combination of audio and video information and end-to-end detection, the prediction function formula for the speaking probability is:

[0075] y = f(F, A; w);

[0076] Where F = [F1, F2, ..., F N [A1, A2, ..., A3] represents the facial features of a user extracted from N consecutive frames of video frame data: A = [A1, A2, ..., A4] T ] represents the frequency domain features extracted from the audio data, y = [y1, y2, ..., y3]. N ] represents the final probability integrated from the speaking probabilities of faces in N consecutive frames of images; w is the parameter for predicting the speaker in the model; the function f is the speech prediction model. In a feasible implementation, the speech prediction model can be a deep learning model, such as a cross-attention deep learning model, a transformer deep learning model, etc.

[0077] Furthermore, the function f can be refined to f(F, A; w) = p(E) v (F;w vE a (A;w a );w), where E v ∈R d It is a video feature extraction network used to extract facial features from video frame data; E a ∈R d This is an audio feature extraction network used to extract frequency domain features from sound data; p is the speech prediction model, which can be a single speech prediction model or a speech prediction model composed of multiple models, used to predict the final speaker outcome. The prediction result output by the speech prediction model p can be a single value, representing the judgment result for the entire video sequence, or it can be N values, representing the judgment result for each frame in the video sequence. When the speech prediction model p outputs N prediction results, a final integration calculation is needed to obtain the final speaking probability for each user.

[0078] Optionally, for ease of understanding, please refer to Figure 3, which is a flowchart of the logical steps of a speech prediction model provided in an embodiment of this application. When p is a standalone speech prediction model, as shown in Figure 3(A), the sound data is input into the audio feature extraction network E. a The frequency domain features are obtained, and the video frame data of each user is input into the video feature extraction network E. v The facial features of the users are obtained. Then, the frequency domain features and the facial features corresponding to each user are concatenated and integrated to obtain the fused features corresponding to each user. The fused features of each user are input into the speech prediction model to obtain the speech probability output by the speech prediction model for each user. Directly fusing the features and using a single model for prediction can reduce the amount of computation to a certain extent.

[0079] In another feasible implementation, when p is a speech prediction model composed of multiple integrated models, as shown in Figure 3(B), the sound data is also first input into the audio feature extraction network E. a The frequency domain features are obtained, and the video frame data of each user is input into the video feature extraction network E. vThe system obtains the user's facial features. Then, it simultaneously inputs the frequency domain features and the facial features corresponding to each user into a first prediction model and a second prediction model, respectively. This yields a first speaking probability output by the first prediction model for each user and a second speaking probability output by the second prediction model. The first and second prediction models are different, and preferably, they predict speaking probabilities from different perspectives based on their own parameters. Based on the first and second speaking probabilities, the first and second speaking probabilities corresponding to the same user are further input into a third prediction model to obtain the final speaking probability output by the third prediction model for each user. This simultaneous prediction of multimodal features using multiple different prediction models better combines the performance and advantages of each model, resulting in more accurate output results.

[0080] This application provides a speaker identification method, which involves collecting audio data and video frame data from a video conference speaking scenario; extracting frequency domain features from the audio data and extracting facial features of each user in the speaking scenario based on the video frame data; combining the frequency domain features and the facial features corresponding to each user and inputting them into a speaking prediction model to obtain the speaking probability output by the speaking prediction model for each user; and determining the user whose speaking probability meets a preset condition among all users as the current speaker. In a conference scenario, audio data and video frame data are collected, frequency domain features are extracted from the audio data, and facial features of each user are extracted from the video frame data. The frequency domain features are combined with the facial features of each user to achieve feature-level fusion. The fused multimodal features corresponding to each user are input into the speaking prediction model, enabling the speaking prediction model to more accurately predict the speaker based on the correlation and fusion information between audio and video, thereby locating the speaker among multiple users.

[0081] To improve the accuracy and efficiency of information transmission in video conferencing and to make online meetings more intelligent, it is also necessary to track each participant so that when a user speaks or when there is a user who needs attention, a close-up view of the target user can be displayed, making it easier for other participants to understand and pay attention to the meeting content and the speech.

[0082] Please refer to Figure 4, which is a flowchart illustrating a method for tracking attendees on screen according to an embodiment of this application. As shown in Figure 4:

[0083] S402. Detect the face position information and body position information of each user in each frame image according to the video frame data, and combine the face position information and body position information of each user in different frames according to the temporal information of the video frame data to obtain the position association model corresponding to each user.

[0084] Optionally, users' behaviors, actions, and expressions during a meeting are not static. Therefore, tracking is a dynamic and continuous process. Thus, tracking requires analyzing the activities of each user in the meeting based on video frame data using visual technology, constructing a tracking model for each user in the meeting, and thereby grasping the dynamic activity characteristics of users. Only then can we continue to accurately identify each user in each newly acquired image frame and achieve tracking of each user.

[0085] Optionally, user tracking primarily relies on user location information, facial features, and body characteristics. Location information is mainly used to identify patterns in user movement, while facial and body characteristics are primarily used to identify the user's identity. Therefore, tracking modeling can be achieved through one or more of the following information from video frame data: location information, facial features, and body characteristics. In a preferred embodiment provided in this application, multimodal association modeling of each user in continuous temporal sequence can be established simultaneously based on the user's location information, facial features, and body characteristics, using multimodal data to support accurate tracking of users in the frame.

[0086] Furthermore, firstly, sufficient video frame data containing multiple frames of images is collected for user tracking modeling. The video frame data with a duration of T can be represented as V = {I1, I2, ..., I...} T}, where I t For a video image at time t, while acquiring video frame data, a unique identifier is assigned to each user sequentially to facilitate distinguishing the tracking data corresponding to each user during calculation. Next, deep learning methods are used to detect the face and body position information of each user in each frame. The position information can be further refined into face and body position information, with body position information including head and / or human shape position information. Taking body position information that includes both head and human shape position information as an example, for video frame data V, the face position information F of each user in each frame can be detected. face (I t ), Head location information F head (I t Human location information F body (I tWhen detecting face, head, and human shape information using deep learning methods, three targeted models—face detection model, head detection model, and human shape detection model—can be used to detect the three aspects separately, resulting in more accurate recognition results. Alternatively, a single overall model can output the results for face, head, and human shape all at once, which is more efficient and saves more computing resources.

[0087] Optionally, the facial and body position information of each user in different frames is combined according to the temporal sequence of the video frame data, that is, the time-series information set of each user in terms of face, head, and human shape is obtained first: P face ={F face (I1), F face (I2), ..., F face (I T )}, P head and P body Similarly, information about all aspects of the same person (P) face P head P body The data are grouped together to obtain the location association model for each user.

[0088] S404. Extract the facial feature information of each user in each frame of the video frame data, match the facial feature information in different frames, and obtain the facial feature association model corresponding to each user.

[0089] Optionally, similar to the construction of location association modeling, for video frame data, facial feature information of each user in each frame is extracted, and the facial feature information in different frames is matched with each other. Specifically, the matching process involves identifying the facial information of all users in each frame, comparing the similarity of all facial information in each frame, and then determining that the facial information with the highest similarity to other frames belongs to the same user, thereby establishing a facial feature association model for each user. For example, if there are 3 people in the first frame and 3 people in the second frame, then each person in the second frame must be matched sequentially with the 3 people in the first frame, with the highest similarity being the same person.

[0090] S406. Extract the body feature information of each user in each frame of the video frame data, match the body feature information in different frames, and obtain the body feature association model corresponding to each user.

[0091] Optionally, similar to the construction of location-related modeling and facial feature-related modeling, for video frame data, the body feature information of each user in each frame is extracted, and the body feature information in different frames is matched with each other. Specifically, the matching process involves identifying the body information of all users in each frame, comparing the similarity of all body information in each frame, and then determining that each body information with the highest similarity to the body information in other frames comes from the same user, thereby establishing a body feature-related model for each user.

[0092] S408. Based on location association modeling, facial feature association modeling and body feature association modeling for each user, a tracking model for each user is obtained.

[0093] Optionally, based on location association modeling, facial feature association modeling, and body feature association modeling corresponding to each user, the same person in different frames can be associated, thus obtaining the tracking model corresponding to each user. By tracking and modeling each user in the meeting, it is possible to quickly identify and track each user in new frames.

[0094] In this embodiment, the face and body positions of each user are identified in continuous images of video frame data, and then the face and body positions of each user in different frame images are associated to construct a position association model for each user; the facial and body features of each user are identified in continuous images of video frame data, and then the facial and body features of each user in different frame images are associated to construct a facial feature association model and a body feature association model for each user; the multimodal temporal association data of body position, facial features, and body features supports accurate tracking of users in the scene, and the continuous data sequence ensures that even if the user's information is inaccurate or incomplete at some moments, it does not affect the tracking effect.

[0095] Please refer to Figure 5, which is a flowchart illustrating a speaker identification method provided in an embodiment of this application.

[0096] As shown in Figure 5, the speaker identification method may include at least:

[0097] S502. Collect audio data and video frame data from the speaking scene in the video conference.

[0098] S504. Extract the frequency domain features of the audio data, and extract the facial features of each user in the speaking scene based on the video frame data.

[0099] S506. Combine the frequency domain features and the facial features corresponding to each user and input them into the speech prediction model to obtain the speaking probability output by the speech prediction model for each user, and determine the user whose speaking probability meets the preset conditions among all users as the current speaker.

[0100] For details regarding steps S502-S506, please refer to the detailed descriptions in steps S202-S206, which will not be repeated here.

[0101] S508. Determine whether there is a close-up shot of the previous speaker in the video conference screen and determine whether the close-up duration of the previous speaker has reached the first preset duration.

[0102] Optionally, when the current speaker is located, the video conference screen may still be displaying a close-up of the previous speaker. If the screen switches to the close-up of the previous speaker and then immediately switches to a new close-up of the current speaker, it can easily lead to screen clutter and a poor user experience. Therefore, to avoid frequently switching between close-up shots of different speakers and affecting the user experience, it can be determined whether a close-up shot of the previous speaker exists and whether the duration of the previous speaker's close-up shot has reached a first preset duration. Based on whether a close-up shot already exists and the duration of the existing close-up shot, it can be determined whether the close-up shot of the current speaker needs to be played. The first preset duration can be 2 seconds, 5 seconds, etc., and can be set according to actual needs. This application embodiment does not impose a unique limitation.

[0103] S510. If there is a close-up shot of the previous speaker and the close-up duration has not reached the first preset duration, then the close-up shot of the previous speaker will continue to be displayed.

[0104] Optionally, if there is a close-up shot of the previous speaker and the close-up duration has not reached the first preset duration, the close-up shot of the previous speaker is displayed and the current speaker's close-up is not switched to temporarily, so as to avoid frequent switching of close-up shots of different users, which would result in a poor user experience.

[0105] S512. If there is no close-up shot of the previous speaker, or if there is a close-up shot of the previous speaker and the close-up duration reaches the first preset duration, then the speaker is tracked by camera based on the tracking model of the current speaker. The tracking model is constructed based on the video frame data of the current speaker in the speaking scene.

[0106] Optionally, if there is no close-up shot of the previous speaker or the playback duration of the previous speaker's close-up shot reaches a first preset duration, the camera can switch to the close-up shot of the current speaker, achieving a smooth switching of close-up shots between multiple speakers. When switching the close-up shot of the current speaker, the camera tracks the current speaker based on the tracking model of the current speaker obtained from video frame data through the steps in the above embodiments.

[0107] Optionally, after confirming that the current speaker needs to be tracked by camera, the video feed switches to a view centered on the current speaker based on tracking modeling, and the speaker's position information in the video also needs to be continuously tracked. For each new frame, firstly, based on face / body position detection, the current position information of each user in the current frame is obtained. Then, the current position information of each user is correlated with the position of the current speaker to calculate the position intersection, thus obtaining the position intersection score S for each user in the current frame. pos For users whose faces are present in the image, extract the current facial feature information of each user in the current frame image, and perform face similarity matching by associating the current facial feature information of each user with the facial features of the current speaker to obtain the face matching score S for each user in the current frame image. face For users whose faces are not present in the image, their facial information S face =0; In addition, it is necessary to obtain the current body feature information of each user in the current frame image, associate the current body feature information of each user with the body feature of the current speaker to perform body similarity matching, and obtain the body matching score S of each user in the current frame image. person Based on the location intersection score, face matching score, and body matching score of each user in the current frame image, multi-feature fusion is performed. Methods such as weighted averaging or Bayesian inference are used for fusion to obtain the final tracking result of the speaker in the current frame image. Taking the weighted averaging method as an example, the formula for determining the similarity score between each user and the current speaker in the current frame image is as follows:

[0108] S obj =W pos *S pos +W face *S face +W person *S person .

[0109] Based on this, the similarity score S is calculated. objSubsequently, users whose similarity scores meet the preset similarity criteria can be further identified as the current speaker in the current frame image, thus enabling the tracking of the speaker in the new current frame image. Furthermore, the current speaker's current location information, current facial feature information, and current body feature information in the current frame image are updated in the corresponding tracking model, achieving continuous tracking of the current speaker in each frame. The tracking result for the current speaker in the current frame is obtained from the matching scores of multiple data sources, ensuring accurate tracking even if the tracked person loses some feature information in a certain frame, thanks to a highly robust tracking algorithm. Moreover, after successful tracking in a new frame, the new location and feature information are updated in the tracking model, ensuring the stability and reliability of subsequent tracking.

[0110] S514. Display a close-up view of the current speaker in the video conference frame.

[0111] Optionally, based on the above tracking calculation, the camera device can be controlled to display a close-up image of the current speaker in the video conference frame.

[0112] S516. If it is detected that the current speaker has been away from the speaking scene for a period of time that reaches the second preset time, or that all speakers in the current scene have finished speaking for a period of time that reaches the second preset time, then the meeting screen will be switched to a panoramic view of the current scene.

[0113] Optionally, if the speaker leaves the speaking scene while displaying a close-up shot, the meeting view can be switched back to a panoramic view after the speaker has left the scene for a second preset duration or after all speaking in the scene has ended for a second preset duration. This avoids lag when no one is speaking or there is no speaker in the frame. The second preset duration can be set to 3 seconds, 5 seconds, etc. Taking a second preset duration of 3 seconds as an example, starting from the moment the speaker leaves the frame (all body features are out of the frame and cannot be tracked anymore), the screen remains for 3 seconds. If the person does not return to the frame after 3 seconds, it is determined that the person has left and the screen switches back to a panoramic view. If the person returns to the frame within 3 seconds, tracking continues.

[0114] This application provides a speaker identification method. It identifies the face and body positions of each user in continuous images within video frame data, and then associates these positions across different frames to construct a positional association model for each user. It also identifies the facial and body features of each user in continuous images within video frame data, and then associates these features across different frames to construct facial and body feature association models for each user. This multimodal temporal association data, including body position, facial features, and body features, supports accurate tracking of users in the video. The continuous data sequence ensures that even if user information is inaccurate or incomplete at certain times, the tracking effect remains unaffected. When determining the current speaker, it first checks whether the close-up of the previous speaker has exceeded a first preset duration. If the duration has not reached the first preset duration, the current speaker's close-up is not switched to temporarily. If there is no close-up of the previous speaker or the previous speaker's close-up has reached the first preset duration, the current speaker's close-up can be switched to, enabling smooth switching between close-up shots of multiple speakers. After identifying the current speaker, a close-up shot of them can be captured and displayed on the video conference screen based on their tracking model. The tracking result of the current speaker in the current frame is obtained by matching scores from multiple data sources. This allows for accurate tracking even if the person being tracked loses some feature information in a certain frame, thanks to a highly robust tracking algorithm. Furthermore, after a new frame is successfully tracked, the new location and feature information are updated in the tracking model, thereby achieving accurate, fast, and stable positioning and display of the speaker.

[0115] Please refer to Figure 6, which is a flowchart illustrating a speaker identification method provided in an embodiment of this application.

[0116] As shown in Figure 6, the speaker identification method may include at least:

[0117] S602. Collect audio data and video frame data from the speaking scene in the video conference.

[0118] For details regarding step S602, please refer to the description in step S202; it will not be repeated here.

[0119] S604. Calculate the sound source location of the sound data based on the position of each microphone device in the microphone array device and the time difference between the received sound data between every two microphone devices.

[0120] Optionally, in a speaking scenario, to better capture the speaker's speech, a microphone array device is typically used to collect sound data. This device allows for accurate identification of the sound source location. Simply put, if only one user is at the sound source location, that user can be directly identified as the speaker; however, if multiple users are at the sound source location, further precise identification is required to pinpoint the speaker. Therefore, considering that determining the sound source location is simpler and faster than predicting using multimodal features, the sound source location can be calculated first upon acquiring the sound data, and then verified whether the unique speaker can be directly located using that location.

[0121] Specifically, please refer to Figure 7, which is a logical step diagram of a speaker identification method provided in an embodiment of this application. As shown in Figure 7, when sound data is collected, the sound source location of the sound data can first be calculated based on the position of each microphone device in the microphone array device and the time difference between the sound data received between every two microphone devices.

[0122] Optionally, the specific method for calculating the sound source location is as follows: Assume there is a microphone array consisting of N microphones in the scene, and use M... i (t) represents the sound signal collected by the i-th microphone at time t. The direction of the sound source is calculated using the time difference of arrival (TDOA) of the sound signal. Assume the sound source location is (x... s y s , z s If the time difference of sound arrival at the i-th microphone is..., then the time difference of sound arrival at the i-th microphone is... Where c is the speed of sound. After the time-difference-based calculation is completed, multiple sound source locations will be obtained. Therefore, further weighting, smoothing, and other processing are needed to obtain a final sound source location.

[0123] S606. Determine whether the duration of the sound at the sound source location has reached the third preset duration; if it has, determine that the sound source location is a valid location, and execute the step of determining whether there are multiple users at the sound source location based on the video frame data.

[0124] Optionally, please refer to Figure 7. Generally, in addition to normal speaking sounds, microphones may also pick up background noise and other disturbances. Therefore, to avoid misjudgments caused by brief, informal conversations, after locating the sound source, it is possible to further determine whether there is continuous sound at that location for a period of time. That is, to determine whether the duration of the sound at the sound source location reaches a third preset duration. Specifically, Pi(t) can be used to represent whether there is a sound signal at time t. If β is a third preset duration threshold (2 seconds, 3 seconds, etc.), the sound source location is determined to be a valid location; if it is not reached, the sound source location is determined to be an invalid location, and no further processing of the sound data is required to avoid wasting unnecessary computing resources.

[0125] S608. Determine whether there are multiple users at the sound source location based on the video frame data; if there is a single user at the sound source location, determine that the user at the sound source location is the current speaker; if there are multiple users at the sound source location, perform the step of extracting the frequency domain features of the sound data.

[0126] Optionally, referring to Figure 7, after confirming the sound source location is valid, further determine whether multiple users exist at that location. If only one user exists at the valid sound source location, they can be directly identified as the current speaker. If multiple users exist at the valid sound source location, it is necessary to combine video frame data and use multimodal features of audio and video to more accurately determine the current speaker. This preliminary judgment based on the sound source location simplifies the speaker determination process and improves the efficiency of speaker localization.

[0127] S610. Encode the sound data to obtain sound features, and generate voiceprint features corresponding to the sound features based on the voiceprint recognition model.

[0128] Optionally, during a video conference, the same user may speak multiple times. If, when a user speaks for the first time, their voiceprint features are entered into the online voiceprint database corresponding to that video conference, when the user speaks again in the same conference, the speaker identification process can be streamlined by directly comparing the voiceprint features with those in the online database, rather than following a specific, complete speaker identification procedure. This improves the efficiency of speaker identification during the conference. Based on this, please refer to Figure 7. The first step should be to encode the audio data using MFCC (Mel Frequency Cepstral Coefficients) to obtain the audio feature X. mfcc Then, a voiceprint recognition model is used to generate voice features X. mfcc Corresponding voiceprint feature X new .

[0129] The voiceprint recognition model is pre-trained and converged using a sample voiceprint dataset D. k ={S k，1 S k2 S k，l The dataset contains L segments of voice data from each of K users. Based on a sample voiceprint dataset, a voiceprint recognition model is trained using a deep learning algorithm, denoted as F. voice(*), This model maps sound data to a voiceprint feature space, and after convergence, it is used to extract the voiceprint features of the sound.

[0130] S612. Compare the similarity of the voiceprint feature with at least one voiceprint feature in the online voiceprint database to determine whether there is a target voiceprint feature in the database whose similarity to the voiceprint feature meets the preset matching conditions. The online voiceprint database is used to store the voiceprint features corresponding to all historical speakers in the video conference.

[0131] Optionally, in this video conference, the online voiceprint database stores the voiceprint features corresponding to all historical speakers in the video conference. Once new voice data is detected, its corresponding voiceprint features are compared with at least one voiceprint feature in the online voiceprint database for similarity. Specifically, this can be achieved by calculating the Euclidean distance d. euclidean (X existing X existing ), or cosine distance d cosine (X existing X existing The similarity is calculated, and then it is further determined whether there is a target voiceprint feature in the database whose similarity to the voiceprint feature meets the preset matching conditions. The preset matching conditions can also be set in a variety of ways, and the embodiments of this application do not limit them.

[0132] S614. If the target's voiceprint features exist in the database, then determine the historical speaker corresponding to the target's voiceprint features as the current speaker; if the target's voiceprint features do not exist in the database, then perform the step of extracting the frequency domain features of the sound data.

[0133] Optionally, if the target voiceprint feature exists in the database, the match is successful, and the historical speaker corresponding to the target voiceprint feature in the database can be identified as the current speaker, improving the speaker identification efficiency. If the target voiceprint feature does not exist in the database, the match fails, and then it is necessary to perform the step of extracting the frequency domain features of the sound data and using multimodal data of audio and video to predict the speaking probability. The online voiceprint database only stores voiceprint data that appeared in the current meeting, enhancing the applicability of the voiceprint database in meetings and avoiding unnecessary storage space occupation due to the presence of voiceprints from people who are not in the current meeting.

[0134] S616. Extract the frequency domain features of the audio data, and extract the facial features of each user in the speaking scene based on the video frame data.

[0135] S618. Combine the frequency domain features and the facial features corresponding to each user and input them into the speech prediction model to obtain the speaking probability output by the speech prediction model for each user, and determine the user whose speaking probability meets the preset conditions among all users as the current speaker.

[0136] For details regarding steps S616-S618, please refer to the detailed descriptions in steps S204-S206, which will not be repeated here.

[0137] S620. Update the voiceprint features and the correspondence between the voiceprint features and the current speaker to the online voiceprint database.

[0138] Optionally, if the current speaker's voiceprint features are not matched in the voiceprint database, the following situations may occur: 1. The current speaker is speaking for the first time in this meeting, and the online voiceprint database has not yet stored the current speaker's voiceprint features. Therefore, the voiceprint features and the correspondence between the voiceprint features and the current speaker need to be updated in the online voiceprint database. 2. The current speaker already has voiceprint features in the database, but due to deviations in voiceprint quality, data collection quality, etc., the two voiceprint features of the speaker did not match. In this case, the current voiceprint can also be added to the database as the corresponding voiceprint of the speaker, that is, the speaker has two voiceprints in the database. Compared with ordinary voiceprint databases, the online voiceprint database used in this application embodiment only stores the voiceprints of users who have spoken in this meeting, which enhances the applicability of the voiceprint database in meetings and avoids occupying unnecessary storage space due to the presence of voiceprints of people who are not in this meeting.

[0139] This application provides a speaker identification method. When sound data is collected, the number of users at the sound source location is first determined. If only one user is present at that location, they can be directly identified as the current speaker. If multiple users are present, the speaker is determined by combining video frame data. This preliminary judgment based on the sound source location simplifies the speaker identification process and improves speaker location efficiency. When locating a speaker based on the sound source location, after locating the sound source location, it can be further determined whether there is continuous sound at that location for a period of time. If so, it indicates that this is a valid location for locating the speaker; if the sound disappears quickly, it indicates that it may be some noise, and the sound source location can be considered invalid, avoiding misjudgment due to brief sounds of informal speeches. The voiceprints of users who have spoken in the current meeting are stored in an online voiceprint database. When new sound data is detected, its corresponding voiceprint features are first compared with the voiceprint features in the online voiceprint database. If a match is successful, the speaker can be directly located, improving the speaker identification efficiency. The online voiceprint database only stores voiceprint data that appeared in the current meeting, enhancing its applicability in meetings and preventing unnecessary storage space from being occupied by voiceprints from people who are not in the current meeting. If the current speaker is not matched in the voiceprint database, the voiceprint data can be updated to the online voiceprint database after the speaker is identified, so that the same speaker can be quickly identified when speaking multiple times in the future.

[0140] Please refer to Figure 8, which is a structural block diagram of a speaker identification device provided in an embodiment of this application. As shown in Figure 8, the speaker identification device 800 includes:

[0141] The data acquisition module 810 is used to acquire audio data and video frame data in the speaking scenario of a video conference;

[0142] The feature extraction module 820 is used to extract the frequency domain features of the sound data and extract the facial features of each user in the speaking scene based on the video frame data.

[0143] The multimodal prediction module 830 is used to combine the frequency domain features and the facial features corresponding to each user and input them into the speech prediction model to obtain the speaking probability output by the speech prediction model for each user, and determine the user whose speaking probability meets the preset conditions among all users as the current speaker.

[0144] Optionally, the speaker identification device 800 further includes: a tracking module for performing video tracking of the current speaker based on the current speaker's tracking model, wherein the tracking model is constructed based on video frame data of the current speaker in the speaking scene; and displaying a close-up image of the current speaker in the video conference screen.

[0145] Optionally, the speaker recognition device 800 further includes: a close-up screen switching module, used to determine whether there is a close-up screen of the previous speaker in the video conference screen and whether the close-up duration of the previous speaker has reached a first preset duration; if there is a close-up screen of the previous speaker and the close-up duration has not reached the first preset duration, then the close-up screen of the previous speaker is kept displayed; if there is no close-up screen of the previous speaker, or if there is a close-up screen of the previous speaker and the close-up duration has reached the first preset duration, then the step of performing camera tracking on the current speaker based on the tracking model of the current speaker is executed.

[0146] Optionally, the speaker recognition device 800 further includes: a tracking model construction module, used to detect the face position information and body position information of each user in each frame of video frame data, combine the face position information and body position information of each user in different frames according to the temporal information of the video frame data to obtain the position association model corresponding to each user; extract the face feature information of each user in each frame of video frame data, match the face feature information in different frames to obtain the face feature association model corresponding to each user; extract the body feature information of each user in each frame of video frame data, match the body feature information in different frames to obtain the body feature association model corresponding to each user; and obtain the tracking model corresponding to each user based on the position association model, face feature association model, and body feature association model corresponding to each user.

[0147] Optionally, the tracking module is further configured to: acquire the current location information of each user in the current frame image; perform position intersection calculation by associating the current location information of each user with the position of the current speaker in the current frame image to obtain the position intersection score of each user in the current frame image; acquire the current facial feature information of each user in the current frame image; perform face similarity matching by associating the current facial feature information of each user with the facial feature of the current speaker in the current frame image to obtain the face matching score of each user in the current frame image; acquire the current body feature information of each user in the current frame image; perform body similarity matching by associating the current body feature information of each user with the body feature of the current speaker in the current frame image to obtain the body matching score of each user in the current frame image; determine the similarity score between each user and the current speaker in the current frame image based on the position intersection score, face matching score, and body matching score of each user in the current frame image; determine the user whose similarity score meets the preset similarity condition as the current speaker in the current frame image; and update the current location information, current facial feature information, and current body feature information of the current speaker in the current frame image to the tracking model corresponding to the current speaker.

[0148] Optionally, the speaker recognition device 800 further includes a panoramic view switching module, which switches the meeting screen to a panoramic view of the current scene if it is detected that the duration of the current speaker leaving the speaking scene reaches a second preset duration, or the duration of all speeches in the current scene ends reaches a second preset duration.

[0149] Optionally, the multimodal prediction module 830 is further configured to determine the user whose speaking probability exceeds a preset probability threshold and is the highest probability value among all users as the current speaker; or, to determine the candidate speaker whose speaking probability exceeds a preset probability threshold and whose location matches the sound source location of the sound data among all users, and to determine the candidate speaker with the highest speaking probability value among all candidate speakers as the current speaker.

[0150] Optionally, the multimodal prediction module 830 is also used to perform feature concatenation and integration on the frequency domain features and the face features corresponding to each user to obtain the fused features corresponding to each user, and input the fused features of each user into the speech prediction model to obtain the speech probability output by the speech prediction model for each user.

[0151] Optionally, the multimodal prediction module 830 is also used to simultaneously input the frequency domain features and the face features corresponding to each user into the first prediction model and the second prediction model respectively, to obtain the first speaking probability output by the first prediction model for each user and the second speaking probability output by the second prediction model for each user, wherein the first prediction model and the second prediction model are different; and input the first speaking probability and the second speaking probability corresponding to the same user into the third speaking model to obtain the final speaking probability output by the third prediction model for each user.

[0152] Optionally, the speaker identification device 800 further includes: a sound source location acquisition module, used to calculate the sound source location of the sound data based on the location of each microphone device in the microphone array device and the time difference between the received sound data between every two microphone devices; determine whether there are multiple users at the sound source location based on the video frame data; if there is a single user at the sound source location, determine that the user at the sound source location is the current speaker; if there are multiple users at the sound source location, perform the step of extracting the frequency domain features of the sound data.

[0153] Optionally, the speaker identification device 800 further includes: a sound source location validity determination module, used to determine whether the duration of the sound at the sound source location reaches a third preset duration; if it does, the sound source location is determined to be a valid location, and the step of determining whether there are multiple users at the sound source location based on video frame data is executed.

[0154] Optionally, the speaker recognition device 800 further includes: a voiceprint database matching module, used to encode sound data to obtain sound features, generate voiceprint features corresponding to the sound features based on a voiceprint recognition model; compare the similarity of the voiceprint features with at least one voiceprint feature in the online voiceprint database, and determine whether there is a target voiceprint feature in the database whose similarity to the voiceprint feature meets a preset matching condition, wherein the online voiceprint database is used to store the voiceprint features corresponding to all historical speakers in the video conference; if a target voiceprint feature exists, then the historical speaker corresponding to the target voiceprint feature is determined as the current speaker; if no target voiceprint feature exists, then the step of extracting the frequency domain features of the sound data is performed.

[0155] Optionally, the speaker identification device 800 further includes a voiceprint database management module, used to update the voiceprint features and the correspondence between the voiceprint features and the current speaker in the online voiceprint database.

[0156] Please refer to Figure 9, which is a schematic diagram of the structure of a terminal provided in an embodiment of this application. As shown in Figure 9, the terminal 900 may include: at least one terminal processor 901, at least one network interface 904, a user interface 903, a memory 905, and at least one communication bus 902.

[0157] The communication bus 902 is used to enable communication between these components.

[0158] The user interface 903 may include a display screen and a camera. Optionally, the user interface 903 may also include a standard wired interface and a wireless interface.

[0159] The network interface 904 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0160] The terminal processor 901 may include one or more processing cores. The terminal processor 901 connects to various parts within the terminal 900 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 905, and by calling data stored in the memory 905. Optionally, the terminal processor 901 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The terminal processor 901 may integrate one or more of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed on the screen; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the terminal processor 901.

[0161] The memory 905 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 905 may include a non-transitory computer-readable storage medium. The memory 905 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 905 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), instructions for implementing the various method embodiments described above, etc.; the data storage area may store data involved in the various method embodiments described above, etc. Optionally, the memory 905 may also be at least one storage device located remotely from the aforementioned terminal processor 901. As shown in FIG9, the memory 905, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a speaker identification program.

[0162] In the terminal 900 shown in Figure 9, the user interface 903 is mainly used to provide an input interface for the user and to obtain the user's input data; while the terminal processor 901 can be used to call the speaker identification program stored in the memory 905 and specifically perform the following operations:

[0163] Collect audio data and video frame data from the speaking scenes in video conferences;

[0164] Extract the frequency domain features of the audio data, and extract the facial features of each user in the speaking scene based on the video frame data;

[0165] The frequency domain features and the facial features corresponding to each user are combined and input into the speech prediction model to obtain the speaking probability output by the speech prediction model for each user. The user whose speaking probability meets the preset condition is determined as the current speaker.

[0166] In some embodiments, after determining that the user whose speaking probability meets the preset conditions among all users is the current speaker, the terminal processor 901 further performs the following steps: performing camera tracking on the current speaker based on the current speaker's tracking model, wherein the tracking model is constructed based on the video frame data of the current speaker in the speaking scene; and displaying a close-up image of the current speaker in the conference screen of the video conference.

[0167] In some embodiments, after determining that the user whose speaking probability meets the preset condition among all users is the current speaker, the terminal processor 901 further performs the following steps: determining whether there is a close-up of the previous speaker in the video conference screen and determining whether the close-up duration of the previous speaker has reached a first preset duration; if there is a close-up of the previous speaker and the close-up duration has not reached the first preset duration, then the close-up of the previous speaker is kept displayed; if there is no close-up of the previous speaker, or if there is a close-up of the previous speaker and the close-up duration has reached the first preset duration, then the step of performing camera tracking on the current speaker based on the tracking model of the current speaker is executed.

[0168] In some embodiments, the terminal processor 901 further performs the following steps: detecting the face location information and body location information of each user in each frame of video frame data, combining the face location information and body location information of each user in different frames according to the temporal information of the video frame data to obtain a location association model corresponding to each user; extracting the face feature information of each user in each frame of video frame data, matching the face feature information in different frames to obtain a face feature association model corresponding to each user; extracting the body feature information of each user in each frame of video frame data, matching the body feature information in different frames to obtain a body feature association model corresponding to each user; and obtaining a tracking model corresponding to each user based on the location association model, face feature association model, and body feature association model corresponding to each user.

[0169] In some embodiments, when the terminal processor 901 performs camera tracking of the current speaker based on tracking modeling, it specifically executes the following steps: acquiring the current position information of each user in the current frame image, performing position intersection calculation by associating the current position information of each user with the position of the current speaker in the current frame image, and obtaining the position intersection score of each user in the current frame image; acquiring the current facial feature information of each user in the current frame image, performing face similarity matching by associating the current facial feature information of each user with the facial feature of the current speaker in the current frame image, and obtaining the face matching score of each user in the current frame image; acquiring the current body feature information of each user in the current frame image, performing body similarity matching by associating the current body feature information of each user with the body feature of the current speaker in the current frame image, and obtaining the body matching score of each user in the current frame image; determining the similarity score between each user in the current frame image and the current speaker based on the position intersection score, face matching score, and body matching score of each user in the current frame image; determining the user whose similarity score meets the preset similarity condition as the current speaker in the current frame image, and updating the current position information, current facial feature information, and current body feature information of the current speaker in the current frame image to the tracking model corresponding to the current speaker.

[0170] In some embodiments, after the terminal processor 901 displays a close-up view of the current speaker in the video conference screen, it further performs the following steps: if it detects that the duration of the current speaker leaving the speaking scene reaches a second preset duration, or the duration of all speaking in the current scene ends reaches a second preset duration, then the conference screen is switched to a panoramic view of the current scene.

[0171] In some embodiments, when the terminal processor 901 determines the user whose speaking probability meets the preset condition among all users as the current speaker, it specifically performs the following steps: determining the user whose speaking probability exceeds the preset probability threshold and is the highest probability value among all users as the current speaker; or, determining the candidate speaker whose speaking probability exceeds the preset probability threshold among all users and whose location matches the sound source location of the sound data, and determining the candidate speaker with the highest speaking probability value among all candidate speakers as the current speaker.

[0172] In some embodiments, when the terminal processor 901 performs the following steps when combining the frequency domain features and the facial features corresponding to each user and inputting them into the speech prediction model to obtain the speech probability output by the speech prediction model for each user: performing feature concatenation and integration on the frequency domain features and the facial features corresponding to each user to obtain the fused features corresponding to each user, and inputting the fused features of each user into the speech prediction model to obtain the speech probability output by the speech prediction model for each user.

[0173] In some embodiments, when the terminal processor 901 combines the frequency domain features and the facial features corresponding to each user and inputs them into the speech prediction model to obtain the speech probability output by the speech prediction model for each user, it specifically performs the following steps: simultaneously inputting the frequency domain features and the facial features corresponding to each user into the first prediction model and the second prediction model to obtain the first speech probability output by the first prediction model for each user and the second speech probability output by the second prediction model for each user, wherein the first prediction model and the second prediction model are different; inputting the first speech probability and the second speech probability corresponding to the same user into the third speech model to obtain the final speech probability output by the third prediction model for each user.

[0174] In some embodiments, after the terminal processor 901 acquires audio data and video frame data in a video conference speaking scenario based on the microphone array device, it further performs the following steps: calculating the sound source location of the audio data according to the position of each microphone device in the microphone array device and the time difference between the audio data received between every two microphone devices; determining whether there are multiple users at the sound source location based on the video frame data; if there is a single user at the sound source location, determining the user at the sound source location as the current speaker; if there are multiple users at the sound source location, performing the step of extracting the frequency domain features of the audio data.

[0175] In some embodiments, after the terminal processor 901 calculates the sound source location of the sound data, it further performs the following steps: determining whether the duration of the sound at the sound source location reaches a third preset duration; if it does, determining the sound source location as a valid location, and performing the step of determining whether there are multiple users at the sound source location based on the video frame data.

[0176] In some embodiments, after the terminal processor 901 collects audio data and video frame data in the speaking scenario of a video conference, it further performs the following steps: encoding the audio data to obtain audio features, generating voiceprint features corresponding to the audio features based on a voiceprint recognition model; comparing the voiceprint features with at least one voiceprint feature in an online voiceprint database to determine whether there is a target voiceprint feature in the database whose similarity to the voiceprint feature meets a preset matching condition, wherein the online voiceprint database is used to store the voiceprint features corresponding to all historical speakers in the video conference; if a target voiceprint feature exists, the historical speaker corresponding to the target voiceprint feature is determined to be the current speaker; if no target voiceprint feature exists, the step of extracting the frequency domain features of the audio data is performed.

[0177] In some embodiments, after determining that the user whose speaking probability meets the preset conditions among all users is the current speaker, the terminal processor 901 further performs the following steps: updating the voiceprint features and the correspondence between the voiceprint features and the current speaker to the online voiceprint database.

[0178] This application also provides a computer-readable storage medium storing computer program code. When the computer program code is run on a computer, the computer executes the aforementioned method steps to implement a speaker identification method provided in the above embodiments.

[0179] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned related steps to achieve the method for adjusting the state of a device provided in the above embodiments.

[0180] In this application, the apparatus, computer-readable storage medium, computer program product or chip provided in the embodiments are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0181] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0182] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0183] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this specification are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0184] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0185] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0186] The above is a description of a speaker identification method, device, storage medium, and terminal provided in this application. For those skilled in the art, based on the ideas of the embodiments of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A speaker identification method, characterized in that, The method includes: collecting audio data and video frame data in a video conference speaking scenario; extracting frequency domain features from the audio data, and extracting facial features of each user in the speaking scenario based on the video frame data; combining the frequency domain features and the facial features corresponding to each user and inputting them into a speaking prediction model to obtain the speaking probability output by the speaking prediction model for each user, and determining the user whose speaking probability meets the preset conditions among all users as the current speaker.

2. The method according to claim 1, characterized in that, After determining that the user whose speaking probability meets the preset condition among all users is the current speaker, the method further includes: performing video tracking on the current speaker based on the tracking modeling of the current speaker, wherein the tracking modeling is constructed based on the video frame data of the current speaker in the speaking scene; and displaying a close-up image of the current speaker in the conference screen of the video conference.

3. The method according to claim 2, characterized in that, After determining that the user whose speaking probability meets the preset condition among all users is the current speaker, the method further includes: determining whether there is a close-up shot of the previous speaker in the video conference screen and determining whether the close-up duration of the previous speaker has reached a first preset duration; if there is a close-up shot of the previous speaker and the close-up duration has not reached the first preset duration, then the close-up shot of the previous speaker is kept displayed; if there is no close-up shot of the previous speaker, or if there is a close-up shot of the previous speaker and the close-up duration has reached the first preset duration, then the step of performing camera tracking on the current speaker based on the tracking model of the current speaker is executed.

4. The method according to claim 2, characterized in that, The method further includes: detecting the face position information and body position information of each user in each frame of the video frame data; combining the face position information and body position information of each user in different frames according to the temporal information of the video frame data to obtain a position association model corresponding to each user; extracting the face feature information of each user in each frame of the video frame data; matching the face feature information in different frames to obtain a face feature association model corresponding to each user; extracting the body feature information of each user in each frame of the video frame data; matching the body feature information in different frames to obtain a body feature association model corresponding to each user; and obtaining a tracking model corresponding to each user based on the position association model, the face feature association model, and the body feature association model corresponding to each user.

5. The method according to claim 4, characterized in that, The step of tracking the current speaker using video modeling based on the current speaker includes: acquiring the current position information of each user in the current frame image; performing position intersection calculation by associating the current position information of each user with the position of the current speaker in the current frame image; acquiring the current facial feature information of each user in the current frame image; performing face similarity matching by associating the current facial feature information of each user with the facial feature of the current speaker in the current frame image; and acquiring the current body feature information of each user in the current frame image. The system associates the previous body feature information with the body feature of the current speaker to perform body similarity matching, and obtains the body matching score of each user in the current frame image. Based on the position intersection score, face matching score and body matching score of each user in the current frame image, the similarity score between each user in the current frame image and the current speaker is determined. The user whose similarity score meets the preset similarity condition is determined as the current speaker in the current frame image, and the current position information, current face feature information and current body feature information of the current speaker in the current frame image are updated to the tracking model corresponding to the current speaker.

6. The method according to claim 2, characterized in that, After displaying a close-up view of the current speaker in the video conference frame, the method further includes: if it is detected that the current speaker has been away from the speaking scene for a period of time that reaches a second preset duration, or if all speaking in the current scene has ended for a period of time that reaches a second preset duration, then the conference frame is switched to a panoramic view of the current scene.

7. The method according to claim 1, characterized in that, The step of determining the user whose speaking probability meets the preset condition among all users as the current speaker includes: determining the user whose speaking probability exceeds the preset probability threshold and is the highest probability value among all users as the current speaker; or, determining the candidate speaker whose speaking probability exceeds the preset probability threshold among all users and whose location matches the sound source location of the sound data, and determining the candidate speaker with the highest speaking probability value among all candidate speakers as the current speaker.

8. The method according to claim 1, characterized in that, The step of combining the frequency domain features and the facial features corresponding to each user and inputting them into the speech prediction model to obtain the speech prediction model outputting the speech probability for each user includes: performing feature concatenation and integration on the frequency domain features and the facial features corresponding to each user to obtain the fused features corresponding to each user, and inputting the fused features of each user into the speech prediction model to obtain the speech probability output by the speech prediction model for each user.

9. The method according to claim 1, characterized in that, The step of combining the frequency domain features and the facial features corresponding to each user and inputting them into the speech prediction model to obtain the speech prediction model's output speech probability for each user includes: simultaneously inputting the frequency domain features and the facial features corresponding to each user into a first prediction model and a second prediction model to obtain a first speech probability output by the first prediction model for each user and a second speech probability output by the second prediction model for each user, wherein the first prediction model and the second prediction model are different; inputting the first speech probability and the second speech probability corresponding to the same user into a third speech model to obtain the third prediction model's output final speech probability for each user.

10. The method according to claim 1, characterized in that, The audio data is acquired based on a microphone array device. After acquiring audio data and video frame data from the speaking scenario of the video conference, the method further includes: calculating the sound source location of the audio data based on the position of each microphone device in the microphone array device and the time difference between receiving the audio data between every two microphone devices; determining whether there are multiple users at the sound source location based on the video frame data; if there is a single user at the sound source location, determining the user at the sound source location as the current speaker; if there are multiple users at the sound source location, performing the step of extracting the frequency domain features of the audio data.

11. The method according to claim 10, characterized in that, After calculating the sound source location of the sound data, the method further includes: determining whether the duration of the sound at the sound source location reaches a third preset duration; if it does, then determining the sound source location as a valid location, and performing the step of determining whether there are multiple users at the sound source location based on the video frame data.

12. The method according to claim 1, characterized in that, After collecting audio data and video frame data from the speaking scene of the video conference, the method further includes: encoding the audio data to obtain audio features, generating voiceprint features corresponding to the audio features based on a voiceprint recognition model; comparing the voiceprint features with at least one voiceprint feature in an online voiceprint database to determine whether there is a target voiceprint feature in the database whose similarity to the voiceprint feature meets a preset matching condition, wherein the online voiceprint database is used to store the voiceprint features corresponding to all historical speakers in the video conference; if the target voiceprint feature exists, then the historical speaker corresponding to the target voiceprint feature is determined to be the current speaker; if the target voiceprint feature does not exist, then the step of extracting the frequency domain features of the audio data is performed.

13. The method according to claim 12, characterized in that, After determining that the user whose speaking probability meets the preset condition among all users is the current speaker, the method further includes: updating the voiceprint features and the correspondence between the voiceprint features and the current speaker to the online voiceprint database.

14. A speaker identification device, characterized in that, The device includes: a data acquisition module for acquiring audio data and video frame data in a video conference speaking scenario; a feature extraction module for extracting frequency domain features from the audio data and extracting facial features of each user in the speaking scenario based on the video frame data; and a multimodal prediction module for combining the frequency domain features and the facial features corresponding to each user and inputting them into a speaking prediction model to obtain the speaking probability output by the speaking prediction model for each user, and determining the user whose speaking probability meets a preset condition among all users as the current speaker.

15. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions adapted for loading by a processor and executing the steps of the method as claimed in any one of claims 1 to 13.

16. A terminal, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the method as described in any one of claims 1 to 13.

Images