Method for training multi-modal speaker verification model and electronic device

Abstract

The application discloses a training method of a multi-modal speaker authentication model and an electronic device, relates to the field of speaker authentication, and comprises the following steps: obtaining an audio-video sequence sample corresponding to a given speaker; performing lip feature coding on each audio frame in the audio-video sequence sample in sequence to determine a corresponding audio sample lip latent vector sequence; performing feature coding on a speaker mouth region in each video frame in the audio-video sequence sample in sequence to determine a corresponding video sample lip latent vector sequence; calculating an audio-video sequence similarity corresponding to the audio sample lip latent vector sequence and the video sample lip latent vector sequence; and training a speaker authentication model according to the audio-video sequence similarity and an audio-video speaker alignment label. Thus, the similarity of the speaker audio and the lip shape graph of the person to be detected is compared by matching the speaker audio and the lip shape graph into the same dimension vector through the latent vector, so that the accuracy of the authentication result can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of speaker authentication technology, and in particular to a multi-modal speaker authentication model training method, a storage medium and an electronic device. BACKGROUND

[0002] With the popularity of the Internet and mobile communication technology, the production and dissemination of digital media content (including audio, video and images) has shown explosive growth. The popularity of smartphones, high-definition camera equipment and social platforms has made it easy for everyone to easily create and disseminate multimedia content.

[0003] In recent years, the rapid development of deep learning-based technologies, especially generative adversarial networks (GAN), has given rise to the "deepfake" phenomenon. Deepfake technology can generate counterfeit audio and video content with extremely high fidelity. Through large-scale data training, the generated counterfeit content is almost indistinguishable in terms of vision and hearing.

[0004] Traditional authentication technology mainly relies on the analysis of a single modality (such as video or audio), but in video speaker authentication or audio speaker authentication, it often focuses on microscopic differences in pixel-level or spectral features, and in the face of carefully post-processed or multiple counterfeit means, it is easy to miss or misjudge.

[0005] In view of the above problems, the industry has not yet proposed a better solution. SUMMARY

[0006] The present application provides a multi-modal speaker authentication model training method, a storage medium and an electronic device to at least solve the problem of misjudgment or missed judgment caused by single modal authentication methods in the related art.

[0007] In a first aspect, the embodiments of the present application provide a multi-modal speaker authentication model training method, comprising: obtaining an audio-video sequence sample corresponding to a given speaker, the audio-video sequence sample comprising a plurality of sequentially combined audio-video frame pairs and a corresponding audio-video speaker alignment label; performing lip feature encoding on each audio frame in the audio-video sequence sample in sequence to determine a corresponding audio sample lip latent vector sequence; performing feature encoding on the speaker mouth region in each video frame in the audio-video sequence sample in sequence to determine a corresponding video sample lip latent vector sequence; calculating the audio-video sequence similarity corresponding to the audio sample lip latent vector sequence and the video sample lip latent vector sequence; and training a speaker authentication model according to the audio-video sequence similarity and the audio-video speaker alignment label.

[0008] In a second aspect, an electronic device is provided, which includes at least one processor, and a memory connected with the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the training method of the multi-modal speaker verification model according to any of the embodiments of the present application.

[0009] In a third aspect, the embodiments of the present application provide a storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the training method of the multi-modal speaker verification model according to any of the embodiments of the present application.

[0010] In a fourth aspect, the embodiments of the present application provide a computer program product, which includes computer programs / instructions, and the computer programs / instructions, when executed by a processor, implement the steps of the training method of the multi-modal speaker verification model according to any of the embodiments of the present application.

[0011] The embodiments of the present application have the following beneficial effects:

[0012] By jointly analyzing the multi-modal features of the audio-video samples, the lip latent vector sequence of the audio sample and the lip latent vector sequence of the video sample are introduced, and the similarity between the two is calculated, so that the model can capture the temporal correlation and semantic consistency between the audio-video content. Fake content is often difficult to maintain high consistency in terms of time alignment, lip movement trajectory and voice characteristics. By sequentially encoding each audio-video frame in the audio-video sequence sample, vector quantization is performed in the same latent space, and the lip feature latent vectors are matched, the temporal information is fully captured, and the accuracy of the verification result can be effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0014] Figure 1 A flowchart of an example of the training method of the multi-modal speaker verification model according to the embodiments of the present application is shown;

[0015] Figure 2 An operation flowchart of an example of obtaining an audio-video sequence sample according to the embodiments of the present application is shown;

[0016] Figure 3An operation flowchart of an example of training a speaker authentication model according to an embodiment of the present application is shown.

[0017] Figure 4 A structural schematic diagram of an embodiment of an electronic device of the present application. DETAILED DESCRIPTION

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0019] Currently, speaker authentication methods have experienced traditional methods of using face lip key point thresholds to simply judge lip movement, combining audio VAD comparison to start point simple recognition. This method relies heavily on single modal accuracy and synchronization between audio and visual modalities. The visual model fails when the mouth does not pronounce when judging lip movement. The auditory modality is difficult to distinguish whether the speaker is the speaker. Now, deep learning is gradually used to determine. However, the existing method still tends to use single modal feature extraction and then send it to a synchronization expert discriminator to determine whether the speaker is speaking. This classifier has a small recognition window and is difficult to capture long-time features. For example, some sounds such as "four", "ten", "fourteen" in audio often correspond to the same mouth shape, which brings great uncertainty to the model. A 5-frame (200ms) or even 15-frame (600ms) window is not enough to determine whether the speaker is speaking. Therefore, a method and device capable of capturing long-time speaking state are urgently needed.

[0020] It should be noted that in order to achieve reliable speaker authentication results, the industry has proposed some related technical solutions:

[0021] In patent application CN118711621A, a method and device for audio-video multi-modal joint authentication, an electronic device and a storage medium are disclosed. The device uses the modality consistency and time synchronization information of the audio and image dimensions in the input video. It proposes to extract the average characteristics of the entire segment from the audio or image, the consistency of the onset / offset of the sound / lip movement, and the consistency of the lip movement generated by the speech and the face lip movement to construct a three-branch audio-video joint authentication method, thereby improving the performance of audio-video joint authentication. (1) In terms of extracting the average characteristics of the entire segment from the audio or image, the Wav2Vec2.0 self-supervised pre-training model is used to extract the entire speech feature representation, the ViT model trained based on the CLIP method is used to extract the face feature, and then the speech and face features are spliced and input into the multi-layer MSTCN for authenticity classification. (2) In terms of the consistency of the onset / offset of the sound / lip movement, the onset and offset of the speech and the onset / offset of the face lip movement are detected, and the audio and lip movement onset / offset time information is vectorized, and the vector cosine similarity is calculated. (3) In terms of the consistency of the lip movement generated by the speech and the face lip movement, the modality conversion of the lip movement image generated based on the speech is performed, the audio and video frame rate are aligned, the Wav2Lip method is used to generate the audio-driven lip movement sequence, the lip movement features generated by the audio and the face lip movement features in the original video are extracted based on the LipNet, and then the multi-layer MSTCN network is used for lip movement consistency classification. (4) Based on the authentication fusion decision of the three-branch module, the weighted fusion of the similarity scores of the authentication of the spliced speech features and face features, the authentication of the consistency of the onset / offset of the speech and the onset / offset of the face lip movement, and the authentication of the consistency of the speech to the lip movement modality conversion and the face lip movement.

[0022] However, in the above patent application CN118711621A, the information after modality conversion (audio to lip movement image) is compared with the to-be-detected speaker lip movement to realize multi-modal information confirmation, but this method ignores the negative effects brought by the conversion model wav2lip (audio to lip movement image), such as the accuracy of generating lip movement images and the overfitting of the wav2lip model itself, which will affect the accuracy of the final judgment. Therefore, Wav2Lip is essentially a generation model, which belongs to an indirect feature comparison method. During the conversion process, some details in the original audio-video may be lost, resulting in a decrease in detection performance. In addition, the use of wav2lip to realize multi-modal feature deep fusion still has insufficient granularity, and wav2lip relies on the pre-trained synchronous expert network (SyncNet), which is sensitive to the frame rate synchronization and alignment errors of audio and video, and has weak robustness in real complex scenarios.

[0023] In view of this, Figure 1 A flowchart showing an example of a training method for a multi-modal speaker authentication model according to an embodiment of the present application is shown.

[0024] As shown in the figure, in step S110, an audio-video sequence sample corresponding to a given speaker is obtained, which contains a plurality of sequentially combined audio-video frame pairs and corresponding audio-video speaker alignment labels.

[0025] It should be noted that the audio-video sequence sample can be obtained from various sources, such as public datasets, real scene collection, or synthetic data generation. In addition, the environmental scenarios covered by the sample can also be diverse, such as different resolutions, environmental noise, and different device acquisition conditions, to improve the robustness and generalization ability of the model.

[0026] In addition, the audio-video speaker alignment label can be a positive label (i.e., sequence alignment) or a negative label (i.e., sequence misalignment), the positive label indicating that each audio-video frame pair in the sequential combination is audio-video aligned, and the negative label indicating that each audio-video frame pair in the sequential combination is not completely audio-video aligned, for example, one or more audio-video frame pairs can be audio-video inconsistent. Accordingly, the audio-video speaker alignment label can be used as a speaker authentication label, for example, the positive label corresponds to a real speaker, and the negative label corresponds to a virtual speaker. It should be understood that the audio-video speaker alignment label can be constructed in various ways, such as manual annotation, to ensure high accuracy of the label information, so that the model can accurately capture the real features of audio-video synchronization during the training process.

[0027] In step S120, the lip feature of each audio frame in the audio-video sequence sample is encoded to determine the corresponding audio sample lip latent vector sequence.

[0028] In some embodiments, an audio feature extraction network (such as Mel spectrum, MFCC feature, or Wav2Vec model) can be used to encode the audio signal. In addition, during the encoding process, the model should focus on the key frequency band related to the mouth pronunciation (such as the speech frequency band of 250Hz-4kHz), to avoid the interference of redundant non-speech information on the model. Further, the encoded features are further mapped to the audio sample lip latent vector sequence, which can capture the lip movement features in the audio signal, such as mouth shape changes, pronunciation rhythm, pitch changes, etc.

[0029] Preferably, a timing model (such as Bi-LSTM, Transformer) can also be introduced to capture more fine-grained audio dynamic features, enhance the ability to capture time-dependent information in the audio sequence, and improve the performance of forgery detection in long-time audio.

[0030] In step S130, the speaker mouth region in each video frame in the audio-video sequence sample is respectively feature-encoded to determine the corresponding video sample lip latent vector sequence.

[0031] In some embodiments, when extracting features of the mouth region in the video frame, the face detection and key point positioning algorithm can be used to accurately extract the mouth region and align its spatial position. For example, the face key point sequence is extracted from the detected face by using the face alignment module, and the Lip (mouth region or lip) image is obtained by framing the face image with specific face key points. More specifically, 68 key points are obtained by face detection technology, and the key point number [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 30] is framed to obtain the Lip image by cutting out the lip region in the face. By accurately positioning the mouth region and extracting the lip motion features, the model can more effectively identify the subtle abnormalities of the mismatch between the mouth shape and the voice in the fake video.

[0032] Here, various deep neural networks (such as ResNet, MobileNet, etc.) can be used to encode the key dynamic features of the mouth region (such as the opening and closing of the lips, the change of the mouth shape). In addition, a time series model (such as LSTM, Transformer, etc.) can be used to capture the dynamic changes of the mouth movement to form the video sample lip latent vector sequence, and the time series modeling is introduced to further enhance the detection performance of the model on fast mouth movements (such as smiling and fast speaking).

[0033] Preferably, the video encoding process can also consider multi-scale feature extraction to ensure that the model has strong adaptability to video samples of different resolutions and shooting angles.

[0034] In step S140, the audio sample lip latent vector sequence and the video sample lip latent vector sequence are calculated to determine the audio-video sequence similarity.

[0035] Here, a variety of similarity calculation methods can be used to compare the latent vector sequences of the audio-video samples, such as cosine similarity, Euclidean distance, etc., to calculate the corresponding audio-video sequence similarity.

[0036] In step S150, the speaker authentication model is trained according to the audio-video sequence similarity and the audio-video speaker alignment label.

[0037] Here, by scoring the similarity of the audio-video sample latent vector sequence, it can to some extent reflect the problems such as audio-video misalignment, mouth shape mismatch, abnormal speech rhythm caused by forgery. In some embodiments, the similarity score and the audio-video speaker alignment label are used as a supervised signal, and the model is trained through a classification loss, thereby improving the understanding ability of the model for real audio-video sequences and ensuring that the model has stronger practicality in real scenarios.

[0038] In the embodiments of the present application, the original audio signal and the mouth region of the original video frame are directly encoded, and the similarity of the speaking audio and the lip shape graph of the person to be detected is compared by matching them into the same dimension vector through the latent vector. No additional intermediate conversion model is introduced to convert the audio to an image, avoiding the noise or information loss that may be introduced in the conversion process. Specifically, the audio encoding part uses lip-related features (such as phoneme timing, sound spectrum features) to extract the lip latent vector sequence of the audio sample; the video encoding part directly extracts the lip latent vector sequence from the mouth region, forming a direct feature matching mechanism to ensure that the key information in the original data is fully retained. Compared with CN118711621A, the conversion bias caused by Wav2Lip is avoided, and the sensitivity and accuracy of the model to forged samples are improved, especially in non-standard scenarios (such as low-resolution video, noise interference).

[0039] Through the embodiments of the present application, multi-modal feature encoding and timing feature capture are used to form a complete audio-video multi-modal feature extraction, similarity analysis and model training closed loop, so that the model can more accurately and stably identify the forged content with inconsistent speakers in the audio-video.

[0040] In some examples of the embodiments of the present application, the audio-video speaker alignment label includes multiple frame alignment labels and sequence alignment labels. Each frame alignment label is used to indicate whether the corresponding audio-video frame pair is aligned, and the sequence alignment label is used to indicate whether all audio-video frame pairs are aligned, and the sequence alignment label is used to indicate whether the audio-video sequence sample corresponds to a real speaker or a virtual speaker. Therefore, by additionally adding frame alignment labels in the audio-video speaker alignment label, the audio-video alignment annotation is further refined based on the original sequence alignment label, which not only improves the discrimination ability of the model, but also significantly enhances the robustness of the model in complex scenarios.

[0041] Specifically, the forged audio-video content is not always full-time distortion, and partial deep fake technology can only forge a segment (such as a few seconds of audio-video misplacement) or local mouth shape (such as misplacement only when pronouncing some words) in the video. In the traditional method based on sequence alignment label, the overall consistency of the entire audio-video sequence is often judged, and this local anomaly can be ignored, resulting in missed detection. After introducing the frame alignment label, each frame is individually assigned an audio-video alignment state, so that the model can learn more fine-grained audio-video synchronization features in training, making it easier to identify short-time forgery, local mouth shape misplacement and other hidden forgery methods. Thus, the sensitivity of the model to local forgery (such as modifying a few frames of mouth shape or audio rhythm) is improved, and the risk of missed detection due to overall judgment errors is reduced.

[0042] In addition, as a more fine-grained supervision signal, the frame alignment label is not only used to judge authenticity, but also further indicates the location of abnormal audio-video frames. In the model inference process, when audio-video forgery is detected, the introduction of the frame alignment label can identify which specific frames have abnormalities, improve the positioning accuracy of the model to the forgery location, and enhance the explainability of the detection result, providing strong support for manual review and further analysis.

[0043] Figure 2 An operation flowchart of an example of obtaining an audio-video sequence sample according to an embodiment of the present application is shown.

[0044] As shown in Figure 2 In step S210, a given speaker video is obtained.

[0045] Here, the given speaker video can be a video about the speaker collected in real time by a camera device, for example, collecting the speaker video using a high-resolution camera device to ensure the video resolution and frame rate. In addition, the given speaker video can also be a recorded video, such as a host explanation video, which contains complete face picture information.

[0046] In some embodiments, the face image can also be compressed and feature extracted, which can effectively compress the original 256*256*3 face information to 512 dimensions or even lower dimensions, so that long-time sequence mapping can be performed.

[0047] In step S220, a positive sample audio-video sequence is formed according to a preset number of audio frames and video frames aligned with the corresponding timestamp in the given speaker video, and a negative sample audio-video sequence is formed according to a preset number of audio frames and video frames misaligned with the corresponding timestamp in the given speaker video.

[0048] As for the construction details of the positive samples, the corresponding audio frames and video frames can be accurately extracted from the video according to a preset number (for example, N consecutive frame pairs) by using video timestamp information. In addition, a fixed sampling window and interframe interval can be designed so that each positive sample covers a complete pronunciation process and a period of lip movement change.

[0049] As for the construction details of the negative samples, a time offset strategy is adopted, and according to a preset number of negative sample generation, a time offset (for example, shifting the audio frame forward or backward by several frames) between the audio frame and the video frame is artificially set to generate an obviously mismatched audio-video sequence. In addition to the fixed offset, a certain random offset can also be introduced to form various mismatch modes, simulating various forgery or mismatch situations that may occur in actual scenarios.

[0050] In some embodiments, the multimodal speaker verification training positive samples and negative samples are selected. The positive samples are selected as follows: 1-30 seconds of audio-video sequence is randomly selected from the given speaker video as the training positive sample. In addition, the negative samples are generated by the following operations: first, part or all of the selected positive sample audio-video sequence is randomly reorganized. Then, two audio-video sequences of different times are randomly selected , and , are reorganized into , or , .

[0051] In addition, the given speaker alignment label is given. If the audio-video content is consistent, it is 1, otherwise it is 0. Therefore, the frame label of a 10-frame sequence example can be marked as 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, and the overall sequence synchronization label is 0. If all frame labels are 1, the overall sequence label is 1, the frame label is recorded as frame_label, and the overall label is recorded as clip_label.

[0052] Through the embodiments of the present application, positive sample audio-video sequences and negative sample audio-video sequences are constructed from given speaker videos. The positive samples constructed by strict timestamp alignment truly reflect the characteristics of audio-video synchronization, providing reliable positive training data for the model, which helps to improve the recognition accuracy of the model for normal audio-video synchronization patterns. By strategically constructing negative samples through time offset, the model can effectively simulate forgery or error synchronization situations and provide key training information for identifying forged samples. In addition, diversified negative samples can be designed through various timestamp offset strategies, so that the model has higher robustness when facing different types of forgery methods.

[0053] Furthermore, joint training with both positive and negative samples enables the model to better balance the feature differences between different modalities during optimization, improving cross-modal matching and anti-spoofing capabilities. Based on a bidirectional supervision mechanism, the model can not only accurately determine whether audio and video are synchronized, but also improve its sensitivity to subtle misalignments, thus demonstrating stronger anti-spoofing capabilities in practical applications.

[0054] Figure 3 A flowchart illustrating an example of training a speaker authentication model according to an embodiment of this application is shown.

[0055] like Figure 3 As shown, in step S310, the InfoNCE loss is calculated based on the similarity of the first audio-video sequence corresponding to the positive sample audio-video sequence and the similarity of the second audio-video sequence corresponding to the negative sample audio-video sequence.

[0056] In some implementations, the audio-visual latent vectors extracted from positive and negative samples are normalized to ensure that similarity calculations are performed at the same scale. Furthermore, cosine similarity or Euclidean distance can be used to calculate the similarity of the first audio-visual sequence corresponding to the positive sample and the similarity of the second audio-visual sequence corresponding to the negative sample. The similarity of the positive samples is used as a positive term, and the similarity of the negative samples as a negative term; the InfoNCE loss is designed using the idea of ​​contrastive learning.

[0057] For example, the InfoNCE loss can be expressed as follows:

[0058] Equation (1)

[0059] In the formula, This represents the similarity between the first audio and video sequences corresponding to the positive samples. Indicates the first The similarity score corresponding to each negative sample represents the number of negative samples.

[0060] By employing InfoNCE loss, the model's ability to align features between audio and video modalities is significantly enhanced, making positive samples more compact in the feature space while negative samples are spaced further apart. Contrastive learning enables the model to better distinguish between genuine synchronization information and forged information, thereby improving the system's defense against forgery attacks.

[0061] In step S320, the speaker authentication classification loss is calculated based on the speaker alignment labels in the audio and video.

[0062] Exemplarily, the category information of "real" or "fake" is labeled for each sample through the audio-video speaker alignment label. In some embodiments, a binary cross-entropy loss can be adopted as the speaker authentication classification loss, and the real label alignment supervision can effectively measure the accuracy of the model in judging the authenticity of the sample.

[0063] In step S330, the speaker authentication model is trained according to the speaker authentication classification loss and the InfoNCE loss.

[0064] In some embodiments, the InfoNCE loss and the speaker authentication classification loss are weighted and summed with a certain weight (for example, the respective contributions are adjusted by a hyperparameter) to form a final joint loss function:

[0065] , Equation (2)

[0066] In the equation, is the classification loss, and is a hyperparameter for balancing the two-part loss.

[0067] Exemplarily, in the actual training operation, a pre-training and fine-tuning combined strategy can be adopted, and the audio and video feature encoders are pre-trained first, and then the entire system is trained end-to-end using the joint loss. Specifically, in the early stage of training, the weight of the InfoNCE loss can be appropriately increased to ensure the feature alignment effect, and after the model is stable, the weight of the classification loss can be increased to refine the discrimination ability.

[0068] Through the embodiments of the present application, by designing a hybrid loss function and joint training, the model can simultaneously consider both the feature alignment and the authenticity classification tasks, which not only ensures the close matching between modalities, but also improves the discrimination accuracy of the fake samples. Through the InfoNCE loss calculation, the distance between positive samples can be effectively reduced, and the negative samples can be pushed away, so as to strengthen the cross-modal alignment of audio-video features. Through the speaker authentication classification loss, the model is provided with a clear discrimination supervision signal to help distinguish real and fake audio-video samples. Based on the multi-task joint optimization strategy, the fake attack can be more effectively prevented in practical applications, and the high accuracy and reliability of the speaker authentication system can be ensured.

[0069] In some examples of the embodiments of the present application, the classification model includes cascaded encoding layers, frame-level MLP layers, and segment-level MLP layers.

[0070] The encoding layer can adopt a convolutional neural network layer structure or a Transformer module, etc., and can adopt a multi-layer encoding structure. Exemplarily, the first layer can focus on extracting low-level local features (such as edges, textures or fine-grained audio spectrum features), and subsequent layers gradually extract higher-level abstract information.

[0071] In the frame-level MLP layer, the features extracted by the cascaded encoding layer are fed into one or more fully connected layers (MLP) for each merged audio-video frame pair. In addition, the frame-level MLP layer can be designed as a lightweight structure to focus on the slight alignment difference features of a single frame. After frame-level MLP processing, each audio-video frame pair can generate an embedding vector of a fixed dimension. The frame-level MLP layer can focus on the slight alignment difference features of a single frame, improve the sensitivity of the model to instantaneous speech and lip movement changes, and enhance the detection ability of slight mismatches in the fake sample.

[0072] By comparing each audio-video frame at the frame level, the system can find slight deviations that are not easily detected on the whole segment, thereby greatly improving the sensitivity and accuracy of the fake detection.

[0073] In the segment-level MLP layer, the continuous frame embedding vectors output by the frame-level MLP layer can be time-aggregated to obtain a vector representing the overall features of a segment, so that the final segment-level discrimination result is more in line with the judgment standard of the overall audio-video synchronization and authenticity.

[0074] At the segment level, by aggregating and modeling the overall frames, the global consistency and trend changes of the audio-video sequence are captured, which helps to smooth local noise or occasional abnormalities, so that the model makes a more stable judgment on the overall audio-video synchronization.

[0075] In some embodiments, the audio_latent and the real to be detected person video_latent are compared in sequence similarity, and the cosine distance can be selected to measure the distance between them. At the same time, the InfoNCE loss is calculated, which will promote the similarity of the positive sample pair (audio_latent and video_latent) to be larger, and the similarity of the negative sample pair (for example, the video_latent of other videos) to be smaller, and the audio-video latent is merged and then passed through the Encoder (1*1 convolutional conv1d) and the frame-level MLP and the segment-level MLP to predict the label through the bce loss (binary cross-entropy loss function optimization).

[0076] Compared with the traditional scheme of classifying only the entire sequence or segment, in the embodiments of the present application, two levels of alignment detection at the frame level and the segment level are introduced, so as to capture the inconsistency of audio-video information at multiple scales, make full use of the complementary advantages of local and global information, and form a more detailed and comprehensive judgment system. In addition, by jointly using the contrast loss and the classification loss, supervision is carried out at different levels, effectively reducing the gap between positive and negative samples, and improving the discrimination ability of the overall model. Thus, the use of multi-scale and multi-level joint optimization strategy not only improves the detection sensitivity of the system to fake samples, but also greatly enhances the robustness and stability of the overall system.

[0077] As to the details of the feature encoding of the audio frame in step S120, in some embodiments, the HuBERT feature of the audio frame is extracted, and the HuBERT feature is encoded based on a speech encoding network to determine the corresponding audio sample lip latent vector, and the speech encoding network is supervised training by using a mean squared error (MSE) loss function. The mean squared error loss has the characteristics of fast convergence speed and gradient smoothing, provides a clear regression target, helps the network to stably converge and efficiently learn, and makes the output feature more discriminative.

[0078] Specifically, a pre-trained HuBERT model is used to extract features from each frame to obtain a high-dimensional vector reflecting speech information, prosody and resonance characteristics. The HuBERT model uses a self-supervised learning method and is trained on large-scale speech data, so the extracted features have strong generalization ability and robustness and can capture fine-grained changes when speaking.

[0079] Here, the structure of the speech encoding network can be diverse, for example, it can include multiple layers of fully connected layers or convolutional layers, which map the HuBERT feature vector to the low-dimensional audio sample lip latent vector in the space consistent with the lip features extracted from the video, facilitating cross-modal alignment.

[0080] In addition, the encoding network is supervised trained by using a mean squared error loss function, the goal of which is to minimize the Euclidean distance between the audio lip latent vector output by the speech encoding network and the pre-labeled or extracted lip target vector. Thus, by using the pre-constructed audio-video alignment data as a supervision signal and using the mean squared error loss function to supervise the training of the network, it is ensured that the encoded audio features can be aligned with the video lip features in the same semantic space, which can greatly improve the accuracy and robustness of the system in fine-grained audio-video synchronization detection.

[0081] Details of the feature encoding of the video frame in step S130, in some embodiments, the speaker's mouth region in the video frame is encoded based on a deep residual network to determine the corresponding video sample lip latent vector, the deep residual network includes a convolution block attention module.

[0082] Here, a deep residual network (ResNet) is used as a video feature encoder, which effectively alleviates the gradient vanishing problem in deep network training through residual connection, thereby realizing deeper feature extraction. Illustratively, the deep residual network includes multiple convolution blocks, each of which includes a convolution layer, a batch normalization layer and an activation function (such as ReLU), and the input is directly passed to the subsequent layer through a residual path. The final output of the deep residual network is a video sample lip latent vector, which compactly and effectively represents the mouth features in the video frame in a low-dimensional latent space, can extract more rich and high-level visual features, and effectively capture the subtle movement and morphological changes of the mouth. In addition, through the residual connection, the network remains stable in a deeper level, which improves the convergence speed and overall accuracy of the model.

[0083] Details of the convolution block attention module, in some embodiments, a convolution block attention module (such as CBAM or SE module) is embedded in each or part of the convolution blocks, which automatically adjusts the importance of different regions and channels in the feature map by calculating spatial attention and channel attention at the same time, and can highlight important features related to the mouth region while suppressing irrelevant information and background noise.

[0084] Illustratively, the attention module can be embedded in several convolution blocks in the deep residual network, which strengthens the expression of key feature information layer by layer. During the training process, the attention mechanism will adaptively learn the optimal weight distribution, so that the final output of the lip latent vector has stronger discriminative ability.

[0085] Through the embodiments of the present application, the speaker's mouth region in the video frame is encoded based on a deep residual network, and a convolution block attention module is integrated, which realizes the goal of extracting high-quality, fine-grained lip features from video data. The deep residual network effectively extracts multi-level visual information, and the attention module strengthens the feature expression of the key region, based on multi-level and multi-module collaborative optimization, which significantly improves the expression ability of the video sample lip latent vector.

[0086] It should be noted that, for the foregoing method embodiments, in order to simply describe, the foregoing method embodiments are all described as a series of actions, but those skilled in the art should know that the present application is not limited to the order of the actions described, because according to the present application, some steps can be performed in other order or at the same time. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily necessary for the present application. In the above embodiments, the description of each embodiment is focused on, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0087] In some embodiments, the embodiments of the present application provide a non-volatile computer readable storage medium, the storage medium stores one or more programs including execution instructions, the execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to execute the training method of the multi-modal speaker verification model of any one of the above embodiments of the present application.

[0088] In some embodiments, the embodiments of the present application also provide a computer program product, the computer program product includes a computer program stored on a non-volatile computer readable storage medium, the computer program includes program instructions, when the program instructions are executed by a computer, the computer program instructions make the computer execute the training method of the multi-modal speaker verification model of any one of the above embodiments.

[0089] In some embodiments, the embodiments of the present application also provide an electronic device, which includes at least one processor, and a memory connected with the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the training method of the multi-modal speaker verification model.

[0090] Figure 4 is a hardware structure schematic diagram of an electronic device for executing the training method of the multi-modal speaker verification model provided by another embodiment of the present application, as shown in Figure 4 The device includes:

[0091] one or more processors 410 and a memory 420, Figure 4 The processor 410 is taken as an example in the memory 420.

[0092] The device for executing the training method of the multi-modal speaker verification model can also include an input device 430 and an output device 440.

[0093] The processor 410, the memory 420, the input device 430 and the output device 440 can be connected through a bus or other means, Figure 4The bus connection is taken as an example.

[0094] The memory 420 is a non-volatile computer readable storage medium, which can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the training method of the multi-modal speaker verification model in the embodiments of the present application. The processor 410 executes various functions and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 420, that is, implements the training method of the multi-modal speaker verification model in the above method embodiments.

[0095] The memory 420 can include a program storage area and a data storage area. The program storage area can store an operating system and application programs required by at least one function; the data storage area can store data created during use of the electronic device, etc. In addition, the memory 420 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory 420 can optionally include a memory disposed remotely with respect to the processor 410, which can be connected to the electronic device through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0096] The input device 430 can receive input digital or character information, and generate signals related to user settings and function control of the electronic device. The output device 440 can include a display device such as a display screen.

[0097] The one or more modules are stored in the memory 420, and when executed by the one or more processors 410, perform the training method of the multi-modal speaker verification model in any of the above method embodiments.

[0098] The above product can perform the method provided in the embodiments of the present application, and has the corresponding function modules and beneficial effects of performing the method. Technical details not described in detail in the embodiments can be referred to the method provided in the embodiments of the present application.

[0099] The electronic device of the embodiments of the present application exists in various forms, including but not limited to:

[0100] (1) Mobile communication device: the characteristic of this kind of device is that it has mobile communication function, and the main target is to provide voice and data communication. This kind of terminal includes: smart phone, multimedia phone, functional phone, and low-end phone, etc.

[0101] (2) Ultra-mobile personal computer devices: These devices belong to the category of personal computers, have computing and processing functions, and generally have mobile Internet access features. Such terminals include PDA, MID and UMPC devices, etc.

[0102] (3) Portable entertainment devices: These devices can display and play multimedia content. Such devices include audio and video players, handheld game consoles, electronic books, and smart toys and portable car navigation devices.

[0103] (4) Other on-board electronic devices with data interaction functions, such as car-mounted devices installed on vehicles.

[0104] The device embodiments described above are merely illustrative, wherein the units illustrated as separate components can or can not be physically separated, and the components illustrated as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment.

[0105] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus a general hardware platform, and of course can also be implemented by hardware. Based on such understanding, the above technical solutions or the part that contributes to the related art can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some part of the embodiments.

[0106] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training a multi-modal speaker verification model, comprising: obtaining audio-video sequence samples corresponding to a given speaker, the audio-video sequence samples comprising a plurality of audio-video frame pairs and corresponding audio-video speaker alignment labels; performing lip feature encoding on each audio frame in the audio-video sequence samples to determine a corresponding audio sample lip latent vector sequence; performing feature encoding on a speaker mouth region in each video frame in the audio-video sequence samples to determine a corresponding video sample lip latent vector sequence; calculating audio-video sequence similarity corresponding to the audio sample lip latent vector sequence and the video sample lip latent vector sequence; training a speaker verification model according to the audio-video sequence similarity and the audio-video speaker alignment labels; wherein the audio-video sequence samples comprise positive sample audio-video sequences and negative sample audio-video sequences, and the training of the speaker verification model according to the audio-video sequence similarity and the audio-video speaker alignment labels comprises: calculating an InfoNCE loss according to a first audio-video sequence similarity corresponding to the positive sample audio-video sequences and a second audio-video sequence similarity corresponding to the negative sample audio-video sequences; calculating a speaker verification classification loss according to the audio-video speaker alignment labels; training the speaker verification model according to the speaker verification classification loss and the InfoNCE loss; wherein the calculation of the speaker verification classification loss according to the audio-video speaker alignment labels comprises: merging the audio sample lip latent vector sequence and the video sample lip latent vector sequence to obtain a corresponding audio-video sample lip latent vector sequence; inputting the audio-video sample lip latent vector sequence into a classification model to output a corresponding audio-video alignment prediction result; calculating a speaker verification classification loss according to the audio-video alignment prediction result and the audio-video speaker alignment labels; the InfoNCE loss is expressed by the following formula: ， In the formula, represents the similarity of the first audio-video sequence corresponding to the positive sample, represents the similarity corresponding to the i th negative sample.

2. The method of claim 1, wherein, the obtaining of the audio-video sequence samples corresponding to the given speaker comprises: obtaining a given speaker video; forming positive sample audio-video sequences according to a preset number of audio frames and video frames corresponding to a timestamp in the given speaker video, and forming negative sample audio-video sequences according to a preset number of audio frames and video frames corresponding to a misaligned timestamp in the given speaker video.

3. The method of claim 1, wherein, the audio-video speaker alignment labels comprise a plurality of frame alignment labels and a sequence alignment label; each frame alignment label is used to indicate whether a corresponding audio-video frame pair is audio-video aligned; the sequence alignment label is used to indicate whether all audio-video frame pairs are audio-video aligned, and the sequence alignment label is used to indicate whether the audio-video sequence sample corresponds to a real speaker or a virtual speaker.

4. The method of claim 1, wherein, the speaker verification classification loss is a binary cross-entropy loss.

5. The method of claim 4, wherein, the classification model comprises a cascaded encoding layer, a frame-level MLP layer, and a segment-level MLP layer.

6. The method of claim 1, wherein, the feature encoding for the audio frame comprises: HuBERT features are extracted for the audio frames and the HuBERT features are encoded based on a speech encoding network that is supervised trained with a mean squared error loss function to determine corresponding audio sample lip latent vectors.

7. The method of claim 1, wherein, For feature encoding of the video frames, including: encoding the speaker mouth region in the video frames based on a deep residual network to determine corresponding video sample lip latent vectors; the deep residual network including a convolutional block attention module.

8. An electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of any one of claims 1-7.

Images