Methods, models, training methods, devices, and electronic equipment for speech lip reading recognition.

Abstract

This disclosure provides a method, apparatus, and electronic device for training a speech lip-reading recognition model. The method involves inputting training speech and video into a video encoder and a speech encoder, respectively, for feature extraction to obtain training speech features and training video features. These training speech and video features are then input into a cross-modal adversarial memory module to generate training multimodal audiovisual features. Based on the training multimodal audiovisual features and a preset loss function, it is determined whether the speech lip-reading recognition model has been successfully trained. Compared with related technologies, this disclosure, by inputting the extracted training speech and video features into a cross-modal adversarial memory module for cross-modal processing, can better integrate the training speech and video features, obtaining more information about the correlation between speech and video features, thereby improving the model's generalization and recognition capabilities.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a method, apparatus, and electronic device for training a speech lip-reading recognition model. Background Technology

[0002] Lip-reading is a technique for reconstructing spoken speech from an input video sequence of a speaker's face. It's a video-to-speech mapping, and the reconstructed speech should include information about the spoken content in the video and speech-related information about the speaker. Initially, researchers chose text as an intermediate representation between video and speech in lip-reading tasks; however, with further research, most researchers have opted to use acoustic features (such as spectrograms, Mel spectrograms, and LPC linear predictive coding) as the intermediate feature representation for video-to-speech.

[0003] Traditional feedforward networks use CNNs, RNNs, etc., to compress the training set into latent vectors for storage; this results in the loss of much useful information during the encoding process and fails to effectively capture latent vectors related to acoustic features. Most existing models focus on speech reconstruction for a single speaker, meaning the model is speaker-dependent. Therefore, achieving effective lip-reading in multi-speaker scenarios is a pressing issue. Summary of the Invention

[0004] This disclosure provides a method, model training method, apparatus, and electronic device for speech lip-reading recognition. Its main purpose is to achieve speech lip-reading recognition under multi-speaker conditions and improve the generalization ability of the speech purity recognition model.

[0005] According to a first aspect of this disclosure, a method for training a speech lip-reading recognition model is provided, comprising:

[0006] The training audio and video are input into the video encoder and the audio encoder respectively for feature extraction, resulting in training audio features and training video features.

[0007] The training speech features and the training video features are input into the cross-modal adversarial memory module to generate training multimodal audiovisual features;

[0008] Based on the multimodal audiovisual features used for training and the preset loss function, determine whether the speech lip reading recognition model has been trained successfully.

[0009] Optionally, the cross-modal adversarial memory module includes key memory and value memory, and the method further includes:

[0010] The training video features are used as the training prior video features of the key memory for updating, wherein the key memory is addressed in the value memory based on the training prior video features;

[0011] The training speech features are updated as the training prior speech features of the value memory, wherein the training prior video features and the training prior speech features are stored in a one-to-one correspondence.

[0012] Optionally, the step of inputting the training audio and video into the video encoder and the audio encoder respectively for feature extraction to obtain training audio features and training video features includes:

[0013] Extract video frame data from the training audio and video;

[0014] Feature extraction is performed on the video frame data to obtain the training video features;

[0015] Speech data is extracted from the training speech video, and a first Mel spectrogram is generated based on the speech data;

[0016] Feature extraction is performed on the first Mel spectrogram to obtain the training speech features.

[0017] Optionally, the cross-modal adversarial memory module further includes a modality classifier;

[0018] The step of inputting the training speech features and the training video features into the cross-modal adversarial memory module to generate training multimodal audiovisual features includes:

[0019] The training speech features and the training video features are input into the modality classifier for cross-modal processing;

[0020] The similarity between the cross-modal processed training video features and the training prior video features is calculated to obtain the first similarity weight;

[0021] The similarity between the training speech features after cross-modal processing and the training prior speech features is calculated to obtain the second similarity weight;

[0022] The first similarity weight and the prior speech features used for training are weighted and summed to obtain the multimodal audiovisual features used for training.

[0023] Optionally, the step of inputting the training speech features and the training video features into the modality classifier for cross-modal processing includes:

[0024] The training speech features are mapped to a training mapping network to obtain a first feature vector;

[0025] The training video features are mapped to the training mapping network to obtain a second feature vector;

[0026] Based on the modality classifier, cross-modal processing is performed on the first feature vector and the second feature vector.

[0027] Optionally, the method further includes:

[0028] Calculate the first feature distance between the first feature vector and the second feature vector;

[0029] After performing cross-modal processing on the first feature vector and the second feature vector, the classification loss is calculated;

[0030] Based on the first feature distance and the classification loss, it is determined whether the modality classifier has been trained successfully.

[0031] Optionally, determining whether the speech lip-reading recognition model has been trained successfully based on the multimodal audiovisual features used for training and a preset loss function includes:

[0032] The training prior speech features and the second similarity weights are weighted and summed to obtain pseudo-speech features;

[0033] Calculate the second feature distance between the pseudo-speech features and the training speech features;

[0034] Calculate the KL divergence between the first similarity weight and the second similarity weight;

[0035] A second Mel spectrogram is generated based on the multimodal audiovisual features used in the training.

[0036] Based on the first Mel spectrum and the second Mel spectrum, calculate the first loss function;

[0037] Calculate the preset loss function, wherein the preset loss function is the sum of the first loss function, the first feature distance, the second feature distance, and the KL divergence, minus the classification loss;

[0038] Based on the calculation result of the preset loss function, it is determined whether the speech lip reading recognition model has been trained successfully.

[0039] Optionally, calculating the first loss function based on the first Mel spectrum and the second Mel spectrum includes:

[0040] Based on the first Mel spectrogram, the first predicted Mel spectrogram, and the second Mel spectrogram, calculate the second loss function;

[0041] Based on the first Mel spectrogram, the second predicted Mel spectrogram, and the third Mel spectrogram, the third loss function is calculated;

[0042] The first loss function is obtained by adding the second loss function and the third loss function.

[0043] Optionally, the method further includes:

[0044] Before inputting the training audio-visual video into the video encoder for feature extraction, the human figures in the training audio-visual video are cropped and aligned.

[0045] According to a second aspect of this disclosure, a method for speech lip reading recognition is provided, the method comprising:

[0046] The video to be identified is input into a video encoder for feature extraction to obtain video features;

[0047] The video features are input into the cross-modal adversarial memory module to generate multimodal audiovisual features;

[0048] The multimodal audiovisual features are reconstructed using a speech decoder to generate a fourth Mel spectrogram;

[0049] Based on the vocoder, the fourth Mel spectrogram is converted to obtain the speech for lip reading recognition.

[0050] Optionally, the cross-modal adversarial memory module further includes a modality classifier, key memory, and value memory;

[0051] The step of inputting the video features into the cross-modal adversarial memory module to generate multimodal audiovisual features includes:

[0052] Calculate the addressing similarity between the video features and the prior video features stored in the key memory;

[0053] Based on the addressing similarity, the corresponding prior speech features in the value memory are searched;

[0054] The video features and the prior speech features are weighted and summed to obtain the multimodal audiovisual features.

[0055] Optionally, the method further includes:

[0056] Before inputting the video to be identified into the video encoder for feature extraction, the human figures in the video to be identified are cropped and aligned.

[0057] According to a third aspect of this disclosure, a training apparatus for a speech lip-reading recognition model is provided, comprising:

[0058] The first training unit is used to input the training audio and video into the video encoder and the audio encoder respectively for feature extraction, so as to obtain the training audio features and the training video features.

[0059] The second training unit is used to input the training speech features and the training video features into the cross-modal adversarial memory module to generate training multimodal audiovisual features.

[0060] The first determining unit is used to determine whether the speech lip reading recognition model has been trained successfully based on the first Mel spectrogram for training, the second Mel spectrogram for training, and a preset loss function.

[0061] Optionally, the device further includes:

[0062] The first update unit is used to update the training video features as the training prior video features of the key memory, wherein the key memory is addressed in the value memory based on the training prior video features; the cross-modal adversarial memory module includes key memory and value memory;

[0063] The second update unit is used to update the training speech features as the training prior speech features of the value memory, wherein the training prior video features and the training prior speech features are stored in a one-to-one correspondence.

[0064] Optionally, the first training unit includes:

[0065] The first extraction module is used to extract video frame data from the training audio and video.

[0066] The second extraction module is used to extract features from the video frame data to obtain the training video features;

[0067] The third extraction module is used to extract speech data from the training speech video and generate a first Mel spectrogram based on the speech data.

[0068] The fourth extraction module is used to extract features from the first Mel spectrogram to obtain the training speech features.

[0069] Optionally, the second training unit includes:

[0070] A cross-modal processing module is used to input the training speech features and the training video features into the modality classifier for cross-modal processing; the cross-modal adversarial memory module further includes a modality classifier;

[0071] The first calculation module is used to calculate the similarity between the cross-modal processed training video features and the training prior video features to obtain the first similarity weight.

[0072] The second calculation module is used to calculate the similarity between the training speech features after cross-modal processing and the training prior speech features to obtain the second similarity weight.

[0073] The third calculation module is used to perform weighted summation of the first similarity weight and the prior speech features used for training to obtain the multimodal audiovisual features used for training.

[0074] Optionally, the cross-modal processing module is further configured to:

[0075] The training speech features are mapped to a training mapping network to obtain a first feature vector;

[0076] The training video features are mapped to the training mapping network to obtain a second feature vector;

[0077] Based on the modality classifier, cross-modal processing is performed on the first feature vector and the second feature vector.

[0078] Optionally, the device further includes:

[0079] The first calculation unit is used to calculate the first feature distance between the first feature vector and the second feature vector;

[0080] The second computing unit is used to calculate the classification loss after performing cross-modal processing on the first feature vector and the second feature vector;

[0081] The second determining unit is used to determine whether the modality classifier has been trained successfully based on the first feature distance and the classification loss.

[0082] Optionally, the first determining unit includes:

[0083] The fourth calculation module is used to perform weighted summation on the prior speech features used for training and the second similarity weight to obtain pseudo-speech features;

[0084] The fifth calculation module is used to calculate the second feature distance between the pseudo-speech features and the training speech features;

[0085] The sixth calculation module is used to calculate the KL divergence between the first similarity weight and the second similarity weight;

[0086] A generation module is used to generate a second Mel spectrogram based on the training multimodal audiovisual features;

[0087] The seventh calculation module is used to calculate the first loss function based on the first Mel spectrogram and the second Mel spectrogram;

[0088] The eighth calculation module is used to calculate the preset loss function, wherein the preset loss function is the sum of the first loss function, the first feature distance, the second feature distance, and the KL divergence, minus the classification loss;

[0089] The determination module is used to determine whether the speech lip reading recognition model has been trained successfully based on the calculation result of the preset loss function.

[0090] Optionally, the seventh calculation module is further configured to:

[0091] Based on the first Mel spectrogram, the first predicted Mel spectrogram, and the second Mel spectrogram, calculate the second loss function;

[0092] Based on the first Mel spectrogram, the second predicted Mel spectrogram, and the third Mel spectrogram, the third loss function is calculated;

[0093] The first loss function is obtained by adding the second loss function and the third loss function.

[0094] Optionally, the device further includes:

[0095] The first cropping unit is used to crop and align the human figures in the training audio-visual video before inputting the training audio-visual video into the video encoder for feature extraction.

[0096] According to a fourth aspect of this disclosure, a speech lip-reading recognition apparatus is provided, the apparatus comprising:

[0097] The extraction unit is used to input the video to be recognized into the video encoder for feature extraction to obtain video features;

[0098] The first generation unit is used to input the video features into the cross-modal adversarial memory module to generate multimodal audiovisual features;

[0099] The second generation unit is used to reconstruct the features of the multimodal audiovisual features using a speech decoder to generate a fourth Mel spectrogram;

[0100] The conversion unit is used to convert the fourth Mel spectrogram based on the vocoder to obtain the speech for lip reading recognition.

[0101] Optionally, the first generation unit includes:

[0102] The ninth calculation module is used to calculate the addressing similarity between the video features and the prior video features stored in the key memory; the cross-modal adversarial memory module also includes a modality classifier, key memory, and value memory;

[0103] The lookup module is used to find the corresponding prior speech features in the value memory based on the addressing similarity.

[0104] The tenth calculation module is used to perform weighted summation of the video features and the prior speech features to obtain the multimodal audiovisual features.

[0105] Optionally, the device further includes:

[0106] The second cropping unit is used to crop and align the human figures in the video to be identified before inputting the video to be identified into the video encoder for feature extraction.

[0107] According to a fifth aspect of this disclosure, an electronic device is provided, comprising:

[0108] At least one processor; and

[0109] A memory communicatively connected to the at least one processor; wherein,

[0110] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods described in the first and / or second aspects.

[0111] According to a sixth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the methods described in the first and / or second aspects described above.

[0112] According to a seventh aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the methods described in the first and / or second aspects described above.

[0113] This disclosure provides a method, apparatus, and electronic device for training a speech lip-reading recognition model. The method involves inputting training speech and video into a video encoder and a speech encoder, respectively, for feature extraction to obtain training speech features and training video features. These training speech and video features are then input into a cross-modal adversarial memory module to generate training multimodal audiovisual features. Based on the training multimodal audiovisual features and a preset loss function, it is determined whether the speech lip-reading recognition model has been successfully trained. Compared with related technologies, this disclosure, by inputting the extracted training speech and video features into a cross-modal adversarial memory module for cross-modal processing, can better integrate the training speech and video features, obtaining more information about the correlation between speech and video features, thereby improving the model's generalization and recognition capabilities.

[0114] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0115] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0116] Figure 1 A flowchart illustrating a training method for a speech lip-reading recognition model provided in this embodiment of the disclosure;

[0117] Figure 2 A flowchart illustrating another training method for a speech lip-reading recognition model provided in this embodiment of the present disclosure;

[0118] Figure 3 A flowchart illustrating another training method for a speech lip-reading recognition model provided in this embodiment of the present disclosure;

[0119] Figure 4 A flowchart for training a cross-modal adversarial memory module;

[0120] Figure 5 A schematic flowchart illustrating a speech lip-reading recognition method provided in this embodiment of the disclosure;

[0121] Figure 6 A flowchart illustrating another speech lip-reading recognition method provided in this embodiment of the disclosure;

[0122] Figure 7 A schematic flowchart of a training device for a speech lip-reading recognition model provided in an embodiment of this disclosure;

[0123] Figure 8 A schematic flowchart of a training device for another speech lip-reading recognition model provided in an embodiment of this disclosure;

[0124] Figure 9 A schematic flowchart of a speech lip-reading recognition device provided in an embodiment of this disclosure;

[0125] Figure 10 A schematic flowchart of another speech lip-reading recognition device provided in an embodiment of this disclosure;

[0126] Figure 11 A schematic block diagram of an example electronic device provided for embodiments of this disclosure. Detailed Implementation

[0127] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0128] The following describes, with reference to the accompanying drawings, a method for training a speech lip-reading recognition model, an apparatus, and an electronic device according to embodiments of the present disclosure.

[0129] Figure 1 This is a flowchart illustrating a training method for a speech lip-reading recognition model provided in an embodiment of this disclosure.

[0130] like Figure 1 As shown, the method includes the following steps:

[0131] Step 101: Input the training audio and video into the video encoder and audio encoder respectively for feature extraction to obtain training audio features and training video features.

[0132] In embodiments of this disclosure, training audio-visual content is used as the training input for the speech lip-reading recognition model. The training audio-visual content includes video information and speech information of the speaker. A video encoder is used to extract video features related to the spoken content from the input training audio-visual content; a speech encoder is used to extract speech-related features of the speaker.

[0133] The speech and video in training data represent two distinct modalities, providing information from different perspectives and in different formats. By extracting features from the training speech and video using separate video and speech encoders, a more comprehensive and multi-dimensional data representation can be obtained, better supporting subsequent analysis tasks. The speech and video in training data contain relevant information about the same event or object. Extracting features from them separately and then fusing the resulting speech and video features improves data utilization. This allows for better use of available data, thereby enhancing model performance and effectiveness.

[0134] In summary, by inputting the training speech and video data into the speech encoder and video encoder respectively for feature extraction, more comprehensive information can be provided, data utilization can be improved, and model performance can be enhanced.

[0135] Step 102: Input the training speech features and the training video features into the cross-modal adversarial memory module to generate training multimodal audiovisual features.

[0136] In the embodiments of this disclosure, the speech and video in the training audio-visual materials represent two different modalities. To establish the differences between these two modalities, it is necessary to eliminate the influence of these modal differences. Therefore, the training speech features and training video features are input into a cross-modal adversarial memory module. This module can eliminate the modal differences between the training speech features and training video features. Furthermore, by calculating the similarity between these features and the priors stored within the module, a representative training multimodal audio-visual feature is output after weighted summation.

[0137] By inputting training speech and video features into a cross-modal adversarial memory module (CMM), feature information from two different modalities can be fused. The CMM can learn the correlations and differences between speech and video, thereby generating fused multimodal audiovisual features. Through adversarial learning, the CMM helps the model learn more generalizable feature representations. This improves the model's performance on unseen data and enhances its generalization ability.

[0138] In summary, by inputting the speech and video features used for training into a cross-modal adversarial memory module, multimodal audiovisual features for training can be generated. This method enables cross-modal fusion, improves feature representation and enhances the model's generalization ability, thereby improving the model's performance and effectiveness.

[0139] Step 103: Based on the multimodal audiovisual features used for training and the preset loss function, determine whether the speech lip reading recognition model has been trained successfully.

[0140] In some embodiments of this disclosure, a preset loss function is used to guide the training of the speech lip-reading recognition model by calculating the feature loss between the multimodal audiovisual features generated by the cross-modal adversarial memory module and the features in the original speech and video. The preset loss function may include, but is not limited to, the loss function used when training a video encoder or a speech encoder, the loss function of the cross-modal adversarial memory module, or the loss function of a module such as a decoder that generates speech information based on the multimodal audiovisual features used for training. The preset loss function may be a combination of these loss functions, or it may be a loss function calculated based on the multimodal audiovisual features used for training and used to calculate the loss of the entire model; this disclosure does not limit this.

[0141] In other embodiments of this disclosure, the training of the speech lip-reading recognition model is guided by a loss function between the Mel spectrum generated from multimodal audiovisual features and the Mel spectrum of the original speech data.

[0142] This disclosure provides a training method for a speech lip-reading recognition model. Training speech and video are input into a video encoder and a speech encoder respectively for feature extraction, resulting in training speech features and training video features. These training speech and video features are then input into a cross-modal adversarial memory module to generate training multimodal audiovisual features. Based on the training multimodal audiovisual features and a preset loss function, it is determined whether the speech lip-reading recognition model has been successfully trained. Compared with related technologies, this disclosure, by inputting the extracted training speech and video features into a cross-modal adversarial memory module for cross-modal processing, can better integrate the training speech and video features, obtaining more information about the correlation between speech and video features, thereby improving the model's generalization recognition ability.

[0143] To clearly illustrate the embodiments of this disclosure, this embodiment provides... Figure 2 The diagram shows a flowchart of another training method for a speech lip-reading recognition model.

[0144] like Figure 2 As shown, the method includes the following steps:

[0145] Step 201: Crop and align the images in the training audio-visual materials.

[0146] Specifically, in the embodiments disclosed herein, such as Figure 3 As shown, a large dataset of speaking videos was collected to generate training audio-visual videos. A face alignment tool was used to detect facial feature points in the video frames and crop them to a preset size. By cropping the images of people in the video frames to a preset size, the video encoder can easily extract features from the training videos.

[0147] Step 202: Input the training audio and video into the video encoder and audio encoder respectively for feature extraction to obtain training audio features and training video features.

[0148] As one possible implementation of this disclosure, the step of inputting the training audio-visual data into a video encoder and a speech encoder respectively for feature extraction to obtain training audio features and training video features includes: extracting video frame data from the training audio-visual data; performing feature extraction on the video frame data to obtain the training video features; extracting audio data from the training audio-visual data and generating a first Mel spectrogram based on the audio data; and performing feature extraction on the first Mel spectrogram to obtain the training audio features.

[0149] Specifically, in this embodiment, video frame data and speech data are obtained from the training audio-visual data. After obtaining the speech data, a corresponding Mel spectrogram is generated. Training video features are extracted from the video frame data using a video encoder, and the video encoder outputs the training video features (lip movement features). The speech encoder extracts training speech features from the Mel spectrogram, and outputs the training speech features.

[0150] Step 203: Input the training speech features and the training video features into the modality classifier for cross-modal processing.

[0151] As one possible implementation of this disclosure, the step of inputting the training speech features and the training video features into the modality classifier for cross-modal processing includes: mapping the training speech features to a training mapping network to obtain a first feature vector; mapping the training video features to the training mapping network to obtain a second feature vector; and performing cross-modal processing on the first feature vector and the second feature vector based on the modality classifier.

[0152] Step 204: Calculate the first feature distance between the first feature vector and the second feature vector.

[0153] Step 205: After performing cross-modal processing on the first feature vector and the second feature vector, calculate the classification loss.

[0154] Step 206: Based on the first feature distance and the classification loss, determine whether the modality classifier has been trained.

[0155] Specifically, in the embodiments disclosed herein, such as Figure 4 As shown, the cross-modal adversarial memory module M = (K, V) includes a modality classifier D and a key memory. Sum value memorization Where n represents the number of video-speech feature pairs (i.e., memory slots) stored within the module. Before addressing the cross-modal adversarial memory module, the training video features (lip movement features) obtained from the video encoder are respectively... and training speech features obtained from the speech encoder Each feature is mapped to the same feature space through a mapping network:

[0156]

[0157] Among them, H c Let c be the first feature vector output by the mapping network. i For lip movement features, n is the number of input video frames. Speech feature a iThe mapping method is the same as the above formula, and the output is represented as H. A This is the second feature vector output by the mapping network.

[0158] To eliminate the differences between video and speech modalities, H is made in the feature space. c and H A The distance between them is getting smaller and smaller:

[0159]

[0160] in, H in the feature space c and H A The distance between the first and second feature vectors. It should be noted that, in this embodiment, the Euclidean distance between the first and second feature vectors is used as the first feature distance. However, this does not constitute a limitation on the calculation method of the first feature distance in this disclosure; for example, the Manhattan distance, Mahalanobis distance, etc., can be used to calculate the first feature distance.

[0161] A modality classifier D is introduced, which is similar to the discriminator in a generative adversarial network. The input to the modality classifier is a transformed feature vector H, and the classifier determines and outputs whether H is a video feature or a speech feature. Utilizing... Classification loss and Train the modality classifier. Train the modality classifier using the adversarial training approach, then calculate the similarity between and , and between and . Finally, use KL divergence to fit the addressing vector and value vector (making them conform to ), enabling the cross-modal adversarial memory module to learn to extract representative multimodal audiovisual information, thereby better reconstructing speech in silent videos. Figure 4 The internal processes of the cross-modal adversarial memory module during the training phase are described in detail.

[0162] Step 207: Calculate the similarity between the cross-modal processed training video features and the training prior video features to obtain the first similarity weight.

[0163] Step 208: Calculate the similarity between the cross-modal processed training speech features and the training prior speech features to obtain the second similarity weight.

[0164] Specifically, in this embodiment of the disclosure, in order to store and process relevant features, the cross-modal adversarial memory module calculates the similarity between the prior lip movement feature K stored in each key memory and the currently input lip movement feature C. During the model's inference phase, the currently input lip movement feature C is first... i As a query vector, the query vector is memorized with each key k. l Compare the results and assign a relevant probability to each candidate memory:

[0165]

[0166] in, It is the similarity between the i-th lip movement feature and the l-th key memory, k l and c i These are the l-th key memory and the i-th lip movement feature, respectively. Simultaneously, the feature components v stored in the value memory were calculated. l With input speech features a i Similarity between

[0167]

[0168] During the addressing process, after comparing the lip movement features with each prior memory in the key memory, a similarity weight is assigned to each prior memory:

[0169]

[0170] Here, k is a constant value for scaling similarity scores on the hypersphere. The first similarity weight; The second similarity weight is calculated using the same formula as above.

[0171] Step 209: Perform a weighted summation of the first similarity weight and the prior speech features used for training to obtain the multimodal audiovisual features used for training.

[0172] Specifically, in this embodiment of the disclosure, the prior v stored in the value memory is accessed through addressing. l and Multiply each component and sum them by weight to output the multimodal audiovisual vector.

[0173]

[0174] And obtain pseudo-speech features in sequence This pseudo-speech feature Used only during training to ensure that the value memory V accurately reconstructs the original speech features A.

[0175] Network output training uses multimodal audiovisual features

[0176]

[0177] Where v l and The value memory stores the similarity weights of the vector and key memory outputs, respectively. Finally, the multimodal audiovisual features are... After feature fusion, the input is given to the speech decoder to generate a speech Mel spectrum.

[0178] Step 210: Determine whether the speech lip reading recognition model has been trained based on the multimodal audiovisual features used for training and the preset loss function.

[0179] As one possible implementation of this disclosure, determining whether the speech lip-reading recognition model has been trained successfully based on the training multimodal audiovisual features and a preset loss function includes: performing a weighted summation process on the training prior speech features and the second similarity weights to obtain pseudo-speech features; calculating a second feature distance between the pseudo-speech features and the training speech features; calculating the KL divergence between the first similarity weights and the second similarity weights; generating a second Mel spectrogram based on the training multimodal audiovisual features; calculating a first loss function based on the first Mel spectrogram and the second Mel spectrogram; calculating the preset loss function, wherein the preset loss function is the sum of the first loss function, the first feature distance, the second feature distance, and the KL divergence minus the classification loss; and determining whether the speech lip-reading recognition model has been trained successfully based on the calculation result of the preset loss function.

[0180] Furthermore, the step of calculating the first loss function based on the first Mel spectrogram and the second Mel spectrogram includes: calculating a second loss function based on the first Mel spectrogram, the first predicted Mel spectrogram, and the second Mel spectrogram; calculating a third loss function based on the first Mel spectrogram, the second predicted Mel spectrogram, and the third Mel spectrogram; and adding the second loss function and the third loss function to obtain the first loss function.

[0181] Specifically, in this embodiment of the disclosure, in order to better fit the output of the key memory to the output of the value memory, the first similarity weight and the second similarity weight need to be trained. Therefore, the KL divergence between the first similarity weight and the second similarity weight is calculated:

[0182]

[0183] Calculate the relationship between real speech features and pseudo-speech features distance:

[0184]

[0185] use The loss function constructs the final output objective function: calculating the predicted spectrogram without post-processing network processing. The difference between (the first predicted Mel spectrogram) and the actual Mel spectrogram (the first Mel spectrogram) M Distance; Mel spectrum output after post-processing network The difference between the second Mel spectrum and the true Mel spectrum M (the first Mel spectrum) As shown in the following formula:

[0186]

[0187] in, This refers to the concatenation of eigenvectors. Therefore, This is the Mel-spectrum output of the network. Loss function. This is to train the video encoder and speech encoder to extract representative features so that they correctly guide the cross-modal adversarial memory module to store and output meaningful features. Furthermore, a [system / mechanism] was constructed. Among them, multimodal audiovisual features It is replaced with the original speech feature A. Therefore, yes and sum.

[0188]

[0189] Finally, the entire model (encoder + cross-modal adversarial memory module + decoder) is trained end-to-end. The loss function for model training is all the loss functions mentioned above. The sum, minus It should be noted that, in the embodiments of this disclosure, the calculation of feature distance is illustrated using Euclidean distance as an example. However, this does not constitute a limitation on the calculation method in this disclosure; for example, Manhattan distance, Mahalanobis distance, etc., can also be used to calculate feature distance.

[0190] Step 211: Update the training video features as the training prior video features of the key memory, wherein the key memory is addressed in the value memory based on the training prior video features.

[0191] Step 212: Update the training speech features as the training prior speech features of the value memory, wherein the training prior video features and the training prior speech features are stored in a one-to-one correspondence.

[0192] Specifically, in this embodiment, in the initial stage, an array for storing prior information is randomly initialized. The vectors in the initialized array must follow a normal distribution. As the network trains, the vectors in the array are continuously updated. The cross-modal adversarial memory module internally stores feature vectors related to a given input video and speech. During the addressing process, the similarity between the current input lip movement features and the vectors (prior information) stored in each key memory is calculated, and the similarity weights are obtained after normalization. The similarity weights calculated during the addressing process are then weighted and summed with the feature vectors stored in the value memory to obtain the multimodal audiovisual features.

[0193] Specifically, the key memory is responsible for addressing, that is, it performs a correlation score on the current input lip movement feature vector and the candidate memory (prior) stored in the memory module and obtains a similarity weight; while the value memory is responsible for weighted summation of the similarity weight and the corresponding speech feature vector in the candidate memory and outputting a multimodal audiovisual vector.

[0194] Figure 5 This is a schematic flowchart of a speech lip-reading recognition method provided in an embodiment of this disclosure.

[0195] like Figure 5 As shown, the method includes the following steps:

[0196] Step 301: Input the video to be identified into the video encoder for feature extraction to obtain video features.

[0197] In the embodiments of this disclosure, the video to be identified is video data without audio data that requires lip-reading recognition. A video encoder is used to extract video features (lip movement features) from the input video to be identified. The trained lip-reading model is then used to perform lip-reading on the input video. During the inference phase of the model, the original audio corresponding to the input video is not required.

[0198] By inputting the video to be identified into a video encoder for feature extraction, a more comprehensive and multi-dimensional data representation can be obtained, providing more comprehensive information and improving data utilization.

[0199] Step 302: Input the video features into the cross-modal adversarial memory module to generate multimodal audiovisual features.

[0200] In embodiments of this disclosure, the cross-modal adversarial memory module is trained to: eliminate the differences between video features and speech features, store and address features, copy addressing vectors, and use addressing vectors to extract features containing multimodal information.

[0201] The cross-modal adversarial memory module stores features from both the source and target modalities. The source modality serves as the network's input, while the target modality features are those the network seeks to obtain from the memory network. A bridge is constructed based on the relationships between the source and target memories within the cross-modal adversarial memory module. This bridge facilitates the learning of intermodal relationships, enabling the proposed framework to acquire relevant target modality features from the memory network even when only the source modality is used as input.

[0202] Step 303: Use a speech decoder to reconstruct the features of the multimodal audiovisual features and generate a fourth Mel spectrogram.

[0203] Step 304: Based on the vocoder, the fourth Mel spectrogram is converted to obtain the speech for lip reading recognition.

[0204] In embodiments of this disclosure, a fourth Mel spectrogram is generated based on a cross-modal adversarial memory module, and a vocoder is used to transform the fourth Mel spectrogram to obtain speech for lip-reading recognition. Corresponding speech can be synthesized from different speakers, or even from a video of a speaker never seen during the training phase.

[0205] This disclosure provides a method for speech lip-reading recognition. The method involves inputting a video to be recognized into a video encoder for feature extraction to obtain video features; inputting the video features into a cross-modal adversarial memory module to generate multimodal audiovisual features; using a speech decoder to reconstruct the multimodal audiovisual features to generate a fourth Mel spectrogram; and using a vocoder to transform the fourth Mel spectrogram to obtain the speech for lip-reading recognition. Compared with related technologies, the cross-modal adversarial memory module of this disclosure stores the relationship information between prior video features and prior speech features. This enables the speech lip-reading recognition model to more accurately generate multimodal audiovisual features based on the input video features, thereby making the generated fourth Mel spectrogram closer to the Mel spectrogram of the original speech data. This improves the generalization ability of the speech lip-reading recognition model.

[0206] To clearly illustrate the embodiments of this disclosure, this embodiment provides... Figure 6 The diagram shows a flowchart of another speech lip-reading recognition method.

[0207] like Figure 6 As shown, the method includes the following steps:

[0208] Step 401: Crop and align the human images in the video to be identified.

[0209] Specifically, in this embodiment, a face alignment tool is used to detect facial feature points in video frames of the video to be identified and crop them to a preset size. By cropping the images of people in the video frames to a preset size, the video encoder can easily extract video features.

[0210] Step 402: Input the video to be identified into the video encoder for feature extraction to obtain video features.

[0211] Specifically, in this embodiment, the video to be identified is video data without audio data that requires lip-reading recognition. A video encoder is used to extract video features (lip movement features) from the input video to be identified. A trained lip-reading model is then used to perform lip-reading on the input video. During the model's inference phase, the original audio corresponding to the input video is not required.

[0212] Step 403: Calculate the addressing similarity between the video features and the prior video features stored in the key memory.

[0213] Step 404: Based on the addressing similarity, search for the corresponding prior speech features in the value memory.

[0214] Step 405: Perform a weighted summation of the video features and the prior speech features to obtain the multimodal audiovisual features.

[0215] Specifically, in this embodiment, the cross-modal adversarial memory module is trained to: eliminate the differences between video features and speech features, store and address features, copy addressing vectors, and use the addressing vectors to extract features containing multimodal information. The cross-modal adversarial memory module receives a video feature and obtains the addressing vector σ by calculating the similarity between the video feature vector and the features stored in the value memory module. key .

[0216] The input video features (lip movement features) c i As a query vector, the query vector is memorized with each key k. l The addressing similarity is obtained by comparison. The prior speech features corresponding to the current video features are obtained through addressing. The video features and prior speech features are then weighted and summed to obtain the multimodal audiovisual features.

[0217] Step 406: Use a speech decoder to reconstruct the features of the multimodal audiovisual features and generate a fourth Mel spectrogram.

[0218] Step 407: Based on the vocoder, the fourth Mel spectrogram is converted to obtain the speech for lip reading recognition.

[0219] Specifically, in this embodiment, a fourth Mel spectrogram is generated based on the cross-modal adversarial memory module, and a vocoder is used to convert the fourth Mel spectrogram to obtain the speech for lip-reading recognition. Corresponding speech can be synthesized from different speakers, or even from a video of a speaker never seen during the training phase.

[0220] In summary, a trained speech lip-reading model is used to perform lip-reading on input videos. During the model's inference phase, the original audio corresponding to the input video is not required. By inputting the video to be recognized into a video encoder for feature extraction, a more comprehensive and multi-dimensional data representation can be obtained, providing more complete information and improving data utilization. A fourth Mel spectrogram is generated based on the cross-modal adversarial memory module, and a vocoder is used to transform the fourth Mel spectrogram to obtain the speech for lip-reading recognition. Corresponding speech can be synthesized from videos of different speakers, even from videos of speakers never seen during the training phase.

[0221] It should be noted that the embodiments of this disclosure may include multiple steps. For ease of description, these steps are numbered, but these numbers are not a limitation on the execution time slots or execution order between the steps; these steps can be implemented in any order, and the embodiments of this disclosure do not limit this.

[0222] Corresponding to the training method of the speech lip-reading recognition model described above, this invention also proposes a training device for the speech lip-reading recognition model. Since the device embodiments of this invention correspond to the method embodiments described above, details not disclosed in the device embodiments can be referred to in the method embodiments described above, and will not be repeated here.

[0223] Figure 7 This is a schematic diagram of the structure of a training device for a speech lip-reading recognition model provided in an embodiment of the present disclosure, as shown below. Figure 7 As shown, it includes:

[0224] The first training unit 51 is used to input the training speech and video into the video encoder and speech encoder respectively for feature extraction, so as to obtain the training speech features and training video features.

[0225] The second training unit 52 is used to input the training speech features and the training video features into the cross-modal adversarial memory module to generate training multimodal audiovisual features;

[0226] The first determining unit 53 is used to determine whether the speech lip reading recognition model has been trained successfully based on the first Mel spectrogram for training, the second Mel spectrogram for training, and the preset loss function.

[0227] This disclosure provides a training device for a speech lip-reading recognition model. Training speech and video are input into a video encoder and a speech encoder respectively for feature extraction, resulting in training speech features and training video features. These training speech and video features are then input into a cross-modal adversarial memory module to generate training multimodal audiovisual features. Based on the training multimodal audiovisual features and a preset loss function, it is determined whether the speech lip-reading recognition model has been successfully trained. Compared with related technologies, this disclosure, by inputting the extracted training speech and video features into the cross-modal adversarial memory module for cross-modal processing, can better integrate the training speech and video features, obtaining more information about the correlation between speech and video features, thereby improving the model's generalization recognition ability.

[0228] Furthermore, in one possible implementation of this embodiment, such as Figure 8 As shown, the device further includes:

[0229] The first update unit 54 is used to update the training video features as the training prior video features of the key memory, wherein the key memory is addressed in the value memory based on the training prior video features; the cross-modal adversarial memory module includes key memory and value memory;

[0230] The second update unit 55 is used to update the training speech features as the training prior speech features of the value memory, wherein the training prior video features and the training prior speech features are stored in a one-to-one correspondence.

[0231] Furthermore, in one possible implementation of this embodiment, such as Figure 8 As shown, the first training unit 51 includes:

[0232] The first extraction module 511 is used to extract video frame data from the training audio and video.

[0233] The second extraction module 512 is used to extract features from the video frame data to obtain the training video features;

[0234] The third extraction module 513 is used to extract speech data from the training speech video and generate a first Mel spectrogram based on the speech data.

[0235] The fourth extraction module 514 is used to extract features from the first Mel spectrogram to obtain the training speech features.

[0236] Furthermore, in one possible implementation of this embodiment, such as Figure 8 As shown, the second training unit 52 includes:

[0237] The cross-modal processing module 521 is used to input the training speech features and the training video features into the modality classifier for cross-modal processing; the cross-modal adversarial memory module further includes a modality classifier;

[0238] The first calculation module 522 is used to calculate the similarity between the cross-modal processed training video features and the training prior video features to obtain the first similarity weight.

[0239] The second calculation module 523 is used to calculate the similarity between the training speech features after cross-modal processing and the training prior speech features to obtain the second similarity weight.

[0240] The third calculation module 524 is used to perform weighted summation of the first similarity weight and the prior speech features used for training to obtain the multimodal audiovisual features used for training.

[0241] Furthermore, in one possible implementation of this embodiment, the cross-modal processing module 521 is further configured to:

[0242] The training speech features are mapped to a training mapping network to obtain a first feature vector;

[0243] The training video features are mapped to the training mapping network to obtain a second feature vector;

[0244] Based on the modality classifier, cross-modal processing is performed on the first feature vector and the second feature vector.

[0245] Furthermore, in one possible implementation of this embodiment, such as Figure 8 As shown, the device further includes:

[0246] The first calculation unit 56 is used to calculate the first feature distance between the first feature vector and the second feature vector;

[0247] The second calculation unit 57 is used to calculate the classification loss after performing cross-modal processing on the first feature vector and the second feature vector;

[0248] The second determining unit 58 is used to determine whether the modality classifier has been trained based on the first feature distance and the classification loss.

[0249] Furthermore, in one possible implementation of this embodiment, the first determining unit 53 includes:

[0250] The fourth calculation module 531 is used to perform weighted summation on the training prior speech features and the second similarity weight to obtain pseudo-speech features;

[0251] The fifth calculation module 532 is used to calculate the second feature distance between the pseudo-speech features and the training speech features;

[0252] The sixth calculation module 533 is used to calculate the KL divergence between the first similarity weight and the second similarity weight;

[0253] The generation module 534 is used to generate a second Mel spectrogram based on the training multimodal audiovisual features;

[0254] The seventh calculation module 535 is used to calculate the first loss function based on the first Mel spectrum and the second Mel spectrum;

[0255] The eighth calculation module 536 is used to calculate the preset loss function, wherein the preset loss function is the sum of the first loss function, the first feature distance, the second feature distance, and the KL divergence minus the classification loss;

[0256] The determination module 537 is used to determine whether the speech lip reading recognition model has been trained successfully based on the calculation result of the preset loss function.

[0257] Furthermore, in one possible implementation of this embodiment, the seventh calculation module 535 is also used for:

[0258] Based on the first Mel spectrogram, the first predicted Mel spectrogram, and the second Mel spectrogram, calculate the second loss function;

[0259] Based on the first Mel spectrogram, the second predicted Mel spectrogram, and the third Mel spectrogram, the third loss function is calculated;

[0260] The first loss function is obtained by adding the second loss function and the third loss function.

[0261] Furthermore, in one possible implementation of this embodiment, such as Figure 8 As shown, the device further includes:

[0262] The first cropping unit 59 is used to crop and align the human figures in the training audio-visual video before inputting the training audio-visual video into the video encoder for feature extraction.

[0263] Corresponding to the aforementioned method for speech lip reading recognition, this invention also proposes a device for speech lip reading recognition. Since the device embodiments of this invention correspond to the aforementioned method embodiments, details not disclosed in the device embodiments can be referred to the aforementioned method embodiments, and will not be repeated here.

[0264] Figure 9This is a schematic diagram of the structure of a speech lip-reading recognition device provided in an embodiment of this disclosure, as shown below. Figure 9 As shown, it includes:

[0265] Extraction unit 61 is used to input the video to be identified into the video encoder for feature extraction to obtain video features;

[0266] The first generation unit 62 is used to input the video features into the cross-modal adversarial memory module to generate multimodal audiovisual features;

[0267] The second generation unit 63 is used to reconstruct the features of the multimodal audiovisual features using a speech decoder to generate a fourth Mel spectrogram;

[0268] The conversion unit 64 is used to convert the fourth Mel spectrogram based on the vocoder to obtain the speech for lip reading recognition.

[0269] This disclosure provides a device for speech lip-reading recognition. The device inputs a video to be recognized into a video encoder for feature extraction, obtaining video features. These video features are then input into a cross-modal adversarial memory module to generate multimodal audiovisual features. A speech decoder is used to reconstruct these multimodal audiovisual features, generating a fourth Mel spectrogram. Based on a vocoder, the fourth Mel spectrogram is transformed to obtain the speech for lip-reading recognition. Compared to related technologies, the cross-modal adversarial memory module of this disclosure stores the relationship information between prior video features and prior speech features. This allows the speech lip-reading recognition model to more accurately generate multimodal audiovisual features based on the input video features, resulting in a fourth Mel spectrogram that more closely approximates the Mel spectrogram of the original speech data. This improves the generalization ability of the speech lip-reading recognition model.

[0270] Furthermore, in one possible implementation of this embodiment, such as Figure 10 As shown, the first generation unit 62 includes:

[0271] The ninth calculation module 621 is used to calculate the addressing similarity between the video features and the prior video features stored in the key memory; the cross-modal adversarial memory module also includes a modality classifier, a key memory, and a value memory;

[0272] The lookup module 622 is used to look up the corresponding prior speech features in the value memory based on the addressing similarity.

[0273] The tenth calculation module 623 is used to perform weighted summation of the video features and the prior speech features to obtain the multimodal audiovisual features.

[0274] Furthermore, in one possible implementation of this embodiment, such as Figure 10 As shown, the device further includes:

[0275] The second cropping unit 65 is used to crop and align the human figures in the video to be identified before inputting the video to be identified into the video encoder for feature extraction.

[0276] It should be noted that the foregoing explanation of the method embodiments also applies to the apparatus of this embodiment, and the principle is the same, so it is not limited in this embodiment.

[0277] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0278] Figure 11 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0279] like Figure 11 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 702 or a computer program loaded from storage unit 708 into RAM (Random Access Memory) 703. RAM 703 can also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. I / O (Input / Output) interface 705 is also connected to bus 704.

[0280] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0281] The computing unit 701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as methods for speech lip reading recognition and methods for training models. For example, in some embodiments, the methods for speech lip reading recognition and methods for training models can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, the computing unit 701 may be configured in any other suitable manner (e.g., by means of firmware) to perform the aforementioned speech lip-reading recognition method and model training method.

[0282] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0283] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0284] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0285] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0286] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0287] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0288] It's important to note that artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0289] The various numerical designations such as "first," "second," etc., used in this disclosure are merely for ease of description and are not intended to limit the scope of the embodiments of this disclosure, nor do they indicate a sequential order.

[0290] At least one of the features described in this disclosure can also be described as one or more, and multiple features can be two, three, four or more, and this disclosure does not impose any limitations. In the embodiments of this disclosure, for a technical feature, the technical features in that technical feature are distinguished by "first", "second", "third", "A", "B", "C" and "D", etc., and there is no sequential order or size order among the technical features described by "first", "second", "third", "A", "B", "C" and "D".

[0291] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0292] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A training method for a speech lip-reading recognition model, characterized in that, The method includes: The training audio and video are input into the video encoder and the audio encoder respectively for feature extraction, resulting in training audio features and training video features. The training speech features and training video features are input into a cross-modal adversarial memory module to generate training multimodal audiovisual features; the cross-modal adversarial memory module includes a modality classifier; the step of inputting the training speech features and training video features into the cross-modal adversarial memory module to generate training multimodal audiovisual features includes: The training speech features and the training video features are input into the modality classifier for cross-modal processing; The similarity between the cross-modal processed training video features and the training prior video features is calculated to obtain the first similarity weight; The similarity between the training speech features after cross-modal processing and the training prior speech features is calculated to obtain the second similarity weight; The first similarity weight and the prior speech features used for training are weighted and summed to obtain the multimodal audiovisual features used for training. The cross-modal adversarial memory module includes key memory and value memory. The training video features are used as the training prior video features of the key memory for updating, wherein the key memory is addressed in the value memory based on the training prior video features; The training speech features are updated as the training prior speech features of the value memory, wherein the training prior video features and the training prior speech features are stored in a one-to-one correspondence. Based on the multimodal audiovisual features used for training and the preset loss function, determine whether the speech lip reading recognition model has been trained successfully.

2. The method according to claim 1, characterized in that, The step of inputting the training audio and video into the video encoder and audio encoder respectively for feature extraction to obtain training audio features and training video features includes: Extract video frame data from the training audio and video; Feature extraction is performed on the video frame data to obtain the training video features; Speech data is extracted from the training speech video, and a first Mel spectrogram is generated based on the speech data; Feature extraction is performed on the first Mel spectrogram to obtain the training speech features.

3. The method according to claim 2, characterized in that, The step of inputting the training speech features and the training video features into the modality classifier for cross-modal processing includes: The training speech features are mapped to a training mapping network to obtain a first feature vector; The training video features are mapped to the training mapping network to obtain a second feature vector; Based on the modality classifier, cross-modal processing is performed on the first feature vector and the second feature vector.

4. The method according to claim 3, characterized in that, The method further includes: Calculate the first feature distance between the first feature vector and the second feature vector; After performing cross-modal processing on the first feature vector and the second feature vector, the classification loss is calculated; Based on the first feature distance and the classification loss, it is determined whether the modality classifier has been trained successfully.

5. The method according to claim 4, characterized in that, The step of determining whether the speech lip-reading recognition model has been trained successfully based on the multimodal audiovisual features used for training and the preset loss function includes: The training prior speech features and the second similarity weights are weighted and summed to obtain pseudo-speech features; Calculate the second feature distance between the pseudo-speech features and the training speech features; Calculate the KL divergence between the first similarity weight and the second similarity weight; A second Mel spectrogram is generated based on the multimodal audiovisual features used in the training. Based on the first Mel spectrum and the second Mel spectrum, calculate the first loss function; Calculate the preset loss function, wherein the preset loss function is the sum of the first loss function, the first feature distance, the second feature distance, and the KL divergence, minus the classification loss; Based on the calculation result of the preset loss function, it is determined whether the speech lip reading recognition model has been trained successfully.

6. The method according to claim 5, characterized in that, The calculation of the first loss function based on the first Mel spectrum and the second Mel spectrum includes: Based on the first Mel spectrogram, the first predicted Mel spectrogram, and the second Mel spectrogram, calculate the second loss function; Based on the first Mel spectrogram, the second predicted Mel spectrogram, and the third Mel spectrogram, the third loss function is calculated; The first loss function is obtained by adding the second loss function and the third loss function.

7. The method according to claim 1, characterized in that, The method further includes: Before inputting the training audio-visual video into the video encoder for feature extraction, the human figures in the training audio-visual video are cropped and aligned.

8. A method for speech lip reading recognition, characterized in that, The method includes: The video to be identified is input into a video encoder for feature extraction to obtain video features; The video features are input into the cross-modal adversarial memory module to generate multimodal audiovisual features; the cross-modal adversarial memory module also includes a modality classifier, key memory, and value memory; The step of inputting the video features into the cross-modal adversarial memory module to generate multimodal audiovisual features includes: Calculate the addressing similarity between the video features and the prior video features stored in the key memory; Based on the addressing similarity, the corresponding prior speech features in the value memory are searched; The video features and the prior speech features are weighted and summed to obtain the multimodal audiovisual features; The multimodal audiovisual features are reconstructed using a speech decoder to generate a fourth Mel spectrogram; Based on the vocoder, the fourth Mel spectrogram is converted to obtain the speech for lip reading recognition.

9. The method according to claim 8, characterized in that, The method further includes: Before inputting the video to be identified into the video encoder for feature extraction, the human figures in the video to be identified are cropped and aligned.

10. A training device for a speech lip-reading recognition model, characterized in that, The device includes: The first training unit is used to input the training audio and video into the video encoder and the audio encoder respectively for feature extraction, so as to obtain the training audio features and the training video features. The second training unit is used to input the training speech features and the training video features into the cross-modal adversarial memory module to generate training multimodal audiovisual features; the cross-modal adversarial memory module includes a modality classifier; the step of inputting the training speech features and the training video features into the cross-modal adversarial memory module to generate training multimodal audiovisual features includes: inputting the training speech features and the training video features into the modality classifier for cross-modal processing; calculating the similarity between the cross-modal processed training video features and the training prior video features to obtain a first similarity weight; calculating the similarity between the cross-modal processed training speech features and the training prior speech features to obtain a second similarity weight; and performing a weighted summation of the first similarity weight and the training prior speech features to obtain the training multimodal audiovisual features; The cross-modal adversarial memory module includes key memory and value memory. The training video features are used as the training prior video features of the key memory for updating, wherein the key memory is addressed in the value memory based on the training prior video features; The training speech features are updated as the training prior speech features of the value memory, wherein the training prior video features and the training prior speech features are stored in a one-to-one correspondence. The first determining unit is used to determine whether the speech lip reading recognition model has been trained successfully based on the first Mel spectrogram for training, the second Mel spectrogram for training, and the preset loss function.

11. A device for speech lip reading recognition, characterized in that, The device includes: The extraction unit is used to input the video to be recognized into the video encoder for feature extraction to obtain video features; The first generation unit is used to input the video features into the cross-modal adversarial memory module to generate multimodal audiovisual features; the cross-modal adversarial memory module further includes a modality classifier, key memory, and value memory; The step of inputting the video features into the cross-modal adversarial memory module to generate multimodal audiovisual features includes: Calculate the addressing similarity between the video features and the prior video features stored in the key memory; Based on the addressing similarity, the corresponding prior speech features in the value memory are searched; The video features and the prior speech features are weighted and summed to obtain the multimodal audiovisual features; The second generation unit is used to reconstruct the features of the multimodal audiovisual features using a speech decoder to generate a fourth Mel spectrogram; The conversion unit is used to convert the fourth Mel spectrogram based on the vocoder to obtain the speech for lip reading recognition.

12. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

13. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-9.

14. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-9.

Images