Self-supervised learning method and system based on audio and video alignment

Abstract

The invention provides a self-supervised learning method and system based on audio and video alignment. The method comprises the following steps: acquiring an audio and video offset data set; obtaining a similarity matrix based on the original audio signal and the video signal, and performing reverse translation alignment on the video features according to the offset in the similarity matrix to obtain aligned audio and video; encoding the aligned audios and videos to obtain multi-modal features of the audios and the videos; the multi-modal features are decoded for downstream speech related tasks. Through the self-supervised learning method which constructs the audio and video similarity matrix, automatically calculates the time sequence offset and carries out feature alignment, the robustness of voice related tasks in a complex scene in which the audio and the video have offset is improved.

Description

Technical Field

[0001] The present application relates to the technical field of speech self-supervised learning, and specifically to a self-supervised learning method and system based on audio and video alignment. Background Art

[0002] Automatic Speech Recognition (ASR) converts the vocabulary of human speech into text. Speech recognition technology is widely used in scenarios such as meeting recording and subtitle production, as well as in devices such as voice assistants, smart homes, and smart cars.

[0003] In recent years, speech recognition has reached a state-of-the-art level. However, the performance of pure speech recognition systems degrades dramatically in noisy environments. Because video is more robust to acoustic noise, lip movements can complement audio modalities and enable speech recognition using both audio and video modalities. Audio-visual speech recognition (AVSR) models have shown superior recognition performance compared to audio-only speech recognition.

[0004] However, most existing audio-visual models assume that audio and video are temporally aligned, and thus their performance suffers when faced with misaligned inputs. In real-world applications, temporal misalignment between audio and video signals is common, such as in television broadcasts, interviews, and social media videos. Furthermore, existing audio-visual datasets lack large-scale, well-synchronized audio-visual data with accurate transcription annotations, hindering further exploration and development of audio-visual models.

[0005] Self-supervised models can be trained using unlabeled audio and video data to train an encoder that learns good audio and video features. By connecting a decoder after the encoder, they can be fine-tuned for various downstream speech-related tasks, such as speech recognition, speech enhancement, and speech generation. Among them, the AV-HuBERT model (Reference 1 Bowen Shi, Wei-Ning Hsu, KushalLakhotia, and Abdelrahman Mohamed, “Learning audio-visual speech representation by masked multimodal cluster prediction,” arXiv preprint arXiv:2201.02184, 2022) excels in capturing audio-visual synchronization features. This property has been successfully applied in multiple studies to enhance multimodal feature alignment. Specific application cases include: semantic-level alignment, where researchers used AV-HuBERT to achieve semantic-level synchronization of lip-reading video generation, significantly improving the naturalness of lip movements in the generated videos; deep fake detection, where AV-HuBERT was used to identify inconsistencies between video and audio modalities, and an efficient deep fake detection system was built; and target speaker extraction, where the pre-trained AV-HuBERT layer was integrated into the target speaker extraction system to effectively learn audio-visual synchronization representations. Summary of the Invention

[0006] The present application provides a self-supervised learning method and system based on audio and video alignment, which can solve the technical problem of reduced recognition performance caused by audio and video timing deviation in the prior art.

[0007] In a first aspect, the present application provides a self-supervised learning method based on audio and video alignment, comprising the following steps: Obtain an audio and video offset dataset and input it into the model for self-supervised training of audio and video alignment. Obtaining a similarity matrix between audio frames in the original audio signal and video frames in the video signal, constructing an alignment loss function based on the similarity matrix, training an audio alignment module and a video alignment module, predicting an audio and video offset using the trained audio alignment module and video alignment module, and performing a reverse frame offset operation on the video features to obtain aligned audio and video; The encoder is trained using a mask prediction loss function, and the trained encoder is used to perform cross-modal feature fusion on the aligned audio and video to obtain multimodal features of audio and video. Combining the mask prediction loss and the alignment loss into a total loss function, jointly training the encoder weights, so that the encoder simultaneously learns the audio and video multimodal features and the alignment features during the training process, and outputs the aligned audio and video multimodal features; Taking the aligned multimodal features as input, the loss function trained according to the corresponding task is used to learn and decode the multimodal features, train the decoder, and output the audio and video decoding results for downstream speech-related tasks.

[0008] Furthermore, an audio and video offset dataset is obtained and input into the model for self-supervised training of audio and video alignment, which specifically includes the following steps: Randomly offset video frames within a preset frame range to construct an audio and video offset dataset; Furthermore, a similarity matrix of audio frames in the original audio signal and video frames in the video signal is obtained, an alignment loss function is constructed based on the similarity matrix, an audio alignment module and a video alignment module are trained, audio and video offsets are predicted by the trained audio alignment module and video alignment module, and a reverse frame offset operation is performed on the video features to obtain aligned audio and video, specifically comprising the following steps: Extracting audio feature vectors and video feature vectors through an audio encoder and a video encoder respectively; Calculate the similarity matrix between the audio feature vector and the video feature vector; constructing an alignment loss function based on the similarity matrix; Based on the alignment loss function training, the audio alignment module and the video alignment module are obtained. The trained audio alignment module and the video alignment module predict the audio and video offset, and perform reverse frame offset operation on the video features to obtain aligned audio and video.

[0009] Furthermore, constructing an alignment loss function based on the similarity matrix specifically includes the following steps: Calculate the average similarity of each diagonal in the similarity matrix; Determine the diagonal sequence number with the smallest average similarity as the audio and video offset; The alignment sub-matrix is cropped from the similarity matrix according to the offset labels provided by the audio and video offset dataset; The cross entropy loss of the audio stream and video stream is calculated based on the row and column directions of the alignment submatrix respectively, and the average of the two is taken to construct the alignment loss function.

[0010] Furthermore, the alignment loss function is shown as follows:

[0011] Where, is the alignment loss, is the cross entropy loss of the audio stream, is the cross entropy loss of the video stream.

[0012] Furthermore, the mask prediction loss function is used to train the encoder, and the trained encoder is used to perform cross-modal feature fusion on the aligned audio and video to obtain multimodal features of audio and video. Specifically, the following steps are included: Input the masked audio and video features into the multimodal Transformer encoder, predict the true target features of the masked part, and construct a mask prediction loss function for the audio and video features based on the prediction results. Train the encoder based on the mask prediction loss function. The trained encoder is used to perform cross-modal fusion of audio features and video features to generate multimodal features.

[0013] Furthermore, the mask prediction loss and alignment loss are combined into a total loss function, and the encoder weights are jointly trained to achieve that during the training process, the encoder simultaneously learns the audio and video multimodal features and the alignment features, and outputs the aligned audio and video multimodal features, which specifically includes the following steps: The mask prediction loss function and the alignment loss function are combined into a total loss function through hyperparameter weighting; The alignment loss and mask prediction loss are optimized simultaneously based on the total loss function to train the encoder. The encoder is a 12-layer Transformer structure, and the output is aligned multimodal features, which are used in the decoder and then applied to downstream speech-related tasks.

[0014] Furthermore, the total loss function is expressed as:

[0015] Where, is the alignment loss, is the mask prediction loss, is the weight hyperparameter of the alignment loss.

[0016] Furthermore, the aligned multimodal features are used as input, and the loss function of the corresponding task's standard training is used to learn to decode the multimodal features. The decoder is trained to obtain the audio and video decoding results, which are used in downstream speech-related task steps. The decoder is a 6-layer Transformer structure, and the downstream speech-related tasks are speech recognition, speech enhancement or speech generation.

[0017] In a second aspect, the present application provides a self-supervised learning system based on audio and video alignment, comprising the following steps: The dataset acquisition module is used to obtain the audio and video offset dataset and input it into the model for self-supervised training of audio and video alignment; An audio and video alignment module is communicatively connected to the data set acquisition module, and is used to obtain a similarity matrix between audio frames in the original audio signal and video frames in the video signal, construct an alignment loss function based on the similarity matrix, train an audio alignment module and a video alignment module, predict the audio and video offset using the trained audio alignment module and video alignment module, and perform a reverse frame offset operation on the video features to obtain aligned audio and video; A self-supervised learning module is communicatively connected to the audio and video alignment module, and is used to construct an audio and video offset dataset based on the similarity matrix, train an encoder based on the audio and video offset dataset using a mask prediction loss function, and use the trained encoder to perform cross-modal feature fusion on the aligned audio and video to obtain multimodal features of audio and video; The audio and video alignment module and the self-supervised learning module are trained to obtain the encoder through the total loss function of the combination of mask prediction loss and alignment loss; After the joint training of the audio and video alignment module and the self-supervised learning module is completed, the fine-tuning module is connected to the decoder. It is used to take the aligned multimodal features as input, use the loss function trained according to the corresponding task's specifications to learn to decode the multimodal features, train the decoder, and output the audio and video decoding results for downstream speech-related tasks.

[0018] Furthermore, the audio and video alignment module includes: A feature vector extraction unit, configured to extract an audio feature vector and a video feature vector through an audio encoder and a video encoder respectively; a similarity matrix acquisition unit, communicatively connected to the feature vector extraction unit, and configured to calculate a similarity matrix between the audio feature vector and the video feature vector; an alignment loss function construction unit, communicatively connected to the similarity matrix acquisition unit, and configured to construct an alignment loss function based on the similarity matrix; The audio and video alignment unit is communicatively connected to the alignment loss function construction unit, and is used to obtain an audio alignment module and a video alignment module based on the alignment loss function training, predict the audio and video offset through the trained audio alignment module and video alignment module, and perform a reverse frame offset operation on the video features to obtain aligned audio and video.

[0019] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least: By constructing an audio and video similarity matrix, automatically calculating timing offsets, and performing feature alignment, a self-supervised learning method is used to improve the robustness of speech-related tasks in complex scenarios where audio and video offsets exist. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of a self-supervised learning method based on audio and video alignment provided in an embodiment of the present application; Figure 2 This is a performance diagram of speech recognition in the audio and video offset scenario provided by an embodiment of the present application; Figure 3 A module structure diagram of a self-supervised learning system based on audio and video alignment provided in an embodiment of the present application; Figure 4 A flowchart for calculating alignment loss provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit the "first", "second" and "third" to different types.

[0023] In the description of the embodiments of this application, the words "exemplary," "for example," or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary," "for example," or "for example" in the embodiments of this application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary," "for example," or "for example" is intended to present the relevant concepts in a concrete manner.

[0024] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0025] In some processes described in the embodiments of the present application, multiple operations or steps are included that appear in a specific order. However, it should be understood that these operations or steps may not be performed in the order in which they appear in the embodiments of the present application or may be performed in parallel. The sequence numbers of the operations are only used to distinguish between different operations, and the sequence numbers themselves do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations or steps may be performed in sequence or in parallel, and these operations or steps may be combined.

[0026] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0027] First, as Figure 1 As shown, the present application provides a self-supervised learning method based on audio and video alignment, comprising the following steps: Step S1: Obtain an audio and video offset dataset, which includes the original audio signal and the offset video, and input it into a model for self-supervised training of audio and video alignment; Step S2: Obtain a similarity matrix between audio frames in the original audio signal and video frames in the video signal, construct an alignment loss function based on the similarity matrix, train an audio alignment module and a video alignment module, predict the audio and video offset using the trained audio alignment module and video alignment module, and perform a reverse frame offset operation on the video features to obtain aligned audio and video; Step S3: Using the mask prediction loss function to train the encoder, the trained encoder is used to perform cross-modal feature fusion on the aligned audio and video to obtain multimodal features of audio and video; Step S4: combining the mask prediction loss and the alignment loss into a total loss function, and jointly training the encoder weights, so that the encoder simultaneously learns the audio and video multimodal features and the alignment features during the training process, and outputs the aligned audio and video multimodal features; Step S5: Using the aligned multimodal features as input, use the loss function of the corresponding task's canonical training to learn and decode the multimodal features, train and obtain the decoder, and output the audio and video decoding results for downstream speech-related tasks; for example, when training the downstream task of speech recognition, use the canonical training method of speech recognition, i.e., sequence classification loss, to fine-tune to the speech recognition task and output the text corresponding to the audio and video.

[0028] This application constructs an alignment loss function and an audio and video offset dataset to enable the model to effectively handle the common audio and video desynchronization problem in practical applications; The similarity matrix is used to accurately calculate the audio and video offset, and the reverse frame offset operation is used to achieve temporal alignment at the feature level, solving the problem of performance degradation in traditional methods when audio and video are out of sync. The encoder is trained using a mask prediction loss function. Based on the trained encoder, deep feature fusion is performed through a cross-modal attention mechanism. The generated joint multimodal features not only retain the temporal characteristics of a single modality, but also contain rich cross-modal complementary information. The decoder design enables the trained model to be directly adapted to a variety of downstream speech tasks, including speech recognition, speech enhancement, and speech separation. This method significantly improves the robustness and adaptability of the model in audio and video offset scenarios while maintaining compatibility with existing audio and video systems.

[0029] In one embodiment, step S1: obtaining an audio and video offset dataset and inputting it into a model for self-supervised training of audio and video alignment; randomly offsetting video frames within a preset frame range to construct an audio and video offset dataset; specifically includes the following steps: Use the torch.roll function to perform frame-level offsets on the video, randomly shifting the video within the range of (-25, 25). When the offset is greater than 0, the video is faster than the audio, and vice versa. Given a video frame rate of 25 fps, the offset corresponds to (-1s, 1s). Through this data augmentation process, the training data is tripled, expanding the dataset to four times its original size without adding any new data. This results in an audio-video offset dataset. This dataset contains audio, offset videos, offset labels for calculating the alignment loss, and target labels for the mask prediction task, which is used to calculate the mask prediction loss.

[0030] The dataset is used to train speech recognition models capable of handling audio and video alignment; training is performed on two dataset sizes: pre-trained on the 433-hour LRS3 dataset, which contains English videos with text transcriptions, and trained on a combined dataset totaling 1,759 hours, which contains all 433 hours of LRS3 data and the English portion of the VoxCeleb2 dataset (1,326 hours).

[0031] The original VoxCeleb2 dataset contains 2,442 hours of unlabeled multilingual videos. Following the method in Reference 1, only English video data was selected for training. This data combination strategy not only ensures the quality of the labeled data, but also fully utilizes the advantages of a large-scale unlabeled dataset.

[0032] In one embodiment, step S2: obtaining a similarity matrix of audio frames in the original audio signal and video frames in the video signal, constructing an alignment loss function based on the similarity matrix, training an audio alignment module and a video alignment module, predicting an audio and video offset using the trained audio alignment module and video alignment module, and performing a reverse frame offset operation on video features to obtain aligned audio and video, specifically includes the following steps: Step S21: extracting audio feature vectors and video feature vectors through audio encoder and video encoder respectively; more specifically, including audio signal and video signals The audio-visual data of ( , ) First, a lightweight modality-specific encoder (audio encoder FFN (linear projection layer) and image encoder ResNet-18, the structure is consistent with the base model of AV-HuBERT) is used to generate feature representations ( , ), is the audio feature vector, is the video feature vector; Step S22: Calculate the similarity matrix between the audio feature vector and the video feature vector; Step S23: constructing an alignment loss function based on the similarity matrix; Step S24: obtaining an audio alignment module and a video alignment module based on the alignment loss function training; Step S25: The audio and video offsets are predicted by the trained audio alignment module and video alignment module, and the reverse frame offset operation is performed on the video features. Specifically, the torch.roll function is used to offset the video at the frame level to obtain the aligned audio and video ( , ),in, is the audio feature vector, is the video feature vector.

[0033] In one embodiment, the step S22 of calculating the similarity matrix between the audio feature vector and the video feature vector specifically includes the following steps: For each audio stream (i.e., original audio signal) and video stream (i.e., video signal corresponding to the original audio signal time), two alignment modules are used and To process, , The module uses a Transformer-based encoder architecture with 12 Transformer blocks, consistent with the AV-HuBERT base model, to obtain audio feature vectors. and video feature vector , and The dimensions are (T, D):

[0034]

[0035] Figure 2 (a) shows the speech recognition performance in the audio and video misalignment scenario. The speech recognition performance is the result of the model pre-trained on the 433-hour dataset and fine-tuned on the 30-hour dataset. The numbers represent the obtained audio and video offset dataset, and the offset data is expanded three times for data enhancement.

[0036] Figure 2 (b) is the speech recognition performance graph in the audio and video offset scenario. The speech recognition performance is the result after the model is pre-trained on a 433-hour dataset and fine-tuned on a 433-hour dataset of the same size.

[0037] Figure 2 (c) is the speech recognition performance graph in the audio and video offset scenario. The speech recognition performance is the result after the model is pre-trained on a 1,759-hour dataset and fine-tuned on a 30-hour dataset.

[0038] Figure 2 (d) is the speech recognition performance graph in the audio and video offset scenario. The speech recognition performance is the result after the model is pre-trained on a 1,759-hour dataset and fine-tuned on a 433-hour dataset.

[0039] like Figure 2 As shown, video features Relative to audio features There is a single frame offset, in which case the alignment diagonal line starts from the second frame Extend to frame N , so n=N−1. Calculate the similarity between each frame of audio and each frame of video to obtain the similarity matrix. The similarity calculation is shown in the following formula:

[0040] Where, and The dimensions are (T, D), Indicates the exponential operation of the temperature coefficient, the temperature coefficient Used to adjust the similarity matrix range, Construct a (T, T)-dimensional matrix (T is the sequence length N).

[0041] In one embodiment, the step S23 of constructing an alignment loss function based on the similarity matrix specifically includes the following steps: Calculate the average similarity of each diagonal in the similarity matrix; Determine the diagonal sequence number with the smallest average similarity as the audio and video offset x; the value range of offset x is limited to (-25, 25) frames, corresponding to the time offset range of (-1s, 1s) When training the model, an (n,n)-dimensional alignment submatrix is cropped from the similarity matrix based on the offset label x' provided by the audio and video offset dataset; where n = N - |x|, and N is the total number of frames; When training the model, the cross entropy loss of the audio stream and the video stream is calculated based on the row and column directions of the aligned submatrix. Specifically, when calculating the audio loss and video loss When , the original (N, N)-dimensional similarity matrix is cropped into an (n, n)-dimensional alignment submatrix so that its main diagonal corresponds to the alignment relationship defined by the target label {0, 1, ..., n-1}, and the cross entropy loss is calculated along the row (audio) and column (video) directions respectively to force the cross-modal temporal alignment to obtain the audio loss and video loss ; Take the average of the two to construct the alignment loss function, as shown below:

[0042] Where, is the alignment loss, is the cross entropy loss of the audio stream, is the cross entropy loss of the video stream.

[0043] Through this alignment loss function, the model can bring the temporally aligned audiovisual feature vectors closer together, thereby determining whether the audiovisual data is synchronized at the frame level.

[0044] In one embodiment, step S24: in the step of obtaining the audio alignment module and the video alignment module based on the alignment loss function training, both modules adopt a 12-layer Transformer structure and share weights with the main encoder; during the training process, the parameters of the audio front-end and the video front-end are kept fixed, and the audio alignment module and the video alignment module obtained through training predict the audio and video offset; during training, the weights of the fifth round base model of AV-HuBERT are used to initialize the model parameters. The model trained on 433h is trained for 40,000 steps, trained on 32 GPUs, and the video frame length range is (5, 500); the model trained on 1759h is trained for 80,000 steps, trained on 64 GPUs, and the video frame length range is (5, 1000). Other hyperparameter settings are consistent with the parameter settings of the pre-training step in Document 1; In one embodiment, step S25: predicting the audio and video offset using the trained audio alignment module and video alignment module, and performing a reverse frame offset operation on the video features, specifically includes the following steps: When the offset frame number is greater than 0, the video is faster than the audio, and vice versa. Specifically, according to the principle that the similarity of the same frame is the highest and the distance is the smallest, the average similarity of each diagonal is calculated. (in Represents the matrix diagonal lines). Index The value range is This is because when loading the data, a maximum translation limit of 25 frames is set for the video frame, and the average similarity of each diagonal is calculated. The formula is as follows:

[0045] Average similarity across all diagonals The smallest diagonal number is determined as the final offset ;according to Move the video in the opposite direction to align the audio and video.

[0046] In one embodiment, step S3: training an encoder using a mask prediction loss function, and using the trained encoder to perform cross-modal feature fusion on the aligned audio and video to obtain multimodal features of the audio and video, specifically includes the following steps: Step S31: Input the masked audio and video features to the multimodal Transformer encoder, predict the true target features of the masked part, and construct a mask prediction loss function for the audio and video features based on the prediction results, and train the encoder based on the mask prediction loss function; Step S32: Use the trained encoder to perform cross-modal fusion of audio features and video features to generate multimodal features.

[0047] In one embodiment, step S31: inputting the masked audio and video features into a multimodal Transformer encoder, predicting the true target features of the masked portion, constructing a mask prediction loss function for the audio and video features based on the prediction results, and training the encoder based on the mask prediction loss function, specifically includes the following steps: Input the masked audio and video features to the multimodal Transformer encoder, and predict the masked audio and video features to obtain the true target features of the audio and video features; According to the predicted true target features of the audio and video, a mask prediction loss function for decoding the audio and video features is constructed, as shown in the following formula:

[0048] Where, is the output probability; Generate target labels for the mask prediction task; and Corresponding to audio streams and video streaming The masked frames in ; α is a hyperparameter used to weight the contribution of the unmasked region in the loss function; In one embodiment, step S4: combining the mask prediction loss and the alignment loss into a total loss function, jointly training the encoder weights, so that the encoder simultaneously learns the audio and video multimodal features and the alignment features during the training process, and outputs the aligned audio and video multimodal features, specifically includes the following steps: Step S41: To achieve simultaneous optimization of multimodal feature self-supervised training and audio-visual feature alignment, define the total loss function It is the weighted sum of the alignment loss and the AV-HuBERT mask prediction loss. The mask prediction loss function and the alignment loss function are weighted and combined into the total loss function through hyperparameters, as shown in the following formula:

[0049] Where, is the alignment loss, is the mask prediction loss, λ is the weight hyperparameter of the alignment loss, and (1−λ) corresponds to the weight of the mask prediction loss; In one embodiment, step S5: using the aligned multimodal features as input, learning to decode the multimodal features using a loss function trained for the corresponding task, training a decoder, and outputting audio and video decoding results for downstream speech-related tasks, specifically includes the following steps: Step S51: The hyperparameter settings are consistent with those in the fine-tuning step in Reference 1. During fine-tuning, the model still uses the dataset from Step S1. The decoder is a 6-layer Transformer structure.

[0050] Step S52: Train downstream speech-related tasks using the corresponding standard training method. For example, when training the downstream speech recognition task, fine-tune the speech recognition task using the standard training method for speech recognition, namely the sequence loss function. The sequence loss function specifically follows the S2S loss function (attention-based sequence-to-sequence cross-entropy loss) described in Reference 1.

[0051] In one embodiment, the downstream speech-related tasks include speech recognition, speech enhancement, or speech generation.

[0052] This paper fine-tunes the downstream tasks of speech recognition. The decoder outputs speech recognition text. Speech recognition performance is calculated using the word error rate (WER) as a core metric. Testing is performed on a dataset with a timing offset within the range of (-25, 25) frames. The model performance is tested for each frame from a timing offset of -25 to 25 frames, assessing the model's robustness to offset data. Lower scores indicate better model performance.

[0053] As attached Figure 3 As shown in the figure, the performance of the self-supervised model AV-HuBERT (solid line) based on mask prediction loss drops significantly when there is frame offset. The model trained using the method of the present invention (dashed line) maintains essentially unchanged performance under various offset conditions. When pre-trained on 433 hours, with no frame offset, its performance is similar to AV-HuBERT, and when pre-trained on 1759 hours, with no frame offset, its performance is superior to AV-HuBERT.

[0054] Aiming at the special demand of speech recognition of complete sentences, the present invention designs the following Figure 2 The cyclic offset feature processing method shown here applies a cyclic offset to the video frame. To eliminate decoding ambiguity caused by the cyclic effect (the model may predict the offset as n or n−N), the following evaluation strategy is implemented: when the actual offset is positive, the prediction error is only calculated for the area to the right of the main diagonal of the matrix; when the actual offset is negative, the prediction error is only calculated for the area to the left of the main diagonal. This targeted evaluation strategy effectively avoids prediction ambiguity between n and n−N.

[0055] As multimodal technology matures and there are more and more sources of audio and lip movement videos, the present invention proposes a method and device for self-supervised learning based on audio and video alignment to address the problem that the current multimodal audio-visual model based on joint audio and video modeling degrades in performance when there is an offset between audio and video. This method proposes a novel self-supervised learning strategy that simultaneously trains alignment loss and mask prediction loss, enabling the model encoder to output aligned multimodal audio and video features. Based on this encoder, the present invention achieves fine-tuning to downstream speech recognition tasks, and the performance remains basically unchanged when the audio and video are offset. The present invention compares the performance of this method in speech recognition with the original self-supervised model AV-HuBERT based on mask prediction loss, proving the robustness of the method, which has great application value for the actual implementation of such technology in actual complex application scenarios.

[0056] Second, as Figure 3 As shown, the present application provides a self-supervised learning system based on audio and video alignment, including a data set acquisition module, an audio and video alignment module, a self-supervised learning module and a fine-tuning module; the data set acquisition module is used to obtain an audio and video offset data set, input a model, and is used for self-supervised training of audio and video alignment; the audio and video alignment module is communicated with the data set acquisition module, and is used to obtain a similarity matrix of audio frames in the original audio signal and video frames in the video signal, construct an alignment loss function based on the similarity matrix, train an audio alignment module and a video alignment module, predict the audio and video offset through the trained audio alignment module and video alignment module, and perform a reverse frame offset operation on the video features to obtain aligned audio and video; self-supervised The learning module is communicatively connected to the audio and video alignment module, and is used to align the audio and video according to the similarity matrix, train the encoder using the mask prediction loss function, and use the trained encoder to perform cross-modal feature fusion on the aligned audio and video to obtain multimodal features of audio and video; the audio and video alignment module and the self-supervised learning module train the encoder through the total loss function of the combination of mask prediction loss and alignment loss; the fine-tuning module is used to access the decoder after the joint training of the audio and video alignment module and the self-supervised learning module is completed, and is used to use the aligned multimodal features as input, use the loss function trained according to the specification of the corresponding task to learn to decode the multimodal features, train the decoder, and output the audio and video decoding results for downstream speech-related tasks.

[0057] The alignment module first calculates the offset between the audio and video. Second, the self-supervised training module applies a mask to the audio and video pairs and processes them through their respective audio and image encoders to obtain audio and video features. The video features are shifted in the opposite direction of the calculated offset to ensure temporal alignment with the audio features. Finally, the fine-tuning module adds a decoder to the trained encoder and applies it to downstream speech-related tasks. The fine-tuning module can be implemented as a decoder.

[0058] In one embodiment, the audio and video alignment module includes: A feature vector extraction unit, configured to extract an audio feature vector and a video feature vector through an audio encoder and a video encoder respectively; a similarity matrix acquisition unit, communicatively connected to the feature vector extraction unit, and configured to calculate a similarity matrix between the audio feature vector and the video feature vector; an alignment loss function construction unit, communicatively connected to the similarity matrix acquisition unit, and configured to construct an alignment loss function based on the similarity matrix; The audio and video alignment unit is communicatively connected to the alignment loss function construction unit, and is used to obtain an audio alignment module and a video alignment module based on the alignment loss function training, predict the audio and video offset through the trained audio alignment module and video alignment module, and perform a reverse frame offset operation on the video features to obtain aligned audio and video.

[0059] like Figure 4 As shown, the audio and video alignment unit can also be implemented as an audio decoder, an audio alignment module, a video decoder, and a video alignment module. Assume that the shift of the input audio and video is one frame. In this case, the diagonal of the audio and video alignment contains n-1 elements, starting from the 2nd frame in the similarity matrix and ending at the Nth frame.

[0060] Among them, the functional implementation of each module in the above-mentioned self-supervised learning system based on audio and video alignment corresponds to the steps in the above-mentioned self-supervised learning method embodiment based on audio and video alignment, and their functions and implementation processes will not be repeated here one by one.

[0061] In a third aspect, an embodiment of the present application provides a self-supervised learning device based on audio and video alignment. The self-supervised learning device based on audio and video alignment can be a personal computer (PC), a laptop, a server, or other device with data processing capabilities.

[0062] Communication interfaces include input / output (I / O) interfaces, physical interfaces, and logical interfaces. These interfaces interconnect components within the self-supervised learning device based on audio and video alignment, as well as interfaces that interconnect the device with other devices (such as other computing devices or user devices). Physical interfaces can include Ethernet, fiber, or ATM interfaces; user devices can include displays and keyboards.

[0063] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0064] The processor can be a general-purpose processor that can call a self-supervised learning program based on audio and video alignment stored in a memory and execute the self-supervised learning method based on audio and video alignment provided in the embodiments of the present application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the self-supervised learning program based on audio and video alignment is called can be referred to in the various embodiments of the self-supervised learning method based on audio and video alignment of the present application and will not be further described here.

[0065] In a fourth aspect, an embodiment of the present application also provides a readable storage medium.

[0066] The readable storage medium of the present application stores a self-supervised learning program based on audio and video alignment, wherein when the self-supervised learning program based on audio and video alignment is executed by a processor, the steps of the self-supervised learning method based on audio and video alignment as described above are implemented.

[0067] Among them, the method implemented when the self-supervised learning program based on audio and video alignment is executed can refer to the various embodiments of the self-supervised learning method based on audio and video alignment in this application, and will not be repeated here.

[0068] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0069] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, or the part that contributes to the existing technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above and includes a number of instructions for enabling a terminal device to execute the methods described in each embodiment of this application.

[0070] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A self-supervised learning method based on audio and video alignment, characterized in that: The following steps are involved: Obtain an audio and video offset dataset and input it into the model for self-supervised training of audio and video alignment. Obtaining a similarity matrix between audio frames in the original audio signal and video frames in the video signal, constructing an alignment loss function based on the similarity matrix, training an audio alignment module and a video alignment module, predicting an audio and video offset using the trained audio alignment module and video alignment module, and performing a reverse frame offset operation on the video features to obtain aligned audio and video; The encoder is trained using a mask prediction loss function, and the trained encoder is used to perform cross-modal feature fusion on the aligned audio and video to obtain multimodal features of audio and video. Combining the mask prediction loss and the alignment loss into a total loss function, jointly training the encoder weights, so that the encoder simultaneously learns the audio and video multimodal features and the alignment features during the training process, and outputs the aligned audio and video multimodal features; Taking the aligned multimodal features as input, the loss function trained according to the corresponding task is used to learn and decode the multimodal features, train the decoder, and output the audio and video decoding results for downstream speech-related tasks.

2. The self-supervised learning method based on audio and video alignment according to claim 1, characterized in that Obtaining the audio and video offset dataset includes the following steps: Randomly offset video frames within a preset frame range to construct an audio and video offset dataset.

3. The self-supervised learning method based on audio and video alignment according to claim 1, characterized in that Obtain a similarity matrix of audio frames in the original audio signal and video frames in the video signal, construct an alignment loss function based on the similarity matrix, train an audio alignment module and a video alignment module, predict the audio and video offset using the trained audio alignment module and video alignment module, and perform a reverse frame offset operation on the video features to obtain aligned audio and video, specifically comprising the following steps: Extracting audio feature vectors and video feature vectors through an audio encoder and a video encoder respectively; Calculate the similarity matrix between the audio feature vector and the video feature vector; constructing an alignment loss function based on the similarity matrix; Based on the alignment loss function training, the audio alignment module and the video alignment module are obtained. The trained audio alignment module and the video alignment module predict the audio and video offset, and perform reverse frame offset operation on the video features to obtain aligned audio and video.

4. The self-supervised learning method based on audio and video alignment according to claim 3, characterized in that The constructing of the alignment loss function based on the similarity matrix specifically includes the following steps: Calculate the average similarity of each diagonal in the similarity matrix; Determine the diagonal sequence number with the smallest average similarity as the audio and video offset; The alignment sub-matrix is cropped from the similarity matrix according to the offset labels provided by the audio and video offset dataset; The cross entropy loss of the audio stream and video stream is calculated based on the row and column directions of the alignment submatrix respectively, and the average of the two is taken to construct the alignment loss function.

5. The self-supervised learning method based on audio and video alignment according to claim 4, characterized in that The alignment loss function is shown as follows: Where, is the alignment loss, is the cross entropy loss of the audio stream, is the cross entropy loss of the video stream.

6. The self-supervised learning method based on audio and video alignment according to claim 1, characterized in that The encoder is trained using the mask prediction loss function. The trained encoder is used to perform cross-modal feature fusion on the aligned audio and video to obtain multimodal features of audio and video. The specific steps include: Input the masked audio and video features into the multimodal Transformer encoder, predict the true target features of the masked part, and construct a mask prediction loss function for the audio and video features based on the prediction results. Train the encoder based on the mask prediction loss function. The trained encoder is used to perform cross-modal fusion of audio features and video features to generate multimodal features.

7. The self-supervised learning method based on audio and video alignment according to claim 1, wherein: The mask prediction loss and alignment loss are combined into a total loss function, and the encoder weights are jointly trained to enable the encoder to simultaneously learn audio and video multimodal features and alignment features during the training process, and output aligned audio and video multimodal features, specifically including the following steps: The mask prediction loss function and the alignment loss function are combined into a total loss function through hyperparameter weighting; The alignment loss and mask prediction loss are optimized simultaneously based on the total loss function to train the encoder. The encoder is a 12-layer Transformer structure, and the output is aligned multimodal features, which are used in the decoder and then applied to downstream speech-related tasks.

8. The self-supervised learning method based on audio and video alignment according to claim 7, characterized in that The total loss function is expressed as: Where, is the alignment loss, is the mask prediction loss, is the weight hyperparameter of the alignment loss.

9. The self-supervised learning method based on audio and video alignment according to claim 1, wherein: The method takes the aligned multimodal features as input, uses the loss function of the corresponding task's standard training to learn and decode the multimodal features, trains to obtain the decoder, and outputs the audio and video decoding results for use in downstream speech-related task steps. The decoder is a 6-layer Transformer structure, and the downstream speech-related tasks are speech recognition, speech enhancement, or speech generation.

10. A self-supervised learning system based on audio and video alignment, characterized in that The following steps are involved: The dataset acquisition module is used to obtain the audio and video offset dataset and input it into the model for self-supervised training of audio and video alignment; An audio and video alignment module is communicatively connected to the data set acquisition module, and is used to obtain a similarity matrix between audio frames in the original audio signal and video frames in the video signal, construct an alignment loss function based on the similarity matrix, train an audio alignment module and a video alignment module, predict the audio and video offset using the trained audio alignment module and video alignment module, and perform a reverse frame offset operation on the video features to obtain aligned audio and video; A self-supervised learning module is communicatively connected to the audio and video alignment module, and is used to construct an audio and video offset dataset based on the similarity matrix, train an encoder based on the audio and video offset dataset using a mask prediction loss function, and use the trained encoder to perform cross-modal feature fusion on the aligned audio and video to obtain multimodal features of audio and video; The audio and video alignment module and the self-supervised learning module are trained to obtain the encoder through the total loss function of the combination of mask prediction loss and alignment loss; The fine-tuning module is used to connect to the decoder after the joint training of the audio and video alignment module and the self-supervised learning module is completed. It is used to take the aligned multimodal features as input, use the loss function trained according to the corresponding task's specifications to learn to decode the multimodal features, train the decoder, and output the audio and video decoding results for downstream speech-related tasks.

Images