Voice and lip movement combined recognition method and system based on multi-mode self-supervised learning

Abstract

The invention belongs to the field of artificial intelligence and natural language processing, and provides a voice and lip movement combined recognition method and system based on multi-mode self-supervised learning, and the method comprises the steps: collecting a to-be-recognized voice signal and a to-be-recognized lip movement video, and carrying out the recognition of a multi-mode combined recognition model; the pre-training process of the multi-modal joint recognition model comprises the steps of training data acquisition, voice feature extraction, motion feature extraction, self-supervised learning, time sequence matching and fusion and model optimization. According to the method, the adaptive capacity to a complex scene can be enhanced by combining two kinds of modal information of voice and lip movement; through self-supervised learning, the missing prediction and masking prediction capabilities of the system are improved.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and natural language processing, and in particular to a method and system for joint recognition of speech and lip movement based on multimodal self-supervised learning. Background Art

[0002] Speech recognition has penetrated into many fields of application. However, in actual use, there are some unfavorable conditions that may lead to a decline in its generalization performance. Especially in noisy environments, the accuracy of speech recognition will be significantly reduced due to the presence of noise. In addition, incompletely audible speech also poses great difficulties for speech recognition, because some words may not be pronounced clearly or without sound.

[0003] Existing solutions mainly include improving voice collection equipment and adopting hybrid voice recognition methods. However, both methods have certain limitations and high costs. Although existing voice collection equipment can reduce some noise interference through high-quality microphones and noise suppression technology, it is still difficult to effectively capture clear voice in complex environments (such as multi-person conversations, background noise, echo, etc.), and the hardware cost is high, the applicable scenarios are limited, and the universality is insufficient. Although the hybrid voice recognition method combines traditional acoustic models and deep learning technology, it has a strong reliance on a large amount of labeled noise data, high training costs, and limited robustness and accuracy when dealing with complex accents, fast speech speeds or non-standard pronunciations.

[0004] Therefore, in order to solve the above problems, there is an urgent need for a speech and lip movement joint recognition method and system based on multimodal self-supervised learning to achieve accurate recognition of speech and lip movement data in complex environments. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a method and system for joint recognition of speech and lip movement based on multimodal self-supervised learning, so as to achieve accurate recognition of speech and lip movement data in complex environments.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A method for joint recognition of speech and lip movement based on multimodal self-supervised learning, comprising:

[0008] Collect the speech signal to be recognized and the lip movement video to be recognized;

[0009] Inputting the speech signal to be recognized and the lip movement video to be recognized into a pre-trained multimodal joint recognition model for recognition, and obtaining a target recognition result;

[0010] The pre-training process of the multimodal joint recognition model includes:

[0011] In a preset noise scene, a microphone and a high-definition camera are used to collect the target user's speech signal to be trained and the lip movement video to be trained; the noise scene includes: office environment, street noise and background music;

[0012] The speech signal to be trained is converted into a time-frequency spectrum graph by using short-time Fourier transform, and the time-frequency spectrum graph is input into a deep convolutional neural network, and the speech features of the time-frequency spectrum graph are extracted by using the convolution layer and the pooling layer in the deep convolutional neural network to obtain a low-dimensional time-frequency feature vector; the speech features include: semantic information data and acoustic change data;

[0013] Extracting motion features of the lip movement video to be trained using a 3D convolutional neural network to obtain a lip movement feature vector;

[0014] Using self-supervised learning, respectively, the low-dimensional time-frequency feature vector and the lip movement feature vector are subjected to missing prediction and occlusion prediction to obtain speech modality information and lip movement modality information;

[0015] Using a self-attention mechanism to perform temporal matching and fusion on the speech modality information and the lip movement modality information to obtain a basic model;

[0016] A multimodal contrastive learning strategy is used to construct positive samples and negative samples, and the positive samples and negative samples are used to optimize the basic model to obtain the multimodal joint recognition model.

[0017] Preferably, the dimension of the low-dimensional time-frequency feature vector is lower than the dimensions of the original convolutional layer and the maximum pooling layer in the deep convolutional neural network.

[0018] Preferably, a speech and lip movement joint recognition system based on multimodal self-supervised learning comprises:

[0019] A speech pattern training module is used to extract feature data of pre-collected speech signals to be trained and lip movement videos to be trained using a deep convolutional neural network and a 3D convolutional neural network, respectively, to obtain low-dimensional time-frequency feature vectors and lip movement feature vectors;

[0020] A self-supervised training module, used to perform missing prediction and occlusion prediction on the low-dimensional time-frequency feature vector and the lip movement feature vector respectively by using self-supervised learning to obtain speech modality information and lip movement modality information;

[0021] The modality integration module is used to construct positive samples and negative samples using a multimodal contrast learning strategy, and optimize the basic model using the positive samples and the negative samples to obtain the multimodal joint recognition model.

[0022] The present invention discloses the following technical effects:

[0023] The present invention provides a method and system for joint recognition of speech and lip movement based on multimodal self-supervised learning. By combining the two modal information of speech and lip movement, the defect of poor adaptability of traditional single-modal systems to complex scenes is solved, and accurate recognition of the target user's speech in complex scenes is achieved; through self-supervised learning, the problem of poor classification and reasoning capabilities of traditional recognition models is solved, and the missing prediction of speech information and the masked prediction of lip movement information are achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0025] Figure 1 A schematic diagram of the process of joint recognition of speech and lip movements by multimodal self-supervised learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The purpose of the present invention is to provide a method and system for joint recognition of speech and lip movement based on multimodal self-supervised learning, so as to realize accurate recognition of speech and lip movement data in complex environments.

[0028] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] Figure 1 A schematic diagram of the speech and lip movement joint recognition process of multimodal self-supervised learning provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the present invention provides a method for joint recognition of speech and lip movement based on multimodal self-supervised learning, comprising:

[0030] Step 100: Collect the speech signal to be recognized and the lip movement video to be recognized;

[0031] Step 200: inputting the speech signal to be recognized and the lip movement video to be recognized into a pre-trained multimodal joint recognition model for recognition, and obtaining a target recognition result;

[0032] The pre-training process of the multimodal joint recognition model includes:

[0033] Step 201: In a preset noise scene, a microphone and a high-definition camera are used to collect a target user's speech signal to be trained and a lip movement video to be trained. The noise scene includes: an office environment, street noise, and background music.

[0034] Step 202: using short-time Fourier transform to convert the speech signal to be trained into a time-frequency spectrum graph, and inputting the time-frequency spectrum graph into a deep convolutional neural network, and using the convolution layer and pooling layer in the deep convolutional neural network to extract speech features of the time-frequency spectrum graph to obtain a low-dimensional time-frequency feature vector; the speech features include: semantic information data and acoustic change data;

[0035] Step 203: extracting motion features of the lip movement video to be trained using a 3D convolutional neural network to obtain a lip movement feature vector;

[0036] Step 204: using self-supervised learning to perform missing prediction and occlusion prediction on the low-dimensional time-frequency feature vector and the lip movement feature vector, respectively, to obtain speech modality information and lip movement modality information;

[0037] Step 205: using a self-attention mechanism to perform temporal matching and fusion on the speech modality information and the lip movement modality information to obtain a basic model;

[0038] Step 206: construct positive samples and negative samples using a multimodal contrastive learning strategy, and optimize the basic model using the positive samples and the negative samples to obtain the multimodal joint recognition model.

[0039] Specifically, the dimension of the low-dimensional time-frequency feature vector is lower than the dimensions of the original convolutional layer and the maximum pooling layer in the deep convolutional neural network.

[0040] Furthermore, a speech and lip movement joint recognition system based on multimodal self-supervised learning includes:

[0041] A speech pattern training module is used to extract feature data of pre-collected speech signals to be trained and lip movement videos to be trained using a deep convolutional neural network and a 3D convolutional neural network, respectively, to obtain low-dimensional time-frequency feature vectors and lip movement feature vectors;

[0042] A self-supervised training module, used to perform missing prediction and occlusion prediction on the low-dimensional time-frequency feature vector and the lip movement feature vector respectively by using self-supervised learning to obtain speech modality information and lip movement modality information;

[0043] The modality integration module is used to construct positive samples and negative samples using a multimodal contrast learning strategy, and optimize the basic model using the positive samples and the negative samples to obtain the multimodal joint recognition model.

[0044] Specifically, the speech pattern training module is used to obtain the user's speech information and the corresponding lip movement video information as a training set, and input it into the convolutional neural network for speech and lip movement feature extraction; the self-supervised training module is used to perform self-supervised tasks to facilitate learning of robust speech representations; the modal integration module is used to establish an interactive channel between the speech pattern and the lip movement pattern, and calculate the similarity between the two to obtain the final output result.

[0045] Furthermore, the training process of the speech pattern training module includes the following steps:

[0046] S1: Short-time Fourier transform (STFT) is used to extract time-frequency domain feature representation;

[0047] S2: converting the user's voice signal into a time-frequency spectrum graph and inputting the result into a convolutional neural network (CNN) composed of a deep convolutional neural network;

[0048] S3: A combination of convolutional layers and pooling layers is used to extract features. The resulting time-frequency feature vector contains the keyword information corresponding to each speech syllable, and its dimension is lower than that of the original convolutional layer and the maximum pooling layer.

[0049] S4: Use the Softmax activation function to classify the time-frequency feature vector and select the correct label to obtain the word with the highest accuracy.

[0050] Preferably, the implementation process of the self-supervised training module includes the following steps:

[0051] S1: In the process of self-supervised learning of speech patterns, it is assumed that the user's speech will be affected by the environment and undergo certain changes, such as lip shape or sound distortion;

[0052] S2: Input the user's real speech signal into the model as a whole, and achieve the difference in audibility and completeness of the speech signal by modifying part of it;

[0053] S3: Reassemble the incomplete signal to restore it to its original state, preserve the waveform and time series features of the speech signal during the reconstruction process, and compare the restored speech segment with the original to determine the accuracy and recall of the model;

[0054] S4: Other motivational parameters are added during the training process, such as loss functions such as cross entropy or mean absolute error, to improve the performance of the algorithm and to check whether its performance level meets the requirements during the validation phase.

[0055] Preferably, the implementation process of the modality integration module includes the following steps:

[0056] S1: Input the information from the speech and lip movement modalities into a common network space and integrate them to form a new high-dimensional feature expression;

[0057] S2: In this process, each subspace influences each other and is correlated with the overall distribution, which significantly improves the generalization performance of the entire network.

[0058] Specifically, there are the following three types of connections between the speech and lip movement modalities: 1) Closely connected interactions in time: When one of the modalities is abnormal, the other modality will also be affected; 2) Spatial overlapping effects: Although the two belong to different spatial ranges, speech and lip movement modalities establish connections through shape matching and other methods; 3) Synergy at the feature level: The information of the two can often be jointly encoded into a unified vector form, which can make up for the negative impact of their respective defects to a certain extent.

[0059] Specifically, data collection and preprocessing. First, the system collects the user's voice signal and the corresponding lip movement video. The voice data is recorded by a high-quality microphone, while the lip movement video records the movement of the speaker's mouth through a high-definition camera. During the recording process, different noise scenes are selected to simulate complex real environments, including office environments, street noise, and background music.

[0060] Furthermore, feature extraction. For the speech modality, the short-time Fourier transform (STFT) is used to convert the original audio signal into a time-frequency spectrum, retaining the time-frequency characteristics of the speech signal. These time-frequency diagrams are then input into a deep convolutional neural network (DCNN), and the important features of the speech signal are extracted using convolutional layers and pooling layers. These features include the semantic information and acoustic changes of the syllables, and ultimately generate a set of low-dimensional time-frequency feature vectors. At the same time, the lip movement modality uses a 3D convolutional neural network (3D-CNN) to extract the movement features of the lips. By analyzing the spatiotemporal changes in the video, 3D-CNN can capture subtle changes in the shape and movement trajectory of the lips, and then generate a lip movement feature vector corresponding to the speech content.

[0061] Furthermore, self-supervised learning. In the process of self-supervised learning, the speech modality and lip movement modality perform independent tasks. The speech modality predicts the missing parts of the speech signal, that is, randomly masks certain parts of the speech segment, and reconstructs the masked segments through the model, thereby improving the model's ability to handle incomplete speech. The lip movement modality performs occlusion prediction of the lip movement video, randomly masks certain frames in the lip movement video, and requires the model to reconstruct the occluded lip movements to improve the model's ability to understand partially visible video data.

[0062] Preferably, multimodal fusion and joint learning are used. After completing the self-supervised learning task, the system uses the self-attention mechanism to perform temporal matching and fusion of the two modal information of speech and lip movement. The self-attention mechanism can dynamically adjust the feature weights of the two modalities, making the speech and lip movement features more coordinated, thereby improving the accuracy of recognition. To further optimize the model, the system adopts a multimodal contrastive learning strategy to optimize the model representation by constructing positive and negative sample pairs. The speech and lip movement samples of the same speaker constitute positive samples, and the samples of different speakers constitute negative samples. The model optimizes the recognition performance by comparing the positive and negative sample pairs.

[0063] Furthermore, testing and verification. During the verification of the system, the public multimodal datasets VoxCeleb and GRID were used. The test scenarios included quiet environments and noisy backgrounds to perform speech and lip movement recognition tasks. The experimental results show that the speech recognition accuracy of the multimodal joint recognition system in noisy environments is significantly improved compared to the traditional single-modal system. Especially in the case of partial speech loss, the introduction of lip movement modality helps the system significantly improve the robustness and accuracy of recognition.

[0064] The beneficial effects of the present invention are as follows:

[0065] The present invention can enhance the adaptability to complex scenes by combining the two modal information of speech and lip movement; through self-supervised learning, the missing prediction and masking prediction capabilities of the system are improved.

[0066] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0067] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for joint recognition of speech and lip movement based on multimodal self-supervised learning, characterized in that: include: Collect the speech signal to be recognized and the lip movement video to be recognized; Inputting the speech signal to be recognized and the lip movement video to be recognized into a pre-trained multimodal joint recognition model for recognition, and obtaining a target recognition result; The pre-training process of the multimodal joint recognition model includes: In a preset noise scene, a microphone and a high-definition camera are used to collect the target user's speech signal to be trained and the lip movement video to be trained; the noise scene includes: office environment, street noise and background music; The speech signal to be trained is converted into a time-frequency spectrum graph by using short-time Fourier transform, and the time-frequency spectrum graph is input into a deep convolutional neural network, and the speech features of the time-frequency spectrum graph are extracted by using the convolution layer and the pooling layer in the deep convolutional neural network to obtain a low-dimensional time-frequency feature vector; the speech features include: semantic information data and acoustic change data; Extracting motion features of the lip movement video to be trained using a 3D convolutional neural network to obtain a lip movement feature vector; Using self-supervised learning, respectively, the low-dimensional time-frequency feature vector and the lip movement feature vector are subjected to missing prediction and occlusion prediction to obtain speech modality information and lip movement modality information; Using a self-attention mechanism to perform temporal matching and fusion on the speech modality information and the lip movement modality information to obtain a basic model; A multimodal contrastive learning strategy is used to construct positive samples and negative samples, and the positive samples and negative samples are used to optimize the basic model to obtain the multimodal joint recognition model.

2. The method for joint recognition of speech and lip movement based on multimodal self-supervised learning according to claim 1, characterized in that: The dimension of the low-dimensional time-frequency feature vector is lower than the dimensions of the original convolutional layer and the maximum pooling layer in the deep convolutional neural network.

3. A speech and lip movement joint recognition system based on multimodal self-supervised learning, characterized in that: include: A speech pattern training module is used to extract feature data of pre-collected speech signals to be trained and lip movement videos to be trained using a deep convolutional neural network and a 3D convolutional neural network, respectively, to obtain low-dimensional time-frequency feature vectors and lip movement feature vectors; A self-supervised training module, used to perform missing prediction and occlusion prediction on the low-dimensional time-frequency feature vector and the lip movement feature vector respectively by using self-supervised learning to obtain speech modality information and lip movement modality information; The modality integration module is used to construct positive samples and negative samples using a multimodal contrast learning strategy, and optimize the basic model using the positive samples and the negative samples to obtain the multimodal joint recognition model.

Images