Audiovisual speech recognition methods, devices, equipment, media and products

Abstract

This application provides an audiovisual speech recognition method, apparatus, device, medium, and product. The method includes: acquiring video features to be recognized and corresponding noisy spectrograms; the video features are features extracted based on a user's lip image; the noisy spectrogram is generated based on the user's audio data; inputting the video features and the noisy spectrogram into a preset audiovisual speech enhancement model for noise reduction processing, and outputting a corresponding enhanced spectrogram; inputting the enhanced spectrogram and the video features into a preset audiovisual speech recognition model for speech recognition, and outputting a corresponding final character sequence. The audiovisual speech recognition method of this application recognizes both the enhanced spectrogram and video features using a preset audiovisual speech recognition model, improving the accuracy of the final character sequence through the complementary nature of the two modalities, thereby enhancing the accuracy of speech recognition.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an audiovisual speech recognition method, apparatus, device, medium and product. Background Technology

[0002] Speech is the most important means of communication in people's daily lives, and it also possesses various attributes unique to humans, such as regional and emotional characteristics. To further explore the secrets contained within speech, speech recognition technology has emerged. Automatic speech recognition technology, in addition to studying linguistics, incorporates theories and practices from various disciplines such as artificial intelligence, digital signal processing, and pattern recognition, making it of significant research value.

[0003] Current mainstream speech recognition technologies, such as single-modal visual lip reading and single-modal auditory speech recognition, have high accuracy rates. However, high accuracy is based on specific speech scenarios, such as pure and noise-free speech datasets, identical distribution of training and test data, and stable speech rate.

[0004] In complex environments with high noise levels, dialects, and accents, current speech recognition methods are less accurate. Summary of the Invention

[0005] This application provides an audiovisual speech recognition method, apparatus, device, medium, and product to solve the problem that current speech recognition methods have poor accuracy in complex environments such as high noise, dialects, and accents.

[0006] The first aspect of this application provides an audiovisual speech recognition method, including:

[0007] The video features to be identified and the corresponding noisy spectrogram are obtained; the video features are extracted based on the user's lip image; the noisy spectrogram is generated based on the user's audio data.

[0008] The video features and the noisy spectrogram are input into a preset audiovisual speech enhancement model for noise reduction, and the corresponding enhanced spectrogram is output.

[0009] The enhanced spectrogram and the video features are input into a preset audiovisual speech recognition model for speech recognition, and the corresponding final character sequence is output.

[0010] Furthermore, in the method described above, obtaining the video features to be identified and the corresponding noisy spectrogram includes:

[0011] Acquire the audio and video data to be identified;

[0012] The audio data is preprocessed to generate a corresponding noisy spectrogram; the preprocessing includes pre-emphasis processing, frame segmentation, windowing, short-time Fourier transform, and Mel filtering.

[0013] The video data is subjected to facial recognition using a preset facial recognition tool to generate corresponding facial images;

[0014] The face image is input into a preset pseudo-3D residual convolutional network to extract the video features.

[0015] Furthermore, in the method described above, the preset audiovisual speech enhancement model includes: a video encoder, an audio encoder, a preset fusion noise reduction module, and an audio decoder;

[0016] The step of inputting the video features and the noisy spectrogram into a preset audiovisual speech enhancement model for noise reduction processing and outputting the corresponding enhanced spectrogram includes:

[0017] The video features are input into the video encoder for encoding processing to generate corresponding first video encoded features;

[0018] The noisy spectrogram is input into the audio encoder for encoding processing to generate a corresponding first audio encoding feature;

[0019] The first video coding feature and the first audio coding feature are input into the preset fusion noise reduction module for fusion noise reduction to generate enhanced coding features after noise reduction.

[0020] The enhanced coding features are input into the audio decoder for decoding to generate the enhanced spectrogram.

[0021] Furthermore, in the method described above, the preset fusion noise reduction module includes: a soft thresholding algorithm and a threshold block;

[0022] The step of inputting the first video coding feature and the first audio coding feature into the preset fusion noise reduction module for fusion noise reduction to generate enhanced coding features after noise reduction includes:

[0023] The first video coding feature and the first audio coding feature are concatenated to generate an intermediate coding feature;

[0024] The intermediate encoded features are input into the threshold block for convolutional pooling and nonlinear transformation to generate a threshold reference value.

[0025] The soft thresholding algorithm is used to denoise the audio coding features based on the threshold reference value, generating denoised enhanced coding features.

[0026] Furthermore, in the method described above, the preset audiovisual speech recognition model includes an audiovisual encoder, a preset audiovisual fusion algorithm, an audiovisual decoder, and a Softmax function;

[0027] The step of inputting the enhanced spectrogram and the video features into a preset audiovisual speech recognition model for speech recognition and outputting the corresponding final character sequence includes:

[0028] The enhanced spectrogram and the video features are input into the audiovisual encoder for encoding processing to generate corresponding second audio encoding features and second video encoding features; the audiovisual encoder includes a temporal convolutional neural network and a skip recurrent neural network SkipRNN;

[0029] The preset audio-visual fusion algorithm is used to perform cross-modal attention mechanism processing and fusion processing on the second audio coding feature and the second video coding feature to generate the final fused coding feature;

[0030] The final fused coding features are input into the audiovisual decoder for decoding processing to generate the final fused decoding features;

[0031] The Softmax function is used to classify the final fused decoding features and output the final character sequence corresponding to the maximum probability.

[0032] Further, in the method described above, the step of using the preset audio-visual fusion algorithm to perform cross-modal attention mechanism processing and fusion processing on the second audio coding feature and the second video coding feature to generate the final fused coding feature includes:

[0033] The preset audio-visual fusion algorithm is used to project the second video coding feature onto the second audio coding feature to generate a second audio coding feature that includes the second video coding feature;

[0034] The preset audio-visual fusion algorithm is used to project the second audio coding feature onto the second video coding feature to generate a second video coding feature containing the second audio coding feature;

[0035] The second audio coding feature containing the second video coding feature and the second video coding feature containing the second audio coding feature are fused together to generate the final fused coding feature.

[0036] A second aspect of this application provides an audiovisual speech recognition device, comprising:

[0037] The acquisition module is used to acquire the video features to be identified and the corresponding noisy spectrogram; the video features are features extracted based on the user's lip image; the noisy spectrogram is generated based on the user's audio data.

[0038] The noise reduction module is used to input the video features and the noisy spectrogram into a preset audiovisual speech enhancement model for noise reduction processing, and output the corresponding enhanced spectrogram.

[0039] The recognition module is used to input the enhanced spectrogram and the video features into a preset audiovisual speech recognition model for speech recognition and output the corresponding final character sequence.

[0040] Furthermore, in the apparatus described above, the acquisition module is specifically used for:

[0041] The process involves acquiring audio and video data to be identified; preprocessing the audio data to generate a corresponding noisy spectrogram; the preprocessing includes pre-emphasis processing, frame segmentation, windowing, short-time Fourier transform, and Mel filtering; using a preset face recognition tool to perform face recognition on the video data to generate a corresponding face image; and inputting the face image into a preset pseudo-3D residual convolutional network to extract the video features.

[0042] Furthermore, in the device described above, the preset audiovisual speech enhancement model includes: a video encoder, an audio encoder, a preset fusion noise reduction module, and an audio decoder;

[0043] The noise reduction module is specifically used for:

[0044] The video features are input into the video encoder for encoding processing to generate corresponding first video encoded features; the noisy spectrogram is input into the audio encoder for encoding processing to generate corresponding first audio encoded features; the first video encoded features and the first audio encoded features are input into the preset fusion noise reduction module for fusion noise reduction to generate noise-reduced enhanced encoded features; the enhanced encoded features are input into the audio decoder for decoding processing to generate the enhanced spectrogram.

[0045] Furthermore, in the device described above, the preset fusion noise reduction module includes: a soft thresholding algorithm and a threshold block;

[0046] When the noise reduction module inputs the first video coding feature and the first audio coding feature into the preset fusion noise reduction module for fusion noise reduction to generate enhanced coding features after noise reduction, it is specifically used for:

[0047] The first video coding feature and the first audio coding feature are concatenated to generate an intermediate coding feature; the intermediate coding feature is input into the threshold block for convolutional pooling and nonlinear transformation processing to generate a threshold reference value; the soft thresholding algorithm is used to denoise the audio coding feature based on the threshold reference value to generate a denoised enhanced coding feature.

[0048] Furthermore, in the device described above, the preset audiovisual speech recognition model includes an audiovisual encoder, a preset audiovisual fusion algorithm, an audiovisual decoder, and a Softmax function;

[0049] The identification module is specifically used for:

[0050] The enhanced spectrogram and the video features are input into the audiovisual encoder for encoding processing to generate corresponding second audio encoding features and second video encoding features. The audiovisual encoder includes a temporal convolutional neural network and a skip recurrent neural network (SkipRNN). The preset audiovisual fusion algorithm is used to perform cross-modal attention processing and fusion processing on the second audio encoding features and the second video encoding features to generate final fused encoding features. The final fused encoding features are input into the audiovisual decoder for decoding processing to generate final fused decoding features. The Softmax function is used to classify the final fused decoding features and output the final character sequence corresponding to the maximum probability.

[0051] Furthermore, in the apparatus described above, when the recognition module performs cross-modal attention mechanism processing and fusion processing on the second audio coding feature and the second video coding feature using the preset audio-visual fusion algorithm to generate the final fused coding feature, it is specifically used for:

[0052] The second video coding feature is projected onto the second audio coding feature using the preset audio-visual fusion algorithm to generate a second audio coding feature containing the second video coding feature; the second audio coding feature is projected onto the second video coding feature using the preset audio-visual fusion algorithm to generate a second video coding feature containing the second audio coding feature; the second audio coding feature containing the second video coding feature and the second video coding feature containing the second audio coding feature are fused to generate the final fused coding feature.

[0053] A third aspect of this application provides an electronic device, including: a memory and a processor;

[0054] The memory stores computer-executed instructions;

[0055] The processor executes computer execution instructions stored in the memory to implement the audiovisual speech recognition method as described in any of the first aspects.

[0056] A fourth aspect of this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the audiovisual speech recognition method described in any of the first aspects.

[0057] The fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the audiovisual speech recognition method described in any of the first aspects.

[0058] This application provides an audiovisual speech recognition method, apparatus, device, medium, and product. The method includes: acquiring video features to be recognized and corresponding noisy spectrograms; the video features are features extracted based on a user's lip image; the noisy spectrogram is generated based on the user's audio data; inputting the video features and the noisy spectrogram into a preset audiovisual speech enhancement model for noise reduction processing, and outputting a corresponding enhanced spectrogram; inputting the enhanced spectrogram and the video features into a preset audiovisual speech recognition model for speech recognition, and outputting a corresponding final character sequence. The audiovisual speech recognition method of this application uses a preset audiovisual speech enhancement model to perform noise reduction processing on the noisy spectrogram based on video features, thereby obtaining an enhanced spectrogram with higher purity. By recognizing both the enhanced spectrogram and video features using a preset audiovisual speech recognition model, the accuracy of the final character sequence is improved through the complementary nature of the two modalities, thus enhancing the accuracy of speech recognition. Attached Figure Description

[0059] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0060] Figure 1 This is a scene diagram illustrating how the audiovisual speech recognition method of the embodiments of this application can be implemented;

[0061] Figure 2 Flowchart of the audiovisual speech recognition method provided in this application Figure 1 ;

[0062] Figure 3 Flowchart of the audiovisual speech recognition method provided in this application Figure 2 ;

[0063] Figure 4 A schematic diagram illustrating the overall process of the audiovisual speech recognition method provided in this application;

[0064] Figure 5 A schematic diagram of the pre-defined pseudo-3D residual convolutional network structure type provided in this application;

[0065] Figure 6 A schematic diagram of the noise reduction process for the audiovisual speech recognition method provided in this application;

[0066] Figure 7 A schematic diagram of the threshold block structure for the audiovisual speech recognition method provided in this application;

[0067] Figure 8 A schematic diagram of the audio-visual encoder structure for the audio-visual speech recognition method provided in this application;

[0068] Figure 9 A schematic diagram of the audiovisual speech recognition device provided in this application;

[0069] Figure 10 A schematic diagram of the structure of the electronic device provided in this application.

[0070] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0071] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0072] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solutions of this application comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0073] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0074] It should be noted that the audiovisual speech recognition methods, devices, equipment, media, and products disclosed herein can be used in the field of data processing technology. They can also be used in any field other than data processing, such as the financial field. The application areas of the audiovisual speech recognition methods, devices, equipment, media, and products disclosed herein are not limited.

[0075] The technical solutions of this application will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0076] To clearly understand the technical solution of this application, we will first provide a detailed introduction to the solutions of existing technologies. Current mainstream speech recognition technologies have achieved an accuracy rate of 98% or higher. However, this high accuracy is contingent on specific speech scenarios, such as a clean, noise-free speech dataset, identical distribution of training and testing data, and stable speech rate. Therefore, the key to the stable performance of current speech recognition technologies is overcoming the influence of various environmental factors. In recent years, with the rapid development of artificial intelligence technology, the trend from single-modal learning to multi-modal learning has become crucial for improving various AI tasks. To address the problem of low speech recognition rates in noisy environments, researchers have proposed the concept of audiovisual speech recognition. This technology is frequently used in daily life; for example, people with hearing impairments often combine lip movements with speech to help understand the speaker's voice. In noisy environments, people with normal hearing typically combine sound and lip movements to achieve an effect that cannot be achieved by simply combining noisy sounds.

[0077] Human perception of the outside world is multidimensional, utilizing not only hearing but also sight, touch, taste, and smell. Sight and hearing are the two most important sensory modalities in our daily lives. Over the past few decades, audiovisual speech recognition methods have seen extensive development in academia. However, the application of audiovisual speech recognition technology is still in its early stages. Many practical problems, such as those in navigation systems, intelligent voice assistants, and autonomous driving, remain to be solved by researchers and developers. These problems include the fusion of noisy video modal information, the inability to converge training due to data complexity, and poor model generalization. These issues significantly hinder the application of audiovisual speech recognition technology in various real-world scenarios. The problem of audio-visual modal information fusion is the most prominent limitation on the performance of audiovisual speech recognition. Problems in audio-visual modal fusion methods can lead to lower accuracy in multimodal audiovisual speech recognition compared to single-modal speech recognition. Given this theoretical and practical background, audiovisual fusion-based speech recognition technology is a topic requiring in-depth research and is also a highly challenging one.

[0078] With the development of artificial intelligence theory, especially deep learning theory, the performance of deep learning-based speech recognition systems has made great progress compared to the past. However, in complex environments such as high noise, dialects, and accents, the accuracy of current speech recognition methods is relatively poor.

[0079] Therefore, addressing the issue of poor accuracy in current speech recognition methods under complex environments such as high noise levels, dialects, and accents, the inventors discovered that since both single-modal visual lip-reading and single-modal auditory speech recognition aim to predict the text information corresponding to the spoken content, visual lip-reading information and auditory speech information can be considered as two heterogeneous modes with essentially the same distribution. To solve this problem, a multimodal approach combining visual and auditory elements can be used to reduce noise in the noisy spectrogram and improve speech recognition accuracy.

[0080] Specifically, the process involves acquiring the video features to be identified and their corresponding noisy spectrograms. The video features are extracted from the user's lip images. The noisy spectrograms are generated from the user's audio data. The video features and noisy spectrograms are then input into a pre-defined audiovisual speech enhancement model for noise reduction, outputting the corresponding enhanced spectrogram. Finally, the enhanced spectrogram and video features are input into a pre-defined audiovisual speech recognition model for speech recognition, outputting the corresponding final character sequence.

[0081] The audiovisual speech recognition method of this application uses a pre-set audiovisual speech enhancement model to denoise noisy spectrograms based on video features, thereby obtaining enhanced spectrograms with higher purity. By recognizing both the enhanced spectrogram and video features using a pre-set audiovisual speech recognition model, the accuracy of the final character sequence is improved through the complementary use of these two modalities, thus enhancing the accuracy of speech recognition.

[0082] Based on the above-mentioned inventive discovery, the inventor has proposed the technical solution of this application.

[0083] The following describes the application scenarios of the audiovisual speech recognition method provided in the embodiments of this application. For example... Figure 1 As shown, 1 represents an electronic device, 2 represents a camera, and 3 represents the user whose speech is to be recognized. The network architecture of the application scenario corresponding to the audiovisual speech recognition method provided in this embodiment includes: electronic device 1 and camera 2. Camera 2 can be a smart terminal with a camera function, such as a mobile phone or tablet.

[0084] For example, while user 3 is speaking, camera device 2 captures video and audio data of user 3. The video data includes user 3's facial features, especially the features corresponding to the lip image. Camera device 3 extracts features from the video and audio data to generate corresponding video features and a noisy spectrogram. Camera device 3 sends the video features and noisy spectrogram to electronic device 1. Electronic device 1 inputs the video features and noisy spectrogram into a preset audiovisual speech enhancement model for noise reduction processing and outputs the corresponding enhanced spectrogram. Simultaneously, electronic device 1 inputs the enhanced spectrogram and video features into a preset audiovisual speech recognition model for speech recognition and outputs the corresponding final character sequence. Electronic device 1 can output the final character sequence to camera device 2 to display the final character sequence to user 3.

[0085] In other application scenarios, the first electronic device 1 can also acquire video features and noisy spectrograms from other electronic devices that store video features and noisy spectrograms. Alternatively, the first electronic device 1 can also function as a camera to capture user 3, generating video and audio data corresponding to user 3, and performing feature extraction to obtain the corresponding video features and noisy spectrograms. This embodiment does not limit this aspect.

[0086] The embodiments of this application are described below with reference to the accompanying drawings.

[0087] The audio-visual speech recognition (AVSR) in this embodiment utilizes both audio and video dual-modal information to recognize speech information.

[0088] Figure 2 Flowchart of the audiovisual speech recognition method provided in this application Figure 1 ,like Figure 2 As shown, in this embodiment, the execution subject of this application embodiment is an audiovisual speech recognition device, which can be integrated into an electronic device. The audiovisual speech recognition method provided in this embodiment includes the following steps:

[0089] Step S101: Obtain the video features to be identified and the corresponding noisy spectrogram. The video features are extracted based on the user's lip image. The noisy spectrogram is generated based on the user's audio data.

[0090] In this embodiment, the data can be obtained directly from an electronic device that stores video features and corresponding noisy spectrograms, or it can be obtained by extracting video and audio data. This embodiment does not limit the specific method used.

[0091] Because users' speech is usually accompanied by interfering noises such as ambient sounds and device noises, the audio data obtained after feature extraction is a noisy spectrogram.

[0092] Step S102: Input the video features and noisy spectrogram into the preset audiovisual speech enhancement model for noise reduction processing, and output the corresponding enhanced spectrogram.

[0093] In this embodiment, a preset audiovisual speech enhancement model performs noise reduction processing on the noisy spectrogram based on video features. By leveraging the incompatibility between video features and the noisy spectrogram, the noise in the noisy spectrogram is further reduced, resulting in an enhanced spectrogram. This leads to higher accuracy in subsequent speech recognition.

[0094] The preset audiovisual and speech enhancement model can employ neural networks, including a video encoder, an audio encoder, a preset fusion and noise reduction module, and an audio decoder. The video encoder and audio encoder are convolutional neural networks, while the audio decoder is a deconvolutional neural network.

[0095] Step S103: Input the enhanced spectrogram and video features into the preset audiovisual speech recognition model for speech recognition, and output the corresponding final character sequence.

[0096] In this embodiment, the preset audiovisual speech recognition model is based on the complementarity of audiovisual multimodal speech recognition to further improve the accuracy of speech recognition.

[0097] This application provides an audiovisual speech recognition method, which includes: acquiring video features to be recognized and corresponding noisy spectrograms; inputting the video features and noisy spectrograms into a preset audiovisual speech enhancement model for noise reduction processing, and outputting corresponding enhanced spectrograms; inputting the enhanced spectrograms and video features into a preset audiovisual speech recognition model for speech recognition, and outputting corresponding final character sequences.

[0098] The audiovisual speech recognition method of this application uses a pre-set audiovisual speech enhancement model to denoise noisy spectrograms based on video features, thereby obtaining enhanced spectrograms with higher purity. By recognizing both the enhanced spectrogram and video features using a pre-set audiovisual speech recognition model, the accuracy of the final character sequence is improved through the complementary use of these two modalities, thus enhancing the accuracy of speech recognition.

[0099] Figure 3 Flowchart of the audiovisual speech recognition method provided in this application Figure 2 ,like Figure 3 As shown, the audiovisual speech recognition method provided in this embodiment is a further refinement based on the audiovisual speech recognition method provided in the previous embodiment of this application. Figure 4This is a schematic diagram of the overall process of the audiovisual speech recognition method provided in this application. The overall process of this embodiment is as follows: Figure 4 As shown, the audiovisual speech recognition method provided in this embodiment includes the following steps.

[0100] Step S201: Obtain the audio and video data to be identified.

[0101] In this embodiment, the data can be obtained from a shooting device or from an electronic device that stores audio and video data.

[0102] Step S202 involves preprocessing the audio data to generate the corresponding noisy spectrogram. Preprocessing includes pre-emphasis processing, frame segmentation, windowing, short-time Fourier transform, and Mel filtering.

[0103] In this embodiment, the audio data is first pre-emphasized. Pre-emphasis is used to increase the energy of the speech, making high-frequency information stand out and solving the problem of high energy in low frequencies in speech, i.e., spectral tilt. To divide the long speech signal into multiple short-time single-frame signals and to utilize the stable characteristics of short-term speech signals, a framing operation is performed. Since the discontinuity between the start and end signals of a speech frame can cause spectral leakage, windowing is required to reduce this effect. Windowing makes the overall signal more continuous. In this embodiment, a Hamming window can be used for windowing operations to improve the windowing effect.

[0104] Meanwhile, to convert the difficult-to-process time-domain speech signal into a frequency-domain signal that is easier to observe and analyze, a short-time Fourier transform (STFT) is performed on the speech frames after the previous processing to obtain the spectrum of each frame. The STFT window size is set to 640 samples, equivalent to 40ms, which is equivalent to the length of a single video frame of 25fps (frames per second). The window shift length is 10ms to ensure that adjacent windows overlap by 75%. The spectrogram obtained by the short-time Fourier transform (STFT) is multiplied by a Mel-scale filter bank to obtain a Mel-scale spectrogram, which yields the noisy spectrogram.

[0105] like Figure 4 As shown, the audio data, after the above preprocessing, especially the short-time Fourier transform, yields a noisy spectrogram.

[0106] Step S203: Use a preset face recognition tool to perform face recognition on the video data and generate the corresponding face image.

[0107] In this embodiment, for video feature extraction, since a video unfolds into a sequence of images along the time dimension, it naturally possesses both spatial and temporal attributes. Spatial attributes refer to the information contained in the lip image region within the image sequence, while temporal attributes refer to the motion and change information of the lip image region over time. Video modal features should simultaneously include both spatial and temporal features. When cropping the video into images, care should be taken to ensure that the images only contain facial images.

[0108] In this embodiment, the preset face recognition tool uses the OpenFace2.x open-source face recognition framework, thereby improving the accuracy of face recognition.

[0109] like Figure 4 As shown, during the feature extraction process from video data, the lip image is obtained from the face image after face recognition in the video data.

[0110] Step S204: Input the face image into a preset pseudo-3D residual convolutional network to extract video features.

[0111] This embodiment employs a pseudo-3D residual convolutional network (Pseudo-3D, abbreviated as P3D). Assuming the size of the 3D convolutional layer is represented as d×k×k, where d represents the temporal depth of the convolutional kernel and k is the spatial size of the kernel, then a 3×3×3 3D convolutional layer can be divided into a 1×3×3 spatial convolution (i.e., a regular 2D convolution) and a 3×1×1 temporal convolution. P3D has three structures, as detailed below. Figure 5 As shown, these are P3D-A, P3D-B, and P3D-C, respectively.

[0112] In this embodiment, a 30-layer P3D network consisting of P3D-A, P3D-B, and P3D-C stacked sequentially is used, with the following structure: P3D-A--P3D-B--P3D-C--P3D-A--P3D-B--P3D-C…...…P3D-A--P3D-B--P3D-C. This further improves the feature extraction effect.

[0113] When a speech signal is in a noisy background or in extreme cases where most information is obscured, the preset audiovisual speech enhancement model extracts the clean audio signal from the complex speech signal. The key to audiovisual speech enhancement is extracting clean audio information from the speaker's audio information containing noisy information using the speaker's lip movement information. The audiovisual speech enhancement model accepts two original modal inputs, namely audio and video modal inputs. After feature extraction by the feature extractor, the extracted features are fused using multimodal methods. The model then decodes the fused features to approximate clean acoustic features. The core step in audiovisual speech enhancement is the fusion of audio and video multimodal features; an excellent fusion algorithm is a prerequisite for good performance. The preset audiovisual speech enhancement model in this embodiment includes: a video encoder, an audio encoder, a preset fusion noise reduction module, and an audio decoder, with the overall structure as follows: Figure 6 As shown.

[0114] Step S205: Input the video features into the video encoder for encoding processing to generate the corresponding first video encoded features.

[0115] Step S206: Input the noisy spectrogram into the audio encoder for encoding processing to generate the corresponding first audio encoding feature.

[0116] Audio encoders and video encoders are composed of multiple layers of convolutional neural networks (CNNs) with different convolutional kernels, while decoders are composed of their corresponding deconvolutions. Figure 6 In this embodiment, fusion refers to fusing the higher representations of the two modalities obtained by the audio and video encoders.

[0117] Step S207: Input the first video coding feature and the first audio coding feature into the preset fusion noise reduction module for fusion noise reduction to generate the noise-reduced enhanced coding feature.

[0118] In this embodiment, the first audio coding feature is subjected to noise reduction processing through the first video coding feature to purify the user's voice features in the first audio coding feature.

[0119] Optionally, in this embodiment, the preset fusion noise reduction module includes: a soft threshold algorithm and a threshold block.

[0120] Step S207 can be specifically as follows:

[0121] The first video coding feature and the first audio coding feature are concatenated to generate the intermediate coding feature.

[0122] The intermediate encoded features are input into the threshold block and subjected to convolutional pooling and nonlinear transformation to generate a threshold reference value.

[0123] A soft thresholding algorithm is used to denoise the audio coding features based on a threshold reference value, generating denoised enhanced coding features.

[0124] The preset fusion noise reduction module is based on the soft thresholding algorithm (ST). The ST algorithm means that the signal values ​​within the absolute value range of the threshold are set to 0, and the signal values ​​outside the threshold are linearly transformed to asymptotically approach 0.

[0125] Soft thresholding is a non-linear transformation process, similar to the ReLU activation function. The ST algorithm can be used in neural networks instead of the ReLU function. The ST algorithm used in this embodiment can be summarized by the following formula.

[0126]

[0127] In the formula, x represents the audio modal feature containing noise information, y represents the enhanced audio feature, and τ represents the threshold.

[0128] As can be seen from the formula, the most crucial step in the ST algorithm is threshold calculation. A good threshold is a prerequisite for effective noise reduction. In ST, the threshold setting must satisfy two conditions: it must be a positive number and not exceed the maximum value of the signal. If the threshold is greater than the maximum value of the signal, the output will be 0. Based on the properties of the threshold, this embodiment designs a threshold block. The input is the higher representation information of the audio and video modalities after CNN processing, and the output is... like Figure 7 As shown.

[0129] In the threshold block, the first video coding feature X extracted from the video encoder and audio encoder is first... v and the first audio coding feature V a The video modality is then stitched together, which can be considered as partially completing the damaged audio modality. After two convolutional max-pooling operations, more shared feature space information between the audio and video modalities is preserved. The value obtained through the Sigmoid function represents the clean information in the audio, i.e., the part that should be enhanced. The Sigmoid function is an activation function that maps a real number to the interval (0, 1), and can be used for binary classification. The threshold should then be the difference between 1 and the Sigmoid value. The above process can be expressed by the following formula.

[0130]

[0131]

[0132] For each different input sample, a corresponding threshold is obtained. This can be seen as a special attention mechanism, noting features relevant to the current task and retaining them, while setting irrelevant features to 0. The audio modal features extracted by the audio encoder are compared with the threshold τ. Feature values ​​outside the absolute value range of τ are processed to approach 0, while those within the absolute value range of τ are set to 0. This significantly reduces the noise information mixed in the audio feature map and increases the display of clean audio features.

[0133] Step S208: Input the enhanced coding features into the audio decoder for decoding processing to generate an enhanced spectrogram.

[0134] After the audio feature map processed by the ST algorithm is deconvolved by the audio decoder, an enhanced spectrogram with the same dimensions as the noisy spectrogram is obtained, which provides a foundation for subsequent speech recognition.

[0135] Step S209: Input the enhanced spectrogram and video features into the preset audiovisual speech recognition model to perform speech recognition and output the corresponding final character sequence.

[0136] The implementation of step S209 in this embodiment is similar to that of step S103 in the previous embodiment, and will not be described in detail here.

[0137] Optionally, in this embodiment, the preset audiovisual speech recognition model includes an audiovisual encoder, a preset audiovisual fusion algorithm, an audiovisual decoder, and a Softmax function. Then step S209 can be refined as follows:

[0138] The enhanced spectrogram and video features are input into the audiovisual encoder for encoding, generating corresponding second audio and second video encoded features. The audiovisual encoder includes a temporal convolutional neural network and a skip recurrent neural network (SkipRNN).

[0139] A preset audio-visual fusion algorithm is used to process and fuse the second audio coding features and the second video coding features through a cross-modal attention mechanism to generate the final fused coding features.

[0140] The final fused encoded features are input into the audiovisual decoder for decoding processing to generate the final fused decoded features.

[0141] The Softmax function is used to classify the final fused decoded features and output the final character sequence corresponding to the maximum probability.

[0142] The preset audiovisual speech recognition model in this embodiment has an overall encoder-decoder structure. Its audio feature input is an enhanced spectrogram after visual perception enhancement, which is close to the pure audio spectrogram. The video input is the video feature. The entire audiovisual encoder structure is as follows: Figure 8 As shown in the figure, X v1 Refers to the video features of the input, X a1 The input audio features are enhanced spectrograms. Mainstream models use RNN structures to represent audio and video modal features in a higher dimension. To better capture the sequence information contained in speech and video modal features while possessing the characteristics of CNNs, this embodiment uses a Temporal Convolutional Network (TCN) instead of CNN. TCN not only has the excellent parallel processing capabilities of CNNs, but also has a flexible receptive field, which is determined by the number of hidden layers, the size of the convolutional kernel, and the dilation coefficient. Moreover, since the underlying structure of TCN is CNN, it does not suffer from gradient explosion and gradient vanishing problems.

[0143] Since this embodiment is based on two models—a preset audiovisual speech enhancement model and a preset audiovisual speech recognition model—the overall model is larger in structure and model parameters than the current audiovisual speech recognition model. Therefore, without reducing model performance, a Skip Recurrent Neural Network (SkipRNN) is creatively adopted. Compared to ordinary RNNs, the binary state update gate in its network structure improves the problems of slow inference speed, vanishing gradients, and difficulty in capturing long-term dependencies. The RNN structure adopts a GRU (Gate Recurrent Unit) structure, which improves the problems of slow inference speed, vanishing gradients, and difficulty in capturing long-term dependencies compared to ordinary RNNs. Therefore, this embodiment mainly uses TCN and GRU as the constituent units of the audiovisual encoder.

[0144] Optionally, in this embodiment, a preset audio-visual fusion algorithm is used to perform cross-modal attention mechanism processing and fusion processing on the second audio coding features and the second video coding features to generate the final fused coding features, including:

[0145] A preset audio-visual fusion algorithm is used to project the second video coding feature onto the second audio coding feature to generate a second audio coding feature that contains the second video coding feature.

[0146] A preset audio-visual fusion algorithm is used to project the second audio coding feature onto the second video coding feature to generate a second video coding feature that contains the second audio coding feature.

[0147] The second audio coding feature containing the second video coding feature and the second video coding feature containing the second audio coding feature are fused together to generate the final fused coding feature.

[0148] Audio and video multimodal information generally possesses high-dimensional properties. Mapping the high-dimensional features of audio and video modalities to a low-dimensional, unified, shared subspace and fusing these features can promote the mutual complementarity and sharing of modal information. Based on this, this embodiment proposes a pre-defined audiovisual fusion algorithm based on two cross-modal attention operations. The first operation projects video modal information into audio modal information, making the audio modal space contain video modal information. The second operation projects audio modal information into video modal information, making the video modal space contain audio modal information. Finally, the audio modal containing video modal information and the video modal containing audio modal information are fused to generate the final fused encoded features.

[0149] In the first use of the cross-modal attention mechanism, the video modality is represented as Query and denoted as Q. v The audio modality is represented as Key and Value, denoted as K. a and V a Then Q v K a and V a The definition is as follows:

[0150]

[0151]

[0152]

[0153] in, and It is a weight vector, where d represents the feature dimension. A higher representation of the audio information output by the audio encoder. This represents a higher-level representation of the video information output by the video encoder, where L represents the sequence length. A relevance score is calculated between the query and the key. A high relevance score indicates that clean audio dominates the audio information; a low score suggests that noise overshadows the original information. This can be represented as follows:

[0154]

[0155] Y a =score a V a

[0156] Here, "score" represents the relevance score of audio and video modal information. This indicates normalization. The obtained correlation score is then compared with V. a Multiply to get At this time Y aAlthough it represents audio modal information, it also contains features representing video modal information, which is equivalent to the video information performing noise reduction processing on the noisy audio information. a This also represents the final impact of the video modality on the audio modality information under different noise conditions. The second cross-modal attention is similar to the first cross-modal attention and can be expressed by the following formula:

[0157]

[0158]

[0159]

[0160]

[0161] Y v =score v V v

[0162] In this case, only the subscripts of the parameters have changed, and their meanings are similar to those of the first cross-modal attention mentioned above. According to the formula above, it is similar to the first use of cross-modal attention, except that the modalities represented by Query and Key have been swapped, and Y... a Similarly, the Y obtained at this time v The video modal space already contains audio modal information. (Y) v and Y a The fused features obtained by feature space fusion are fed into the decoder as context vectors. At this time, the fused features include enhanced audio and video modal shared features as well as audio and video dissimilar features. After decoding and Softmax operation, the final character sequence is obtained.

[0163] The audiovisual speech recognition method of this embodiment has the following effects:

[0164] Better video feature extraction results are achieved by using a pseudo-3D neural network as the video feature extraction network, which performs better in extracting spatial and temporal features of videos compared to using CNN alone.

[0165] For better fusion results, audio and video modal information is fused more fully. When fusioning audio and video, two cross-modal attention operations are used to map the high-dimensional features of audio and video modalities to a low-dimensional unified shared subspace for feature fusion, so as to achieve better fusion results.

[0166] Higher recognition accuracy is achieved by using audiovisual speech enhancement methods, which utilize visual information to enhance the clean audio in noisy audio information. In subsequent audiovisual fusion algorithms, the clean audio information and the corresponding video information are better integrated, resulting in higher recognition accuracy.

[0167] Figure 9 The schematic diagram of the audiovisual speech recognition device provided in this application is as follows: Figure 9 As shown, in this embodiment, the audiovisual speech recognition device 300 can be installed in an electronic device, and the audiovisual speech recognition device 300 includes:

[0168] The acquisition module 301 is used to acquire the video features to be identified and the corresponding noisy spectrogram. The video features are extracted based on the user's lip image. The noisy spectrogram is generated based on the user's audio data.

[0169] The noise reduction module 302 is used to input video features and noisy spectrograms into a preset audiovisual speech enhancement model for noise reduction processing and output the corresponding enhanced spectrogram.

[0170] The recognition module 303 is used to input the enhanced spectrogram and video features into a preset audiovisual speech recognition model for speech recognition and output the corresponding final character sequence.

[0171] The audiovisual speech recognition device provided in this embodiment can perform... Figure 2 The technical solution of the method embodiment shown has the same implementation principle and technical effect as... Figure 2 The methods and embodiments shown are similar and will not be described in detail here.

[0172] The audiovisual speech recognition device provided in this application is a further refinement of the audiovisual speech recognition device provided in the previous embodiment. The audiovisual speech recognition device 300 includes:

[0173] Optionally, in this embodiment, the acquisition module 301 is specifically used for:

[0174] The system acquires the audio and video data to be identified. The audio data is preprocessed to generate a noisy spectrogram. Preprocessing includes pre-emphasis, frame segmentation, windowing, short-time Fourier transform, and Mel-filtering. A pre-defined face recognition tool is used to perform face recognition on the video data, generating corresponding face images. These face images are then input into a pre-defined pseudo-3D residual convolutional network to extract video features.

[0175] Optionally, in this embodiment, the preset audiovisual speech enhancement model includes: a video encoder, an audio encoder, a preset fusion noise reduction module, and an audio decoder.

[0176] The noise reduction module 302 is specifically used for:

[0177] Video features are input into a video encoder for encoding to generate corresponding first video encoded features. A noisy spectrogram is input into an audio encoder for encoding to generate corresponding first audio encoded features. The first video encoded features and the first audio encoded features are input into a preset fusion and denoising module for fusion and denoising to generate denoised enhanced encoded features. The enhanced encoded features are input into an audio decoder for decoding to generate an enhanced spectrogram.

[0178] Optionally, in this embodiment, the preset fusion noise reduction module includes: a soft threshold algorithm and a threshold block.

[0179] When the noise reduction module 302 inputs the first video coding features and the first audio coding features into the preset fusion noise reduction module for fusion noise reduction to generate enhanced coding features after noise reduction, it is specifically used for:

[0180] The first video coding features and the first audio coding features are concatenated to generate intermediate coding features. These intermediate coding features are then input into a threshold block for convolutional pooling and nonlinear transformation processing to generate a threshold reference value. A soft thresholding algorithm is then used to denoise the audio coding features based on the threshold reference value, generating denoised enhanced coding features.

[0181] Optionally, in this embodiment, the preset audiovisual speech recognition model includes an audiovisual encoder, a preset audiovisual fusion algorithm, an audiovisual decoder, and a Softmax function.

[0182] The identification module 303 is specifically used for:

[0183] The enhanced spectrogram and video features are input into the audiovisual encoder for encoding, generating corresponding second audio and second video encoded features. The audiovisual encoder includes a temporal convolutional neural network and a skip recurrent neural network (SkipRNN). A pre-defined audiovisual fusion algorithm is used to process and fuse the second audio and second video encoded features through a cross-modal attention mechanism, generating the final fused encoded features. The final fused encoded features are then input into the audiovisual decoder for decoding, generating the final fused decoded features. The Softmax function is used to classify the final fused decoded features, outputting the final character sequence corresponding to the highest probability.

[0184] Optionally, in this embodiment, when the recognition module 303 performs cross-modal attention mechanism processing and fusion processing on the second audio coding feature and the second video coding feature using a preset audio-visual fusion algorithm to generate the final fused coding feature, it is specifically used for:

[0185] A preset audio-visual fusion algorithm is used to project the second video coding feature onto the second audio coding feature, generating a second audio coding feature that includes the second video coding feature. The preset audio-visual fusion algorithm is then used to project the second audio coding feature onto the second video coding feature, generating a second video coding feature that includes the second audio coding feature. The second audio coding feature containing the second video coding feature and the second video coding feature containing the second audio coding feature are then fused to generate the final fused coding feature.

[0186] The audiovisual speech recognition device provided in this embodiment can perform... Figures 2-8 The technical solution of the method embodiment shown has the same implementation principle and technical effect as... Figures 2-8 The methods and embodiments shown are similar and will not be described in detail here.

[0187] According to embodiments of this application, this application also provides an electronic device, a computer-readable storage medium, and a computer program product.

[0188] like Figure 10 As shown, Figure 10 This is a schematic diagram of the electronic device provided in this application. The electronic device is intended for various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, blade servers, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described and / or claimed herein.

[0189] like Figure 10 As shown, the electronic device includes a processor 401 and a memory 402. The various components are interconnected via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processor can process instructions executed within the electronic device.

[0190] The memory 402 is the non-transitory computer-readable storage medium provided in this application. The memory stores instructions executable by at least one processor to cause at least one processor to perform the audiovisual speech recognition method provided in this application. The non-transitory computer-readable storage medium of this application stores computer instructions for causing a computer to perform the audiovisual speech recognition method provided in this application.

[0191] Memory 402, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the audiovisual speech recognition method in the embodiments of this application (e.g., attached...). Figure 9The acquisition module 301, noise reduction module 302, and recognition module 303 are shown. The processor 401 executes various functional applications and data processing of the electronic device by running non-transient software programs, instructions, and modules stored in the memory 402, thereby realizing the audiovisual speech recognition method in the above method embodiments.

[0192] In addition, this embodiment also provides a computer product, which, when the instructions in the computer product are executed by the processor of an electronic device, enables the electronic device to perform the audiovisual speech recognition method of the above embodiment.

[0193] Other embodiments of the present application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the embodiments of this application that follow the general principles of the embodiments of this application and include common knowledge or customary techniques in the art not disclosed in the embodiments of this application.

[0194] It should be understood that the embodiments of this application are not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from their scope. The scope of the embodiments of this application is limited only by the appended claims.

Claims

1. An audiovisual speech recognition method, characterized by, include: Obtain the video features to be identified and the corresponding noisy spectrogram; the video features are features extracted based on the user's lip image; The noisy spectrogram is generated based on the user's audio data; The video features and the noisy spectrogram are input into a preset audiovisual speech enhancement model for noise reduction, and the corresponding enhanced spectrogram is output. The enhanced spectrogram and the video features are input into a preset audiovisual speech recognition model for speech recognition, and the corresponding final character sequence is output. The preset audiovisual speech enhancement model includes: a video encoder, an audio encoder, a preset fusion denoising module, and an audio decoder; the preset fusion denoising module includes: a soft thresholding algorithm and a threshold block; the step of inputting the video features and the noisy spectrogram into the preset audiovisual speech enhancement model for denoising processing and outputting the corresponding enhanced spectrogram includes: The video features are input into the video encoder for encoding processing to generate corresponding first video encoded features; The noisy spectrogram is input into the audio encoder for encoding processing to generate a corresponding first audio encoding feature; The first video coding feature and the first audio coding feature are concatenated to generate an intermediate coding feature; the intermediate coding feature is input into the threshold block for convolutional pooling and nonlinear transformation processing to generate a threshold reference value; the soft thresholding algorithm is used to denoise the first audio coding feature based on the threshold reference value to generate a denoised enhanced coding feature; The enhanced coding features are input into the audio decoder for decoding to generate the enhanced spectrogram.

2. The method of claim 1, wherein, The process of obtaining the video features to be identified and the corresponding noisy spectrogram includes: Acquire the audio and video data to be identified; The audio data is preprocessed to generate a corresponding noisy spectrogram; the preprocessing includes pre-emphasis processing, frame segmentation, windowing, short-time Fourier transform, and Mel filtering. The video data is subjected to facial recognition using a preset facial recognition tool to generate corresponding facial images; The face image is input into a preset pseudo-3D residual convolutional network to extract the video features.

3. The method of claim 1, wherein, The preset audiovisual speech recognition model includes an audiovisual encoder, a preset audiovisual fusion algorithm, an audiovisual decoder, and a Softmax function; The step of inputting the enhanced spectrogram and the video features into a preset audiovisual speech recognition model for speech recognition and outputting the corresponding final character sequence includes: The enhanced spectrogram and the video features are input into the audiovisual encoder for encoding processing to generate corresponding second audio encoding features and second video encoding features; the audiovisual encoder includes a temporal convolutional neural network and a skip recurrent neural network SkipRNN; The preset audio-visual fusion algorithm is used to perform cross-modal attention mechanism processing and fusion processing on the second audio coding feature and the second video coding feature to generate the final fused coding feature; The final fused coding features are input into the audiovisual decoder for decoding processing to generate the final fused decoding features; The Softmax function is used to classify the final fused decoding features and output the final character sequence corresponding to the maximum probability.

4. The method of claim 3, wherein, The step of using the preset audio-visual fusion algorithm to perform cross-modal attention mechanism processing and fusion processing on the second audio coding features and the second video coding features to generate the final fused coding features includes: The preset audio-visual fusion algorithm is used to project the second video coding feature onto the second audio coding feature to generate a second audio coding feature that includes the second video coding feature; The preset audio-visual fusion algorithm is used to project the second audio coding feature onto the second video coding feature to generate a second video coding feature containing the second audio coding feature; The second audio coding feature containing the second video coding feature and the second video coding feature containing the second audio coding feature are fused together to generate the final fused coding feature.

5. An audiovisual speech recognition apparatus, characterized in that include: The acquisition module is used to acquire the video features to be identified and the corresponding noisy spectrogram; the video features are features extracted based on the user's lip image; The noisy spectrogram is generated based on the user's audio data; The noise reduction module is used to input the video features and the noisy spectrogram into a preset audiovisual speech enhancement model for noise reduction processing, and output the corresponding enhanced spectrogram. The recognition module is used to input the enhanced spectrogram and the video features into a preset audiovisual speech recognition model for speech recognition and output the corresponding final character sequence; The preset audiovisual speech enhancement model includes: a video encoder, an audio encoder, a preset fusion noise reduction module, and an audio decoder; the noise reduction module is specifically used for: inputting the video features into the video encoder for encoding processing to generate corresponding first video encoded features; inputting the noisy spectrogram into the audio encoder for encoding processing to generate corresponding first audio encoded features; inputting the first video encoded features and the first audio encoded features into the preset fusion noise reduction module for fusion noise reduction to generate noise-reduced enhanced encoded features; and inputting the enhanced encoded features into the audio decoder for decoding processing to generate the enhanced spectrogram; The preset fusion noise reduction module includes a soft thresholding algorithm and a threshold block. When the noise reduction module inputs the first video coding feature and the first audio coding feature into the preset fusion noise reduction module for fusion noise reduction to generate enhanced denoised coding features, it specifically performs the following steps: concatenating the first video coding feature and the first audio coding feature to generate intermediate coding features; inputting the intermediate coding features into the threshold block for convolutional pooling and nonlinear transformation processing to generate a threshold reference value; and using the soft thresholding algorithm to perform noise reduction processing on the first audio coding feature based on the threshold reference value to generate enhanced denoised coding features.

6. An electronic device, comprising: include: Memory and processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the audiovisual speech recognition method as described in any one of claims 1 to 4.

7. A computer readable storage medium characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the audiovisual speech recognition method as described in any one of claims 1 to 4.

8. A computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the audiovisual speech recognition method as described in any one of claims 1 to 4.

Images