Audio and video dual-mode speech recognition method based on attention mechanism

Abstract

The invention discloses an audio and video dual-mode speech recognition method based on an attention mechanism. The audio and video dual-mode speech recognition method comprises the following steps: S1, acquiring audio and video data; s2, extracting video features; s3, audio feature extraction; s4, after the audio features and the video features are coded, inputting the audio features and the video features into a multi-modal attention layer for fusion, and realizing alignment correction of the audio features and the video features through an attention mechanism; s5, decoding to obtain audio recognition text data; s6, decoding to obtain video identification text data; s7, acquiring a deep semantic relationship between the audio recognition text data and the video recognition text data based on a multi-head attention mechanism, and fusing the audio recognition text data and the video recognition text data based on semantic information; and S8, outputting to obtain a speech recognition result. Compared with the prior art, the problems that audio and video information processing is disordered, the recognition matching degree is low and the accuracy is not high in the voice recognition process are solved.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to an audio and video dual-modal speech recognition method based on an attention mechanism. Background Art

[0002] The speech recognition model combined with neural network is widely used in life. However, in noisy environment, multi-person background, noisy or missing sound, the recognition ability of single-modal speech recognition cannot meet the demand, and other information is needed to improve it, such as the addition of video information. The performance of audio speech recognition drops significantly in the case of severe noise pollution, and there is uncertainty in video speech recognition due to homophones, which can be compensated in dual mode.

[0003] There are also difficulties in audio and video dual-modal speech recognition. On the one hand, the characterization of visual information is relatively diverse, and it is difficult to determine which visual features help improve speech recognition performance. On the other hand, the audio and video rates do not match and the information lengths are different, making it difficult to effectively fuse the information. This makes it easy for audio and video information processing to be confused, recognition matching to be low, and accuracy to be low during speech recognition. Summary of the invention

[0004] The purpose of the present invention is to provide an audio and video dual-modal speech recognition method based on an attention mechanism to solve the problems of confusion in audio and video information processing, low recognition matching degree and low accuracy that are prone to occur during speech recognition.

[0005] In order to achieve the above object, the present invention adopts the following technical solution: an audio and video dual-modal speech recognition method based on an attention mechanism, comprising the following steps:

[0006] S1. Acquire audio data and video data collected by the collection terminal;

[0007] S2, performing denoising preprocessing on the video image in the video data, extracting video features using a residual network architecture, and obtaining a video feature vector;

[0008] S3, extracting audio features from the audio data through a neural network to obtain an audio feature vector;

[0009] S4, after encoding the audio features and the video features, input them into a multimodal attention layer for fusion, and realize the alignment correction of the audio features and the video features through the attention mechanism, obtain the corrected audio features, and realize the early fusion of the video features and the audio features;

[0010] S5, decoding the audio feature vector through an audio decoder to obtain audio recognition text data;

[0011] S6, decoding the video feature vector through a video decoder to obtain video recognition text data;

[0012] S7, obtaining the deep semantic relationship between the audio recognition text data and the video recognition text data based on the multi-head attention mechanism, fusing the audio recognition text data and the video recognition text data based on the semantic information, and obtaining the fused recognition text data;

[0013] S8. Output the speech recognition result.

[0014] As a further description of the above technical solution:

[0015] In step S2, the image is pre-processed for denoising by using a frequency domain filtering formula.

[0016] As a further description of the above technical solution:

[0017] In step S2, for the video image obtained from the GRID database, a weight matrix is ​​constructed according to the correlation attributes between the image pixels, and a graph Fourier transform is performed in combination with the graph Laplace matrix, and a frequency domain filtering formula is obtained according to the signal prior knowledge.

[0018] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0019] 1. In the present invention, the use of the attention mechanism in the process of audio and video speech recognition effectively solves the information fusion problem caused by the mismatch of audio and video rates and the different information lengths. The two auxiliary functions of video information in the early and late stages of features effectively improve the recognition results of the model under noise and increase the robustness of the model to noise.

[0020] 2. In the present invention, a high recognition rate and a low error rate are achieved in complex environmental interference, especially in a background of multiple people, mixed or missing sounds, which provides a good recognition basis for subsequent audio recognition and quickly and accurately recognizes the video information to be extracted. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 This is a flowchart of an audio and video dual-modal speech recognition method based on the attention mechanism. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0024] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Example 1

[0026] See also Figure 1 The present invention provides a technical solution: an audio and video dual-modal speech recognition method based on an attention mechanism, comprising the following steps:

[0027] S1. Acquire audio data and video data collected by the collection terminal;

[0028] S2. Perform denoising preprocessing on the video image in the video data, use the residual network architecture to extract video features, and obtain video feature vectors. Compared with Wiener filtering, Gaussian filtering and basic frequency domain filtering methods, the pixel spacing of the image filtering is smaller;

[0029] S3, extracting audio features from the audio data through a neural network to obtain an audio feature vector;

[0030] S4, after encoding the audio features and the video features, input them into a multimodal attention layer for fusion, and realize the alignment correction of the audio features and the video features through the attention mechanism, obtain the corrected audio features, and realize the early fusion of the video features and the audio features;

[0031] S5, decoding the audio feature vector through an audio decoder to obtain audio recognition text data;

[0032] S6, decoding the video feature vector through a video decoder to obtain video recognition text data;

[0033] S7, based on the multi-head attention mechanism, obtain the deep semantic relationship between the audio recognition text data and the video recognition text data, and fuse the audio recognition text data and the video recognition text data based on the semantic information, that is, select the audio recognition text data and the video recognition text data with a higher probability of corresponding text, and obtain the fused recognition text data to achieve late fusion;

[0034] S8. Output the speech recognition result.

[0035] In step S2, the image is pre-processed for denoising by using a frequency domain filtering formula to improve the quality of the video image and prepare for extracting clear and discernible video features.

[0036] In step S2, for the video image obtained from the GRID database, a weight matrix is ​​constructed according to the correlation attributes between the image pixels, and the graph Fourier transform is performed in combination with the graph Laplacian matrix, and the frequency domain filtering formula is obtained according to the signal prior knowledge. When optimizing the frequency domain filtering formula, the GRID video image is used and corrupted with additive Gaussian noise with a mean of 0 and a standard deviation of 0.1 to obtain a noisy signal. The color image has three channels of R, G, and B, so they are processed separately, and each pixel is connected to its horizontal, vertical, and diagonal elements to construct the underlying image structure, so that the proposed filtering formula can be denoised to improve the image quality.

[0037] In step S6, the video feature vector is decoded through a multi-layer Transformer deep network model to obtain video recognition text data, thereby improving the decoding recognition accuracy.

[0038] In step S3, before audio feature extraction, the audio data is preprocessed, and the preprocessing includes data cleaning, format conversion and storage.

[0039] Working principle: In the process of audio and video speech recognition, the use of the attention mechanism effectively solves the information fusion problem caused by the mismatch of audio and video rates and the different information lengths. The two auxiliary functions of video information in the early and late stages of features effectively improve the recognition results of the model under noise and increase the robustness of the model to noise. This speech recognition method achieves high recognition rate and low error rate in complex environmental interference, especially in multi-person backgrounds, mixed or missing sounds, which provides a good recognition basis for subsequent audio recognition and quickly and accurately recognizes the video information to be extracted.

[0040] Example 2

[0041] This embodiment further makes the following technical solution as an improvement on the above embodiment: in step S4, when the audio features and video features are fused by the attention mechanism method, when the signal-to-noise ratio of the estimated audio data is lower than the set threshold D1, the weight of the video information fusion is adjusted according to the signal-to-noise ratio of the audio data. Experimental data analysis found that excessive involvement of video information will also reduce the speech recognition performance in a clean situation. Therefore, the signal-to-noise ratio of the audio data is estimated based on the signal-to-noise ratio estimation method. When it is lower than the set threshold D1, the weight of the video information fusion is adjusted in the early fusion process of the dual-modal speech recognition, so as to effectively cope with the recognition tasks in different environments and reduce the interference of video information on audio information recognition.

[0042] Example 3

[0043] This embodiment further provides the following improved technical solutions based on the above embodiment: in step S5, when the signal-to-noise ratio of the audio data is lower than the set threshold D2, and the threshold D2 is lower than D1, the audio recognition text data is decoded and then directly proceeded to step S8 to output the speech recognition result. When the sound signal is clean enough and the signal-to-noise ratio is low enough, the video information is directly cancelled in the later stage of speech recognition to avoid interfering with the speech recognition result.

[0044] Experiments were conducted on the GRID database to explore the impact of visual information on speech recognition under different noise pollution conditions. Compared with the single-modality model, the performance of the dual-modality speech recognition model has been improved, not only achieving better recognition results under severe noise pollution, but also achieving good results under relatively clean sound signals.

[0045] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An audio and video dual-modal speech recognition method based on attention mechanism, characterized in that: The following steps are involved: S1. Acquire audio data and video data collected by the collection terminal; S2, performing denoising preprocessing on the video image in the video data, extracting video features using a residual network architecture, and obtaining a video feature vector; S3, extracting audio features from the audio data through a neural network to obtain an audio feature vector; S4, after encoding the audio features and the video features, input them into a multimodal attention layer for fusion, and realize the alignment correction of the audio features and the video features through the attention mechanism, obtain the corrected audio features, and realize the early fusion of the video features and the audio features; S5, decoding the audio feature vector through an audio decoder to obtain audio recognition text data; S6, decoding the video feature vector through a video decoder to obtain video recognition text data; S7, obtaining the deep semantic relationship between the audio recognition text data and the video recognition text data based on the multi-head attention mechanism, fusing the audio recognition text data and the video recognition text data based on the semantic information, and obtaining the fused recognition text data; S8. Output the speech recognition result.

2. According to the method of claim 1, the audio and video dual-modal speech recognition method based on the attention mechanism is characterized in that: In step S2, the image is pre-processed for denoising by using a frequency domain filtering formula.

3. The method for audio and video dual-modal speech recognition based on attention mechanism according to claim 2, characterized in that: In step S2, for the video image obtained from the GRID database, a weight matrix is ​​constructed according to the correlation attributes between the image pixels, and a graph Fourier transform is performed in combination with the graph Laplace matrix, and a frequency domain filtering formula is obtained according to the signal prior knowledge.

4. The method for audio and video dual-modal speech recognition based on attention mechanism according to claim 1, characterized in that: In step S4, when the audio features and video features are fused through the attention mechanism method, when it is estimated that the signal-to-noise ratio of the audio data is lower than the set threshold D1, the weight of the video information fusion is adjusted according to the signal-to-noise ratio of the audio data.

5. The method for audio and video dual-modal speech recognition based on attention mechanism according to claim 4 is characterized in that: In step S5, when the signal-to-noise ratio of the audio data is lower than the set threshold D2, and the threshold D2 is smaller than D1, the audio recognition text data is decoded and then directly processed to step S8 to output the speech recognition result.

6. The method for audio and video dual-modal speech recognition based on attention mechanism according to claim 1, characterized in that: In step S6, the video feature vector is decoded by a multi-layer Transformer deep network model to obtain video recognition text data.

7. The method for audio and video dual-modal speech recognition based on attention mechanism according to claim 1, characterized in that: In step S3, the audio data is preprocessed before audio feature extraction, and the preprocessing includes data cleaning, format conversion and storage.

Images