Voice interaction recognition enhancement method and device and storage medium

Abstract

The invention proposes a voice interaction recognition enhancement method, and the method comprises the steps: collecting a video of voice interaction of a speaker facing a camera, dividing the video into N segments of to-be-recognized voices and N frames of to-be-recognized images, and forming to-be-recognized data; inputting the to-be-recognized data into a preset voice interaction recognition enhancement model which comprises a lip language feature extraction network, a voice feature extraction network, a time feature extraction network and an activation network, the time feature extraction network is used for adding time sequence information to a lip language feature matrix extracted by the lip language feature extraction network and a voice feature matrix extracted by the voice feature extraction network, and the activation network is used for simulating an activation-suppression mechanism of visual information in a biological brain on an auditory neural circuit, so that interaction between a visual mode and an auditory mode is realized. And thus, a speech recognition result is obtained. According to the invention, visual and auditory dual-mode speech recognition is realized, lip language and speech are accurately recognized in a noisy environment, and the response capability and accuracy meet requirements.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech interaction recognition, and in particular relates to a speech interaction recognition enhancement method, device and storage medium. Background Art

[0002] In public noisy environments such as airports and train stations, the accuracy and robustness of speech recognition are crucial. Traditional single-modal speech recognition technology is difficult to work effectively in noisy environments, affecting the efficiency of human-computer interaction. Improving speech recognition capabilities in noisy environments can significantly improve the user experience and application effects of public service robots and mobile terminals.

[0003] Currently, neuroscience research shows that the interaction between visual and auditory information in the biological brain plays an important role in speech recognition. However, most existing research remains at the level of phenomenon observation, and there is insufficient understanding of the specific fusion mechanism. If this interaction mechanism can be implemented in neural networks, it will help improve speech recognition performance.

[0004] Visual and auditory dual-modal speech recognition adds visual information such as lip reading. The speech modality is very accurate in a quiet environment, but as the noise increases, the recognition accuracy drops significantly; although the recognition accuracy of the lip reading modality is not affected by the signal-to-noise ratio, since lip reading to text is a one-to-many mapping, lip reading feature extraction is not well studied, and the recognition accuracy of the lip reading modality in a quiet or high signal-to-noise ratio environment is not as good as that of the speech modality. Therefore, in an environment with a variable signal-to-noise ratio, how to combine the accuracy of the speech modality and the robustness of the lip reading modality to achieve accurate and robust speech recognition is a challenging research.

[0005] Most existing speech recognition models are based on the Transformer architecture, such as OpenAI's Whisper model, which has a large number of parameters and requires extremely high memory and computing power on mobile terminals. This model runs slowly on mobile devices, occupies a large amount of memory, and cannot meet low latency requirements. Based on research on biological brains, the structure of biological brains is similar to a parallel recurrent neural network (RNN). The RNN architecture is closer to the actual brain structure and is suitable for achieving efficient speech recognition on mobile terminals with limited resources. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and to propose a method, device and storage medium for enhancing speech interaction recognition in a noisy environment. Based on the study of the interaction of visual and auditory information in the biological brain, the present invention constructs a small speech recognition model based on RNN to achieve low-latency, real-time speech recognition, which is suitable for the application of mobile terminals in public noisy environments, especially in airports and railway stations, to enhance the speech interaction recognition between robots and pedestrians. Before the interaction of visual and auditory information, it is first necessary to extract features from these two modalities. With reference to the progress of single-modal feature extraction in previous studies, a lip reading recognition network and a speech recognition network were designed. Then, a time feature extraction network was designed, and then an activation network was designed to realize the interaction of the two modalities. Finally, the result is input into the classifier to obtain the final recognition result. In order to achieve enhanced speech interaction recognition accuracy in a noisy environment.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A first aspect of the present invention provides a method for enhancing speech interaction recognition, comprising:

[0009] Collecting a video to be recognized, in which a speaker faces a camera for voice interaction, splitting the video to be recognized into N segments of voice to be recognized and N frames of images to be recognized, and forming data to be recognized;

[0010] The data to be recognized is input into a preset speech interaction recognition enhancement model, which includes a lip reading feature extraction network, a speech feature extraction network, a time feature extraction network and an activation network. The lip reading feature extraction network is used to extract the lip reading feature matrix of N frames of images to be recognized, the speech feature extraction network is used to extract the speech feature matrix of N segments of speech to be recognized, the time feature extraction network is used to obtain a lip reading feature matrix containing timing information and a speech feature matrix containing timing information based on the lip reading feature matrix and the speech feature matrix, and the activation network is used to simulate the activation-inhibition mechanism of visual information on auditory neural circuits in the biological brain, so as to realize the interaction of the two modes of the lip reading feature matrix containing timing information and the speech feature matrix containing timing information, thereby obtaining a speech recognition result.

[0011] In some embodiments, the N segments of speech to be recognized and the N frames of images to be recognized are randomly ordered.

[0012] In some embodiments, the lip reading feature extraction network adopts a 3D convolutional neural network, including a three-dimensional convolution layer, a maximum pooling layer, a batch normalization layer and an activation function layer connected in sequence.

[0013] In some embodiments, the speech feature extraction network adopts a 2D convolutional neural network, including a Mel-frequency cepstral coefficient module, a two-dimensional convolution layer, a maximum pooling layer, a batch normalization layer and an activation function layer connected in sequence.

[0014] In some embodiments, the temporal feature extraction network uses two bidirectional gated recurrent units, one of the bidirectional gated recurrent units is used to process forward timing information when forward splicing feature matrices input at different moments, and the other gated recurrent unit is used to process reverse timing information when reverse splicing feature matrices input at different moments, thereby achieving the capture and utilization of dynamically changing bidirectional contextual information and enhancing the feature representation capability.

[0015] In some embodiments, the activation network includes an activation module, a fusion layer, a second fully connected layer and a classifier connected in sequence, and the activation module is composed of an attention layer and a first fully connected layer; the dimension of the lip reading feature matrix containing timing information is N×28, and the activation module first uses the attention mechanism to calculate the attention weights on 28 lip reading categories, and the 28 lip reading categories are composed of lip shapes corresponding to 26 English letters, 1 space and 1 placeholder, and then uses the first fully connected layer to convert the attention weights obtained by the attention layer into probability distributions of each category, thereby obtaining an activation matrix; the fusion layer is used to multiply the speech feature matrix containing timing information with the elements at corresponding positions in the activation matrix to obtain fusion features; the fusion features are linearly transformed by the second fully connected layer and classified by the classifier to obtain the speech recognition result.

[0016] In some embodiments, the preset voice interaction recognition enhancement model is obtained by the following steps:

[0017] 1) Construct training set:

[0018] Obtain M videos of a speaker, split each video into N audio segments of equal length and N image frames of equal size, use the N audio segments as a group of audio data, perform ROI extraction processing on the N image frames respectively, extract the area near the lips in the image, obtain N frames of ROI image data, and use the N frames of ROI image data as a group of image data; correspond the group of audio data and the group of image data obtained by splitting each video, respectively as a speech recognition training sample and a lip reading recognition training sample, so as to construct a training set;

[0019] 2) Build an enhanced voice interaction recognition model:

[0020] The constructed speech interaction recognition enhancement model includes a lip reading feature extraction network, a speech feature extraction network, a time feature extraction network and an activation network:

[0021] The lip reading feature extraction network adopts a 3D convolutional neural network, takes the lip reading recognition training sample as input, extracts image features, and outputs a lip reading feature matrix V;

[0022] The speech feature extraction network adopts a 2D convolutional neural network, takes the speech recognition training sample as input, extracts speech features, and outputs a speech feature matrix A;

[0023] The temporal feature extraction network adopts a bidirectional gated recurrent unit, takes the lip reading feature matrix V and the speech feature matrix A as input, and outputs a lip reading feature matrix T containing temporal information. V and the speech feature matrix T containing temporal information A ;

[0024] The activation network includes an activation module, a fusion layer, a second fully connected layer and a classifier connected in sequence. The activation network is based on the lip reading feature matrix T containing time sequence information. V and the speech feature matrix T containing the timing information A As input, the lip reading feature matrix T containing time sequence information V Through the attention mechanism of the activation module and the first fully connected layer, the activation matrix T is obtained M , the speech feature matrix T containing time sequence information is transformed through the fusion layer A With the activation matrix T A The elements at corresponding positions are multiplied to obtain a fusion matrix, and the fusion matrix passes through the second fully connected layer and the classifier in sequence to obtain a speech recognition result P.

[0025] 3) Training the voice interaction recognition enhancement model:

[0026] A loss function is set, and the speech interaction recognition enhancement model constructed in step 2) is trained using the training set constructed in step 1) until the loss function converges, thereby obtaining a trained speech interaction recognition enhancement model.

[0027] In some embodiments, in step 1), each audio segment in the set of audio data and the N frames of ROI image data in the set of image data are randomly sorted;

[0028] In step 3), the loss function adopts the CTC loss function.

[0029] The second aspect of the present invention provides a speech interaction recognition enhancement device based on the method described in any embodiment of the first aspect of the present invention, comprising:

[0030] A data acquisition module is used to collect a video to be identified, in which a speaker faces a camera for voice interaction, split the video to be identified into N segments of voice to be identified and N frames of images to be identified, and form data to be identified;

[0031] The recognition module contains a preset speech interaction recognition enhancement model, which includes a lip reading feature extraction network, a speech feature extraction network, a time feature extraction network and an activation network. The lip reading feature extraction network is used to extract the lip reading feature matrix of N frames of images to be recognized, the speech feature extraction network is used to extract the speech feature matrix of N segments of speech to be recognized, the time feature extraction network is used to obtain a lip reading feature matrix containing timing information and a speech feature matrix containing timing information according to the lip reading feature matrix and the speech feature matrix, and the activation network is used to simulate the activation-inhibition mechanism of visual information on auditory neural circuits in the biological brain, so as to realize the interaction of the two modes of the speech feature matrix containing lip timing information and the speech feature matrix containing lip timing information, thereby obtaining a speech recognition result.

[0032] A third aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method described in any embodiment of the first aspect of the present invention.

[0033] Compared with the prior art, the present invention has the following characteristics and beneficial effects:

[0034] (1) The present invention proposes a new modal interaction strategy. In the inter-modal information interaction stage, an activation network is used to multiply the elements of the speech feature matrix and the corresponding positions of the activation matrix. In this way, the model can accurately predict and recognize voice commands in noisy environments without relying on complex detectors, significantly improving the robustness and accuracy of speech recognition.

[0035] (2) The present invention combines convolutional neural networks with bidirectional gated recurrent neural networks, and uses a combined structure of convolutional neural networks (CNN) and bidirectional gated recurrent neural networks (Bi-GRU) in the feature extraction stage. This design not only controls the scale of the network, but also effectively improves the feature extraction capability of the model, enabling it to run on low-computing power devices and maintain a high recognition rate;

[0036] (3) The present invention realizes efficient single-frame image processing. Different from the multi-frame image processing method, the model of the present invention processes a single-frame image, avoiding the deviation introduced in the time series due to dynamic problems during the data acquisition process, thereby improving the accuracy and stability of the recognition results;

[0037] (4) The present invention has significant performance improvement. Compared with the existing excellent models, the present invention has a significant improvement in word error rate (WER) compared with the baseline model and robustness.

[0038] In summary, the present invention proposes a speech recognition enhancement method, which adopts a bottom-up modal interaction strategy and a CNN+Bi-GRU feature extraction structure, can efficiently process single-frame images, and achieve high robustness and high accuracy speech recognition in noisy environments. The present invention has verified its superior performance on multiple data sets, and is particularly suitable for enhancing speech interaction recognition between robots and pedestrians in noisy public environments such as airports and railway stations, meeting the accuracy and efficiency requirements of automated equipment for speech recognition technology, and is easy to integrate into various mobile terminals. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a structural block diagram of the voice interaction recognition enhancement method provided by an embodiment of the present invention.

[0040] Figure 2 It is a schematic diagram of ROI processing of video frames by the voice interaction recognition enhancement method provided in an embodiment of the present invention, wherein the left side is the video frame before processing, and the right side is the video frame after ROI processing.

[0041] Figure 3 It is a schematic diagram of the attention mechanism adopted by the speech interaction recognition enhancement method provided in an embodiment of the present invention.

[0042] Figure 4 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0044] On the contrary, the present application covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present application as defined by the claims. Further, in order to make the public have a better understanding of the present application, some specific details are described in detail in the detailed description of the present application below. Those skilled in the art can fully understand the present application without the description of these details.

[0045] A method for enhancing speech interaction recognition proposed in an embodiment of the first aspect of the present invention includes:

[0046] Collect a video to be recognized, in which a speaker faces a camera for voice interaction, split the video to be recognized into N segments of voice to be recognized and N frames of images to be recognized, and form data to be recognized;

[0047] The data to be recognized is input into a preset voice interaction recognition enhancement model, which includes a lip reading feature extraction network, a voice feature extraction network, a time feature extraction network and an activation network. The lip reading feature extraction network is used to extract the lip reading feature matrix V of N frames of images to be recognized. * , the speech feature extraction network is used to extract the speech feature matrix A of N segments of speech to be recognized * , the temporal feature extraction network is used to extract the lip reading feature matrix V * And the speech feature matrix A * Get the lip reading feature matrix T containing time sequence information V * and the speech feature matrix T containing temporal information A * The activation network is used to simulate the activation-inhibition mechanism of visual information on auditory neural circuits in the biological brain, and realize the lip reading feature matrix T V * and speech feature matrix T A * The interaction of the two modes results in the speech recognition result P * .

[0048] A method for enhancing speech interaction recognition provided by an embodiment of the first aspect of the present invention is shown in the overall flow chart. Figure 1 , which is divided into training phase and testing phase, including the following steps:

[0049] 1) Training phase

[0050] 11) Constructing training set

[0051] Obtain videos of different speakers, split them into audio data and multiple single-frame image data, use the audio data to build a speech recognition training set, and for each single-frame image data, extract the image data of the area near the speaker's lips to build a lip recognition training set.

[0052] In a specific embodiment of the present invention, a training set is constructed using the GRID dataset. GRID is a large audio-visual sentence corpus containing videos of multiple speakers, which is designed to support joint computation and behavioral research on speech perception. The GRID dataset download link is: https: / / spandh.dcs.shef.ac.uk / / gridcorpus / . The M videos in the GRID dataset are split into audio data and single-frame image data, and each video is split into N audio segments of equal length and N image frames of equal size. The N audio segments are randomly sorted as a group of audio data, and the N image frames are subjected to ROI (Region of Interest) extraction processing respectively, and the area near the lip in the image is extracted to obtain N frames of ROI image data, and the N frames of ROI image data are randomly sorted as a group of image data; a group of audio data and a group of image data obtained by splitting each video are made to correspond to each other. Since the data in each group of audio data and each group of image data are randomly arranged, a segment of audio data and a frame of image data in a corresponding group of audio data and a group of image data do not need to correspond one to one. A corresponding set of audio data and a set of image data are used as a speech recognition training sample and a lip reading recognition training sample to construct a training set, which contains M×N audio segments and M×N images.

[0053] In this embodiment, the duration of each video is 3 seconds. Each video is split into N = 90 audio segments of equal duration (each audio segment is 1 / 30 seconds long) and N = 90 image frames of equal size (i.e., the images are split according to 30 frames per second). The 90 audio data segments obtained by splitting each video are directly used as a speech recognition training sample. For the image data obtained by splitting each video, it contains the entire face of the speaker, such as Figure 2 As shown in the left figure, the truly effective lip reading information is mainly hidden in an area near the lips, such as Figure 2 As shown in the figure on the right, it is necessary to perform ROI extraction processing on each split image data, and only extract the area near the lips in the image, so that a large amount of invalid information in the video can be removed. 90 frames of ROI images are obtained for each video, and the 90 frames of ROI image data are used to construct lip recognition training samples. The training set is constructed using one-to-one corresponding speech recognition training samples and lip reading recognition training samples.

[0054] 12) Build a voice interaction recognition model

[0055] The constructed speech interaction recognition model includes a lip reading feature extraction network, a speech feature extraction network, a temporal feature extraction network, and an activation network. The lip reading feature extraction network uses a 3D convolutional neural network (3D CNN) to extract image features, takes the lip reading recognition training samples in the training set as input, extracts image features, and outputs a lip reading feature matrix V. The speech feature extraction network uses a 2D convolutional neural network (2D CNN), takes the speech recognition training samples in the training set as input, extracts speech features, and outputs a speech feature matrix A. The temporal feature extraction network uses a bidirectional gated recurrent unit (Bi-GRU), takes the lip reading feature matrix V and the speech feature matrix A as input, and outputs a lip reading feature matrix T containing temporal information. V and the speech feature matrix T containing temporal information A . The activation network is based on the lip reading feature matrix T V and speech feature matrix T A As input, the lip reading feature matrix T V Through an attention and fully connected (FC) layer, we get the activation matrix T M , the speech feature matrix T A With the activation matrix T M Multiply the elements at the same coordinates in to obtain the fused features, then perform linear transformation on the fused features, and classify the features after linear transformation to obtain the speech recognition result P.

[0056] The structure of each network is described in detail:

[0057] 121) Lip reading feature extraction network

[0058] The lip reading recognition training samples in the training set (i.e., the split single-frame images) are used as the input of the lip reading feature extraction network. The lip reading feature extraction network of this embodiment includes the following modules: first, the spatiotemporal features are extracted from the continuous single-frame images of the input through a 7×7×7 three-dimensional convolutional layer Conv3D with a step size of 2, and then a 2×2 maximum pooling layer MaxPool is used to reduce the size of the feature map and retain the key features. Then, the pooled feature map is batch normalized (BN) processed, and nonlinearity is introduced through an activation function layer (ReLU). After the image passes through the above modules, it is finally output as a lip reading feature matrix V. In this embodiment, the dimension of the lip reading feature matrix V for the entire training set is M×90×4512, where M represents the number of videos, 90 represents the number of image frames split from each video, and 4512 is a hyperparameter that can be adjusted freely. The 4512 used in this example is determined empirically.

[0059] 122) Speech feature extraction network

[0060] The speech recognition training samples in the training set are used as the input of the speech feature extraction network. The speech feature extraction network of this embodiment includes the following modules: first, the audio signal is converted into MFCC through the Mel Frequency Cepstral Coefficients (MFCC) module, and then the speech features are extracted from the continuously input audio through the 7×7 two-dimensional convolution layer Conv2D with a step size of 2, and then the size of the speech features is reduced using the 2×2 maximum pooling layer Max Pool, and then the pooled speech features are batch normalized (BN) and nonlinearity is introduced through the activation function layer (ReLU). After the audio data passes through the above modules, it is finally output as a speech feature matrix A. In this embodiment, the dimension of the speech feature matrix A for the entire training set is M×90×4512, where M represents the number of videos, 90 represents the number of audio segments split from each video, and 4512 is a hyperparameter that can be adjusted freely. The 4512 used in this example is determined empirically.

[0061] 123) Temporal feature extraction network

[0062] The temporal feature extraction network in this example takes the lip reading feature matrix V and the speech feature matrix A as input, and outputs the lip reading feature matrix T containing the temporal information after adding the temporal information. V and the speech feature matrix T containing temporal information A Specifically, the temporal feature extraction network uses two bidirectional gated recurrent units (Bi-GRU). Each Bi-GRU combines two GRU units. The hidden state dimension size of each GRU unit is 256. One GRU unit is used to process the forward time sequence information when the lip reading feature matrix V / speech feature matrix A at different input moments is forward concatenated, and the other GRU unit is used to process the reverse time sequence information when the lip reading feature matrix V / speech feature matrix A at different input moments is reverse concatenated, thereby capturing and utilizing the dynamically changing bidirectional context information, enhancing the representation ability of the features, and thus improving the performance of the model in tasks such as lip reading recognition and speech recognition.

[0063] The timing information is used to reflect the dynamic changes and dependencies of lip reading features and speech features over time, including the change pattern of features over time, contextual information, and the correlation between different time steps. For example, suppose a person's lips gradually open from closed to round when he says "hello". This change is a continuous and gradual process. If the model only considers the lip shape features of a single frame and ignores the information of the previous and next frames, it may not be able to accurately recognize the syllable. Therefore, the timing information captures the continuous change process from closed lips to open lips. Similarly, in a speech signal, the pronunciation of a syllable may gradually rise from low frequency to high frequency. If this change trend in the time series is not considered, the model may not be able to accurately recognize the syllable.

[0064] 124) Activate the network

[0065] After completing the extraction of lip reading and speech features containing time sequence information, this example realizes the interaction of the two modes through an activation network. V and the speech feature matrix T containing temporal information A As the input of the activation network, the lip reading feature matrix T V The activation matrix T is obtained through an activation module M The activation module consists of an attention layer and a first fully connected layer (FC). The lip reading feature matrix T A With the activation matrix T M The elements at corresponding positions (i.e., at the same coordinates) are multiplied to obtain fused features. This stage simulates the activation-inhibition mechanism of visual information on auditory neural circuits in the biological brain. This is a new modal interaction strategy that realizes the interaction between the two modal features of lip reading features and speech features. The fused features are then linearly transformed through the second fully connected layer. Finally, the linearly transformed features are classified by a classifier to obtain the speech recognition result P.

[0066] Next, from the lip reading feature matrix T V To the activation matrix T M The generation mechanism of is described in detail:

[0067] Lip reading feature matrix T V is an N×28 matrix, where N is the total number of frames of the image split from a single video, and “28” represents the total number of lip shapes, 1 space, and 1 placeholder corresponding to 26 English letters, corresponding to 28 lip reading feature categories; the lip reading feature matrix T VEach row in is a vector of length 28, and each element in the vector corresponds to the probability that this frame belongs to one of the 28 categories in the lip reading mode. For each frame, the Attention mechanism can be used to calculate the attention weight of the model on each category, and the probability of each frame belonging to each category is determined by analyzing the lip reading information. In order to normalize these probabilities, the activation function of the first fully connected layer of this embodiment adopts the softmax function to convert the calculated attention weights into probability distributions of each category, thereby more clearly reflecting the degree of "attention" concentration of the input image on each category. The greater the calculated probability, the greater the "attention" focused on this category, as shown in the schematic diagram Figure 3 As shown, Figure 3 The color depth indicates the probability. The darker the color, the greater the probability. Therefore, this "attention" can reflect the "activation degree" of the lip reading information for each category in this frame. Therefore, for the lip reading feature matrix T V Each row of is processed by Attention, and then passes through a first fully connected layer to generate the activation matrix T M .

[0068] 13) Model training.

[0069] Initialize the network parameters of the speech interaction recognition model constructed in step 12), set the loss function, and use the training set constructed in step 11) to train the speech interaction recognition enhanced model.

[0070] Considering that the image data and audio data in each training sample of the training set used in this embodiment are randomly sorted, that is, the data of the visual and auditory modes are not aligned. Data misalignment means that it is impossible to know which category each time window corresponds to, so the traditional loss function is invalid in this problem. In order to solve this problem, this embodiment adopts the CTC (Connectionist Temporal Classification) loss function, which enables the model to learn useful information directly from these misaligned data, greatly simplifying the preprocessing steps. It should be noted that the essence of lip reading recognition is a 27 classification, that is, to judge which category of 26 letters and a space this frame belongs to on each frame. CTC introduces the concept of blank characters, turning 27 classifications into 28 classifications, so that the model can predict a state where no characters are output, helping to solve the problem of misalignment of time windows and characters. The speech interaction recognition model constructed in step 12) is trained using the CTC loss function and the training set constructed in step 11) until the total loss function converges to obtain a trained speech interaction recognition model.

[0071] 2) Testing phase

[0072] In a noisy public environment, a vehicle-mounted camera is used to collect a video of a pedestrian interacting with the camera and performing voice recognition. The video is split into N voice segments and N image frames according to step 11), and the video is input into the trained voice interaction recognition enhancement model to obtain the voice recognition output result P. * .

[0073] In summary, the present invention aims at the problem that the existing RNN-based visual and auditory bimodal speech recognition has poor robustness in noisy environments, and proposes a speech recognition method with high robustness, high accuracy and low latency. The present invention adopts a structure combining CNN and Bi-GRU to extract features of the visual and auditory modes, and simulates the activation-inhibition mechanism of the visual information in the biological brain on the auditory neural circuit in the inter-modal information interaction stage, and proposes a new modal interaction strategy. The present invention has been verified on the GRID data set, and the results show that its recognition accuracy and robustness are better than the benchmark model (such as the unimodal lip reading recognition model LipNet, the speech recognition model DeepSpeech, the speech recognition model Whisper, and the bimodal model AV-HuBert), and the running time is constant, which can meet the voice interaction needs between robots and pedestrians in public noisy environments such as airports and train stations, and is easy to integrate into mobile terminals, which has important reference value for speech recognition research in noisy environments.

[0074] A second aspect of the present invention provides a speech interaction recognition enhancement device, comprising:

[0075] The data acquisition module is used to collect a video to be identified, in which a speaker faces a camera for voice interaction, split the video to be identified into N segments of voice to be identified and N frames of images to be identified, and form data to be identified;

[0076] The recognition module contains a preset voice interaction recognition enhancement model, which includes a lip reading feature extraction network, a voice feature extraction network, a time feature extraction network and an activation network. The lip reading feature extraction network is used to extract the lip reading feature matrix V of N frames of images to be recognized. * , the speech feature extraction network is used to extract the speech feature matrix A of N segments of speech to be recognized * , the temporal feature extraction network is used to extract the lip reading feature matrix V * And the speech feature matrix A * Get the lip reading feature matrix T containing time sequence information V * and the speech feature matrix T containing temporal information A *The activation network is used to simulate the activation-inhibition mechanism of visual information on auditory neural circuits in the biological brain, and realize the lip reading feature matrix T V * and speech feature matrix T A * The interaction of the two modes results in the speech recognition result P * .

[0077] It should be noted that the aforementioned explanation of an embodiment of a voice interaction recognition enhancement method is also applicable to a voice interaction recognition enhancement device of this embodiment, and will not be repeated here.

[0078] In order to implement the above embodiment, the embodiment of the present invention further proposes a computer-readable storage medium, on which a computer program is stored. The program is executed by a processor to execute the voice interaction recognition enhancement method of the above embodiment.

[0079] Reference below Figure 4 , which shows a schematic diagram of the structure of an electronic device suitable for implementing an embodiment of the present invention. It should be noted that the electronic device in the embodiment of the present invention may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, servers, etc. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0080] like Figure 4 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 102 or a program loaded from a storage device 108 to a random access memory (RAM) 103. In the RAM 103, various programs and data required for the operation of the electronic device are also stored. The processing device 101, the ROM 102, and the RAM 103 are connected to each other via a bus 104. An input / output (I / O) interface 105 is also connected to the bus 104.

[0081] Typically, the following devices may be connected to the I / O interface 105: an input device 106 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, etc.; an output device 107 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 108 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 109. The communication device 109 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Figure 4 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0082] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the present embodiment includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 109, or installed from the storage device 108, or installed from the ROM 102. When the computer program is executed by the processing device 101, the above-mentioned functions defined in the method of the embodiment of the present invention are executed.

[0083] It should be noted that the computer-readable medium of the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0084] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0085] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the voice interaction recognition enhancement method.

[0086] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, python, and conventional procedural programming languages ​​such as "C-" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0087] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0088] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0089] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0090] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0091] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0092] A person of ordinary skill in the art may understand that all or part of the steps carried out in the above-mentioned embodiment method may be completed by instructing the relevant hardware through a program, and the developed program may be stored in a computer-readable storage medium, which, when executed, includes one of the steps of the method embodiment or a combination thereof.

[0093] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0094] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for enhancing speech interaction recognition, characterized in that: include: Collecting a video to be recognized, in which a speaker faces a camera for voice interaction, splitting the video to be recognized into N segments of voice to be recognized and N frames of images to be recognized, and forming data to be recognized; The data to be recognized is input into a preset speech interaction recognition enhancement model, which includes a lip reading feature extraction network, a speech feature extraction network, a time feature extraction network and an activation network. The lip reading feature extraction network is used to extract the lip reading feature matrix of N frames of images to be recognized, the speech feature extraction network is used to extract the speech feature matrix of N segments of speech to be recognized, the time feature extraction network is used to obtain a lip reading feature matrix containing timing information and a speech feature matrix containing timing information based on the lip reading feature matrix and the speech feature matrix, and the activation network is used to simulate the activation-inhibition mechanism of visual information on auditory neural circuits in the biological brain, so as to realize the interaction of the two modes of the lip reading feature matrix containing timing information and the speech feature matrix containing timing information, thereby obtaining a speech recognition result.

2. The method for enhancing speech interaction recognition according to claim 1, characterized in that: The N segments of speech to be recognized and the N frames of images to be recognized are randomly ordered.

3. The method for enhancing speech interaction recognition according to claim 1, characterized in that: The lip reading feature extraction network adopts a 3D convolutional neural network, including a three-dimensional convolution layer, a maximum pooling layer, a batch normalization layer and an activation function layer connected in sequence.

4. The method for enhancing speech interaction recognition according to claim 1, characterized in that: The speech feature extraction network adopts a 2D convolutional neural network, including a Mel-frequency cepstral coefficient module, a two-dimensional convolution layer, a maximum pooling layer, a batch normalization layer and an activation function layer connected in sequence.

5. The method for enhancing speech interaction recognition according to claim 1, characterized in that: The temporal feature extraction network adopts two bidirectional gated recurrent units, one of which is used to process forward timing information when forward splicing feature matrices input at different moments, and the other gated recurrent unit is used to process reverse timing information when reverse splicing feature matrices input at different moments, thereby capturing and utilizing dynamically changing bidirectional contextual information and enhancing the feature representation capability.

6. The method for enhancing speech interaction recognition according to claim 1, characterized in that: The activation network includes an activation module, a fusion layer, a second fully connected layer and a classifier connected in sequence, and the activation module is composed of an attention layer and a first fully connected layer; the dimension of the lip reading feature matrix containing time series information is N×28, and the activation module first uses the attention mechanism to calculate the attention weights on 28 lip reading categories, and the 28 lip reading categories are composed of lip shapes corresponding to 26 English letters, 1 space and 1 placeholder, and then uses the first fully connected layer to convert the attention weights obtained by the attention layer into probability distributions of each category, thereby obtaining an activation matrix; the fusion layer is used to multiply the speech feature matrix containing time series information with the elements at corresponding positions in the activation matrix to obtain fusion features; the fusion features are linearly transformed by the second fully connected layer and classified by the classifier to obtain the speech recognition result.

7. The method for enhancing speech interaction recognition according to claim 1, characterized in that: The preset voice interaction recognition enhancement model is obtained through the following steps: 1) Construct training set: Obtain M videos of a speaker, split each video into N audio segments of equal length and N image frames of equal size, use the N audio segments as a group of audio data, perform ROI extraction processing on the N image frames respectively, extract the area near the lips in the image, obtain N frames of ROI image data, and use the N frames of ROI image data as a group of image data; A set of audio data and a set of image data obtained by splitting each video are matched, and used as a speech recognition training sample and a lip reading recognition training sample respectively, so as to construct a training set; 2) Build an enhanced voice interaction recognition model: The constructed speech interaction recognition enhancement model includes a lip reading feature extraction network, a speech feature extraction network, a time feature extraction network and an activation network: The lip reading feature extraction network adopts a 3D convolutional neural network, takes the lip reading recognition training sample as input, extracts image features, and outputs a lip reading feature matrix V; The speech feature extraction network adopts a 2D convolutional neural network, takes the speech recognition training sample as input, extracts speech features, and outputs a speech feature matrix A; The temporal feature extraction network adopts a bidirectional gated recurrent unit, takes the lip reading feature matrix V and the speech feature matrix A as input, and outputs a lip reading feature matrix T containing temporal information. V and the speech feature matrix T containing temporal information A ; The activation network includes an activation module, a fusion layer, a second fully connected layer and a classifier connected in sequence. The activation network is based on the lip reading feature matrix T containing time sequence information. V and the speech feature matrix T containing the timing information A As input, the lip reading feature matrix T containing time sequence information V Through the attention mechanism of the activation module and the first fully connected layer, the activation matrix T is obtained M , the speech feature matrix T containing time sequence information is transformed through the fusion layer A With the activation matrix T A The elements at corresponding positions are multiplied to obtain a fusion matrix, and the fusion matrix passes through the second fully connected layer and the classifier in sequence to obtain a speech recognition result P. 3) Training the voice interaction recognition enhancement model: A loss function is set, and the speech interaction recognition enhancement model constructed in step 2) is trained using the training set constructed in step 1) until the loss function converges, thereby obtaining a trained speech interaction recognition enhancement model.

8. The method for enhancing speech interaction recognition according to claim 7, characterized in that: In step 1), each audio segment in the set of audio data and the N frames of ROI image data in the set of image data are randomly sorted; In step 3), the loss function adopts the CTC loss function.

9. A speech interaction recognition enhancement device based on the method according to any one of claims 1 to 8, characterized in that: include: A data acquisition module is used to collect a video to be identified, in which a speaker faces a camera for voice interaction, split the video to be identified into N segments of voice to be identified and N frames of images to be identified, and form data to be identified; The recognition module contains a preset speech interaction recognition enhancement model, which includes a lip reading feature extraction network, a speech feature extraction network, a time feature extraction network and an activation network. The lip reading feature extraction network is used to extract the lip reading feature matrix of N frames of images to be recognized, the speech feature extraction network is used to extract the speech feature matrix of N segments of speech to be recognized, the time feature extraction network is used to obtain a lip reading feature matrix containing timing information and a speech feature matrix containing timing information according to the lip reading feature matrix and the speech feature matrix, and the activation network is used to simulate the activation-inhibition mechanism of visual information on auditory neural circuits in the biological brain, so as to realize the interaction of the two modes of the speech feature matrix containing lip timing information and the speech feature matrix containing lip timing information, thereby obtaining a speech recognition result.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 8.

Images