A speech recognition method and device based on feature fusion

Abstract

The application discloses a speech recognition method based on feature fusion, comprising the following steps: step 1, obtaining speech data, labeling the text content of the speech data, and forming a training set by combining the speech data and the labeled tags; step 2, constructing a speech recognition network based on feature fusion, wherein the speech recognition network comprises a feature extraction module, a data processing module, a feature fusion module, an attention module and a recognition module; step 3, training the speech recognition network by using the training set of step 1 to obtain a speech recognition model for speech-to-text conversion; and step 4, inputting speech data to be recognized into the speech recognition model obtained in step 3, and outputting high-quality recognized text corresponding to the speech data after recognition and analysis. The application further provides a speech recognition device. The method provided by the application can improve the accuracy of speech recognition by extracting multiple features in speech, thereby obtaining high-quality speech-to-text content.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech recognition, and in particular to a speech recognition method and device based on feature fusion. BACKGROUND

[0002] With the continuous development of science and technology, speech recognition technology is being used by more and more people. For example, in the mobile terminal, with the strengthening of speech recognition algorithms, models, and adaptability, speech recognition systems have been widely used in mobile terminals, such as Apple's Siri, Microsoft's Xiaona, and Xiaomi's Xiaoai products have realized commercialization. Speech recognition technology simplifies the communication between people and machines, eliminates the intermediate keyboard input and handwriting steps, and brings great convenience to human-computer interaction.

[0003] For example, in the field of smart home, all electrical devices at home can be controlled through speech recognition; in the field of voice car interconnection, the speech recognition system enables the driver to focus on the driving process of the car, and the driver can control various devices in the car through voice commands: the heating of the seat, the ventilation of the seat, the temperature of the air conditioner, the opening and closing of the sunshade curtain, etc. can be controlled through voice commands; in the field of aerospace, pilots can use the speech recognition system to realize multi-thread control of the aircraft during flight, thereby reducing the operating pressure of the pilots. If the accuracy of the speech recognition model cannot be guaranteed, it may cause danger in some scenarios, such as in a vehicle environment, if the vehicle speech recognition system cannot accurately recognize the user's voice, it may cause an accident.

[0004] Patent document CN114333782A discloses a speech recognition method, device, equipment and storage medium, which includes: acquiring initial audio data, preprocessing the initial audio data, transforming and filtering the preprocessed audio data to obtain filter bank acoustic features; calling a one-dimensional convolution layer and a multi-resolution convolution block to sequentially perform convolution processing and normalization processing on the filter bank acoustic features to obtain a target convolution result; calling an attention mechanism layer and a feature fusion layer to sequentially perform pooling processing and feature fusion processing on the target convolution result to obtain a feature fusion result; calling a pre-set language model to predict the feature fusion result to obtain a target speech recognition result. This method uses a convolution network, but has high requirements for environmental factors and is not suitable for use in noisy environments.

[0005] Patent document CN113192489A discloses a kind of based on multi-scale enhanced BiLSTM model's paint robot speech recognition method. 1) the commonly used spraying sound instruction is collected using signal acquisition system, wherein the data acquisition card selects NI-9234;2) 100 times Gaussian white noise is repeatedly added to the collected audio signal, to generate noisy signal and solve the corresponding mel spectrum sequence, and then solve the average sequence of 100 mel spectrum sequences;3) using multi-scale convolution filter to extract features from the average mel spectrum sequence, then using BiLSTM model to further mine the extracted features to obtain the corresponding output;4), the output of BiLSTM model is spliced together, and then input to the fully connected layer and Softmax layer for processing, and finally the speech recognition is realized by combining CTC algorithm;5), the model trained in steps 1-4 is embedded in the spraying robot, and the corresponding spraying task is intelligently realized. The BiLSTM model selected by the method cannot well identify the timing characteristics of the input speech, resulting in the possibility of information disconnection in the final output result, so that there may be errors or multiple sound word usage errors in the final recognition result.

[0006] In recent years, the application of deep learning has brought very convenient training steps to the speech recognition system, and the accuracy of the speech recognition model based on deep learning has been greatly improved compared with the speech recognition model based on statistics. However, different languages have their own word habits, especially the multiple sound word problem in Chinese. If only single words / characters are recognized and fed back as text without considering the timing relationship between words, syntax errors or multiple sound word recognition errors may occur, especially in vehicle-mounted voice systems or aerospace fields. Since there will be bumps during driving, noise during equipment operation, which will pollute the driver's input voice information, if the recognized text corresponding to the voice has a large number of errors or multiple sound word usage errors, it will affect the system's recognition of the driver's instructions, resulting in incorrect operation. SUMMARY

[0007] To solve the above problems, the present application provides a speech recognition method based on feature fusion, which extracts multiple features in the speech for joint recognition analysis, thereby improving the accuracy of speech recognition and obtaining high-quality speech-to-text content.

[0008] A speech recognition method based on feature fusion, comprising:

[0009] Step 1, obtain speech data, label the text content of the speech data, and form a training set with the speech data and the labeled tags;

[0010] Step 2, a speech recognition network based on feature fusion is constructed, the speech recognition network comprises a feature extraction module, a data processing module, a feature fusion module, an attention module and a recognition module, the feature extraction module comprises a multi-feature extractor, the multi-feature extractor is used for extracting speech features and speech timing features in input speech data and inputting to the data processing module, the data processing module is used for unifying dimensions of input features, the feature fusion module is used for fusing the speech features and the speech timing features after unifying dimensions, and the fused features obtained by fusion are input to the attention module, the attention module is used for attention enhancement on the input fused features, and the recognition module is used for prediction calculation on the fused features after attention enhancement, so as to output a recognition result, the recognition result comprises text content corresponding to the input speech data;

[0011] Step 3, the training set in step 1 is used to train the speech recognition network, and a speech recognition model for speech to text is obtained;

[0012] Step 4, input the speech data to be recognized into the speech recognition model obtained in step 3, and output high-quality recognized text corresponding to the speech data after recognition analysis.

[0013] The present application is directed to speech features and speech timing features in speech data, fused features with timing features are obtained by splicing the two kinds of features, and the information of the fused features is enhanced by using an attention mechanism, so as to improve the accuracy of speech recognition, reduce the probability of wrong characters and polyphones in the recognized text, and obtain high-quality speech to text content.

[0014] Specifically, the speech data comprises an AISHELL-1 Chinese speech data set and a LibriSpeech English data set.

[0015] Specifically, before the speech data is input into the speech recognition network, the speech data is preprocessed, including pre-emphasis, framing, windowing, filtering processing and data transformation, and the preprocessed speech data is input into the speech recognition network.

[0016] Specifically, the specific process of the preprocessing is as follows:

[0017] Step 1-1, pre-emphasizing the speech data, so as to remove the influence of lip radiation and increase the high-frequency resolution of the speech;

[0018] Step 1-2, framing the pre-emphasized speech data;

[0019] Step 1-3, windowing the framed speech data, in order to solve the leakage problem that the frequency spectrum is tailing in the whole frequency band due to non-periodic truncation of the signal;

[0020] Steps 1-4: Decompose the windowed speech data using Fast Fourier Transform and filter the decomposed data using Mel filter bank.

[0021] Steps 1-5: Process the data obtained from filtering in step 4 using logarithmic operations and discrete cosine transform to obtain the corresponding Mel-frequency cepstral coefficients (MFCC).

[0022] Preferably, the speech recognition network in step 2 is constructed by introducing an attention mechanism on the basis of mutual supervision between the CNN backbone network and the RNN backbone network. Temporal features are added to the speech features, and the fusion features are enhanced through the attention mechanism, so that the recognition accuracy of the final recognition model is improved and it has strong robustness.

[0023] Specifically, the unified feature dimension of the data processing module is to reconstruct speech features until they are consistent with the speech temporal feature dimension, and its expression is as follows:

[0024] Length CNN Width CNN *Chan=Length RNN *Unite

[0025] In the formula, Length CNN Width represents the length of a speech feature. CNN The width of the speech feature is represented by 'Chan', the number of channels for the speech feature is represented by 'Length'. RNN This indicates the length of the speech temporal feature, and Unite indicates the number of units in the speech temporal feature.

[0026] Specifically, the feature fusion module performs matrix concatenation between the unified-dimensional speech features and the speech temporal features to obtain the fused features under the target dimension. The expression for the target dimension is as follows:

[0027] Size FCRNN =[Batchsize,Length RNN [Unite*2]

[0028] In the formula, Batchsize represents the number of input speech samples during training, and Length... RNN This indicates the length of the speech temporal feature, and Unite indicates the number of units in the speech temporal feature.

[0029] Specifically, the attention module enhances attention by applying it to the number of units of the speech temporal features in the fused features, and its expression is as follows:

[0030] F′ CRNN =F CRNN ·Att

[0031] In the formula, F′ CRNN F represents the fusion feature after attention enhancement. CRNN Let Att represent the fusion features without attention enhancement, and Att represent the attention mechanism matrix, with dimension Size. Att = [Unite*2, N], where N is the system's hyperparameter.

[0032] Specifically, the recognition module consists of multiple fully connected layers, and the number of neurons in each fully connected layer is determined by the training set.

[0033] If the training set is English speech, the number of neurons is the number of English letters plus one.

[0034] If the training set is Chinese speech, then the number of neurons is the number of Chinese characters plus one.

[0035] Preferably, during the training in step 3, the speech recognition network is trained using the CTC loss function as the objective function. The parameters of the speech recognition network are updated by comprehensively considering the loss function between the prediction results of the speech recognition network and the training set. The CTC loss function does not require the alignment of speech and text when training the speech recognition model (the CTC loss function is the sum of the probabilities of outputting the correct label after a given sample), which speeds up the training process.

[0036] The present invention also provides a speech recognition device, including a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, wherein the computer memory employs the aforementioned speech recognition model; when the computer processor executes the computer program, it performs the following steps: inputting the speech to be recognized into the speech recognition model, and outputting the recognized text corresponding to the speech after recognition analysis.

[0037] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0038] (1) The speech recognition model proposed in this invention uses a fusion feature that combines speech features and speech temporal features for recognition, which can improve the accuracy of speech recognition and reduce the probability of misspellings and polyphonic characters in the recognized text.

[0039] (2) The speech recognition model proposed in this invention is constructed by introducing an attention mechanism on the basis of mutual supervision between the CNN backbone network and the RNN backbone network, which can not only ensure the robustness of speech recognition, but also improve the accuracy of the final speech recognition. Attached Figure Description

[0040] Figure 1 A schematic flowchart of the feature fusion-based speech recognition method provided by the present invention;

[0041] Figure 2 This is a schematic diagram of the structure of a speech recognition network provided in this embodiment. Detailed Implementation

[0042] Because external noise may affect the speech output process, the accuracy of the final recognition result is not high, resulting in a large number of typos in the recognized text content.

[0043] like Figure 1 As shown, this embodiment proposes a speech recognition method based on feature fusion, including:

[0044] Step 1: Obtain the AISHELL-1 Chinese speech dataset, label the text content of the speech data, and combine the speech data with the labeled data to form a training set. The speech data needs to be preprocessed before use.

[0045] Step 1-1: Pre-emphasize the speech data to remove the influence of lip radiation and increase the high-frequency resolution of the speech.

[0046] Steps 1-2: Perform frame segmentation on the pre-emphasized speech data;

[0047] Steps 1-3: Window the framed audio data to solve the problem of spectrum leakage caused by non-periodic truncation of the signal, resulting in a trailing phenomenon throughout the frequency band.

[0048] Steps 1-4: Decompose the windowed speech data using Fast Fourier Transform and filter the decomposed data using Mel filter bank.

[0049] Steps 1-5: Process the data obtained from filtering in step 4 using logarithmic operations and discrete cosine transform to obtain the corresponding Mel-frequency cepstral coefficients (MFCCs).

[0050] Step 2, as follows Figure 2 As shown, a speech recognition network is constructed using a basic network framework of mutual supervision between CNN and RNN backbones, and an attention mechanism is introduced. This speech recognition network includes a feature extraction module, a data processing module, a feature fusion module, an attention module, and a recognition module.

[0051] The feature extraction module includes a multi-feature extractor, which is used to extract speech features F from the input speech data. CNN and speech temporal features F RNN Speech features F are extracted through the CNN backbone network. CNN This ensures the robustness of the model by extracting speech temporal features F through the RNN backbone network. RNNThis can improve the model's recognition accuracy;

[0052] The data processing module is used to process speech features F CNN Reconstruction is performed until it matches the speech temporal features F. RNN The dimensions are consistent, and its expression is as follows:

[0053] Length CNN Width CNN *Chan=Length RNN *Unite

[0054] In the formula, Length CNN Width represents the length of a speech feature. CNN The width of the speech feature is represented by 'Chan', the number of channels for the speech feature is represented by 'Length'. RNN This indicates the length of the speech temporal feature, and Unite indicates the number of units in the speech temporal feature;

[0055] The feature fusion module is used to process the unified-dimensional speech features F CNN and speech temporal features F RNN Perform fusion and obtain the fusion feature F CRNN The input attention module has the following dimensions for the fused features:

[0056] Size FCRNN =[Batchsize,Length RNN [Unite*2]

[0057] In the formula, Batchsize represents the number of input speech samples during training, and Length... RNN This indicates the length of the speech temporal feature, and Unite indicates the number of units in the speech temporal feature;

[0058] The attention module is used to fuse the input features F CRNN Attention enhancement is applied to the number of units representing temporal features in Chinese speech:

[0059] F′ CRNN =F CRNN ·Att

[0060] In the formula, F′ CRNN F represents the fusion feature after attention enhancement. CRNN Let Att represent the fusion features without attention enhancement, and Att represent the attention mechanism matrix, with dimension Size. Att = [Unite*2, N], where N is the system's hyperparameter;

[0061] The recognition module consists of multiple fully connected layers, used to process the enhanced fusion features F′.CRNN Perform predictive calculations to output the recognition results.

[0062] The number of neurons in a fully connected layer is determined by the training set:

[0063] If the training set is English speech, the number of neurons is the number of English letters plus one.

[0064] If the training set is Chinese speech, then the number of neurons is the number of Chinese characters plus one.

[0065] The specific expression for this prediction calculation is as follows:

[0066] f(x) = T

[0067] Where f(·) represents the constructed model, x represents the preprocessed speech features, and T represents the model's recognition result for x.

[0068] Step 3: Using the training set from Step 1, train the speech recognition network with the CTC loss function as the objective function. The CTC loss function can be expressed as follows:

[0069] CTCLoss(f(x),T′)=-logP(T′|f(x))

[0070] Where T′ represents the text corresponding to the input speech feature x, and P(T′|f(x)) represents the probability that the constructed model can be identified as T′ when the input speech feature is x.

[0071] By comprehensively considering the loss function between the prediction results of the speech recognition network and the training set, the parameters of the speech recognition network are updated to obtain a speech recognition model for speech-to-text conversion.

[0072] Step 4: Input the speech data to be recognized into the speech recognition model obtained in Step 3. After recognition and analysis, output the recognized text corresponding to the speech data.

[0073] This embodiment also provides a speech recognition device, including a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, wherein the speech recognition model described above is used in the computer memory.

[0074] When in use, the speech is input into the speech recognition model. After calculation and recognition, the model outputs high-quality recognized text corresponding to the speech. By adding temporal features to the speech during the recognition process, the accuracy of speech recognition is improved, thereby reducing the probability of typos and polyphonic characters in the recognized text.

Claims

1. A speech recognition method based on feature fusion, characterized in that, include: Step 1: Acquire speech data, label the text content of the speech data, and combine the speech data and the labeled data to form a training set; Step 2: Construct a speech recognition network based on feature fusion. This network is built upon the mutual supervision of a CNN backbone network and an RNN backbone network, incorporating an attention mechanism. The network includes a feature extraction module, a data processing module, a feature fusion module, an attention module, and a recognition module. The feature extraction module includes a multi-feature extractor, which extracts speech features and temporal features from the input speech data and inputs them to the data processing module. The data processing module unifies the dimension of the input features by reconstructing the speech features until they match the temporal feature dimension. The expression for this unification is as follows: Length CNN *Width CNN *Chan=Length RNN *Unite In the formula, Length CNN Width represents the length of a speech feature. CNN The width of the speech feature is represented by , Chan by the number of channels in the speech feature, and Length by . RNN This indicates the length of the speech temporal feature, and Unite indicates the number of units in the speech temporal feature; The feature fusion module is used to fuse the unified-dimensional speech features and speech temporal features, and input the fused features to the attention module. The feature fusion module performs matrix concatenation of the unified-dimensional speech features and speech temporal features to obtain the fused features under the target dimension. The expression of the target dimension is as follows: SizeF CRNN =[Batchsize,Length RNN ,Unite*2] In the formula, Batchsize represents the number of input speech samples during training, and Length... RNN This indicates the length of the speech temporal feature, and Unite indicates the number of units in the speech temporal feature; The attention module is used to enhance the attention of the input fusion features, and the recognition module is used to perform prediction calculations on the attention-enhanced fusion features to output a recognition result. The recognition result includes the text content corresponding to the input speech data. The attention module enhances the attention of the number of units of the speech temporal features in the fusion features, and its expression is as follows: F′ CRNN =F CRNN ·To In the formula, F' CRNN F represents the fusion feature after attention enhancement. CRNN Let Att represent the fusion features without attention enhancement, and Att represent the attention mechanism matrix, with dimension Size. Att = [Unite*2, N], where N is the system's hyperparameter; Step 3: Using the training set from Step 1, train the speech recognition network to obtain a speech recognition model for speech-to-text conversion. Step 4: Input the speech data to be recognized into the speech recognition model obtained in Step 3. After recognition and analysis, output the high-quality recognized text corresponding to the speech data.

2. The speech recognition method based on feature fusion according to claim 1, characterized in that, Before the speech data is input into the speech recognition network, it undergoes preprocessing, including pre-emphasis, framing, windowing, filtering, and data transformation. The preprocessed speech data is then input into the speech recognition network.

3. The speech recognition method based on feature fusion according to claim 1, characterized in that, The recognition module consists of multiple fully connected layers, and the number of neurons in each fully connected layer is determined by the training set. If the training set is English speech, the number of neurons is the number of English letters plus one. If the training set is Chinese speech, then the number of neurons is the number of Chinese characters plus one.

4. The speech recognition method based on feature fusion according to claim 1, characterized in that, In step 3, the speech recognition network is trained using the CTC loss function as the objective function. The parameters of the speech recognition network are updated by comprehensively considering the loss function between the prediction results of the speech recognition network and the training set.

5. A voice recognition device, comprising a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, characterized in that, The computer memory uses the speech recognition model as described in claim 1; when the computer processor executes the computer program, it performs the following steps: inputting the speech to be recognized into the speech recognition model, and outputting the recognized text corresponding to the speech after recognition analysis.

Images