A Chinese speech offline recognition method and system based on a Transformer

Abstract

The application discloses a Chinese speech offline recognition method and system based on a Transformer, and the Chinese speech offline recognition method comprises the following steps: inputting speech, extracting Mel frequency cepstrum coefficients in the speech and converting the Mel frequency cepstrum coefficients into an input sequence, and inputting a neural network model; an encoder of the neural network model encodes the input sequence to obtain a character vector; a decoder of the neural network model decodes the character vector to obtain output features; the output features are fused to obtain integrated features; the integrated features are subjected to softmax processing to obtain a predicted character set; and the predicted character set is subjected to weight calculation based on a word group dictionary to obtain accurate characters. In the application, features in the decoder of the Transformer are fused to the top layer, the generalization features of an audio signal are better extracted, and weight calculation is performed through the word group dictionary, so that the problem of homophonic words can be effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of speech recognition, and in particular to a Transformer-based Chinese speech offline recognition method and system. Background Art

[0002] Automatic speech recognition (ASR) technology converts speech signals into text and is widely used in various scenarios, such as healthcare, speech-to-image generation, and aviation voice control. ASR is also a foundational technology for a wide range of other research areas, such as semantic recognition and speaker emotion recognition. For the reasons mentioned above, much research has recently focused on ASR, with significant progress achieved. Consequently, many competing approaches have been proposed. The recently proposed Transformer network has become very popular in the field of natural language processing (NLP) due to its outstanding parallel computing capabilities. In addition to its strong performance in NLP, the Transformer model has also demonstrated its powerful capabilities in speech recognition. Consequently, much research has focused on the Transformer network, and numerous modifications have been proposed to improve model performance.

[0003] Existing technical solutions generally use only features extracted from the top layer for prediction. In this case, the features used for prediction are not sufficiently representative, and optimal model performance cannot be achieved through this operation. Furthermore, most existing solutions are general solutions for all languages ​​and are not optimized for Chinese speech recognition. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings of the existing technology. The present invention provides a Chinese speech offline recognition method and system based on Transformer, which integrates the features in the Transformer decoder into the top layer to better extract the generalized features of the audio signal. The weight calculation is performed through the phrase dictionary, which can effectively solve the problem of homophones.

[0005] In order to solve the above technical problems, an embodiment of the present invention provides a Chinese speech offline recognition method based on Transformer, and the Chinese speech offline recognition method includes:

[0006] Input speech, extract Mel frequency cepstral coefficients from the speech and convert them into input sequence, which is then input into the neural network model;

[0007] The encoder of the neural network model encodes the input sequence to obtain a character vector;

[0008] The decoder of the neural network model decodes the character vector to obtain output features;

[0009] The output features are fused to obtain integrated features;

[0010] Perform softmax processing on the integrated features to obtain the predicted character set;

[0011] The predicted character set is weighted based on the phrase dictionary to obtain the accurate characters.

[0012] Preferably, the step of performing weight calculation on the predicted character set to obtain accurate characters further includes:

[0013] Compose accurate sentences based on accurate characters and character vectors.

[0014] Preferably, the extracting of Mel-frequency cepstral coefficients from speech includes:

[0015] After inputting a piece of speech, a 25ms window is used, shifted every 10ms, to extract the 40-dimensional Mel-frequency cepstral coefficient features, convert the 40-dimensional Mel-frequency cepstral coefficient features into an input sequence, and input the input sequence into the neural network model.

[0016] Preferably, the encoder of the neural network model encodes the input sequence, comprising:

[0017] The encoder encodes characters through self-attention and feedforward neural networks. Self-attention adds word vectors to characters based on the matrix to obtain character vectors.

[0018] Preferably, the fusion processing of the output features includes:

[0019] Connect the output features of the first five decoding layers to obtain the concatenated output features;

[0020] The concatenated output features are compressed based on the convolution operation to obtain fused features;

[0021] Add the fusion features to the output features of the sixth layer to obtain the integrated features.

[0022] Preferably, weight calculation is performed on the predicted character set based on a phrase dictionary, including:

[0023] extracting a weight dictionary based on the predicted characters of the predicted character set in the phrase dictionary;

[0024] In the predicted character set of the current predicted word, the samples are sorted in descending order of probability, and the first k samples are selected to form the sample set;

[0025] Form a prediction list based on the phrase of the previous predicted word;

[0026] The sample set is matched one by one with the characters in the prediction list, and the weight of the matched characters is corrected in the weight dictionary.

[0027] Compare the weights of each character, and the character with the largest weight is the correct character.

[0028] Preferably, the weight calculation formula is:

[0029] ρ(x)=min{ε+μx,β}

[0030] Where ε represents the initial weight; μ is a proportional coefficient used to adjust the impact of the frequency of the primitive; x is the number of occurrences of the corresponding tuple; and a maximum value β is set to limit the range of the final weight.

[0031] Accordingly, the present invention also provides a Transformer-based Chinese speech offline recognition system, which is used to implement the above-mentioned Chinese speech offline recognition method. The Chinese speech offline recognition system includes an input module, an encoding module, a decoding module, an enhanced feature fusion module, a normalization module, and a Mandarin auxiliary recognition module, wherein:

[0032] The input module is used to input speech, extract Mel frequency cepstral coefficients from the speech and convert them into input sequences, which are then input into the neural network model;

[0033] The encoding module encodes the input sequence according to the encoder of the neural network model to obtain a character vector;

[0034] The decoding module decodes the character vector according to the decoder of the neural network model to obtain output features;

[0035] The enhanced feature fusion module performs fusion processing on the output features to obtain integrated features;

[0036] The normalization module performs softmax processing on the integrated features to obtain a predicted character set;

[0037] The Mandarin auxiliary recognition module is used to perform weight calculation on the predicted character set based on the phrase dictionary to obtain accurate characters.

[0038] Preferably, the enhanced feature fusion module includes:

[0039] Connect the output features of the first five decoding layers to obtain the concatenated output features;

[0040] The concatenated output features are compressed based on the convolution operation to obtain fused features;

[0041] Add the fusion features to the output features of the sixth layer to obtain the integrated features.

[0042] Preferably, the Mandarin auxiliary recognition module includes:

[0043] extracting a weight dictionary based on the predicted characters of the predicted character set in the phrase dictionary;

[0044] In the predicted character set of the current predicted word, the samples are sorted in descending order of probability, and the first k samples are selected to form the sample set;

[0045] Form a prediction list based on the phrase of the previous predicted word;

[0046] The sample set is matched one by one with the characters in the prediction list, and the weight of the matched characters is corrected in the weight dictionary.

[0047] Compare the weights of each character, and the character with the largest weight is the correct character.

[0048] This invention provides a Transformer-based offline Chinese speech recognition method and system. This method integrates features from the Transformer decoder into the top layer, making the top layer features more representative and better extracting generalized features of the audio signal, thereby improving efficiency. Furthermore, this method uses a phrase dictionary to perform weight calculations, and weights homophones based on contextual relationships, effectively resolving the homophone problem. This invention significantly improves the model's feature fusion capabilities and effectively mitigates speech recognition errors caused by homophones. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 4 is a flow chart of a method for offline Chinese speech recognition based on Transformer in an embodiment of the present invention.

[0051] Figure 2 2 is a schematic diagram of the decoder fusion process in an embodiment of the present invention.

[0052] Figure 3 3 is a schematic diagram of the structure of the Chinese speech offline recognition system based on Transformer in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0054] Example

[0055] Figure 1 The figure shows a flow chart of the offline Chinese speech recognition method based on Transformer in an embodiment of the present invention. Figure 2 The figure shows a schematic diagram of the decoder fusion process in an embodiment of the present invention. The Chinese speech offline recognition method includes:

[0056] S1: Input speech, extract Mel frequency cepstral coefficients from the speech and convert them into input sequence, which is then input into the neural network model:

[0057] After inputting a speech segment, a 25ms window is used, shifted every 10ms, to extract 40-dimensional Mel-frequency cepstrum coefficient features (Mel Frequency Cepstrum Coefficient, abbreviated as MFCC). The 40-dimensional Mel-frequency cepstrum coefficient features are converted into an input sequence that can be calculated by the neural network, and the input sequence is input into the neural network model.

[0058] S2: The encoder of the neural network model encodes the input sequence to obtain a character vector:

[0059] The encoder encodes characters through self-attention and feedforward neural networks. Self-attention adds word vectors to characters based on the matrix to obtain character vectors. Character vectors can identify the position of characters and recognize characters.

[0060] The encoder has six encoding layers, each of which has a self-attention layer and a feedforward neural network. Self-attention adds word vectors to characters based on a matrix. Each encoding layer of the encoder encodes characters through self-attention and a feedforward neural network to obtain their own character vectors.

[0061] S3: The decoder of the neural network model decodes the character vector to obtain output features;

[0062] The decoder has six decoding layers, each of which has self-attention, encoding-decoding attention layer and feedforward neural network. Each decoding layer of the decoder decodes characters through self-attention, encoding-decoding attention layer and feedforward neural network to obtain respective output features.

[0063] S4: Fuse the output features to obtain integrated features:

[0064] like Figure 2 As shown in Figure 2, the output features of the first five decoding layers are connected to obtain the concatenated output features. Figure 2 Where Layer_i represents the output features of the decoder layer i. If d_model represents the dimension of the output features, that is, the output features of each layer, then the dimension of the concatenated fusion vector is 5×d_model.

[0065] The concatenated output features are compressed using a convolution operation to produce fused features. Convolution is used for compression here, namely comprrss in the figure. Convolution has several advantages in this area: First, it ensures that the output features have the same dimensions as the features of other layers. After the convolution operation, the dimensions of the fused vector return to d_model, facilitating subsequent fusion with the output features of the highest layer. Second, a learnable matrix can be used to determine the most appropriate feature weights. Finally, convolution can capture local information.

[0066] The fused features are added to the output features of the sixth layer to obtain the integrated features. After fusing the features of the previous layers and compressing the dimensionality of the fused features, the top layer becomes more representative. The output features of each layer are added to the output features of the top layer, allowing the model to fully utilize the output features of all layers.

[0067] S5: Perform softmax processing on the integrated features to obtain the predicted character set;

[0068] The output features are calculated through the normalized reference function of softmax, and the predicted output Chinese character sequence, that is, the predicted character set, can be obtained one by one.

[0069] S6: Calculate the weight of the predicted character set based on the phrase dictionary to obtain the accurate character:

[0070] Extract the weight dictionary D2 from the phrase dictionary D1 based on the predicted word of the predicted character set; create the phrase dictionary D1 and the weight dictionary D2: input the label sequence of the predicted character set; extract the sequence M from the label sequence of the predicted character set, and perform weight calculation on the sequence M to obtain the weight dictionary D2.

[0071] The weight calculation formula is:

[0072] ρ(x)=min{ε+μx,β}

[0073] In the formula, ε represents the initial weight; μ is a proportional coefficient used to adjust the impact of the frequency of the primitive; x is the number of occurrences of the corresponding tuple; in order to prevent the final weight from increasing endlessly, a maximum value β is set to limit the range of the final weight.

[0074] In the candidate character set A of the current predicted word, the samples are sorted in descending order of probability, and the first k samples are selected to form the sample set A1;

[0075] The prediction list L is formed based on the phrase of the previous predicted word; i-1 ) and phrase dictionary D1, obtain y i-1 The phrase or list of words that may appear next is L;

[0076] The sample set A1 is matched one by one with the characters in the prediction list L, and the weight of the matched characters is corrected in the weight dictionary. Next, A1 is matched one by one with the characters in L. If a match is found, the corresponding probability is found in the dictionary D2, and the weight of the candidate character A1 is changed, thereby affecting the next recognized Chinese character y i , correct homophone errors. The weights of the corrected characters in the weight dictionary are corrected based on the weight formula.

[0077] After correcting the weights of all characters, compare the weights of each character, and the character with the largest weight is the correct character.

[0078] S7: Form accurate sentences based on accurate characters and character vectors:

[0079] After all predicted characters are corrected to accurate characters, the accurate characters are combined into a complete accurate sentence based on the character vectors of the predicted characters obtained by encoding with the encoder.

[0080] The present invention provides a Transformer-based offline Chinese speech recognition method. The method integrates the features in the Transformer decoder into the top layer, making the top layer features more representative and better extracting the generalized features of the audio signal, thereby improving efficiency. The method also calculates weights through a phrase dictionary and weights homophones based on contextual relationships, effectively solving the homophone problem.

[0081] Figure 3A structural schematic diagram of a Transformer-based Chinese speech offline recognition system in an embodiment of the present invention is shown. The Chinese speech offline recognition system is used to implement the above-mentioned Chinese speech offline recognition method. The Chinese speech offline recognition system includes an input module 201, an encoding module 202, a decoding module 203, an enhanced feature fusion module 204, a normalization module 205 and a Mandarin auxiliary recognition module 206, wherein: the input module 201 is used to input speech, extract Mel-frequency cepstral coefficients from the speech and convert them into an input sequence, which is input into the neural network model; the encoding module 202 encodes the input sequence according to the encoder of the neural network model to obtain a character vector; the decoding module 206 decodes the character vector according to the decoder of the neural network model to obtain output features; the enhanced feature fusion module 204 fuses the output features to obtain integrated features; the normalization module 205 performs softmax processing on the integrated features to obtain a predicted character set; the Mandarin auxiliary recognition module 206 is used to calculate the weights of the predicted character set based on a phrase dictionary to obtain accurate characters.

[0082] The encoding module 202 has six encoding layers, each of which has a self-attention layer and a feedforward neural network. The self-attention layer adds word vectors to the characters based on the matrix. Each encoding layer of the encoder encodes the characters through the self-attention layer and the feedforward neural network to obtain their own character vectors.

[0083] The decoding module 206 has six decoding layers, each of which has self-attention, encoding-decoding attention layer and feedforward neural network. Each decoding layer of the decoder decodes characters through self-attention, encoding-decoding attention layer and feedforward neural network to obtain respective output features.

[0084] The full name of the reinforcement feature fusion module 204 is Reinforcement Feature Fusion, abbreviated as RFF. The reinforcement feature fusion module 204 is only applied to the decoder in the Transformer model, combining the features of all other layers in the decoder, that is, the decoding module 206. The reinforcement feature fusion module 204 includes: connecting the output features of the decoding layers of the first five layers to obtain serial output features; compressing the serial output features based on convolution operations to obtain fused features; and adding the fused features to the output features of the sixth layer to obtain integrated features.

[0085] The Mandarin Aided Recognition module 206 is called Mandarin Aided Recognition, or MAR for short. The module can effectively alleviate the recognition problem caused by homophones. The module 206 includes: extracting a weight dictionary based on the predicted characters in the predicted character set in the phrase dictionary; sorting the samples in the predicted character set of the current predicted character in descending order of probability, and selecting the first k samples to form a sample set; forming a prediction list based on the phrase of the previous predicted character; matching the sample set with the characters in the prediction list one by one, and correcting the weight of the matched characters in the weight dictionary; and comparing the weights of the individual characters, with the character with the largest weight being the correct character.

[0086] This invention provides a Transformer-based offline Chinese speech recognition method and system. This method integrates features from the Transformer decoder into the top layer, making the top layer features more representative and better extracting generalized features of the audio signal, thereby improving efficiency. Furthermore, this method uses a phrase dictionary to perform weight calculations, and weights homophones based on contextual relationships, effectively resolving the homophone problem. This invention significantly improves the model's feature fusion capabilities and effectively mitigates speech recognition errors caused by homophones.

[0087] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), a disk or an optical disk, etc.

[0088] In addition, the embodiments of the present invention are introduced in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A Chinese speech offline recognition method based on Transformer, characterized in that: The Chinese speech offline recognition method comprises: Input speech, extract Mel frequency cepstral coefficients from the speech and convert them into input sequence, which is then input into the neural network model; The encoder of the neural network model encodes the input sequence to obtain a character vector; The decoder of the neural network model decodes the character vector to obtain output features; The output features are fused to obtain integrated features; Perform softmax processing on the integrated features to obtain the predicted character set; The predicted character set is weighted based on the phrase dictionary to obtain the accurate characters; The predicted character set is weighted based on the phrase dictionary, including: extracting a weight dictionary based on the predicted characters of the predicted character set in the phrase dictionary; In the predicted character set of the current predicted word, the samples are sorted in descending order of probability, and the first k samples are selected to form the sample set; Form a prediction list based on the phrase of the previous predicted word; The sample set is matched one by one with the characters in the prediction list, and the weight of the matched characters is corrected in the weight dictionary; Compare the weights of each character, and the character with the largest weight is the correct character.

2. The Chinese speech offline recognition method according to claim 1, characterized in that: The weight calculation of the predicted character set is performed to obtain the accurate character, and then the following steps are further included: Compose accurate sentences based on accurate characters and character vectors.

3. The Chinese speech offline recognition method according to claim 1, characterized in that: The step of extracting Mel-frequency cepstral coefficients from speech comprises: After inputting a piece of speech, a 25ms window is used, shifted every 10ms, to extract the 40-dimensional Mel-frequency cepstral coefficient features, convert the 40-dimensional Mel-frequency cepstral coefficient features into an input sequence, and input the input sequence into the neural network model.

4. The Chinese speech offline recognition method according to claim 1, characterized in that: The encoder of the neural network model encodes the input sequence, including: The encoder encodes characters through self-attention and feedforward neural networks. Self-attention adds word vectors to characters based on the matrix to obtain character vectors.

5. The Chinese speech offline recognition method according to claim 1, characterized in that: The fusing process of the output features includes: Connect the output features of the first five decoding layers to obtain the concatenated output features; The concatenated output features are compressed based on the convolution operation to obtain fused features; Add the fusion features to the output features of the sixth layer to obtain the integrated features.

6. The Chinese speech offline recognition method according to claim 1, characterized in that: The weight calculation formula is: In the formula, ε represents the initial weight; μ is a scaling factor used to adjust the effect of the primitive frequency; x is the number of occurrences of the corresponding tuple; a maximum value β is set to limit the range of the final weight.

7. A Chinese speech offline recognition system based on Transformer, characterized by: The Chinese speech offline recognition system is used to implement the Chinese speech offline recognition method according to any one of claims 1 to 6, and the Chinese speech offline recognition system includes an input module, an encoding module, a decoding module, an enhanced feature fusion module, a normalization module, and a Mandarin auxiliary recognition module, wherein: The input module is used to input speech, extract Mel frequency cepstral coefficients from the speech and convert them into input sequences, which are then input into the neural network model; The encoding module encodes the input sequence according to the encoder of the neural network model to obtain a character vector; The decoding module decodes the character vector according to the decoder of the neural network model to obtain output features; The enhanced feature fusion module performs fusion processing on the output features to obtain integrated features; The normalization module performs softmax processing on the integrated features to obtain a predicted character set; The Mandarin auxiliary recognition module is used to calculate the weight of the predicted character set based on the phrase dictionary to obtain the accurate character; The Mandarin auxiliary recognition module includes: extracting a weight dictionary based on the predicted characters of the predicted character set in the phrase dictionary; In the predicted character set of the current predicted word, the samples are sorted in descending order of probability, and the first k samples are selected to form the sample set; Form a prediction list based on the phrase of the previous predicted word; The sample set is matched one by one with the characters in the prediction list, and the weight of the matched characters is corrected in the weight dictionary; Compare the weights of each character, and the character with the largest weight is the correct character.

8. The Chinese speech offline recognition system according to claim 7, characterized in that: The enhanced feature fusion module includes: Connect the output features of the first five decoding layers to obtain the concatenated output features; The concatenated output features are compressed based on the convolution operation to obtain fused features; Add the fusion features to the output features of the sixth layer to obtain the integrated features.

Images