End-to-end non-autoregressive Chinese speech recognition method integrating pinyin and character information

Abstract

The present invention discloses an end-to-end non-autoregressive Chinese speech recognition method that integrates pinyin and character information. The method learns pinyin and character information simultaneously and applies them to three components. The first is an encoder module based on wav2vec2, which encodes the original audio signal into a speech feature vector; the second is a frame-word alignment module based on an attention mechanism, which aligns the audio vector with the expected output text; and the last is a BERT decoder module based on pinyin phoneme pre-training, which fuses the pinyin and character-level feature vectors obtained in the previous step into the BERT decoder for decoding and outputs text. Compared with the prior art, the present invention starts with the characteristics of Chinese, introduces pinyin-level feature vectors, and uses a BERT decoder based on large-scale pinyin phoneme pre-training to strengthen speech representation, effectively improving the accuracy of Chinese speech recognition.

Description

Technical Field

[0001] The present invention belongs to the field of speech signal semantic representation and speech signal recognition, and in particular relates to an end-to-end non-autoregressive Chinese speech recognition method integrating pinyin information. Background Art

[0002] With the continuous development of artificial intelligence and big data, speech recognition has become a key technology in the current development of artificial intelligence. Its primary research goal is to enable machines to recognize and understand speech signals and convert them into corresponding commands or text. The continuous advancement and development of speech recognition is key to computer intelligence. With the continuous improvement of technology, speech recognition technology has been widely used in daily life, such as smart homes, intelligent customer service, and voice assistants. At the same time, speech recognition technology is playing an increasingly important role in human-computer interaction, natural language processing, intelligent driving and other fields. It is foreseeable that with the further development and application of technology, speech recognition technology will be widely used and promoted in even more fields.

[0003] It's estimated that there are over 7,000 languages ​​spoken worldwide, each with its own unique characteristics and modes of expression. Therefore, it's difficult to directly migrate speech recognition systems based on one language to another. The ever-changing nature of language expressions and the isolation between languages ​​also contribute to the challenges of speech recognition. Currently, most mainstream speech recognition model frameworks are based on English grammatical patterns, which presents numerous challenges when migrating to Chinese language recognition. Therefore, designing a speech recognition algorithm specifically for Chinese is crucial. Chinese has complex prosody, intonation, and tone, and its word combinations are also flexible. Therefore, Chinese speech recognition algorithms must fully consider these characteristics and be designed in conjunction with Chinese grammar and language patterns. Research and development of speech recognition algorithms tailored to Chinese can effectively improve the accuracy and stability of Chinese speech recognition, further promoting the application and development of speech recognition technology in Chinese environments.

[0004] Recently, much work on end-to-end speech recognition has utilized an encoder-decoder module to directly input raw speech audio and generate the final output, thus avoiding the complex steps of building language and dictionary models. This approach not only simplifies the model structure but also effectively reduces the problem of error propagation in speech recognition, improving recognition accuracy and stability. Furthermore, with the popularity of transformers, many studies have introduced attention-based modules, achieving excellent results. This attention mechanism not only helps the model better understand the input speech signal but also effectively processes input sequences of varying lengths. Compared to traditional HMM- or DNN-based speech recognition algorithms, end-to-end and transformer-based speech recognition algorithms offer stronger generalization and adaptability, and therefore hold broader application prospects in practical applications.

[0005] While end-to-end and transformer-based speech recognition algorithms have achieved promising results, they still have some limitations. Chinese speech recognition is unique in its expression. Unlike English, the same text in Chinese can convey completely different meanings through different pronunciations. Therefore, in Chinese speech recognition, directly translating only by incorporating character information will reduce the expressive power of the original audio, ultimately affecting speech recognition accuracy. Summary of the Invention

[0006] To address the shortcomings of the existing technology, this paper proposes an end-to-end non-autoregressive Chinese speech recognition method that integrates phonetic and character information based on BERT phonetic-phoneme pre-training. This method simultaneously extracts phonetic and character features, fully considering the semantic information of the Chinese language. Furthermore, parallel decoding is achieved using BERT, which is based on large-scale phonetic-phoneme pre-training, as a decoder. This decoder learns the relationships between a large number of phonetic phonemes and ultimately outputs text. This approach achieves improved recognition accuracy and real-time rate.

[0007] The present invention proposes an end-to-end non-autoregressive Chinese speech recognition method that integrates pinyin and character information, comprising the following steps:

[0008] Step 1: Take the original speech signal as input and pass it through the self-supervised wav2vec2 speech encoder to obtain the encoded audio vector H a ={h1,h2,h3,…,h T}, where h i Represents the feature vector expression of each speech frame, i = 1, 2, ..., T, T represents the number of speech frames.

[0009] Step 2: First, use the CTC greedy search algorithm to predict the length of the target text, and perform position encoding on the predicted target text of length L to obtain H p =PositionalEncoding({x1,x2,x2,…,x L}). (PositionalEncoding(.) indicates position encoding calculation, H p Represents the text position encoding vector, x j Represents the position code of the jth character, j = 1, 2, ..., L, L represents the length of the character) Then, the obtained character position code vector H p and the audio vector H encoded in step 1 a The frame-word length alignment module is used to align the speech frames and characters to obtain the word-level audio vector H = {h′1,h′2,h′3,…,h′ L}. Among them, h′ j Represents the j-th audio feature expression after alignment with the output text.

[0010] The specific operations of the frame-word length alignment module are as follows:

[0011] The text position encoding vector H with a length of L after position encoding is p As the Query in the attention mechanism, the audio vector H encoded by wav2vec2 is a As the Key and Value in the attention mechanism. Perform attention operation on Query, Key, and Value to obtain the word-level audio vector H = Attention (H p ,H a ,H a ). The attention operation uses a dot-product attention mechanism:

[0012] Among them, softmax(.) represents the normalized exponential activation function; Q, K, and V are the abbreviations of Query, Key, and Value respectively; K T Represents the transpose of Key; d k Represents the dimension of the hidden layer of the audio vector.

[0013] Step 3: The word-level audio vector H obtained above is processed through two different fully connected layers: Through the fully connected layer 1 (FullConnectLayer1, hereinafter referred to as FC1), the character-level feature embedding vector H is obtained char(char_embedding); through the full connection layer 2 (FullConnectLayer2, hereinafter referred to as FC2), the pinyin level feature embedding vector H is obtained pinyin (pinyin_embedding). Note: This paper uses the open source pypinyin toolkit to convert the input word-level audio vector H into the corresponding pinyin-level feature embedding vector H pinyin .

[0014] Step 4: H obtained in step 3 char (char_embedding) and H pinyin (pinyin_embedding) performs feature fusion through the pinyin-character fusion layer, i.e., the concat operation, to obtain the feature embedding vector H that simultaneously integrates character and pinyin information. fusion (fusion_embedding). It is input into the BERT non-autoregressive decoder based on pinyin phoneme pre-training for decoding, and finally the predicted text is output.

[0015] The present invention constructs an end-to-end non-autoregressive Chinese speech recognition algorithm model, which includes three components: a wav2vec2 speech encoder based on self-supervision, a frame-word length alignment module, and a BERT non-autoregressive decoder based on pinyin phoneme pre-training. The method constructed by the present invention can directly read the original audio signal and obtain the audio vector representation of each speech frame through wav2vec2 encoding; then, through the frame-word length alignment module, the attention mechanism is used to align the length of the speech frame with the length of the output character; then, the word-level audio vector obtained by the attention mechanism is passed through different fully connected layers to obtain the character-level feature embedding vector H char (char_embedding) and the pinyin-level feature embedding vector H pinyin (pinyin_embedding). Finally, H char and H pinyin Get H through the pinyin-character fusion layer fusion The input is fed into the BERT non-autoregressive decoder based on large-scale pinyin phoneme pre-training for parallel decoding, and the output is text.

[0016] The present invention provides a special input method for Chinese speech recognition. The same word is pronounced differently in different occasions and can contain different meanings. In order to enhance the information content, pinyin feature embedding is introduced on the basis of ordinary character feature embedding. The pinyin feature embedding and character feature embedding are fused and input into the decoder, so that the entire audio signal is enriched with more semantic information. At the same time, the present invention uses a BERT decoder based on large-scale pinyin phoneme pre-training in the decoder stage. Compared with the traditional BERT model, the BERT model introduces the learning task of pinyin phonemes and learns the relationship between pinyin contexts and the association between pinyin semantics through a large amount of pinyin text. The use of the pinyin phoneme-based BERT decoder can better decode feature vectors and improve the accuracy of speech model recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of the speech recognition method of the present invention;

[0018] Figure 2 is a schematic diagram of a model adopted by the speech recognition method of the present invention;

[0019] Figure 3 It is a model structure diagram of the frame-word length alignment module of the present invention;

[0020] Figure 4 It is a structural diagram of the implementation of pinyin embedding of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be further described below in conjunction with the brief description of the above invention content and the accompanying drawings.

[0022] The present invention proposes an end-to-end non-autoregressive Chinese speech recognition method that integrates pinyin and character information. The method is implemented based on an end-to-end non-autoregressive Chinese speech recognition model. The model includes a wav2vec2 speech encoder based on self-supervision, a frame-word length alignment module, and a BERT non-autoregressive decoder based on pinyin phoneme pre-training. The Chinese speech recognition method specifically includes the following steps:

[0023] Step 1: Take the original speech signal as input and pass it through the self-supervised wav2vec2 speech encoder to obtain the encoded audio vector H a ={h1,h2,h3,…,h T} (T×d) , where h i Represents the feature vector expression of each speech frame, i = 1, 2, ..., t, T represents the number of speech frames, and d represents the hidden layer dimension of each audio vector.

[0024] Step 2: First, use the CTC greedy search algorithm to predict the length of the target text, and perform position encoding H on the predicted target text of length L. p =PositionalEncoding({x1,x2,x3,…,x L}). (PositionalEncoding(.) indicates position encoding calculation, H p Represents the text position encoding vector, x j Represents the position code of the jth word, L represents the length of the word) Then, the obtained position code vector H p and the audio vector H encoded in step 1 a The frame-word length alignment module performs frame-word alignment operation to obtain the character-level audio vector H = {h′1, h′2, h′3, …, h′ L}. h′ j Represents the j-th audio feature expression after alignment with the output text, j = 1, 2, …, L.

[0025] The specific operations of using the frame-word alignment module are as follows:

[0026] The text position encoding vector H p (L×d) As the Query in the attention module, the encoded audio vector H a (T×d) As Key and Value. Computational operations.

[0027] Among them, L represents the length of the text, T represents the length of the speech frame, d represents the hidden layer dimension of the audio vector and the text vector; Q, K, and V are the abbreviations of Query, Key, and Value in the attention module respectively; d k Represents the hidden layer dimension of the audio vector; softmax(.) is the normalized exponential activation function.

[0028] pass Operation, each output position x will be obtained j All different input speech frames h on (j=1,2,…,L) i (i=1,2,3,…,T) attention scores. In particular, for each x j , corresponding to all speech frames h i The sum of the attention scores is 1. The purpose of this operation is to get each text output step x j and different input speech frames h i Then, multiply it by V, that is, each x j Corresponding different input speech frames h iThe weights and each input speech frame h i The value obtained in each dimension contains the current output x j And the relationship between all speech frames. Through this operation, the output vector H = {h′1,h′2,h′3,…,h′ L} (L×d) . Each h′ j (j=1,2,3,…,L) is all x j Corresponding input speech frame h i The weighted sum of weights, L represents the length of the output text, and d represents the dimension of the hidden layer.

[0029] Take the output vector x0 at the 0th position as an example: the vector x0 at the 0th position and the audio vector H a The inner product of each audio vector in is performed, and then T attention weights are obtained through the softmax activation function, where the sum of T attention weights is 1. The advantage of the frame-word alignment module is that when the proposed network framework is trained to convergence, that is, the audio vector of the speech frame is learned, each output vector x j The attention weights can learn the corresponding input speech frame h according to their position in the output text i The output vector x0 corresponds to the first position of the output vector. The attention weight corresponding to x0 may be expressed as follows: the first three to five frames of speech frames contribute more than 90% of the weight, while the remaining speech frames contribute less than 10%. This operation allows the input speech frame to learn the connection between the position of the output vector. Then, the attention weight is weighted and summed with Value (T audio vectors) to obtain the word-level audio vector h corresponding to the output vector x0. ′ 0. By performing the same attention operation on the output vector at each output step and obtaining the attention weight of the corresponding speech frame, we can obtain the L-word-level audio vector H = {h′1,h′2,h′3,…,h′ L}, each word-level audio vector represents the importance of different speech frames corresponding to the current output word.

[0030] Step 3: Substitute the above-obtained word-level audio vector H = {h′1, h′2, h′3, …, h′ L} respectively through two different fully connected layers. Figure 1 As shown, the dimension of the fully connected layer 1 (FC1) is (hidden_size, hidden_size), where hidden_size is the user-defined hidden layer dimension, which is used to obtain the character-level feature embedding vector H char(char_embedding); the dimension of the fully connected layer 2 (FC2) is (hidden_size, max_pinyin_num), where max_pinyin_num is the maximum length in the pinyin table, in order to obtain the pinyin-level H pinyin (pinyin_embedding).

[0031] Among them, the open source pypinyin toolkit is used in the process of obtaining the pinyin feature embedding vector, and the implementation method is as follows Figure 3 As shown, first, argmax is taken through a fully connected layer of dimension (hidden_size, max_pinyin_num) to obtain seq_len pinyins, where seq_len represents the length of the output text. Second, the obtained pinyins are converted into a letter matrix of dimension (seq_len, pinyin_locs), where each letter in the pinyin occupies one position, and pinyin_locs is the set maximum letter dimension. Then, the obtained matrix is ​​input into the letter embedding layer to obtain a letter embedding matrix of dimension (seq_len, pinyin_locs, embedding_size). Finally, the last two dimensions of the letter embedding matrix are swapped, and a one-dimensional convolution is used to slide the last dimension of the letter embedding matrix, and maximum pooling is performed. The input and output channels of the convolution are set to embedding_size and pinyin_out_dim respectively, and the final output is a dimension of (seq_len, pinyin_out_dim), obtaining the pinyin embedding corresponding to each character.

[0032] Step 4: The H obtained by different fully connected layers are char (char_embedding) and H pinyin (pinyin_embedding) is spliced ​​through the pinyin-character fusion layer to obtain the final fusion embedding H of the fusion character and pinyin information fusion (fusion_embedding)

[0033] H fusion =W[H char :H pinyin ]

[0034] Among them H char The dimensions are (L×d), H pinyin The dimension is (L×d), W is the learnable parameter of the pinyin-character fusion layer, the dimension is (L×2L), H fusion The dimension is (L×d), and [a:b] represents the concat operation.

[0035] Step 5: H fusionThe input is decoded in the BERT non-autoregressive decoder pre-trained based on pinyin phonemes. Using the BERT bidirectional parallel decoding mechanism and the model parameters pre-trained based on a large amount of pinyin and text data, the speech feature H that combines characters and pinyin is decoded. fusion Decode, output text, and complete the speech recognition task.

[0036] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. An end-to-end non-autoregressive Chinese speech recognition method integrating pinyin and character information, characterized in that: The method is implemented based on an end-to-end non-autoregressive Chinese speech recognition model, which includes a wav2vec2 speech encoder based on self-supervision, a frame-word length alignment module, and a BERT non-autoregressive decoder based on pinyin phoneme pre-training. The Chinese speech recognition method includes the following steps: Step 1: Take the original speech signal as input and pass it through the self-supervised wav2vec2 speech encoder to obtain the encoded audio vector H a ={h1,h2,h3,…,h T }, where h i represents the feature vector representation of the i-th speech frame, i = 1, 2, ..., T, T represents the number of speech frames, and the hidden layer dimension of each audio vector is d; Step 2: First, use the CTC greedy search algorithm to predict the length of the target text, and perform position encoding on the predicted target text of length L to obtain H p =PositionalEncoding({x1,x2,x3,…,x L }), where PositionalEncoding(.) represents position encoding calculation, H p Represents the text position encoding vector, x j Represents the position code of the jth character, j = 1, 2, ..., L, L represents the length of the character; then, the obtained character position code vector H p and the audio vector H encoded in step 1 a The frame-word length alignment module is used to align the speech frames and characters to obtain the word-level audio vector H = {h′1,h′2,h′3,…,h′ L }, where h′ j Represents the j-th audio feature expression after alignment with the output text; Step 3: Process the above-obtained word-level audio vector H through two different fully connected layers: Through fully connected layer 1, the character-level feature embedding vector H is obtained. char ; Through the fully connected layer 2, the pinyin-level feature embedding vector H is obtained pinyin ; Step 4: H obtained in step 3 char and H pinyin The feature fusion is performed through the pinyin-character fusion layer, i.e., the concat operation, to obtain the feature embedding vector H that simultaneously integrates the character and pinyin information. fusion ; Step 5: H fusion The input is decoded in the BERT non-autoregressive decoder based on pinyin phoneme pre-training, and the feature embedding vector H that integrates character and pinyin information is decoded using the BERT bidirectional parallel decoding mechanism and the model parameters that have been pre-trained based on a large amount of pinyin and text data. fusion Decode and output predicted text to complete the speech recognition task.

2. The end-to-end non-autoregressive Chinese speech recognition method integrating pinyin and character information according to claim 1, characterized in that: The frame-word length alignment module specifically operates as follows: The text position encoding vector H with a length of L after position encoding is p As the Query in the attention mechanism, the audio vector H encoded by wav2vec2 is a As the Key and Value in the attention mechanism, the attention operation is performed on Query, Key, and Value to obtain the word-level audio vector H corresponding to the original speech frame and the output text. p ,H a ,H a ), the attention operation uses the dot product attention mechanism: Among them, softmax(.) represents the normalized exponential activation function; Q, K, and V are the abbreviations of Query, Key, and Value respectively; K T Represents the transpose of Key; d k Represents the dimension of the hidden layer of the audio vector.

3. The end-to-end non-autoregressive Chinese speech recognition method integrating pinyin and character information according to claim 2, characterized in that: The step 3 specifically includes: The obtained word-level audio vector H = {h′1,h′2,h′3,…,H′ L } respectively through two different fully connected layers. The dimension of the fully connected layer 1 is (hidden_size, hidden_size), where hidden_size is the user-defined hidden layer dimension, which is used to obtain the character-level feature embedding vector H char The dimension of the fully connected layer 2 is (hidden_size, max_pinyin_num), where max_pinyin_num is the maximum length in the pinyin table, which is used to obtain the pinyin-level feature embedding vector H pinyin ; Among them, the open source pypinyin toolkit is used to obtain the pinyin feature embedding vector. First, argmax is taken through the fully connected layer 2 with a dimension of (hidden_size, max_pinyin_num) to obtain seq_len pinyins, where seq_len represents the output text length; secondly, the obtained pinyin is converted into a letter matrix with a dimension of (seq_len, pinyin_locs), where each letter in the pinyin occupies one position and pinyin_locs is the set maximum letter dimension; then, the obtained matrix is ​​input into the letter embedding layer to obtain a letter embedding matrix with a dimension of (seq_len, pinyin_locs, embedding_size); finally, the last two dimensions of the letter embedding matrix are exchanged, and a one-dimensional convolution is used to slide on the last dimension of the letter embedding matrix, and maximum pooling is performed; the input channel and output channel of the convolution are set to embedding_size and pinyin_out_dim respectively, and finally the output is output with a dimension of (seq_len, pinyin_out_dim) to obtain the pinyin embedding corresponding to each character.

4. The end-to-end non-autoregressive Chinese speech recognition method integrating pinyin and character information according to claim 3, characterized in that: The step 4 specifically includes: H fusion Specifically expressed as: H fusion =W[H char ;H pinyin ] Among them H char Dimensions are L×d, H pinyin The dimension is L×d, W is the learnable parameter of the pinyin-character fusion layer, the dimension is L×2L, H fusion The dimension is L×d, and [a:b] represents the concat operation.

Images