A method for mixed English-Chinese speech recognition

Abstract

The present application relates to the technical field of speech processing, in particular to a Chinese-English mixed speech recognition method.The present application comprises the following steps: step S1, obtaining acoustic features through preprocessing on Chinese-English mixed audio, using a Conformer model to respectively encode and learn the embedding of relative position information of Chinese and English feature parts, and then mixing the features after encoding and learning into a decoder part; step S2, using the decoder part of a pre-trained generation model as a GPT-Decoder decoder, and simultaneously using a GPT-Decoder decoder, a connection time sequence classification CTC module and a long short-term memory recurrent neural network LSTM-RNNLM model to form a joint decoder; wherein the connection time sequence classification CTC module and the long short-term memory recurrent neural network LSTM-RNNLM model assist the GPT-Decoder decoder in decoding.The encoder part in the present application introduces a Conformer model as a Chinese-English double encoder, replacing the traditional Transformer encoder.The decoder part in the present application adopts a joint decoder form instead of the original single decoder structure.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, and specifically to a method for mixed Chinese and English speech recognition. Background Technology

[0002] Mixed Chinese-English speech recognition refers to the technology of speech recognition when Chinese and English are mixed in the same speech. The challenges of mixed Chinese-English speech recognition mainly come from two aspects: first, the diversity of speech signals, as Chinese and English speech signals differ in acoustic features, such as pitch, timbre, and speech rate; second, the establishment of language models, as Chinese and English grammar and vocabulary also differ, such as word order, parts of speech, and grammatical rules.

[0003] To address the challenges of mixed Chinese-English speech recognition, researchers have proposed numerous methods. One common approach is to use a bilingual hybrid model, which combines speech signals from both languages ​​with a language model for recognition. This method improves accuracy but requires substantial training data and computational resources. Another approach utilizes the features of the speech signals. Researchers have discovered differences in the acoustic features of Chinese and English speech signals. Therefore, by analyzing and extracting these features, the speech signals can be distinguished between Chinese and English. This method reduces reliance on language models but requires more sophisticated feature extraction and classification algorithms. Besides these two methods, other approaches exist, such as using deep learning algorithms for speech recognition and employing speech segmentation and alignment techniques. These methods address the challenges of mixed-language speech recognition to varying degrees, but further research and improvement are still needed. In conclusion, mixed Chinese-English speech recognition is a challenging problem but also a field with broad application prospects. Continued research and exploration are needed to develop more sophisticated feature extraction and classification algorithms, as well as more efficient language model building methods and new modular structures to improve the accuracy and efficiency of mixed Chinese-English speech recognition. Summary of the Invention

[0004] To overcome the shortcomings of the existing technology, this invention proposes a Chinese-English mixed speech recognition method.

[0005] The technical solution adopted in this invention is: a Chinese-English mixed speech recognition method, comprising the following steps:

[0006] Step S1: Preprocess the Chinese and English mixed audio to obtain acoustic features, use the Conformer model to encode and learn the relative position information embedded in the Chinese and English feature parts respectively, and then input the encoded and learned features into the decoder part.

[0007] Step S2: Use the decoder part of the pre-trained generative model as the GPT-Decoder, and at the same time, the joint decoder is composed of the GPT-Decoder, the Connected Temporal Classification (CTC) module, and the Long Short-Term Memory Recurrent Neural Network (LSTM-RNNLM) model; wherein, the Connected Temporal Classification (CTC) module and the Long Short-Term Memory Recurrent Neural Network (LSTM-RNNLM) model assist the GPT-Decoder in decoding.

[0008] As a preferred technical solution of the present invention: In step S1, the preprocessed acoustic features are synchronously trained using a Conformer encoder to generate two independent Chinese encoders and English encoders, and the location information of each language is combined with the encoding during training.

[0009] As a preferred technical solution of the present invention: in step S2, the sequence output by each trained language encoder is combined with the language position information as the input of the joint decoder.

[0010] As a preferred technical solution of the present invention: in step S2, the Connected Temporal Classification (CTC) module is responsible for the alignment task of sequence decoding, and the Long Short-Term Memory Recurrent Neural Network (LSTM-RNNLM) model is responsible for assisting the GPT-Decoder in adapting to the decoding of variable-length input sequences.

[0011] As a preferred technical solution of the present invention: In step S2, the GPT-Decoder first performs unsupervised pre-training on the input sequence, and at the same time uses the Connected Temporal Classification (CTC) module to perform alignment operation on the decoded sequence.

[0012] As a preferred technical solution of the present invention: in step S2, the GPT-Decoder performs supervised fine-tuning, while the LSTM-RNNLM module assists the GPT-Decoder in decoding and outputting the recognized text sequence.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0014] (1) The encoder part of this invention uses the Conformer model as the encoder. By introducing multi-layer convolutional modules to extract deeper feature information, the sequence length is reduced, thereby reducing the complexity of the model and facilitating the rapid training and convergence of the model. The encoder part adopts a joint decoding architecture. The CTC and LSTM-RNNLM models assist the GPT-Decoder in decoding the serialized input, which significantly speeds up the inference speed of the model.

[0015] (2) The RNN structure of the LSTM-RNNLM model of the present invention can pass information from the past to the present time, solve the problem of context dependency, and make the model more adaptable to decoding of variable-length sequence inputs; while multiple LSTM unit networks can control the flow of information through the gate mechanism, thereby avoiding the gradient vanishing problem. Attached Figure Description

[0016] Figure 1 This is a flowchart of the Chinese-English mixed speech recognition method of the present invention;

[0017] Figure 2 This is a structural diagram of the single-layer Conformer model encoder of the present invention;

[0018] Figure 3 This is a structural diagram of the GPT model decoder of the present invention;

[0019] Figure 4 This is a structural diagram of the LSTM-RNNLM model of the present invention. Detailed Implementation

[0020] The present invention will be further explained in detail below with reference to the accompanying drawings, so that those skilled in the art can better understand and implement the present invention. However, the following examples are only used to explain the present invention and are not intended to limit the present invention.

[0021] like Figure 1 As shown, this invention proposes a method for mixed Chinese and English speech recognition, comprising the following steps:

[0022] S1: Using the TALCS mixed Chinese-English audio set as the training dataset, the audio data is first preprocessed and acoustic features are extracted. Then, the Conformer model is used as a dual Chinese-English encoder to serialize the input features and combine them with positional information. Figure 1 The hybrid encoder is shown in the figure;

[0023] S101: Performs preprocessing operations such as pre-filtering, signal conversion, pre-emphasis, framing, windowing, and endpoint detection on mixed Chinese and English audio, and then extracts acoustic features.

[0024] S102: A 12-layer continuous Conformer block structure is used to encode the language features of Chinese and English languages ​​in combination with positional information. The Conformer block structure is as follows: Figure 2 As shown, a Conformer block structure includes: a feedforward network module, a multi-head self-attention module, a convolution module, and a second feedforward network module. Figure 2 The mathematical representation of the quartic residual connection is as follows:

[0025]

[0026] Here, FFN refers to the feedforward network module; MHSA represents the multi-head self-attention mechanism module; Conv is the convolution module; Layernorm indicates layer normalization; x i , x′ i 、x″ i Indicates intermediate output, y i This indicates the final output.

[0027] S2: The language encoder output sequence, plus language location information, is used as input. A joint decoder is used to decode the input sequence, where the CTC module is responsible for sequence alignment during decoding, and the LSTM-RNNLM module assists the GPT-Decoder in adapting to the decoding of variable-length input sequences and resolving context dependency issues.

[0028] The S201:GPT decoder first performs unsupervised pre-training on the input sequence, and then uses the CTC module to align the decoded sequence. The GPT decoder employs a 12-group two-layer Transformer structure, as follows: Figure 3 As shown, mathematically it can be expressed as:

[0029]

[0030] where U=(u -k , ..., u -1 ) is the context token of the current time slice, n is the layer number, and W is the layer number. e It is a word embedding matrix, W p It is the position embedding matrix, where P(u) represents the probability of candidate word u, and softmax is the normalization.

[0031] GPT's unsupervised pre-training is based on a language model, which uses an LSTM-RNNLM model consisting of 512... Figure 4 The unit structure shown is composed of... Given an unlabeled sequence u = {u1, ..., u...} n The optimization objective of the language model is to maximize the following likelihood value:

[0032]

[0033] Where k is the size of the sliding window, P is the conditional probability, and Θ is the model parameters. These parameters are optimized using the Stochastic Gradient Descent (SGD) algorithm.

[0034] S202: After unsupervised pre-training, supervised fine-tuning is then performed. For a labeled dataset... Each instance has m input tokens: {x 1 The tokens {x, ..., xw} form a tag sequence. First, these tokens are input into a pre-trained model to obtain the final feature vector. Then, the predicted result y is obtained through a fully connected layer:

[0035]

[0036] Among them W y These are the parameters for the fully connected layer.

[0037] The optimization objective of the language model at this point is:

[0038]

[0039] However, instead of directly using the L2 loss function, we incorporate the L1 loss function and use λ to adjust the weights for both tasks. This better unifies and integrates the audio feature sequences, thereby improving the model's generalization ability. The value of λ is set to 0.5.

[0040]

[0041] When performing supervised fine-tuning, only the output layer W needs to be trained. y The embedding values ​​of the delimiter reduce computational overhead to some extent, which helps to shorten model training time.

[0042] The encoder part of this invention introduces the Conformer model as a Chinese-English dual encoder, replacing the traditional Transformer encoder.

[0043] The decoder part of this invention adopts a joint decoder form consisting of a GPT-Decoder, a connection-based temporal classification (CTC) module, and a long short-term memory recurrent neural network (LSTM-RNNLM) model, instead of the original single decoder structure.

[0044] The specific implementation schemes described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific implementation schemes of the present invention and are not intended to limit the scope of the present invention. Any equivalent changes and modifications made by those skilled in the art without departing from the concept and principles of the present invention should fall within the scope of protection of the present invention.

Claims

1. A method for mixed Chinese and English speech recognition, characterized in that, Includes the following steps: Step S1: Preprocess the Chinese and English mixed audio to obtain acoustic features, use the Conformer model to encode and learn the relative position information embedded in the Chinese and English feature parts respectively, and then input the encoded and learned features into the decoder part. Step S2: Use the decoder part of the pre-trained generative model as the GPT-Decoder, and at the same time, the joint decoder is composed of the GPT-Decoder, the Connected Temporal Classification (CTC) module, and the Long Short-Term Memory Recurrent Neural Network (LSTM-RNNLM) model; wherein, the Connected Temporal Classification (CTC) module and the Long Short-Term Memory Recurrent Neural Network (LSTM-RNNLM) model assist the GPT-Decoder in decoding; In step S1, the Conformer encoder is used to synchronously train two independent Chinese encoders and English encoders from the preprocessed acoustic features, and the location information of each language is combined with the encoding during training. In step S2, the sequence output by each trained language encoder, plus the language location information, is used as the input to the joint decoder. In step S2, the Connected Temporal Classification (CTC) module is responsible for the alignment task of sequence decoding, and the Long Short-Term Memory Recurrent Neural Network (LSTM-RNNLM) model is responsible for assisting the GPT-Decoder in adapting to the decoding of variable-length input sequences. In step S2, the GPT-Decoder first performs unsupervised pre-training on the input sequence, and at the same time uses the Connected Temporal Classification (CTC) module to perform alignment operations on the decoded sequence; In step S2, the GPT-Decoder performs supervised fine-tuning, while the LSTM-RNNLM module assists the GPT-Decoder in decoding and outputting the recognized text sequence.

Images