A speech recognition method based on phoneme embedding

Abstract

The invention discloses a speech recognition method based on phoneme embedding, which comprises: taking the first m layers of an m+n-layer encoder Encoder as an acoustic model for phoneme information learning, and using target language data and other language data to train the acoustic model; taking the last n layers and a decoder Decoder as a language model, and combining the contextual semantic information output by the first m layers for the recognition of the target language, and using the plain text data of the target language to train the language model; learning the output of the middle layer of the encoder by learning the phoneme information embedding, and after the phoneme information embedding learning, using a large amount of plain text data to produce a phoneme sequence, and generating a feature sequence close to the middle output of the encoder through the phoneme information embedding, and passing it into the language model, and outputting a real text label through the training of the decoder. A large amount of text data is effectively used to train the model, and the problem that the decoder depends on the output of the encoder is solved.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition, and in particular to a speech recognition method based on phoneme embedding. Background Art

[0002] Speech recognition is the use of computers to automatically map speech sequences to text sequences. It is one of the most natural ways of human-computer interaction. Speech interaction plays an important role in search, navigation, control, and entertainment. There are more than 5,000 languages ​​in the world, but only Chinese, English, and Russian are used by a large population. Other languages ​​are difficult to collect manually or have high labor costs, resulting in a lack of speech resources. Speech recognition technology has gone through three stages: the traditional GMM-HMM (Gaussian mixture-hidden Markov model) model, DNN-HMM (deep neural network-hidden Markov model hybrid model), and end-to-end model. The end-to-end model uses the Encoder-Decoder (encoder-decoder) structure or the Encoder-CTC (continuous time classification) structure. The end-to-end model can solve the problem of unequal length between sequences. The Encoder encodes the acoustic signal into effective features, and the Decoder can use the Encoder output to iteratively generate the final text. CTC solves the problem of audio-to-text alignment by introducing blank labels. End-to-end can directly convert audio data into text, and integrate the acoustic model and language model into a unified model. The end-to-end model has greatly improved speech recognition performance, achieved a lower recognition error rate, and can overcome certain noise, accent, dialect and other problems.

[0003] However, the current end-to-end model training relies on a large amount of manually annotated paired data, and the recognition effect is not ideal in languages ​​with scarce data. How to solve the problem of data scarcity has become a research focus.

[0004] To address the problem of data scarcity, commonly used methods include: 1. Transfer learning, which involves migrating a model trained in a language with abundant data resources to a similar language with low data resources, often leading to improved recognition accuracy; 2. Self-supervised pre-training models, which utilize a large amount of available unlabeled audio data for self-supervised training, allowing the model to learn good speech representations before supervised fine-tuning; 3. Allowing the model to utilize a large amount of available unpaired text data, for example: using a language model, using speech synthesis technology to generate audio data, and using text data to pre-train the Decoder that acts as the language model.

[0005] The purpose of text pre-training decoder is to make the decoder independent of audio data so that the model can learn more text information and make the output of the speech recognition model more grammatical. However, in the past text pre-training decoder experiments, the problem of decoder dependence on encoder output has not been solved well. The previous method used artificial features in encoder output or used the embedding sequence of text in decoder instead of encoder output. This is very different from the real encoder output, which reduces the efficiency of using text to train decoder.

[0006] In the past, when using other multilingual data to improve the effect of low-resource language recognition, transfer learning was often used. Phonemes are shared between multiple languages, which is conducive to the learning of the acoustic part of the model. The more pronunciation units are shared between languages, the greater the performance hint. However, in the past, the model did not achieve the structure of the acoustic model and the language model during transfer learning. The text information of multilingual data has differences in grammar, phonetics and semantic mapping, which will lead to the degradation of the performance of the part of the language model in the speech recognition model. Summary of the invention

[0007] The present invention provides a speech recognition method based on phoneme embedding. The present invention decouples the encoder, uses the first m layers as phoneme extractors, and the last n layers as text prediction modules that consider context information. The PhoneEmbedding generation network is designed to fit the output of the first m layers of the encoder. The present invention provides a method for pre-training a model using unpaired text data and other language data, as described below:

[0008] A speech recognition method based on phoneme embedding, the method comprising:

[0009] The first m layers of the m+n-layer encoder are used as acoustic models for phoneme information learning. The target language data and other language data can be used to train the acoustic model. The last n layers and the decoder are used as language models. The contextual semantic information output by the first m layers can be combined for target language recognition. The language model can be trained using plain text data of the target language.

[0010] By learning the phoneme information embedding, we learn the output of the Encoder's intermediate layer. After the phoneme information embedding is learned, a large amount of plain text data is used to produce phoneme sequences. After the phoneme information is embedded, a feature sequence close to the Encoder's intermediate output is generated, which is passed into the language model and outputs the real text label through Decoder training.

[0011] The method adds a phoneme-based CTC classification network to the middle layer of the Encoder, so that the Encoder can display the learning phoneme information.

[0012] Furthermore, the method further comprises:

[0013] During the training process, the "audio-phoneme label" data of other languages ​​participate in the training together with the "audio-phoneme label-text label" data of the target language, and the data appear alternately in different batches.

[0014] The output of the encoder intermediate layer is learned by learning the phoneme information embedding:

[0015] The phoneme sequence generated by the text is encoded and passed to the generative network for phoneme information embedding. The output features of the generative network are passed to the linear classifier, which classifies the output features to obtain the probability distribution of each feature belonging to the phoneme.

[0016] After the Softmax function, the MSE loss function is calculated for the one-hot vector corresponding to each phoneme, and the feature parameters of the Phone Embedding generation network are updated during the gradient backpropagation.

[0017] Furthermore, the method further includes: a fine-tuning stage for embedding the phoneme information, specifically:

[0018] The generative network is added to the network of the first stage. The output of the first m layers of the encoder will pass through the linear classifier of Phone CTC and the Gumbel softmax function to classify each output feature. The classification result is a phoneme sequence corresponding to the input audio. The phoneme sequence passes through the generative network to generate a Phone Embedding sequence. The generated Phone Embedding sequence is compared with the output of the m layers of the encoder to calculate the SME loss, making the Phone Embedding sequence and the output of the middle layer of the encoder closer.

[0019] The beneficial effects of the present invention are as follows:

[0020] 1. The present invention effectively utilizes other multilingual data to assist the phoneme extractor module, and can improve the performance of the target language through phoneme sharing;

[0021] 2. The present invention effectively utilizes a large amount of text data to train the model, solving the problem that the Decoder depends on the Encoder output;

[0022] 3. The present invention brings regularization to Encodder during network training through Phone Embedding to improve performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Schematic diagram of introducing Phone CTC (continuous time classification network with phonemes as labels) into the middle layer of Encoder;

[0024] Figure 2 A diagram of how to use a trained linear classifier to help Phone Embedding generate network initialization.

[0025] Figure 3 A diagram showing the network output and the encoder intermediate layer output being close to each other for Phone Embedding generation;

[0026] Figure 4 A schematic diagram of training the Encoder high-level and Decoder using plain text data;

[0027] Figure 5 This is the overall structure diagram of the model. DETAILED DESCRIPTION

[0028] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0029] Example 1

[0030] The purpose of the embodiment of the present invention is to explore a method for improving the model recognition rate by using unpaired text of the target language and speech data of other languages. The Encoder in the end-to-end model acts as an acoustic model, while the Decoder acts as a language model. A large amount of text data can be used to pre-train the Decoder so that the model Decoder obtains more text information. Other multilingual data can improve the recognition performance of a single language, especially low-resource languages, because there is phoneme sharing between languages. In previous work, the entire Encoder is often regarded as part of the acoustic model, and the Decoder is regarded as a language model. This study further refines the model, treating the first m layers of the Encoder (a total of m+n layers) as acoustic models, and the last n layers and the Decoder as language models, because in the Encoder+CTC speech recognition framework, even without a Decoder, the model can recognize the correct homophones in combination with semantic information after recognizing the pronunciation. Past studies have confirmed that the Encoder will first recognize the phoneme information representing the speech and then recognize the text in combination with the context when converting audio data into text data.

[0031] When using text pre-training models in past work, only the decoder obtained text information during the training process, while the encoder, which also needs to learn contextual semantics, was not pre-trained with text. Decoder training depends on the encoder output. Due to the lack of audio data, it is impossible to produce encoder output that matches the text. In past research work, artificial encoder output was used, which caused a large difference in encoder output between the pre-training stage and the fine-tuning stage using labeled data, resulting in performance degradation. In the past, when using other multilingual data for transfer learning, data in other languages ​​would train the entire network, causing the language model part of the model to learn linguistic information in other languages, and the linguistic information in other languages ​​would cause confusion to the language model, resulting in performance degradation.

[0032] In view of the shortcomings of previous studies, this study decouples the encoder into the phoneme information learning part (the first m layers) and the text recognition part (the last n layers) combined with contextual semantics. The target language text data is used to learn the last n layers of the encoder and the decoder, and other multilingual data is used to learn the first m layers of the encoder. The output of the middle layer of the encoder is learned by learning Phone Embedding (phoneme information embedding). After Phone Embedding is learned, the text can be used to produce phoneme sequences, and then Phone Embedding is used to generate feature sequences close to the middle output of the encoder, which are then passed to the encoder (last n layers) and decoder, so as to achieve the purpose of using only the text encoder (last n layers) and decoder, and the decoder training process obtains a more realistic encoder output.

[0033] The overall structure of the speech recognition model baseline is Conformer (CTC+Attention). Conformer is the mainstream model structure in the end-to-end speech recognition model. Here, Attention represents the attention mechanism between the Decoder and the Encoder. The Decoder generates a text sequence iteratively based on the attention mechanism. After the model training is completed, the word error rate will be used as the evaluation criterion for the recognition result. The method of the embodiment of the present invention is to allow the model to learn phoneme information, Phone Embedding close to the output of the Encoder intermediate layer, and text information on the target language after 4 stages of training on the basis of the baseline model. The first stage is to add the phoneme sequence CTC loss function (Phone CTC) to the output of the Encoder intermediate layer, forcing the Encoder to learn the phoneme information in the intermediate layer. The second stage is to use the linear layer in the first stage network that acts as a phoneme classifier to help learn the Phone Embedding generation network, and then fuse the network that generates the Phone Embedding into the first stage network to further make the Phone Embedding sequence and the Encoder intermediate layer output close to each other. The third stage is to use text data to generate phoneme sequences and add them to the Phone Embedding generation network to generate outputs that are very close to the Encoder middle layer, and then add them to the Encoder back (n) layer to pre-train the model with pure text data. The fourth stage is to fine-tune the entire model using labeled audio data in the target language.

[0034] The training method based on phoneme embedding proposed in the embodiment of the present invention effectively utilizes a large amount of text data and other multilingual data, and obtains a relative performance improvement of 10% on the test set of Librispeech.

[0035] Example 2

[0036] Combine the following Figure 1-Figure 5 , the calculation formula further introduces the scheme in Example 1, as described below:

[0037] 1. Phoneme sequence CTC in the middle layer of Encoder

[0038] In past research on Chinese speech recognition, the embodiments of the present invention found that when a pure (Encoder+CTC) network training model is used, the (Encoder+CTC) model can distinguish homophones with different meanings. Based on this phenomenon, the embodiments of the present invention believe that the Encoder considers the semantic information of the context to distinguish polyphones with the same pronunciation but different meanings, that is, the Encoder network will first learn phoneme information with a relatively low granularity, and then consider the context to distinguish homophones. Therefore, the first few layers of the Encoder will recognize the phoneme information, and the later layers will predict the real text based on the context, but there is no clear phoneme learning layer inside the Encoder network. Therefore, the embodiments of the present invention add a phoneme-based CTC classification network to the middle layer of the Encoder, allowing the Encoder to explicitly learn phoneme information, and bring performance improvements in experiments. After adding the phoneme sequence CTC network, the output features of the middle layer of the Encoder are features that can represent phonemes. The model structure with the Phone CTC loss function added is as follows: Figure 1 shown.

[0039] exist Figure 1 In the figure, Fbank Seq represents the Fbank (filter bank feature) feature extracted from the audio data. The feature is sent to the first 8 layers of the Encoder, and then the output of the 8th layer is sent to the Phone CTC network for phoneme classification to calculate the loss function of Phone CTC. The phoneme classification is done by a linear network. At the same time, the output of the 8th layer is transmitted to the last n layers of the Encoder network, and the output of the last layer of the Encoder is sent to the CTC network to calculate the CTC loss function of the text. At the same time, the output of the last layer of the Encoder is transmitted to the Decoder network, and the output of the Decoder is used to predict the text data.

[0040] The embodiment of the present invention uses labeled speech data in other languages ​​to help pre-train the first m layers of the model network. There are often shared pronunciation units between different languages, and data in other languages ​​can help train the Encoder to act as the first m layers of phoneme feature extraction. The training method is to use the G2P (word to phoneme mapping) model to convert the text sequence of each language into a phoneme sequence, and convert the phoneme into the International Phonetic Alphabet IPA (International Phonetic Alphabet) according to the mapping table of the notes of each language and the International Phonetic Alphabet. Use audio data and the corresponding International Phonetic Alphabet sequence to pre-train the first m layers of the model Encoder. During the training process, the "audio-phoneme label" data of other multiple languages ​​participate in the model training together with the "audio-phoneme label-text label" data of the target language, and the data appears alternately in different batches (a separate batch of data passed to the model).

[0041] 2. How to learn Phone Embedding

[0042] Phone Embedding (phoneme information embedding) encodes all pronunciation units in the target language. This encoding should be similar to the output of the middle layer of the encoder representing the phoneme features. The purpose is to select phone Embedding from the phone Embedding to replace the output of the first m layers of the encoder to train the last m layers of the encoder and the decoder after the phone sequence is generated from the text sequence. Phone Embedding is generated by a Phone Embedding generation network.

[0043] The learning of the Phone Embedding generation network goes through two stages: the initialization stage and the fine-tuning stage after loading into the model.

[0044] 1. Introduction to the initialization phase:

[0045] When calculating the Phone CTC loss, the output of the Encoder intermediate layer passes through a linear classification layer to obtain the posterior probability of the phoneme to which each feature belongs, and then the loss function is obtained through the CTC calculation formula. The linear classification layer is a linear network followed by a softmax layer. Its function is to classify features of dimension N. These features are points distributed in the N-dimensional space. The linear classifier divides the N-dimensional space into a small space for each phoneme category. Features belonging to the same subspace will be classified into the same category by the classifier. In the previous experiment, the embodiment of the present invention added the Phone CTC loss function to the model training convergence and learned a linear phoneme classification network layer. The method of initializing Phone Embedding is to find a center point in the subspace divided by the linear classifier into the N-dimensional space as the Embedding of the phone corresponding to this subspace. The example of the present invention will construct a Phone Embedding generation network so that the Embedidng corresponding to each Phone is in the subspace divided by the linear classifier into the N-dimensional space.

[0046] The learning process is to encode the phoneme sequence generated by the text and pass it to the Phone Embedding generation network. Then the output features of the generation network will be passed to the linear classifier. The linear classifier will classify the features to obtain the probability distribution of the phoneme to which each feature belongs. After the Softmax function, the MSE loss function will be calculated for the one-hot (unique hot encoding) vector corresponding to each phoneme. During the gradient back-conduction, only the feature parameters of the Phone Embedding generation network are updated, and the parameters of the linear classifier are not updated. During the training process, the features of each phoneme of the Phone Embedding generation network are forced to move to the subspace divided by the linear classifier. Based on this, an initialized Phone Embedding generation network will be obtained. In addition, during the training process, part of the phonemes in the input phoneme sequence of the Phone Embedding generation network will be replaced with blank phonemes, but the Phone Embedidng generation network will restore the replaced phonemes, so that the Phone Embedding generation network has a certain degree of robustness. Phone Embedding initialization network is as follows Figure 2 As shown in the figure, the result of the PhoneEmbedding generated network is a layer of Conformer.

[0047] 2. Phone Embedding fine-tuning stage:

[0048] In the Phone Embedding fine-tuning stage, the Phone Embedding generation network is added to the network of the first stage. The output of the first m layers of the Encoder will pass through the linear classifier of Phone CTC to classify each output feature. The classification result is a phoneme sequence corresponding to the input audio. The phoneme sequence passes through the Phone Embedding generation network to generate a Phone Embedding sequence. At this time, the generated Phone Embedding sequence and the output of the Encoder's m layers are used to calculate the SMEloss to make the Phone Embedding sequence and the output of the Encoder's middle layer closer.

[0049] There are some differences between the Phone Embedding sequence generated by the Phone Embedding generation network and the output of the Encoder middle layer. The reason for the difference is that the previous phoneme classifier cannot completely and correctly classify the Encoder output as Phone. In addition, all possible alignments are considered during CTC training. The output features of the Encoder middle layer will be divided into different labels in different possible alignments, which will cause the output features of the Encoder middle layer to have low discrimination.

[0050] In order to reduce this difference, the example of the present invention performs MSEloss between Phone Embedding and the output of the Encoder middle layer. The method adopted by the embodiment of the present invention is to use the Gumbel softmax function to select the corresponding Phone Embedding from the Phone Embedding set according to the output of the Encoder middle layer. The Gumbel softmax function can select the Phone Embedding according to the probability, and the selection process is differentiable. In the fine-tuning stage, the Phone Embedding and the output of the Encoder middle layer are closer. When the output of the Encoder middle layer approaches the screened Phone Embedding, the Encoder is forced to generate phoneme features with higher discrimination. Because the output of the model middle layer is regularized by Phone Embedding, the recognition accuracy is further improved. The process of selecting the Embedding sequence from the Phone Embedding according to the output of the Encoder middle layer is as follows. Figure 3 shown.

[0051] 3. Pre-training network methods using unpaired text data

[0052] Using text data to pre-train the model decoder allows the model to learn linguistic information in advance, but decoder training depends on encoder output, and in the absence of audio data, there is no encoder output. Existing methods use random noise or word embedding for encoder output, which is very different from the actual encoder output, resulting in greater adjustments to model parameters in the fine-tuning phase to adapt to the new encoder output.

[0053] The method of using the embodiment of the present invention is to convert the text sequence into a phoneme sequence according to the G2P (word to phoneme mapping) model, and then the Phone Embedding generation network generates a corresponding Phone Embedding sequence according to the phoneme sequence. The generated Phone Embedding sequence is sent to the Encoder high-level layer, and then the Encoder high-level layer and the Decoder will predict the text sequence, so that a large amount of text data can be used to train the model. Using this method can make the Encoder output that the Decoder training depends on more meaningful. The training process is as follows Figure 4 shown.

[0054] 4. Fine-tuning using labeled audio data in the target language

[0055] In the previous pre-training phase, we used other multilingual data to train the first m layers of the Encoder model, and also used text data in the target language to pre-train the last n layers of the Encoder and the Decoder. Pre-training can give the model a good starting point for training when fine-tuning with labeled data, and prevent the model from entering a local optimum.

[0056] 5. Model configuration and dataset introduction

[0057] The model uses the Conformer result, where the Encoder has 12 layers and the Decoder has 6 layers. It uses Librispeech clean-100h data as a low-resource dataset and 960h audio corresponding text data as additional text data. The model is trained in 4 stages:

[0058] In the first stage of training, the Phone CTC loss function is added to the mixed multi-language data after the first 8 layers of the Encoder to enhance the ability of the first 8 layers of the Encoder to extract phoneme features. The loss function is as follows:

[0059] Loss = 0.7 * Loss att +0.3(Loss ctc +Loss phone-ctc ) (1)

[0060] The second stage is to learn Phone Embedding. After the first stage of training, a separate Phone Embedding generation network is constructed to generate Phone Embedding. During the training process, the parameters of the linear classification layer are not updated, and only the parameters of the Phone Embedding generation network are updated. After the parameters of the Phone Embedding generation network are initialized with the help of the linear classifier and the phoneme sequence, the Phone Embedding generation network is added to the network of the first stage, and the Gumbelsoftmax method is used to identify the phoneme category to which each feature of the encoder output belongs. Then the Phone Embedding generation network generates the Phone Embedding sequence. The Phone Embedding sequence and the encoder output are subjected to MSEloss (mean loss function) to make the Phone Embedding close to the output of the encoder middle layer. This allows the Embedding selected in the subsequent text pre-training stage to be close to the actual encoder middle layer output. At the same time, the output of the encoder middle layer is close to the phone Embedding with higher discrimination, which will make the model output features with high discrimination and obtain regularization.

[0061] The third stage of training is the text pre-training stage. The model modules and parameters are part of the modules and parameters of the second stage. At this time, the text data of Librispeech-960h is used to pre-train the first m layers and decoder of the model. The model input is the phone Embedding sequence selected from the Phone Embedding by the G2P model, and the output is the real text sequence. In order to consider that the text length is less than the encoder output length, the phonemes in the phoneme sequence will be repeated twice, and the phonemes in the phoneme sequence will be replaced or deleted with a certain probability. This makes the training decoder have a certain robustness, and can predict the real text sequence in combination with the context when the Phone Embedding sequence is wrong.

[0062] The fourth stage is the fine-tuning stage, in which the model is fine-tuned using librispeeh-clean-100-hour data.

[0063] Example 3

[0064] The feasibility of the schemes in Examples 1 and 2 is verified in combination with specific data, as described below:

[0065] The experimental process of using multilingual data to train the first 8 layers of the Encoder. In the second stage mentioned above, other multilingual data and low-resource data will appear alternately in adjacent batches, and the batches contain language ids. During the training process, if it is other languages, the model forward calculation will only calculate the first 8 layers of the Encoder and calculate the phone CTC loss function. If it is a low-resource language, the forward calculation process will pass through the entire network. Based on this process, other multilingual data can help the model learn a better phoneme classifier.

[0066] Table 1

[0067] Model Test set WER Baseline test-clean 8.56 Baseline+Phone CTC test-clean 8.49 Baseline+Phone CTC+Phone Embedding test-clean 8.14 Fine-turn after text pretraining test-clean 7.69

[0068] In the table above, Baseline is the training result of the Conformer model using Librispeech 100 hours of data, and Baseline+Phone CTC is the experimental result of adding Phone CTC to the model and training it using multilingual data. Baseline+Phone CTC+Phone Embedding is the result of the model's intermediate output being regularized by Phone Embedding. Fine-turn after text pretraining is the result of the model being fine-tuned after training the model with Librispeech-960h corresponding text data and then with Librispeech 100 hours of data. The results of each stage of the experiment show that the best result achieved a relative improvement of 10.1%.

[0069] Compared with the baseline, the model adds the CTC phone loss function module and the Phone Embedding generation network module in the middle layer of the Encoder. In the four stages of model training, we will focus on learning different modules of the model. In the first stage, we use other multilingual data to enhance the ability of the first 8 layers of the Encoder to learn phoneme features. In the second stage, we learn the Phone Embedding generation network module to make the Embedding close to the output of the 8th layer of the Encoder and make the 8th layer output more distinguishable. In the third stage, we use a large amount of text to train the model's Encoder's last n layers and Decoder, so that the model can learn more text information. In the fourth stage, we use labeled data to fine-tune the overall model parameters.

[0070] In the application phase, the trained model has the same structure as the baseline model except for the Phone Embedding module. This training method can be applied to the training process of various other models. In the test phase, the voice features are passed to the model, and the model generates the corresponding text label sequence. The model performance is estimated by the word error rate. This method achieves consistent performance improvement in other experimental data.

[0071] Unless otherwise specified, the models of the components in the embodiments of the present invention are not limited, and any device that can perform the above functions may be used.

[0072] Those skilled in the art will appreciate that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0073] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A speech recognition method based on phoneme embedding, characterized in that: The method comprises: The first m layers of the m+n-layer encoder are used as acoustic models for phoneme information learning. The target language data and other language data can be used to train the acoustic model. The last n layers and the decoder are used as language models. The contextual semantic information output by the first m layers can be combined for target language recognition. The language model can be trained using plain text data of the target language. By learning the phoneme information embedding, we learn the output of the Encoder's intermediate layer. After the phoneme information embedding is learned, a large amount of plain text data is used to produce phoneme sequences. After the phoneme information is embedded, a feature sequence close to the Encoder's intermediate output is generated and passed into the language model. The real text label is output through the Decoder training.

2. A speech recognition method based on phoneme embedding according to claim 1, characterized in that: The method adds a phoneme-based CTC classification network to the middle layer of the Encoder, so that the Encoder can explicitly learn phoneme information.

3. A speech recognition method based on phoneme embedding according to claim 1, characterized in that: The method further comprises: During the training process, the "audio-phoneme label" data of other languages ​​participate in the training together with the "audio-phoneme label-text label" data of the target language, and the data appear alternately in different batches.

4. A speech recognition method based on phoneme embedding according to claim 2, characterized in that: The output of the intermediate layer of the encoder is learned by learning the phoneme information embedding: The phoneme sequence generated by the text is encoded and passed to the generative network for phoneme information embedding. The output features of the generative network are passed to the linear classifier, which classifies the output features to obtain the probability distribution of each feature belonging to the phoneme. After the Softmax function, the MSE loss function is calculated for the one-hot vector corresponding to each phoneme, and the feature parameters of the Phone Embedding generation network are updated during the gradient backpropagation.

5. A speech recognition method based on phoneme embedding according to claim 4, characterized in that: The method further comprises: a fine-tuning stage for embedding the phoneme information, specifically: The generative network is added to the network of the first stage. The output of the first m layers of the encoder will pass through the linear classifier of Phone CTC and the Gumbel softmax function to classify each output feature. The classification result is a phoneme sequence corresponding to the input audio. The phoneme sequence passes through the generative network to generate a Phone Embedding sequence. The generated Phone Embedding sequence is compared with the output of the m layers of the encoder to calculate the SME loss, making the Phone Embedding sequence and the output of the middle layer of the encoder closer.

Images