A multi-language speech recognition model based on multi-head experts and its training method

Abstract

The present invention relates to the field of speech recognition technology of deep learning, and in particular to a multilingual speech recognition model based on multiple experts and a training method thereof. The present invention builds and trains a single multilingual speech recognition model that can be trained end-to-end based on the mainstream convolutional neural network and attention mechanism of deep learning, and then fine-tunes and tests the initially trained speech recognition model on a new language. The beneficial effects of the present invention are: it can effectively improve the training efficiency of the multilingual speech recognition model, while improving the accuracy of multilingual speech recognition, and effectively solve the problem of excessive forgetting of previously trained content during the training process.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning speech recognition, and in particular to a multi-language speech recognition model based on multiple experts and a training method thereof. Background Art

[0002] The purpose of speech recognition is to identify the corresponding text from the speech in audio or video. In real life, the application scenarios of multilingual speech are increasing. However, most of the current single speech recognition models often only support the recognition of one language, which is very limited for many application scenarios. In recent years, there have been many advances in the research of multilingual end-to-end models. The mainstream multilingual ASR model structures include: such as connection temporal classification (CTC) model, long short-term memory (LSTM) model and attention-based model. However, many models that support multilingual speech recognition need to explicitly give language information or design multiple encoders, which increases the complexity of the model, increases the computational cost, and is not conducive to end-to-end model training. How to design a single model that can be trained end-to-end without explicitly giving language information has attracted much attention. In order to solve the above problems, there are currently several solutions: 1. Large-scale training sets: Using a large enough training set to train the model can make up for the problem of reduced accuracy caused by a simple model structure or the lack of specific structures for certain specific problems. For example, the concept of the whisper speech recognition model released by OpenAI is to use a large-scale data set for training. The whisper model has not made much change compared to the traditional transformer model. However, the problem with this method is that compared with the improvement of the model by research, large-scale data sets are expensive. 2. Mixture of Experts (MoE) model: This method can solve the problem of displaying language information when training the model. This method generally uses a mixed expert model to replace the FFN layer in the original model. The feature vector input to the MoE layer will score multiple experts and select the experts with the highest scores. The feature vector will pass through these experts and then be combined to obtain the result as the input for the next step. However, the current method for selecting experts is to use ordinary softmax for selection. How to provide a method to select more suitable experts is the problem of this method. 3. Excessive forgetting problem: Directly using new data to perform full fine-tuning on the basis of a pre-trained model often leads to excessive forgetting of previous knowledge. How can we provide a method to alleviate the problem of excessive forgetting? Summary of the invention

[0003] The object of the present invention is to provide a multi-language speech recognition model based on multiple experts and a training method thereof, so as to solve at least one problem existing in the prior art mentioned in the above background technology.

[0004] To achieve the above object, the present invention adopts the following technical solution: a training method of a multi-language speech recognition model based on multiple experts, comprising the following steps:

[0005] S01. Obtain audio data sets for training, verification and testing, verify the audio data sets and modify the audio sampling rate to obtain the preprocessed audio data sets, construct audio-text pair standardization processing files, construct the dictionary file of the speech recognition model, and construct the global mean variance normalization file of the trained audio data sets; construct a speech recognition model including the LMoEConformer audio feature encoder and the MHR-LMoE Attention decoder.

[0006] S02. Use the audio feature extraction tool to extract 80-dimensional Fbank audio features containing frequency and time information from the trained audio dataset as preliminary audio features.

[0007] S03. Perform two layers of two-dimensional convolution with a step size of 2 on the preliminary audio features to obtain four-fold downsampled audio features. Then, perform linear mapping on the four-fold downsampled audio features and add position encoding, and then input them into the LMoE Conformer audio feature encoder to obtain high-dimensional audio features. The four-fold downsampled audio features after adding position encoding pass through the FFN layer, multi-head self-attention layer, and causal convolution layer in the LMoEConformer audio feature encoder in sequence, and finally enter the LMoE layer to perform weighted scoring on multiple experts, and select two experts with the highest weighted scores from them. After being processed by these two experts with the highest weighted scores, they are combined to obtain high-dimensional audio features.

[0008] S04, using the start mark of the target text sequence corresponding to the trained audio data set as the first input, then performing character embedding processing and adding position coding, inputting it into the MHR-LMoE Attention decoder for normalization processing and then inputting it into the multi-head self-attention layer with mask of the MHR-LMoE Attention decoder, then performing residual and normalization operations and inputting it into the multi-head cross attention layer, and at the same time, inputting the high-dimensional audio features obtained in step S03 into the multi-head cross attention layer for attention calculation, combining the feature vectors obtained in this way, and inputting the residual and normalized feature vectors into the MHR-LMoE layer of the MHR-LMoE Attention decoder for block processing, and performing weighted scoring for multiple experts for each feature block, and then combining the weighted scores of all blocks to obtain two experts with the highest weighted scores, and finally combining the feature vectors before block processing through the two experts with the highest weighted scores and performing residual calculation to obtain the feature vector output by the MHR-LMoE Attention decoder.

[0009] S05. The feature vector output by the MHR-LMoE Attention decoder is normalized and linearly mapped, and then a Softmax operation is performed to obtain the probability distribution of each character. The beam search method is used to obtain the final recognition prediction result.

[0010] S06. Compare the final recognition prediction result with the target text sequence of step S04 and calculate the loss value. Update the parameters of the LMoE Conformer audio feature encoder and the MHR-LMoE Attention decoder by backpropagation according to the loss value, and repeat steps S03 to S05 until the final recognition prediction result obtained in step S05 converges with the loss value of the target text sequence, thereby completing the preliminary training of the speech recognition model.

[0011] S07. Fine-tune the speech recognition model obtained by the preliminary training in step S06 for multiple times on the new language until the accuracy of the fine-tuned speech recognition model meets the requirements under the test of the tested audio data set, thereby completing the final training of the speech recognition model; during fine-tuning, freeze the parameters except the LMoE layer of the LMoE Conformer audio feature encoder and the MHR-LMoE layer of the MHR-LMoE Attention decoder, and use the LoRA module of the LMoE layer to fine-tune the weight matrices of multiple experts in the LMoE layer, and use the LoRA module of the MHR-LMoE layer to fine-tune the weight matrices of multiple experts in the MHR-LMoE layer in multiple routing heads.

[0012] Specifically, the audio dataset for training and testing in step S01 is a multilingual speech dataset that is self-built or obtained from a public audio corpus.

[0013] Specifically, in step S03, the audio features with four times downsampling after adding position encoding are first passed through the FFN layer in the LMoE Conformer audio feature encoder, and then the residual operation is performed after passing through the multi-head self-attention layer, and then the residual operation is performed after passing through the causal convolution layer. After being processed by the two experts with the highest weight scores in the LMoE layer, the residual operation is combined and performed, and finally the high-dimensional audio features of the final output are obtained after the layer normalization operation.

[0014] Specifically, the calculation formula for the residual operation of the audio feature with four times downsampling after adding position encoding in the LMoE Conformer audio feature encoder after passing through the FFN layer is: , the calculation formula for the residual operation after the multi-head self-attention layer is: , the calculation formula for the residual operation after the causal convolution layer is: , after being processed by the two experts with the highest weight scores at the LMoE layer, the calculation formula for the combination and residual operation is: ,in, ,in , are the weights of the two experts with the highest weight scores, , is the output of the two experts with the highest corresponding weight scores; when weighted scores are performed on multiple experts, Softmax is used to calculate the weight of each expert, and the two experts with the highest weights are selected from them. The calculation formula for the weight of the i-th expert is as follows: ,in This is the result of the residual operation after the causal convolution layer, W i is the weight matrix of the i-th expert.

[0015] Specifically, in step S04, the calculation process of the MHR-LMoE layer of the MHR-LMoE Attention decoder is as follows: first, the feature vector x input to the MHR-LMoE layer is divided into k blocks, and x 1 , x 2 , ..., x k , each block corresponds to a routing head of the MHR-LMoE layer. When calculating the weight score of each expert, the weight of the j-th expert in the m-th block is calculated first: ,in is the weight matrix of the routing head corresponding to the jth expert in the mth block, and then the weights of the k blocks of the jth expert are added together and then a Softmax operation is performed to obtain the weight score of the jth expert: , after selecting the two experts with the highest weight scores, the output of the MHR-LMoE layer is obtained: ,in , are the weights of the two experts with the highest weight scores, , are the outputs of the two experts with the highest corresponding weight scores.

[0016] Specifically, when the weight matrices of multiple experts in the LMoE layer are fine-tuned using the LoRA module of the LMoE layer in step S07, the weight matrix of the i-th expert after fine-tuning is: W' i =W i +ΔW i , where W i is the original weight matrix, ΔW iis the incremental weight matrix calculated according to the parameter matrix of the i-th expert; when using the LoRA module of the MHR-LMoE layer to fine-tune the weight matrices of multiple experts in the MHR-LMoE layer for multiple routing heads, the weight matrix of the j-th expert corresponding to the m-th block in the routing head after fine-tuning is: W’ j,m =W j,m +ΔW j,m , where W j,m is the original weight matrix, and ΔW j,m is the incremental weight matrix calculated according to the parameter matrix of the j-th expert corresponding to the m-th block in the routing head.

[0017] Specifically, ΔW i =A i B i , where A i ∈R din×r , B i ∈R r×dout , where r << min(din, dout) is the low-rank number, and din and dout are the rank numbers of the parameter matrix of the i-th expert; ΔW j,m =A j,m B j,m , where A j,m ∈R din’×r’ , B j,m ∈R r ’×dout’ , where r’ << min(din’, dout’) is the low-rank number, and din’ and dout’ are the rank numbers of the parameter matrix of the parameter matrix of the j-th expert corresponding to the m-th block in the routing head.

[0018] A multi-language speech recognition model based on multi-head experts, including an LMoE Conformer audio feature encoder and an MHR-LMoE Attention decoder. The LMoE Conformer audio feature encoder includes an FFN layer, a multi-head self-attention layer, a causal convolution layer, and an LMoE layer. The MHR-LMoE Attention decoder includes a first normalization layer, a multi-head self-attention layer with a mask, a second normalization layer, a multi-head cross-attention layer, a third normalization layer, and an MHR-LMoE layer.

[0019] The beneficial effects of the present invention are as follows: Based on the mainstream convolutional neural network and attention mechanism in deep learning, a multi-language speech recognition model that can be trained end-to-end through a single one is built, which can effectively improve the training efficiency of the multi-language speech recognition model and the accuracy of multi-language speech recognition, and by fine-tuning the preliminarily trained speech recognition model on a new language, the problem of excessive forgetting of the previously trained content during the training process is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Attached Figure 1 is a flow chart of a training method for a multi-language speech recognition model based on multiple experts in an embodiment;

[0021] Attached Figure 2 The figure is an overall architecture diagram of a multi-language speech recognition model based on multiple experts during the training process in an embodiment. DETAILED DESCRIPTION

[0022] Example 1, reference Figure 1 , a training method for a multi-language speech recognition model based on multiple experts, comprising the following steps:

[0023] S01. Obtain the audio data sets for training, verification and testing, verify the audio data sets and modify the audio sampling rate to obtain the preprocessed audio data sets, construct the audio-text pair standardization processing file, construct the dictionary file of the speech recognition model, and construct the global mean variance normalization file of the trained audio data set; construct the speech recognition model including the LMoEConformer audio feature encoder and the MHR-LMoE Attention decoder. Among them, the role of the global mean variance normalization file of the trained audio data set is to preprocess the speech features of the input speech recognition model. By normalizing the features, it helps to improve the robustness of the speech recognition model, reduce the impact of changes in recording conditions and other reasons on the features, and enable the model to have a better generalization ability.

[0024] S02. Use the audio feature extraction tool to extract 80-dimensional Fbank audio features containing frequency and time information from the trained audio dataset as preliminary audio features.

[0025] S03. Perform two layers of two-dimensional convolution with a step size of 2 on the preliminary audio features to obtain four-fold downsampled audio features. Then, perform linear mapping on the four-fold downsampled audio features and add position encoding, and then input them into the LMoE Conformer audio feature encoder to obtain high-dimensional audio features. The four-fold downsampled audio features after adding position encoding pass through the FFN layer, multi-head self-attention layer, and causal convolution layer in the LMoEConformer audio feature encoder in sequence, and finally enter the LMoE layer to perform weighted scoring on multiple experts, and select two experts with the highest weighted scores from them. After being processed by these two experts with the highest weighted scores, they are combined to obtain high-dimensional audio features.

[0026] S04, using the start mark of the target text sequence corresponding to the trained audio data set as the first input, then performing character embedding processing and adding position coding, inputting it into the MHR-LMoE Attention decoder for normalization processing and then inputting it into the multi-head self-attention layer with mask of the MHR-LMoE Attention decoder, then performing residual and normalization operations and inputting it into the multi-head cross attention layer, and at the same time, inputting the high-dimensional audio features obtained in step S03 into the multi-head cross attention layer for attention calculation, combining the feature vectors obtained in this way, and inputting the residual and normalized feature vectors into the MHR-LMoE layer of the MHR-LMoE Attention decoder for block processing, and performing weighted scoring for multiple experts for each feature block, and then combining the weighted scores of all blocks to obtain two experts with the highest weighted scores, and finally combining the feature vectors before block processing through the two experts with the highest weighted scores and performing residual calculation to obtain the feature vector output by the MHR-LMoE Attention decoder.

[0027] S05. The feature vector output by the MHR-LMoE Attention decoder is normalized and linearly mapped, and then a Softmax operation is performed to obtain the probability distribution of each character. The beam search method is used to obtain the final recognition prediction result.

[0028] S06. Compare the final recognition prediction result with the target text sequence of step S04 and calculate the loss value. Update the parameters of the LMoE Conformer audio feature encoder and the MHR-LMoE Attention decoder by backpropagation according to the loss value, and repeat steps S03 to S05 until the final recognition prediction result obtained in step S05 converges with the loss value of the target text sequence, thereby completing the preliminary training of the speech recognition model.

[0029] S07. Fine-tune the speech recognition model obtained by the preliminary training in step S06 for multiple times on the new language until the accuracy of the fine-tuned speech recognition model meets the requirements under the test of the tested audio data set, thereby completing the final training of the speech recognition model; during fine-tuning, freeze the parameters except the LMoE layer of the LMoE Conformer audio feature encoder and the MHR-LMoE layer of the MHR-LMoE Attention decoder, and use the LoRA module of the LMoE layer to fine-tune the weight matrices of multiple experts in the LMoE layer, and use the LoRA module of the MHR-LMoE layer to fine-tune the weight matrices of multiple experts in the MHR-LMoE layer in multiple routing heads.

[0030] Specifically, the audio dataset for training and testing in step S01 is a multilingual speech dataset that is self-built or obtained from a public audio corpus.

[0031] Specifically, in step S03, the audio features with four times downsampling after adding position encoding are first passed through the FFN layer in the LMoE Conformer audio feature encoder, and then the residual operation is performed after passing through the multi-head self-attention layer, and then the residual operation is performed after passing through the causal convolution layer. After being processed by the two experts with the highest weight scores in the LMoE layer, the residual operation is combined and performed, and finally the high-dimensional audio features of the final output are obtained after the layer normalization operation.

[0032] Specifically, the calculation formula for the residual operation of the audio feature with four times downsampling after adding position encoding in the LMoE Conformer audio feature encoder after passing through the FFN layer is: , the calculation formula for the residual operation after the multi-head self-attention layer is: , the calculation formula for the residual operation after the causal convolution layer is: , after being processed by the two experts with the highest weight scores at the LMoE layer, the calculation formula for the combination and residual operation is: ,in, ,in , are the weights of the two experts with the highest weight scores, , is the output of the two experts with the highest corresponding weight scores; when weighted scores are performed on multiple experts, Softmax is used to calculate the weight of each expert, and the two experts with the highest weights are selected from them. The calculation formula for the weight of the i-th expert is as follows: ,in This is the result of the residual operation after the causal convolution layer, W i is the weight matrix of the i-th expert.

[0033] Specifically, in step S04, the calculation process of the MHR-LMoE layer of the MHR-LMoE Attention decoder is as follows: first, the feature vector x input to the MHR-LMoE layer is divided into k blocks, and x 1 , x 2 , ..., x k , each block corresponds to a routing head of the MHR-LMoE layer. When calculating the weight score of each expert, the weight of the j-th expert in the m-th block is calculated first: ,in is the weight matrix of the routing header corresponding to the m-th block for the j-th expert. Then, the weights of the k blocks of the j-th expert are added and then a Softmax operation is performed to obtain the weight score of the j-th expert: After selecting the two experts with the highest weight scores from them, the output of the MHR-LMoE layer is obtained: where 、 are the weights of the two experts with the highest weight scores, 、 are the outputs of the two experts with the highest corresponding weight scores.

[0034] Specifically, when fine-tuning the weight matrices of multiple experts in the LMoE layer using the LoRA module of the LMoE layer, the weight matrix of the i-th expert after fine-tuning is: W’ i =W i +ΔW i where, W i is the original weight matrix, and ΔW i is the incremental weight matrix calculated according to the parameter matrix of the i-th expert; when fine-tuning the weight matrices of multiple experts in multiple routing headers in the MHR-LMoE layer using the LoRA module of the MHR-LMoE layer, the weight matrix of the j-th expert corresponding to the m-th block in the routing header after fine-tuning is: W’ j,m =W j,m +ΔW j,m where, W j,m is the original weight matrix, and ΔW j,m is the incremental weight matrix calculated according to the parameter matrix of the j-th expert corresponding to the m-th block in the routing header.

[0035] Specifically, ΔW i =A i B i where A i ∈R din×r and B i ∈R r×dout where r << min(din, dout) is the low-rank number, and din and dout are the rank numbers of the parameter matrix of the i-th expert; ΔW j,m =A j,m B j,m where A j,m ∈R din’×r’ and B j,m ∈R r ’×dout’ where r’ << min(din’, dout’) is the low-rank number, and din’ and dout’ are the rank numbers of the parameter matrix of the parameter matrix of the j-th expert corresponding to the m-th block in the routing header.

[0036] Reference Figure 2 This embodiment also provides a multi-head expert-based multilingual speech recognition model to adapt to the above training method, which includes an LMoE Conformer audio feature encoder and an MHR-LMoE Attention decoder. The LMoE Conformer audio feature encoder includes an FFN layer, a multi-head self-attention layer, a causal convolution layer and an LMoE layer. The MHR-LMoE Attention decoder includes a first normalization layer, a multi-head self-attention layer with a mask, a second normalization layer, a multi-head cross-attention layer, a third normalization layer and an MHR-LMoE layer.

[0037] Of course, the above are only preferred embodiments of the present invention, and are not intended to limit the scope of use of the present invention. Therefore, any equivalent changes made to the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A training method for a multi-language speech recognition model based on multiple experts, characterized in that: The steps include: S01. Obtain the audio data sets for training and testing, verify the audio data sets and modify the audio sampling rate to obtain the preprocessed audio data sets, construct the audio-text pair standardization processing file, construct the dictionary file of the speech recognition model, and construct the global mean variance normalization file of the trained audio data sets; construct a speech recognition model including the LMoE Conformer audio feature encoder and the MHR-LMoE Attention decoder; S02. Use the audio feature extraction tool to extract 80-dimensional Fbank audio features containing frequency and time information from the trained audio dataset as preliminary audio features; S03, performing two layers of two-dimensional convolution with a step size of 2 on the preliminary audio features to obtain four-fold downsampled audio features, then performing linear mapping and adding position coding on the four-fold downsampled audio features and inputting them into the LMoE Conformer audio feature encoder to obtain high-dimensional audio features, the four-fold downsampled audio features after adding position coding are sequentially passed through the FFN layer, the multi-head self-attention layer and the causal convolution layer in the LMoE Conformer audio feature encoder, and finally enter the LMoE layer to weight and score multiple experts, and select two experts with the highest weight scores from them, and after being processed by these two experts with the highest weight scores, they are combined to obtain high-dimensional audio features; S04, using the start mark of the target text sequence corresponding to the trained audio data set as the first input, then performing character embedding processing and adding position coding, inputting it into the MHR-LMoE Attention decoder for normalization processing, and then inputting it into the multi-head self-attention layer with mask of the MHR-LMoE Attention decoder, and then performing residual and normalization operations, and inputting it into the multi-head cross attention layer, and at the same time, the high-dimensional audio features obtained in step S03 are input into the multi-head cross attention layer for attention calculation, and the feature vectors obtained in this way are combined, and the residual and normalized feature vectors are input into the MHR-LMoE layer of the MHR-LMoE Attention decoder for block processing, and each block feature is weighted to score multiple experts respectively, and then the weight scores of all blocks are combined to obtain two experts with the highest weight scores, and finally the feature vectors before block processing are combined by two experts with the highest weight scores, and residual calculation is performed to obtain the feature vector output by the MHR-LMoE Attention decoder; S05. The feature vector output by the MHR-LMoE Attention decoder is normalized and linearly mapped, and then the Softmax operation is performed to obtain the probability distribution of each character, and the beam search method is used to obtain the final recognition prediction result; S06, compare the final recognition prediction result with the target text sequence of step S04 and calculate the loss value, update the parameters of the LMoE Conformer audio feature encoder and the MHR-LMoE Attention decoder according to the loss value back-propagation, and repeat steps S03 to S05 until the final recognition prediction result obtained in step S05 converges with the loss value of the target text sequence, thereby completing the preliminary training of the speech recognition model; S07. Fine-tune the speech recognition model obtained by the preliminary training in step S06 for multiple times on the new language until the accuracy of the fine-tuned speech recognition model meets the requirements under the test of the tested audio data set, thereby completing the final training of the speech recognition model; during fine-tuning, freeze the parameters except the LMoE layer of the LMoE Conformer audio feature encoder and the MHR-LMoE layer of the MHR-LMoEAttention decoder, and use the LoRA module of the LMoE layer to fine-tune the weight matrices of multiple experts in the LMoE layer, and use the LoRA module of the MHR-LMoE layer to fine-tune the weight matrices of multiple experts in the MHR-LMoE layer in multiple routing heads.

2. The training method of a multi-language speech recognition model based on multiple experts according to claim 1, characterized in that: The audio data set for training and testing in step S01 is a multilingual speech data set that is self-built or obtained from a public audio corpus.

3. The training method of a multi-language speech recognition model based on multiple experts according to claim 1, characterized in that: In the step S03, the audio features of the four-fold downsampling after adding the position encoding are first subjected to residual operation after passing through the FFN layer, then subjected to residual operation after passing through the multi-head self-attention layer, then subjected to residual operation after passing through the causal convolution layer, and then subjected to residual operation after being processed by the two experts with the highest weight scores in the LMoE layer, and then combined and subjected to residual operation, and finally subjected to layer normalization operation to obtain the high-dimensional audio features of the final output.

4. The training method of a multi-language speech recognition model based on multiple experts according to claim 3, characterized in that: The calculation formula for the residual operation of the audio feature of the four-fold down-sampling after adding the position encoding in the LMoE Conformer audio feature encoder after passing through the FFN layer is: , the calculation formula for the residual operation after the multi-head self-attention layer is: , the calculation formula for the residual operation after the causal convolution layer is: , after being processed by the two experts with the highest weight scores at the LMoE layer, the calculation formula for the combination and residual operation is: ,in, ,in , are the weights of the two experts with the highest weight scores, , is the output of the two experts with the highest corresponding weight scores; when weighted scores are performed on multiple experts, Softmax is used to calculate the weight of each expert, and the two experts with the highest weights are selected from them. The calculation formula for the weight of the i-th expert is as follows: ,in This is the result of the residual operation after the causal convolution layer, W i is the weight matrix of the i-th expert.

5. The method for training a multi-language speech recognition model based on multiple experts according to claim 1, characterized in that: In step S04, the calculation process of the MHR-LMoE layer of the MHR-LMoE Attention decoder is as follows: first, the feature vector x input to the MHR-LMoE layer is divided into k blocks to obtain x1, x2, ..., x k , each block corresponds to a routing head of the MHR-LMoE layer. When calculating the weight score of each expert, the weight of the j-th expert in the m-th block is calculated first: ,in is the weight matrix of the routing head corresponding to the jth expert in the mth block, and then the weights of the k blocks of the jth expert are added together and then a Softmax operation is performed to obtain the weight score of the jth expert: , after selecting the two experts with the highest weight scores, the output of the MHR-LMoE layer is obtained: ,in , are the weights of the two experts with the highest weight scores, , are the outputs of the two experts with the highest corresponding weight scores.

6. The method for training a multi-language speech recognition model based on multiple experts according to claim 1, characterized in that: When the weight matrices of multiple experts in the LMoE layer are fine-tuned using the LoRA module of the LMoE layer in step S07, the weight matrix of the i-th expert after fine-tuning is: W' i =W i +ΔW i , where W i is the original weight matrix, ΔW i is the incremental weight matrix calculated based on the parameter matrix of the i-th expert; when the weight matrices of multiple experts in the MHR-LMoE layer in multiple routing heads are fine-tuned using the LoRA module of the MHR-LMoE layer, the weight matrix of the routing head corresponding to the j-th expert in the m-th block after fine-tuning is: W' j,m =W j,m +ΔW j,m , where W j,m is the original weight matrix, ΔW j,m is the incremental weight matrix calculated based on the parameter matrix of the routing head corresponding to the mth block by the jth expert.

7. The method for training a multi-language speech recognition model based on multiple experts according to claim 6, characterized in that: The said ΔW i = A i B i , where A i ∈ R din×r , and B i ∈ R r×dout , where r << min(din, dout) is the low-rank rank number, and din and dout are the rank numbers of the parameter matrices of the i-th expert; ΔW j,m = A j,m B j,m , where A j,m ∈ R din’×r’ , and B j,m ∈ R r ’×dout’ , where r' << min(din', dout') is the low-rank rank number, and din' and dout' are the rank numbers of the parameter matrices of the parameter matrices of the routing header corresponding to the j-th expert in the m-th block.

8. A multi-language speech recognition system based on multiple experts, characterized by: It includes an LMoE Conformer audio feature encoder and an MHR-LMoE Attention decoder, wherein the LMoE Conformer audio feature encoder includes an FFN layer, a multi-head self-attention layer, a causal convolution layer and an LMoE layer, and the FFN layer, the multi-head self-attention layer, the causal convolution layer and the LMoE layer are connected in sequence; the MHR-LMoE Attention decoder includes a first normalization layer, a multi-head self-attention layer with a mask, a second normalization layer, a multi-head cross-attention layer, a third normalization layer and an MHR-LMoE layer, and the first normalization layer, the multi-head self-attention layer with a mask, the second normalization layer, the multi-head cross-attention layer, the third normalization layer and the MHR-LMoE layer are connected in sequence; the output of the LMoE Conformer audio feature encoder is connected to the multi-head cross-attention layer of the MHR-LMoEAttention decoder.

Images