Multi-dialect speech recognition model training method and device

Abstract

The embodiment of the present application provides a kind of multilingual speech recognition model training method and device, it is related to artificial intelligence technical field, the method comprises: obtaining the pre-training model of speech recognition model;Wherein, the pre-training model includes multiple network layers, specified linear layer in the network layer is replaced by expert routing network, and the expert routing network includes routing gate and expert network;At least one dialect vectorized dialect data set is obtained;The expert routing network in the pre-training model is trained using the dialect data set, and when the training model reaches preset convergence condition, the speech recognition model of training completion is obtained;The speech recognition model carries out speech recognition to the dialect speech of at least one dialect.This embodiment of the present application uses unified speech recognition model, can let different regions deploy same speech recognition model, effectively reduces operating cost.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application relates to the technical field of artificial intelligence, in particular to a multi-dialect speech recognition model training method, a multi-dialect speech recognition model training device, an electronic device and a computer readable storage medium. BACKGROUND

[0002] ASR (Automatic Speech Recognition, speech recognition technology) can convert speech into computer-readable text.

[0003] At present, speech recognition technology plays an important role in the field of artificial intelligence and the like, however, most of the speech recognition models based on speech recognition technology can accurately recognize standard Mandarin, but for dialects such as Cantonese, Minnan dialect, Chaoshan dialect and the like, the speech recognition model usually needs to train a small model for each dialect separately, which increases the operation cost. SUMMARY

[0004] In view of the above problems, a multi-dialect speech recognition model training method and device are provided to overcome the above problems or at least partially solve the above problems. The specific technical solutions are as follows:

[0005] In the first aspect of the present application, a multi-dialect speech recognition model training method is first provided, which comprises:

[0006] obtaining a pre-training model of a speech recognition model; wherein the pre-training model comprises a plurality of network layers, a specified linear layer in the network layer is replaced by an expert routing network, and the expert routing network comprises a routing gate and an expert network;

[0007] obtaining at least one dialect data set after vectorization of a dialect;

[0008] training the expert routing network in the pre-training model using the dialect data set, and obtaining a trained speech recognition model when the training model reaches a preset convergence condition; the speech recognition model performs speech recognition on the dialect speech of at least one dialect.

[0009] In an embodiment of the present application, the network layer of the pre-training model comprises a feedforward layer and a multi-head self-attention layer, and the specified linear layer comprises at least a linear layer in the feedforward layer and a linear layer in the multi-head self-attention layer.

[0010] In an embodiment of the present application, before obtaining the pre-training model of the speech recognition model, the method further comprises:

[0011] acquire full data, extract acoustic features from the full data;

[0012] train the speech recognition model to be trained using the acoustic features to obtain a pre-training model of the speech recognition model.

[0013] In an embodiment of the present application, the dialect data set includes dialect data and dialect labels corresponding to the dialect data; and training an expert routing network in the pre-training model using the dialect data set includes:

[0014] training the expert routing network in the pre-training model using the dialect data and the dialect labels in the dialect data set is first-stage training;

[0015] training the expert routing network in the pre-training model using the dialect data in the dialect data set is second-stage training.

[0016] In an embodiment of the present application, the first-stage training of the expert routing network in the pre-training model using the dialect data and the dialect labels in the dialect data set includes:

[0017] training the expert routing network in the pre-training model using the dialect data and the dialect labels in the dialect data set;

[0018] when the number of iterations of training the expert routing network in the pre-training model using the dialect data and the dialect labels in the dialect data set reaches a preset number of iterations, it is determined that the first-stage training of the pre-training model is complete.

[0019] In an embodiment of the present application, the second-stage training of the expert routing network in the pre-training model using the dialect data in the dialect data set includes:

[0020] inputting the dialect data in the dialect data set into the routing gate and the expert network in the pre-training model to obtain an output result of the pre-training model;

[0021] calculating a loss value of the pre-training model according to the output result and the dialect labels of the dialect data set; the loss value is used to adjust parameters of the routing gate and the expert network in the pre-training model;

[0022] when the loss value meets a preset convergence condition, it is determined that the second-stage training of the pre-training model is complete.

[0023] In an embodiment of the present application, the output result is calculated using the following formula:

[0024]

[0025] G(x) = Softmax(W g *x)

[0026] wherein o is an output result of the pre-training model, G(x) is a routing score of the routing gate output, x is a dialect data set, W0 is an original weight matrix, a and r are constant hyperparameters, B and A are learnable low-quality matrices, and W g is a weight network of the routing gate.

[0027] The embodiment of the present application also discloses a multi-dialect speech recognition model training device, the device comprising:

[0028] a pre-training model acquisition module, configured to acquire a pre-training model of a speech recognition model; wherein a plurality of network layers are included in the pre-training model, a specified linear layer in the network layers is replaced by an expert routing network, and the expert routing network includes a routing gate and an expert network;

[0029] a dialect data set acquisition module, configured to acquire at least one dialect data set after vectorization of a dialect;

[0030] a speech recognition model completion module, configured to train the expert routing network in the pre-training model by using the dialect data set, and obtain a trained speech recognition model when the training model reaches a preset convergence condition; the speech recognition model is used for speech recognition of dialect speech of at least one dialect.

[0031] In another aspect of the embodiment of the present application, a computer readable storage medium is also provided, the computer readable storage medium stores instructions, when the instructions are run on a computer, the computer executes the multi-dialect speech recognition model training method described above.

[0032] In another aspect of the embodiment of the present application, a computer program product containing instructions is also provided, when the instructions are run on a computer, the computer executes the multi-dialect speech recognition model training method described above.

[0033] Compared with the related art, the embodiment of the present application has at least the following advantages:

[0034] In the embodiment of the present application, a pre-training model of a speech recognition model is obtained, wherein the pre-training model includes a plurality of network layers, a specified linear layer in the network layer is replaced with an expert routing network, the expert routing network includes a routing gate and an expert network, at least one dialect vectorized dialect dataset is obtained, the expert routing network in the pre-training model is trained using the dialect dataset, and when the trained model reaches a preset convergence condition, a trained speech recognition model is obtained. After the deployment of the speech recognition model, the speech recognition model can recognize the dialect speech of at least one dialect. The pre-training model of the speech recognition model in the embodiment of the present application has certain speech recognition capability, the part of the pre-training model replaced with the expert routing network is trained to adjust the parameters, and each expert network in the expert routing network is implicitly focused on learning the respective dialect, so that the same speech recognition model has the recognition capability of multiple dialects. In this way, the same speech recognition model can be deployed in different regions using a unified speech recognition model, effectively reducing the operating cost, and avoiding the problem that the learning effect of the dialect with small data volume is suppressed by the dialect with large data volume due to the unbalanced dialect dataset used for training of different dialects, thereby ensuring the speech recognition effect of the speech recognition model on different dialects. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced as follows.

[0036] Figure 1 A step flow chart of a multi-dialect speech recognition model training method provided in the embodiment of the present application;

[0037] Figure 2 A structure diagram of an MOE-LORA module in a pre-training model provided in the embodiment of the present application;

[0038] Figure 3 A structure block diagram of a multi-dialect speech recognition model training device provided in the embodiment of the present application;

[0039] Figure 4 A structure block diagram of an electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

[0041] In order to make those skilled in the art better understand the embodiments of the present application, some terms and names related to the embodiments of the present application are described as follows.

[0042] Conformer: a model structure of a neural network;

[0043] LORA (Low-Rank Adaptation): A technique for fine-tuning large models by introducing low-rank matrices to reduce the number of parameters and improve training efficiency while maintaining the model's expressive power.

[0044] Adapter: A technique for fine-tuning large models by inserting additional modules (Adapter modules) into certain layers of the pre-trained model to adapt to new tasks, thereby reducing the number of parameters and improving training efficiency while maintaining the model's expressive power.

[0045] MOE (Mixture of Experts): A large model architecture idea that can improve model performance by mixing multiple expert networks. Specifically, the main idea of MOE is to assign input data to multiple expert networks, each of which focuses on handling specific tasks or subtasks, and then use a gating mechanism (Router) to determine the weight of each expert network to obtain the final output result.

[0046] The target is a speech recognition model for dialect recognition. One of the current methods uses full-quantity dialect data to fine-tune the speech recognition model with full-quantity parameters. The disadvantages are as follows: First, the current speech recognition model has a large number of parameters, and if full-quantity parameter fine-tuning is used, the training speed of the speech recognition model will be slow. Second, full-quantity parameter fine-tuning will change all the parameters of the pre-trained model of the speech recognition model, which is prone to catastrophic forgetting, i.e., when adapting to new tasks, the speech recognition model loses memory of the original knowledge. Finally, generally speaking, dialect data is not balanced, and full-quantity parameter fine-tuning will make the speech recognition model biased towards the dialect with more data, while reducing the recognition effect of the dialect with less data, and balancing the proportion of dialect data is a difficult decision.

[0047] In some cases, the current method also uses multiple dialect sub-models to apply to their respective scenarios to improve dialect recognition in various regions. The disadvantage of this method is high operating cost, and the user needs to select the dialect region in advance to perform corresponding dialect recognition, which has poor flexibility.

[0048] In some cases, the current method also uses LORA and Adapter fine-tuning to improve the recognition effect of the speech recognition model for corresponding dialects. Although this fine-tuning method is faster than full-quantity parameter fine-tuning and can alleviate catastrophic forgetting, it only uses a fixed set of fine-tuning parameters to process multi-dialect data, which cannot fully learn the complexity of the data and is prone to local optimal solution.

[0049] In some cases, the current method is to add a MOE module in the base model to improve the recognition effect of multiple dialects, and the base model needs to be retrained. The data volume used by the base model is huge, and it is time-consuming and money-consuming.

[0050] To solve the above problems, the embodiment of the application provides a fine-tuning method for a multi-dialect speech recognition model. The method is more flexible and can fine-tune only the dialect data or newly added dialect data. The base model structure of most ASR models (speech recognition models) (as long as the model includes a linear layer, and basically all models have a linear layer) can be used to further improve the recognition rate of the model in multiple dialects.

[0051] Specifically, the embodiment of the application translates the multi-dialect dialogue audio into text. First, the embodiment of the application only needs one model to complete multi-dialect speech recognition, rather than multiple dialect small models to complete the speech recognition task. Second, the embodiment of the application focuses on fine-tuning the ASR base model using multi-dialect data, so that the speech recognition model achieves better speech recognition effect on multi-dialect data. The embodiment of the application can be applied to various businesses, such as the following two businesses: 1. Intelligent voice assistant: In a multi-dialect family, family members may use different dialects to communicate with the voice assistant. The system can recognize their voice requirements according to their dialects or accents. 2. Customer service and call center: The customer service department needs to handle customer requests from different regions and language backgrounds. Traditional single language or dialect speech recognition systems may not accurately understand the speech expressions of users in different regions, especially in multi-language countries or regions, where dialect differences can be very large. The current method is to deploy corresponding dialect small models in different regions to achieve the required recognition rate. However, the method of the embodiment of the application can deploy the same speech recognition model in different regions by using a unified multi-dialect recognition model, effectively reducing the operating cost.

[0052] Reference Figure 1 A multi-dialect speech recognition model training method provided in the embodiment of the application is shown in the step flow chart as shown in Figure 1 The method can specifically include the following steps:

[0053] Step 101, obtaining a pre-training model of a speech recognition model; wherein the pre-training model includes a plurality of network layers, a specified linear layer in the network layer is replaced with an expert routing network, and the expert routing network includes a routing gate and an expert network;

[0054] The speech recognition model (multi-dialect recognition model, ASR model) can be a conformer-based speech recognition model. Conformer is a neural network architecture specially designed for speech recognition, which skillfully combines CNNs (Convolution Neural Networks) and Transfommers (a neural network architecture based on self-attention mechanism) together, thereby optimizing the understanding of speech data. Compared with the speech recognition model based on Transfommers or CNNs, the speech recognition model based on Conformer performs better and can achieve better speech recognition effect with lower computing resources.

[0055] In an embodiment of the present application, the network layer of the pre-training model can at least include a feedforward layer and a multi-head self-attention layer, and the specified linear layer can at least include a linear layer in the feedforward layer and a linear layer in the multi-head self-attention layer. The linear layer, also known as the fully connected layer, is one of the most basic layer types in neural networks. The linear layer can map the input features to the output feature space through linear transformation. The linear layer is usually used to convert the input features into higher-dimensional or lower-dimensional feature representations.

[0056] In an embodiment of the present application, the pre-training model of the speech recognition model is a specific speech recognition capability-based model trained by full data. For the pre-training model, the linear layers in the specified linear layer, i.e., the linear layers in the feedforward layer and the linear layers in the multi-head self-attention layer, can be replaced by expert routing networks (MOE-LORA modules / lora modules and router modules). The expert routing network can include a router gate (router G) and a plurality of expert networks (E1,…,E n ).

[0057] Specifically, the role of the router gate is to select the appropriate expert network according to the input feature vector (vectorized dialect data set). The router gate can calculate the score of the input feature vector for each expert network, and the score can determine the weight of each expert network. The expert network with a higher score will be given a higher weight, so as to play a greater role in the final output result of the speech recognition model. Each expert network (Expert) is a smaller neural network that focuses on processing specific tasks or subtasks. For example, in the multi-dialect speech recognition task of the embodiment of the present application, different expert networks can focus on recognizing different dialects or accents. In addition, since each expert network can learn different feature representations, the expression ability and generalization ability of the speech recognition model are improved.

[0058] It should be noted that, no matter what network structure the pre-training model of the speech recognition model is, as long as there is a linear layer, the expert routing network of the embodiment of the application can be used to replace the corresponding linear layer (such as the linear layer in the feedforward layer and the linear layer of the multi-head self-attention layer) in the pre-training model, and then the pre-training model after replacing the linear layer is fine-tuned for multiple dialects, so that the final speech recognition model can have better and more balanced effects on the dialect test set.

[0059] Step 102, obtaining at least one dialect data set after vectorization of dialects;

[0060] In the embodiment of the application, all dialect data sets of at least one dialect used for fine-tuning the pre-training model are arranged, for example, if the pre-training model needs to have or optimize the speech recognition effect for dialect A, dialect B and dialect C, the corresponding dialect data sets of dialect A, dialect B and dialect C are obtained, wherein the dialect data sets need to be vectorized before being input into the pre-training model. The dialect data set can include dialect data and a dialect label corresponding to the dialect data, wherein the dialect label refers to a label used to identify the dialect to which the speech data belongs in the multi-dialect speech recognition task. The dialect label is a clear supervision signal in the training process of the speech recognition model, which can help the speech recognition model learn how to predict the corresponding dialect label according to the input speech data. Exemplarily, the dialect label can include but is not limited to: Cantonese, Minnan, Chaoshan dialect, etc.

[0061] Step 103, training the expert routing network in the pre-training model by using the dialect data set, and obtaining a trained speech recognition model when the training model reaches a preset convergence condition; the speech recognition model performs speech recognition on dialect speech of at least one dialect.

[0062] In the embodiment of the application, after obtaining the dialect data set, the expert routing network (routing gate and expert network) in the pre-training model can be trained by using the dialect data set, and when the training model reaches a preset convergence condition (for example, the number of iteration training of the pre-training model reaches a preset number or the loss value of the pre-training model is a preset loss value), a trained speech recognition model can be obtained. Subsequently, the speech recognition model can be deployed in the corresponding system to perform speech recognition on dialect speech of at least one dialect based on the speech recognition model.

[0063] In the above multi-dialect speech recognition model training, a pre-training model of the speech recognition model is obtained, wherein the pre-training model includes a plurality of network layers, a specified linear layer in the network layer is replaced with an expert routing network, the expert routing network includes a routing gate and an expert network, at least one vectorized dialect data set is obtained, the expert routing network in the pre-training model is trained using the dialect data set, and when the trained model reaches a preset convergence condition, a trained speech recognition model is obtained. After the speech recognition model is deployed, it can perform speech recognition on the dialect speech of at least one dialect. The pre-training model of the speech recognition model in the embodiment of the present application has certain speech recognition capability. The part of the pre-training model replaced by the expert routing network is trained to adjust the parameters, so that each expert network in the expert routing network focuses on learning the respective dialect, so that the same speech recognition model has the recognition capability of multiple dialects. In this way, using a unified speech recognition model can deploy the same speech recognition model in different regions, effectively reducing the operating cost, and at the same time avoiding the problem that the learning effect of the dialect with small data volume is suppressed by the dialect with large data volume due to the unbalanced dialect data set used for training of different dialects, thereby ensuring the speech recognition effect of the speech recognition model on different dialects.

[0064] In an embodiment of the present application, before obtaining the pre-training model of the speech recognition model, the method can further include:

[0065] Obtaining full data, extracting acoustic features from the full data;

[0066] Training the speech recognition model to be trained using the acoustic features to obtain the pre-training model of the speech recognition model.

[0067] Wherein, the pre-training model refers to a base model (Base Model) pre-trained on a large amount of data (full data). The pre-training model is trained using a large amount of data, which can learn general feature representation. Subsequently, the parameters are fine-tuned on a specific task (such as dialect recognition task), so as to adapt to different application scenarios. Specifically, the full data refers to the speech data that can be collected to train the speech recognition model to be trained to obtain the pre-training model. It should be noted that the full data can include dialect and non-dialect corresponding speech data.

[0068] In the embodiment of the present application, 80-dimensional FBANK (Filter Bank, a speech feature representation method) acoustic features are extracted from the speech data (full data) as input data of the speech recognition model to be trained. Of course, other speech feature representation methods can also be selected in the embodiment of the present application, and the embodiment of the present application does not need to be limited thereto. Then, the conformer-based speech recognition model can be trained using the acoustic features extracted from the full data for the ASR speech recognition task to obtain a pre-trained model of the speech recognition model. After obtaining the pre-trained model, the parameters of the expert routing network in the pre-trained model can be fine-tuned according to the dialect data set of the dialect to be recognized, and finally a speech recognition model capable of recognizing different dialects is obtained.

[0069] In an embodiment of the present application, the dialect data set can include dialect data and dialect labels corresponding to the dialect data; training the expert routing network in the pre-trained model using the dialect data set can include:

[0070] Training the expert routing network in the pre-trained model using the dialect data and the dialect labels in the dialect data set is first-stage training;

[0071] Training the expert routing network in the pre-trained model using the dialect data in the dialect data set is second-stage training.

[0072] In the embodiment of the present application, before fine-tuning the pre-trained model of the speech recognition model, the weights of the pre-trained model are loaded into a new structure, i.e., an expert routing network, which includes a lora module (expert network) and a router module (routing gate). The lora module and the router module are randomly initialized. When fine-tuning the pre-trained model of the speech recognition model, the linear layer module parameters of the pre-trained model are not updated, but only the parameters of the lora module and the router module are updated.

[0073] In practical applications, if the router module is directly allowed to learn autonomously, the utilization rate of different expert networks in the pre-training model may be uneven and the expert networks may be homogenized, and some expert networks may be selected too frequently, resulting in overfitting or uneven load. Therefore, in the embodiments of the present application, the learning of the router module is limited. Therefore, in the embodiments of the present application, the training of the router module consists of two stages. The multi-dialect data has corresponding dialect labels, so when the first stage of training is started, multi-task training of the speech recognition task and the dialect recognition task is performed. The extra dialect recognition task uses a cross-entropy loss function (dialect task loss function) to bind the expert network of the lora module with the dialect data set. After certain conditions are met, for example, after the lora module and the router module have certain selection capabilities, the second stage of training is entered, and the dialect label is no longer used, that is, only the dialect data in the dialect data set can be used for model training, and the dialect recognition task and the cross-entropy loss function of the dialect recognition are removed, so that the pre-training model actively learns and improves the generalization ability of the pre-training model.

[0074] In an embodiment of the present application, the expert routing network in the pre-training model is trained using the dialect data in the dialect data set and the dialect label to perform first stage training, comprising:

[0075] The expert routing network in the pre-training model is trained using the dialect data in the dialect data set and the dialect label.

[0076] When the number of iterations of training the expert routing network in the pre-training model using the dialect data in the dialect data set and the dialect label reaches a preset number of iterations, it is determined that the first stage training of the pre-training model is complete.

[0077] In specific implementation, supervised learning is a machine learning method that trains a model using training data with labels (such as dialect labels). In the process of supervised learning, the goal of the model is to learn the relationship between the input data and the label, so that it can predict the corresponding label according to the input data.

[0078] In the embodiment of the present application, in the first stage training process, the expert network and the routing gate of the pre-training model are supervised learning by using the dialect data and the dialect labels corresponding to the dialect data, so as to help the expert network to bind the respective dialects for efficient learning. After a certain number of iterations (iterations) of supervised learning of the expert network and the routing gate of the pre-training model by using the dialect data and the dialect labels corresponding to the dialect data, for example, after 5 rounds of supervised learning of the expert network and the routing gate of the pre-training model by using the dialect data and the dialect labels corresponding to the dialect data, it can be considered that the expert network and the routing gate have certain selection ability, and then the second training stage of the pre-training model can be further trained. In the second training stage, the dialect labels are no longer used as input data, that is, only the dialect data is used as input data.

[0079] In an embodiment of the present application, the second stage training of the expert routing network in the pre-training model by using the dialect data in the dialect data set can include:

[0080] The dialect data in the dialect data set is input into the routing gate and the expert network in the pre-training model to obtain an output result of the pre-training model.

[0081] The loss value of the pre-training model is calculated according to the output result and the dialect labels of the dialect data set. The loss value is used to adjust the parameters of the routing gate and the expert network in the pre-training model.

[0082] When the loss value meets the preset convergence condition, it is determined that the second stage training of the pre-training model is completed.

[0083] In an embodiment of the present application, the output result can be calculated by using the following formula:

[0084]

[0085] G(x)=Softmax(W g *x)

[0086] Wherein, o is the output result of the pre-training model, G(x) is the routing score output by the routing gate, x is the dialect data set, W0 is the original weight matrix, α and r are constant hyperparameters, B and A are learnable low-quality matrices, and W g is the weight network of the routing gate.

[0087] Wherein, Lora is a technology for fine-tuning of a large model (such as a pre-training model of a speech recognition model), which reduces the parameter quantity by introducing a low-rank matrix (A / B) to improve the training efficiency while maintaining the expression ability of the model. Specifically, the main idea of Lora is to decompose the weight matrix of the original model into the product of two low-rank matrices, so that only a small number of parameters need to be updated during fine-tuning, avoiding the catastrophic forgetting problem brought by full-parameter fine-tuning.

[0088] Referring to Figure 2 , a structure diagram of a MOE-LORA module in a pre-training model provided in an embodiment of the present application, specifically, the MOE-LORA module (expert routing network) is composed of a router and some expert networks (E1, …, E n ), the output result h of the MOE-LORA module can be represented by the following formula: For the router, an expert network score can be calculated for each expert network using the input x, and the formula is as follows: G(x) = Softmax(W g *x), where W g is the weight network of the router G, and x is the input (dialect data set after vectorization processing). All lora modules share a router module, wherein the expert network uses a lora layer, and the overall MOE-LORA can use learnable low-quality matrices (A1, A2, …, An, and B1, B2, …, Bn) to reduce the parameter quantity and improve the training efficiency of the pre-training model.

[0089] As can be seen from the above, in the pre-training model of the ASR model (speech recognition model) in the embodiment of the present application, the MOE-LORA module is added for fine-tuning, wherein the MOE-LORA module can include a routing gate and multiple expert networks, can introduce additional parameters, increase the parameter quantity of the speech recognition model, and improve the learnable ability of the speech recognition model. Without retraining the base model of the speech recognition model, by implicitly allowing each expert network to focus on learning its own dialect, the same speech recognition model has multiple dialect recognition capabilities, solving the problem of learning effect of small dialects suppressed by large dialects due to unbalanced data.

[0090] In the embodiment of the present application, the routing gate of the pre-training model adopts a two-step training strategy. Specifically, in the first stage of training, the routing gate is supervised to learn using dialect data and dialect labels corresponding to the dialect data, thereby helping the expert network to bind the respective dialects for efficient learning. In the second stage of training, the learning is performed without the dialect labels, improving the generalization ability of the speech recognition model and enabling the learned knowledge of the expert networks to help each other, thereby enabling the speech recognition model to achieve better speech recognition rate for multiple dialects and optimizing the speech recognition experience of users using different dialects.

[0091] It should be noted that, for the method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the embodiments of the present application are not limited by the order of the described actions, because according to the embodiments of the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present application.

[0092] Reference Figure 3 The structure block diagram of a multi-dialect speech recognition model training device provided in the embodiment of the present application is shown in Figure 3 The device can specifically include the following modules:

[0093] The pre-training model acquisition module 301 is configured to acquire a pre-training model of a speech recognition model. The pre-training model includes a plurality of network layers, a specified linear layer in the network layer is replaced by an expert routing network, and the expert routing network includes a routing gate and an expert network.

[0094] The dialect data set acquisition module 302 is configured to acquire a dialect data set of at least one dialect after vectorization.

[0095] The speech recognition model completion module 303 is configured to train the expert routing network in the pre-training model using the dialect data set, and obtain a trained speech recognition model when the training model reaches a preset convergence condition. The speech recognition model performs speech recognition on dialect speech of at least one dialect.

[0096] In an embodiment of the present application, the network layer of the pre-training model includes a feedforward layer and a multi-head self-attention layer, and the specified linear layer includes at least a linear layer in the feedforward layer and a linear layer in the multi-head self-attention layer.

[0097] In an embodiment of the present application, the device further includes:

[0098] Acquiring full-amount data, and extracting acoustic features from the full-amount data.

[0099] The acoustic features are used to train a voice recognition model to be trained to obtain a pre-trained model of the voice recognition model.

[0100] In an embodiment of the present application, the dialect data set includes dialect data and dialect labels corresponding to the dialect data; and the expert routing network in the pre-trained model is trained using the dialect data set, including:

[0101] The expert routing network in the pre-trained model is trained using the dialect data and the dialect labels in the dialect data set to perform first-stage training;

[0102] The expert routing network in the pre-trained model is trained using the dialect data in the dialect data set to perform second-stage training.

[0103] In an embodiment of the present application, the expert routing network in the pre-trained model is trained using the dialect data and the dialect labels in the dialect data set to perform first-stage training, including:

[0104] The expert routing network in the pre-trained model is trained using the dialect data and the dialect labels in the dialect data set;

[0105] When the number of iterations of training the expert routing network in the pre-trained model using the dialect data and the dialect labels in the dialect data set reaches a preset number of iterations, it is determined that the first-stage training of the pre-trained model is completed.

[0106] In an embodiment of the present application, the expert routing network in the pre-trained model is trained using the dialect data in the dialect data set, including:

[0107] The dialect data in the dialect data set is input into the routing gate and the expert network in the pre-trained model to obtain an output result of the pre-trained model;

[0108] A loss value of the pre-trained model is calculated according to the output result and the dialect labels of the dialect data set; the loss value is used to adjust parameters of the routing gate and the expert network in the pre-trained model;

[0109] When the loss value meets a preset convergence condition, it is determined that the second-stage training of the pre-trained model is completed.

[0110] In an embodiment of the present application, the output result is calculated using the following formula:

[0111]

[0112] G(x) = Softmax(W g *x)

[0113] Where o is the output of the pre-trained model, G(x) is the routing score output by the routing gate, x is the dialect dataset, W0 is the original weight matrix, α and r are constant hyperparameters, B and A are learnable low-quality matrices, and W g The weighted network for the routing gate.

[0114] In this embodiment of the invention, a pre-trained model for a speech recognition model is obtained. This pre-trained model includes multiple network layers, with designated linear layers replaced by an expert routing network. The expert routing network includes a routing gate and an expert network. A dialect dataset, vectorized from at least one dialect, is obtained. The expert routing network in the pre-trained model is trained using this dialect dataset. When the trained model reaches a preset convergence condition, a trained speech recognition model is obtained. After deployment, the speech recognition model can perform speech recognition on at least one dialect. In this embodiment, the pre-trained model already possesses a certain level of speech recognition capability. The portion of the pre-trained model that is replaced with the expert routing network is trained to adjust parameters. Implicitly, each expert network in the expert routing network focuses on learning its own dialect, enabling the same speech recognition model to recognize multiple dialects. This allows the same speech recognition model to be deployed in different regions, effectively reducing operating costs. It also avoids the problem of dialects with larger datasets suppressing the learning effect of dialects with smaller datasets due to imbalanced training datasets, ensuring the speech recognition model's effectiveness in recognizing different dialects.

[0115] The above-described apparatus embodiments are basically similar to the method embodiments, so they are described in a relatively simple manner. For relevant details, please refer to the description of the method embodiments.

[0116] It should be noted that the embodiments of the present invention may involve the use of user data. In practical applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, provided that it complies with the applicable laws and regulations of the country (e.g., with the user's explicit consent, with the user being properly notified, etc.).

[0117] This invention also provides an electronic device, such as... Figure 4 As shown, it includes a processor 501, a communication interface 502, a memory 503, and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.

[0118] Memory 503 is used to store computer programs;

[0119] The processor 501 is configured to implement the multi-dialect speech recognition model training method according to any one of the above embodiments when executing the program stored in the memory 503.

[0120] obtaining a pre-training model of a speech recognition model; wherein the pre-training model comprises a plurality of network layers, a specified linear layer in the network layers is replaced with an expert routing network, and the expert routing network comprises a routing gate and an expert network;

[0121] obtaining a dialect data set after vectorization of at least one dialect;

[0122] training the expert routing network in the pre-training model using the dialect data set, and obtaining a trained speech recognition model when the trained model reaches a preset convergence condition; the speech recognition model is used for speech recognition of dialect speech of at least one dialect. The network layers of the pre-training model comprise a feedforward layer and a multi-head self-attention layer, and the specified linear layer comprises at least a linear layer in the feedforward layer and a linear layer in the multi-head self-attention layer.

[0123] Optionally, before obtaining the pre-training model of the speech recognition model, the method further comprises:

[0124] obtaining full-amount data, and extracting acoustic features from the full-amount data;

[0125] training the speech recognition model to be trained using the acoustic features to obtain the pre-training model of the speech recognition model.

[0126] Optionally, the dialect data set comprises dialect data and dialect labels corresponding to the dialect data; and the training of the expert routing network in the pre-training model using the dialect data set comprises:

[0127] performing first stage training of the expert routing network in the pre-training model using the dialect data and the dialect labels in the dialect data set;

[0128] performing second stage training of the expert routing network in the pre-training model using the dialect data in the dialect data set.

[0129] Optionally, the first stage training of the expert routing network in the pre-training model using the dialect data and the dialect labels in the dialect data set comprises:

[0130] training the expert routing network in the pre-training model using the dialect data and the dialect labels in the dialect data set;

[0131] When the number of iterations of training the expert routing network in the pre-training model by using the dialect data in the dialect data set and the dialect labels reaches a preset number of iterations, it is determined that the first stage training of the pre-training model is completed.

[0132] Optionally, the expert routing network in the pre-training model is trained in a second stage by using the dialect data in the dialect data set, comprising:

[0133] The dialect data in the dialect data set is input into the routing gate and the expert network in the pre-training model to obtain an output result of the pre-training model;

[0134] A loss value of the pre-training model is calculated according to the output result and the dialect labels of the dialect data set; the loss value is used to adjust the parameters of the routing gate and the expert network in the pre-training model;

[0135] When the loss value meets a preset convergence condition, it is determined that the second stage training of the pre-training model is completed.

[0136] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0137] The communication interface is used for communication between the terminal and other devices.

[0138] The memory can include a Random Access Memory (RAM) and can also include a non-volatile memory, such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0139] The processor described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; or can be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0140] In a further embodiment provided by the present application, a computer readable storage medium is also provided, which stores instructions, when executed on a computer, cause the computer to perform the multi-dialect speech recognition model training method according to any one of the above embodiments.

[0141] In a further embodiment provided by the present application, a computer program product containing instructions is also provided, which, when executed on a computer, cause the computer to perform the multi-dialect speech recognition model training method according to any one of the above embodiments.

[0142] In the above embodiments, the implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented by using software, the implementation can be achieved entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the entire or partial process or function described in the embodiments of the present application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0143] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0144] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0145] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for training a multi-dialect speech recognition model, characterized in that, The method includes: Obtain a pre-trained model of a speech recognition model; wherein the pre-trained model includes multiple network layers, and a specified linear layer in the network layer is replaced with an expert routing network, the expert routing network including a routing gate and an expert network; Obtain at least one dialect vectorized dialect dataset, wherein the dialect dataset includes dialect data and dialect labels corresponding to the dialect data; The expert routing network in the pre-trained model is trained using the dialect dataset, and a trained speech recognition model is obtained when the trained model reaches a preset convergence condition; the speech recognition model performs speech recognition on at least one dialect. The step of training the expert routing network in the pre-trained model using the dialect dataset includes: The first stage of training is performed by using dialect data and dialect labels from the dialect dataset to train the expert routing network in the pre-trained model. The dialect data in the dialect dataset is used to perform a second-stage training of the expert routing network in the pre-trained model.

2. The method according to claim 1, characterized in that, The network layers of the pre-trained model include feedforward layers and multi-head self-attention layers, and the specified linear layers include at least the linear layers in the feedforward layers and the linear layers in the multi-head self-attention layers.

3. The method according to claim 1, characterized in that, Before obtaining the pre-trained model of the speech recognition model, the method further includes: Acquire the full dataset and extract acoustic features from it; The acoustic features are used to train the speech recognition model to be trained, resulting in a pre-trained model of the speech recognition model.

4. The method according to claim 1, characterized in that, The first stage of training involves using dialect data and dialect labels from the dialect dataset to train the expert routing network in the pre-trained model, including: The expert routing network in the pre-trained model is trained using dialect data and dialect labels from the dialect dataset. When the number of iterations for training the expert routing network in the pre-trained model using dialect data and dialect labels in the dialect dataset reaches a preset number of iterations, the first stage of training of the pre-trained model is determined to be completed.

5. The method according to claim 1, characterized in that, The second stage of training is performed on the expert routing network in the pre-trained model using dialect data from the dialect dataset, including: The dialect data in the dialect dataset is input into the routing gating and the expert network in the pre-trained model to obtain the output of the pre-trained model. The loss value of the pre-trained model is calculated based on the output and the dialect labels of the dialect dataset; the loss value is used to adjust the parameters of the routing gating and the expert network in the pre-trained model. When the loss value meets the preset convergence condition, the second stage of training of the pre-trained model is determined to be completed.

6. The method according to claim 5, characterized in that, The output result is calculated using the following formula: in, The output of the pre-trained model. The routing score output by the routing gate. For dialect datasets, This is the original weight matrix. and Let B and A be constant hyperparameters, and B and A be learnable low-quality matrices. The weighted network for the routing gate.

7. A multi-dialect speech recognition model training device, characterized in that, The device includes: A pre-trained model acquisition module is used to acquire a pre-trained model of a speech recognition model; wherein, the pre-trained model includes multiple network layers, and a specified linear layer in the network layer is replaced with an expert routing network, the expert routing network including a routing gate and an expert network; A dialect dataset acquisition module is used to acquire at least one vectorized dialect dataset, wherein the dialect dataset includes dialect data and dialect labels corresponding to the dialect data; The speech recognition model completion module is used to train the expert routing network in the pre-trained model using the dialect dataset, and to obtain the trained speech recognition model when the trained model reaches a preset convergence condition; the speech recognition model performs speech recognition on at least one dialect. The step of training the expert routing network in the pre-trained model using the dialect dataset includes: The first stage of training is performed by using dialect data and dialect labels from the dialect dataset to train the expert routing network in the pre-trained model. The dialect data in the dialect dataset is used to perform a second-stage training of the expert routing network in the pre-trained model.

8. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

Images