Multi-dialect speech recognition model training method and device

Abstract

The embodiment of the invention provides a multi-dialect speech recognition model training method and device, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining a pre-training model of a speech recognition model; wherein the pre-training model comprises a plurality of network layers, a specified linear layer in the network layers is replaced by an expert routing network, and the expert routing network comprises a routing gate and an expert network; obtaining at least one dialect data set after dialect vectorization; training an expert routing network in the pre-training model by adopting the dialect data set, and obtaining a trained speech recognition model when the training model reaches a preset convergence condition; voice recognition is carried out on dialect voice of at least one dialect in the voice recognition model. According to the embodiment of the invention, the same voice recognition model can be deployed in different regions by using the unified voice recognition model, so that the operation cost is effectively reduced.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of artificial intelligence technology, and in particular to a multi-dialect speech recognition model training method, a multi-dialect speech recognition model training device, an electronic device, and a computer-readable storage medium. Background Art

[0002] ASR (Automatic Speech Recognition) can convert speech into computer-readable text.

[0003] At present, speech recognition technology plays an important role in a wide range of fields such as artificial intelligence. However, most speech recognition models based on speech recognition technology can recognize standard Mandarin relatively accurately. However, for dialects such as Cantonese, Minnan, Chaoshan, etc., speech recognition models usually need to train a small model separately for each dialect, which increases operating costs. Summary of the invention

[0004] In view of the above problems, a method and device for training a multi-dialect speech recognition model is proposed to overcome the above problems or at least partially solve the above problems. The specific technical solution is as follows:

[0005] In a first aspect of the present invention, a multi-dialect speech recognition model training method is first provided, the method comprising:

[0006] Obtain a pre-trained model of a speech recognition model; wherein the pre-trained model includes multiple network layers, a specified linear layer in the network layer is replaced with an expert routing network, and the expert routing network includes routing gating and an expert network;

[0007] Obtain a dialect dataset after vectorization of at least one dialect;

[0008] The expert routing network in the pre-trained model is trained using the dialect data set, and when the training model reaches a preset convergence condition, a trained speech recognition model is obtained; the speech recognition model performs speech recognition on dialect speech of at least one dialect.

[0009] In one embodiment of the present invention, the network layer of the pre-trained model includes a feedforward layer and a multi-head self-attention layer, and the specified linear layer includes at least a linear layer in the feedforward layer and a linear layer in the multi-head self-attention layer.

[0010] In one embodiment of the present invention, before obtaining the pre-trained model of the speech recognition model, the method further includes:

[0011] Acquiring full data, and extracting acoustic features from the full data;

[0012] The acoustic features are used to train the speech recognition model to be trained to obtain a pre-trained model of the speech recognition model.

[0013] In one embodiment of the present invention, the dialect data set includes dialect data and dialect labels corresponding to the dialect data; and using the dialect data set to train the expert routing network in the pre-trained model includes:

[0014] Using the dialect data in the dialect data set and the dialect label to train the expert routing network in the pre-trained model for the first stage of training;

[0015] The expert routing network in the pre-trained model is trained in the second stage using the dialect data in the dialect data set.

[0016] In one embodiment of the present invention, the first stage of training is performed by using the dialect data in the dialect data set and the dialect label to train the expert routing network in the pre-trained model, including:

[0017] Using the dialect data in the dialect data set and the dialect label to train the expert routing network in the pre-trained model;

[0018] When the number of iterations of training the expert routing network in the pre-trained model using the dialect data in the dialect data set and the dialect label reaches a preset number of iterations, it is determined that the first stage training of the pre-trained model is completed.

[0019] In one embodiment of the present invention, the second stage training of the expert routing network in the pre-trained model is performed using the dialect data in the dialect data set, including:

[0020] Input the dialect data in the dialect data set into the routing gating in the pre-trained model and the expert network to obtain the output result of the pre-trained model;

[0021] Calculating the loss value of the pre-trained model according to the output result and the dialect label of the dialect data set; the loss value is used to adjust the parameters of the routing gating and the expert network in the pre-trained model;

[0022] When the loss value satisfies a preset convergence condition, it is determined that the second stage training of the pre-trained model is completed.

[0023] In one embodiment of the present invention, the output result is calculated using the following formula:

[0024]

[0025] G(x)=Softmax(W g *x)

[0026] Wherein, o is the output result of the pre-trained model, G(x) is the routing score output by the routing gating, x is the dialect dataset, W0 is the original weight matrix, α and r are constant hyperparameters, B and A are learnable low-quality matrices, and W g is the weight network of the routing gating.

[0027] The embodiment of the present invention also discloses a multi-dialect speech recognition model training device, the device comprising:

[0028] A pre-trained model acquisition module, used to acquire a pre-trained model of a speech recognition model; wherein the pre-trained model includes a plurality of network layers, a specified linear layer in the network layer is replaced with an expert routing network, and the expert routing network includes routing gating and an expert network;

[0029] A dialect data set acquisition module, used to acquire a vectorized dialect data set of at least one dialect;

[0030] The speech recognition model completion module is used to use the dialect data set to train the expert routing network in the pre-trained model, and when the training model reaches a preset convergence condition, obtain a trained speech recognition model; the speech recognition model performs speech recognition on dialect speech of at least one dialect.

[0031] In another aspect of the present invention, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes any of the multi-dialect speech recognition model training methods described above.

[0032] In another aspect of the present invention, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the multi-dialect speech recognition model training methods described above.

[0033] Compared with the related art, the embodiments of the present invention have at least the following advantages:

[0034] In an embodiment of the present invention, a pre-trained model of a speech recognition model is obtained, wherein the pre-trained model includes multiple network layers, a designated linear layer in the network layer is replaced by an expert routing network, the expert routing network includes routing gating and an expert network, a dialect data set after vectorization of at least one dialect is obtained, the expert routing network in the pre-trained model is trained using the dialect data set, and when the training model reaches a preset convergence condition, a trained speech recognition model is obtained, and after the speech recognition model is deployed, speech recognition can be performed on dialect speech of at least one dialect. The pre-trained model of the speech recognition model in the embodiment of the present invention already has a certain speech recognition capability, and the part of the pre-trained model replaced by the expert routing network is trained to adjust parameters, implicitly allowing each expert network in the expert routing network to focus on learning their own dialects, so that the same speech recognition model has the recognition capability of multiple dialects. In this way, the use of a unified speech recognition model allows the same speech recognition model to be deployed in different regions, effectively reducing operating costs, while avoiding the problem that the dialect with a large amount of data suppresses the learning effect of the dialect with a small amount of data due to the imbalance of dialect data sets used for training different dialects, thereby ensuring the speech recognition effect of the speech recognition model on different dialects. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art.

[0036] Figure 1 A flowchart of a multi-dialect speech recognition model training method provided in an embodiment of the present invention;

[0037] Figure 2 A structural diagram of a MOE-LORA module in a pre-training model provided in an embodiment of the present invention;

[0038] Figure 3 A structural block diagram of a multi-dialect speech recognition model training device provided in an embodiment of the present invention;

[0039] Figure 4 The present invention is a block diagram of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0041] In order to enable those skilled in the art to better understand the embodiments of the present invention, some terms and names involved in the embodiments of the present invention are explained below.

[0042] Conformer: A neural network model structure;

[0043] LORA (Low-Rank Adaptation): is a technology used for fine-tuning large models. It reduces the number of parameters and improves training efficiency by introducing low-rank matrices while maintaining the expressiveness of the model.

[0044] Adapter: It is a technique for fine-tuning large models. It adapts to new tasks by inserting additional modules (Adapter modules) into certain layers of the pre-trained model, thereby reducing the number of parameters and improving training efficiency while maintaining the expressiveness of the model.

[0045] MOE (Mixture of Experts): A large model architecture idea that can improve the performance of the model by mixing multiple expert networks. Specifically, the main idea of ​​MOE is to distribute the input data to multiple expert networks, each of which focuses on processing a specific task or subtask, and then use a gating mechanism (Router) to determine the weight of each expert network to obtain the final output result.

[0046] The goal is to use the full amount of dialect data to fine-tune the full amount of parameters of the speech recognition model. One of the current methods is to use the full amount of dialect data to fine-tune the full amount of parameters of the speech recognition model. There are two shortcomings. First, the current speech recognition model parameters are very large. If the full amount of parameter fine-tuning is used, the training speed of the speech recognition model will be relatively slow. Secondly, the full amount of parameter fine-tuning will change all the parameters of the pre-trained model of the speech recognition model, which is easy to cause catastrophic forgetting, that is, when adapting to new tasks, the speech recognition model loses its memory of the original knowledge. Finally, in general, there is data imbalance in dialect data. Full parameter fine-tuning will make the speech recognition model biased towards the side with more data, while reducing the recognition effect of the dialect with less data. Balancing the proportion of dialect data is a difficult trade-off.

[0047] In some cases, the current method also uses multiple dialect sub-models and applies them to respective scenarios to improve the dialect recognition effect in various regions. The disadvantage of this method is that the operating cost is high, and the user needs to select the dialect area in advance to perform the corresponding dialect recognition, which has poor flexibility.

[0048] In some cases, current methods also include using LORA and Adapter fine-tuning to improve the recognition effect of the speech recognition model for the corresponding dialect. Although this fine-tuning method is faster than full parameter fine-tuning training and alleviates catastrophic forgetting, it only uses a fixed set of fine-tuning parameters to process multi-dialect data. It cannot fully learn the complexity of the data and can easily cause the speech recognition model to fall into a local optimal solution.

[0049] In some cases, the current method is to add MOE modules to the basic model to improve the recognition of multiple dialects. The basic model needs to be retrained. Generally, the amount of data used by the basic model is huge, which is time-consuming and costly.

[0050] In response to the above problems, an embodiment of the present invention proposes a method for fine-tuning a multi-dialect speech recognition model. The method is more flexible and can be fine-tuned only for dialect data or newly added dialect data. The basic model structure of most ASR models (speech recognition models) (as long as the model includes a linear layer, and basically all models have a linear layer) can use the method of the embodiment of the present invention to further improve the recognition rate of the model in multiple dialects.

[0051] Specifically, the embodiment of the present invention translates the multi-dialect conversation audio into text. First, the embodiment of the present invention only needs one model to complete the multi-dialect speech recognition, rather than multiple dialect small models to complete the speech recognition task; secondly, the embodiment of the present invention focuses on using multi-dialect data to fine-tune the ASR basic model, so that the speech recognition model can achieve better speech recognition effect on multi-dialect data. The embodiment of the present invention can be applied to a variety of businesses, such as the following two businesses: 1. Intelligent voice assistant: In a multi-dialect family, family members may use different dialects to communicate with the voice assistant, and the system can recognize their voice needs based on each person's dialect or accent. 2. Customer service and call center: The customer service department needs to handle customer requests from different regions and language backgrounds. Traditional single language or dialect speech recognition systems may not be able to accurately understand the speech expressions of users in different regions, especially in multilingual countries or regions, where dialect differences may be very large. The current method is to deploy corresponding dialect small models in different regions to achieve the required recognition rate, while the method of the embodiment of the present invention allows different regions to deploy the same speech recognition model by using a unified multi-dialect recognition model, effectively reducing operating costs.

[0052] Reference Figure 1 , is a flowchart of the steps of a multi-dialect speech recognition model training method provided in an embodiment of the present invention, such as Figure 1 As shown, the method may specifically include the following steps:

[0053] Step 101, obtaining a pre-trained model of a speech recognition model; wherein the pre-trained model includes multiple network layers, a specified linear layer in the network layer is replaced with an expert routing network, and the expert routing network includes routing gating and an expert network;

[0054] Among them, the speech recognition model (multi-dialect recognition model, ASR model) can be a speech recognition model based on a conformer. The conformer is a neural network architecture designed specifically for speech recognition, which cleverly combines CNNs (Convolution Neural Networks) and Transformers (Transformers is a neural network architecture based on the self-attention mechanism) to optimize the understanding of speech data. Compared with speech recognition models based on Transformers or CNNs, the conformer-based speech recognition model performs better and can achieve better speech recognition effects with lower computing resources.

[0055] In one embodiment of the present invention, the network layer of the pre-trained model may include at least a feedforward layer and a multi-head self-attention layer, and the specified linear layer includes at least a linear layer in the feedforward layer and a linear layer in the multi-head self-attention layer. Among them, the linear layer, also known as the fully connected layer, is one of the most basic layer types in a neural network. The linear layer can map the input features to the output feature space through a linear transformation. The linear layer is usually used to convert the input features into a higher-dimensional or lower-dimensional feature representation.

[0056] In the embodiment of the present invention, the pre-trained model of the speech recognition model is a basic model with a specific speech recognition ability after training with full data. For the pre-trained model, its specified linear layer, that is, the linear layer in the feedforward layer and the linear layer of the multi-head self-attention layer, can be replaced with an expert routing network (MOE-LORA module / lora module and router module), wherein the expert routing network may include routing gating (router G / router) and multiple expert networks (E1, ..., E n ).

[0057] Specifically, the role of routing gating is to select a suitable expert network based on the input feature vector (the vectorized dialect data set). Routing gating can calculate the score of the input feature vector for each expert network. The score can determine the weight of each expert network, wherein the expert network with a high score will be assigned a higher weight, thereby playing a greater role in the final output result of the speech recognition model. Each expert network (Expert) is a smaller neural network that focuses on processing a specific task or subtask. For example, in the multi-dialect speech recognition task of the embodiment of the present invention, different expert networks can focus on recognizing different dialects or accents. In addition, since each expert network can learn different feature representations, the expressiveness and generalization ability of the speech recognition model are improved.

[0058] It should be noted that no matter what network structure the pre-trained model of the speech recognition model has, as long as there is a linear layer, the expert routing network of the embodiment of the present invention can be used to replace the corresponding linear layer in the pre-trained model (for example, the linear layer in the feedforward layer and the linear layer in the multi-head self-attention layer), and then the pre-trained model after replacing the linear layer can be fine-tuned in multiple dialects, so that the final speech recognition model can have better and more balanced effects on the dialect test set.

[0059] Step 102: Obtain at least one dialect vectorized dialect data set;

[0060] In an embodiment of the present invention, all dialect data sets of at least one dialect used to fine-tune the pre-trained model are collated. For example, if the pre-trained model is required to have or optimize the speech recognition effects of dialect A, dialect B, and dialect C, then the dialect data sets corresponding to dialect A, dialect B, and dialect C are obtained, wherein the dialect data sets need to be vectorized before being input into the pre-trained model. The dialect data set may include dialect data and dialect labels corresponding to the dialect data, wherein the dialect label refers to a label used to identify the dialect to which the speech data belongs in a multi-dialect speech recognition task, and the dialect label is a clear supervisory signal in the training process of the speech recognition model, which can help the speech recognition model learn how to predict the corresponding dialect label based on the input speech data. Exemplarily, dialect labels may include, but are not limited to, Cantonese, Minnan, Chaoshan, and other labels.

[0061] Step 103: Use the dialect data set to train the expert routing network in the pre-trained model, and when the training model reaches a preset convergence condition, obtain a trained speech recognition model; in the speech recognition model, speech recognition is performed on dialect speech of at least one dialect.

[0062] In an embodiment of the present invention, after obtaining a dialect data set, the dialect data set can be used to train the expert routing network (routing gating and expert network) in the pre-trained model, and when the training model reaches a preset convergence condition (for example, the number of iterative training of the pre-trained model reaches a preset number of times or the loss value of the pre-trained model is a preset loss value), a trained speech recognition model can be obtained. Subsequently, the speech recognition model can be deployed to a corresponding system to perform speech recognition on dialect speech of at least one dialect based on the speech recognition model.

[0063] In the above-mentioned multi-dialect speech recognition model training, a pre-trained model of the speech recognition model is obtained, wherein the pre-trained model includes multiple network layers, the specified linear layer in the network layer is replaced by an expert routing network, the expert routing network includes routing gating and an expert network, a dialect data set after vectorization of at least one dialect is obtained, the dialect data set is used to train the expert routing network in the pre-trained model, and when the training model reaches a preset convergence condition, a trained speech recognition model is obtained. After the speech recognition model is deployed, speech recognition can be performed on dialect speech of at least one dialect. The pre-trained model of the speech recognition model in the embodiment of the present invention already has a certain speech recognition ability. The part of the pre-trained model that is replaced by the expert routing network is trained to adjust parameters, implicitly allowing each expert network in the expert routing network to focus on learning their own dialects, so that the same speech recognition model has the recognition ability of multiple dialects. In this way, the use of a unified speech recognition model can allow the same speech recognition model to be deployed in different regions, effectively reducing operating costs. At the same time, it avoids the problem that the learning effect of the dialect with a large amount of data suppresses the dialect with a small amount of data due to the imbalance of dialect data sets used for training different dialects, thereby ensuring the speech recognition effect of the speech recognition model on different dialects.

[0064] In one embodiment of the present invention, before obtaining the pre-trained model of the speech recognition model, the method may further include:

[0065] Acquiring full data, and extracting acoustic features from the full data;

[0066] The acoustic features are used to train the speech recognition model to be trained to obtain a pre-trained model of the speech recognition model.

[0067] Among them, the pre-trained model refers to a base model (Base Model) pre-trained on large-scale data (full data). The pre-trained model can learn general feature representations by using large-scale data for training, and then fine-tune parameters on specific tasks (such as dialect recognition tasks) to adapt to different application scenarios. Specifically, full data refers to the speech data that can be collected to train the speech recognition model to be trained to obtain the pre-trained model. It should be noted that the full data can include speech data corresponding to dialects and non-dialects.

[0068] In the embodiment of the present invention, 80-dimensional FBANK (Filter Bank, a method of representing speech features) acoustic features are extracted from speech data (full data) as input data for the speech recognition model to be trained. Of course, the embodiment of the present invention may also use other speech feature representation methods, and the embodiment of the present invention does not need to be limited to this. Then, the conformer-based speech recognition model can be trained for the ASR speech recognition task using the acoustic features extracted from the full data to obtain a pre-trained model of the speech recognition model. After obtaining the pre-trained model, the parameters of the expert routing network in the pre-trained model can be fine-tuned according to the dialect data set of the dialect to be recognized, and finally a speech recognition model capable of recognizing different dialects can be obtained.

[0069] In an embodiment of the present invention, the dialect data set may include dialect data and dialect labels corresponding to the dialect data; and using the dialect data set to train the expert routing network in the pre-trained model may include:

[0070] Using the dialect data in the dialect data set and the dialect label to train the expert routing network in the pre-trained model for the first stage of training;

[0071] The expert routing network in the pre-trained model is trained in the second stage using the dialect data in the dialect data set.

[0072] In an embodiment of the present invention, before fine-tuning the pre-trained model of the speech recognition model, the weights of the pre-trained model are loaded into a new structure, namely, an expert routing network, the expert routing network includes a LoRa module (expert network) and a router module (routing gating), the LoRa module and the router module are randomly initialized, and when fine-tuning the pre-trained model of the speech recognition model, the linear layer module parameters of the pre-trained model are not updated, but only the parameters of the LoRa module and the router module are updated.

[0073] In practical applications, if the router module is allowed to learn autonomously directly, the utilization rates of different expert networks in the pre-trained model will be unbalanced and the expert networks will be homogenized. Some expert networks may be selected too frequently, resulting in overfitting or uneven load. Therefore, in an embodiment of the present invention, the learning of the router module is restricted. Therefore, in an embodiment of the present invention, the training of the router module consists of two stages, and the multi-dialect data has corresponding dialect labels. Therefore, at the beginning of the training (first stage training), multi-task training of speech recognition tasks and dialect recognition tasks will be performed, wherein the extra dialect recognition tasks use the cross entropy loss function (dialect task loss function), and the dialect labels are used to bind the dialect data set to the expert network of the lora module. After meeting certain conditions, for example, after the lora module and the router module have a certain selection ability, the second stage of training is entered, and the dialect labels are no longer used, that is, only the dialect data in the dialect data set can be used for model training, and the dialect recognition task and the cross entropy loss function of dialect recognition are removed, so that the pre-trained model can learn actively and improve the generalization ability of the pre-trained model.

[0074] In one embodiment of the present invention, the first stage of training is performed by using the dialect data in the dialect data set and the dialect label to train the expert routing network in the pre-trained model, including:

[0075] Using the dialect data in the dialect data set and the dialect label to train the expert routing network in the pre-trained model;

[0076] When the number of iterations of training the expert routing network in the pre-trained model using the dialect data in the dialect data set and the dialect label reaches a preset number of iterations, it is determined that the first stage training of the pre-trained model is completed.

[0077] In specific implementations, supervised learning is a machine learning method that trains a model by using training data with labels (such as dialect labels). In the supervised learning process, the goal of the model is to learn the relationship between input data and labels so that the corresponding labels can be predicted based on the input data.

[0078] In an embodiment of the present invention, during the first stage of training, the expert network and routing gating of the pre-trained model are supervised by using the dialect data and the dialect labels corresponding to the dialect data, so as to help the expert network bind their respective dialects for efficient learning. After a certain number of rounds (iterations at this time) of supervised learning of the expert network and routing gating of the pre-trained model using the dialect data and the dialect labels corresponding to the dialect data are achieved, for example, after 5 rounds of supervised learning of the expert network and routing gating of the pre-trained model using the dialect data and the dialect labels corresponding to the dialect data, it can be considered that the expert network and routing gating have a certain selection ability, and then the second training stage of the pre-trained model can be performed, wherein the dialect labels are no longer used as input data in the second training stage, that is, only the dialect data is used as input data.

[0079] In one embodiment of the present invention, using the dialect data in the dialect data set to perform the second-stage training on the expert routing network in the pre-trained model may include:

[0080] Input the dialect data in the dialect data set into the routing gating in the pre-trained model and the expert network to obtain the output result of the pre-trained model;

[0081] Calculating the loss value of the pre-trained model according to the output result and the dialect label of the dialect data set; the loss value is used to adjust the parameters of the routing gating and the expert network in the pre-trained model;

[0082] When the loss value satisfies a preset convergence condition, it is determined that the second stage training of the pre-trained model is completed.

[0083] In one embodiment of the present invention, the output result can be calculated using the following formula:

[0084]

[0085] G(x)=Softmax(W g *x)

[0086] Wherein, o is the output result of the pre-trained model, G(x) is the routing score output by the routing gating, x is the dialect dataset, W0 is the original weight matrix, α and r are constant hyperparameters, B and A are learnable low-quality matrices, and W g is the weight network of the routing gating.

[0087] Among them, Lora is a technology used for fine-tuning large models (such as pre-trained models of speech recognition models). It reduces the number of parameters and improves training efficiency by introducing low-rank matrices (A / B) while maintaining the expressiveness of the model. Specifically, the main idea of ​​Lora is to decompose the weight matrix of the original model into the product of two low-rank matrices, so that only a small number of parameters need to be updated during fine-tuning, avoiding the catastrophic forgetting problem caused by fine-tuning of all parameters.

[0088] Reference Figure 2 , is a structural diagram of a MOE-LORA module in a pre-training model provided in an embodiment of the present invention. Specifically, the MOE-LORA module (expert routing network) is composed of a router and some expert networks (E1, ..., E n ), the output result h of the MOE-LORA module can be expressed by the following formula: For the router, we can use the input x to calculate an expert network score for each expert network, as follows: G(x) = Softmax(W g *x), where W g is the weight network of router G, and x is the input (the dialect dataset after vectorization). All lora modules share a router module, among which the expert network uses the lora layer, and the overall MOE-LORA can use learnable low-quality matrices (A1, A2...An, and B1, B2...Bn) to reduce the number of parameters and improve the training efficiency of the pre-trained model.

[0089] In summary, in an embodiment of the present invention, a MOE-LORA module is added to the pre-trained model of the ASR model (speech recognition model) for fine-tuning, wherein the MOE-LORA module may include routing gating and multiple expert networks, may introduce additional parameters, increase the parameter amount of the speech recognition model, and improve the learnability of the speech recognition model without retraining the basic model of the speech recognition model. By implicitly allowing each expert network to focus on learning their own dialects, the same speech recognition model has the ability to recognize multiple dialects, thereby solving the problem of dialects with large data volumes suppressing the learning effect of dialects with small data volumes due to data imbalance.

[0090] In an embodiment of the present invention, the routing gating for the pre-trained model adopts a two-step training strategy. Specifically, in the first stage of training, the routing gating uses the dialect data and the dialect labels corresponding to the dialect data for supervised learning at the beginning, thereby helping the expert networks to bind their respective dialects for efficient learning. In the second stage of training, the dialect labels are removed for learning to improve the generalization ability of the speech recognition model, so that the knowledge learned between the expert networks can help each other, thereby allowing the speech recognition model to achieve better speech recognition rate for multiple dialects and optimize the speech recognition experience of users using different dialects.

[0091] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0092] Reference Figure 3 , is a structural block diagram of a multi-dialect speech recognition model training device provided in an embodiment of the present invention, such as Figure 3 As shown, the device may specifically include the following modules:

[0093] A pre-training model acquisition module 301 is used to acquire a pre-training model of a speech recognition model; wherein the pre-training model includes a plurality of network layers, a specified linear layer in the network layer is replaced with an expert routing network, and the expert routing network includes routing gating and an expert network;

[0094] The dialect data set acquisition module 302 is used to acquire a dialect data set after vectorization of at least one dialect;

[0095] The speech recognition model completion module 303 is used to use the dialect data set to train the expert routing network in the pre-trained model, and when the training model reaches a preset convergence condition, obtain a trained speech recognition model; the speech recognition model performs speech recognition on dialect speech of at least one dialect.

[0096] In one embodiment of the present invention, the network layer of the pre-trained model includes a feedforward layer and a multi-head self-attention layer, and the specified linear layer includes at least a linear layer in the feedforward layer and a linear layer in the multi-head self-attention layer.

[0097] In one embodiment of the present invention, the device further comprises:

[0098] Acquiring full data, and extracting acoustic features from the full data;

[0099] The acoustic features are used to train the speech recognition model to be trained to obtain a pre-trained model of the speech recognition model.

[0100] In one embodiment of the present invention, the dialect data set includes dialect data and dialect labels corresponding to the dialect data; and using the dialect data set to train the expert routing network in the pre-trained model includes:

[0101] Using the dialect data in the dialect data set and the dialect label to train the expert routing network in the pre-trained model for the first stage of training;

[0102] The expert routing network in the pre-trained model is trained in the second stage using the dialect data in the dialect data set.

[0103] In one embodiment of the present invention, the first stage of training is performed by using the dialect data in the dialect data set and the dialect label to train the expert routing network in the pre-trained model, including:

[0104] Using the dialect data in the dialect data set and the dialect label to train the expert routing network in the pre-trained model;

[0105] When the number of iterations of training the expert routing network in the pre-trained model using the dialect data in the dialect data set and the dialect label reaches a preset number of iterations, it is determined that the first stage training of the pre-trained model is completed.

[0106] In one embodiment of the present invention, the second stage training of the expert routing network in the pre-trained model is performed using the dialect data in the dialect data set, including:

[0107] Input the dialect data in the dialect data set into the routing gating in the pre-trained model and the expert network to obtain the output result of the pre-trained model;

[0108] Calculating the loss value of the pre-trained model according to the output result and the dialect label of the dialect data set; the loss value is used to adjust the parameters of the routing gating and the expert network in the pre-trained model;

[0109] When the loss value satisfies a preset convergence condition, it is determined that the second stage training of the pre-trained model is completed.

[0110] In one embodiment of the present invention, the output result is calculated using the following formula:

[0111]

[0112] G(x)=Softmax(W g *x)

[0113] Wherein, o is the output result of the pre-trained model, G(x) is the routing score output by the routing gating, x is the dialect dataset, W0 is the original weight matrix, α and r are constant hyperparameters, B and A are learnable low-quality matrices, and W g is the weight network of the routing gating.

[0114] In an embodiment of the present invention, a pre-trained model of a speech recognition model is obtained, wherein the pre-trained model includes multiple network layers, a designated linear layer in the network layer is replaced by an expert routing network, the expert routing network includes routing gating and an expert network, a dialect data set after vectorization of at least one dialect is obtained, the expert routing network in the pre-trained model is trained using the dialect data set, and when the training model reaches a preset convergence condition, a trained speech recognition model is obtained, and after the speech recognition model is deployed, speech recognition can be performed on dialect speech of at least one dialect. The pre-trained model of the speech recognition model in the embodiment of the present invention already has a certain speech recognition capability, and the part of the pre-trained model replaced by the expert routing network is trained to adjust parameters, implicitly allowing each expert network in the expert routing network to focus on learning their own dialects, so that the same speech recognition model has the recognition capability of multiple dialects. In this way, the use of a unified speech recognition model allows the same speech recognition model to be deployed in different regions, effectively reducing operating costs, while avoiding the problem that the dialect with a large amount of data suppresses the learning effect of the dialect with a small amount of data due to the imbalance of dialect data sets used for training different dialects, thereby ensuring the speech recognition effect of the speech recognition model on different dialects.

[0115] As for the above-mentioned device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0116] It should be noted that the embodiments of the present invention may involve the use of user data. In actual applications, user-specific personal data can be used in the scheme described herein within the scope permitted by applicable laws and regulations, subject to the requirements of applicable laws and regulations of the country where the user is located (for example, with the user's explicit consent, effective notification to the user, etc.).

[0117] The embodiment of the present invention further provides an electronic device, such as Figure 4 As shown, it includes a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.

[0118] Memory 503, used for storing computer programs;

[0119] The processor 501 is used to implement the multi-dialect speech recognition model training method described in any one of the above embodiments when executing the program stored in the memory 503:

[0120] Obtain a pre-trained model of a speech recognition model; wherein the pre-trained model includes multiple network layers, a specified linear layer in the network layer is replaced with an expert routing network, and the expert routing network includes routing gating and an expert network;

[0121] Obtain a dialect dataset after vectorization of at least one dialect;

[0122] The expert routing network in the pre-trained model is trained using the dialect data set, and when the training model reaches a preset convergence condition, a trained speech recognition model is obtained; the speech recognition model performs speech recognition on dialect speech of at least one dialect. The network layer of the pre-trained model includes a feedforward layer and a multi-head self-attention layer, and the specified linear layer includes at least a linear layer in the feedforward layer and a linear layer in the multi-head self-attention layer.

[0123] Optionally, before obtaining the pre-trained model of the speech recognition model, the method further includes:

[0124] Acquiring full data, and extracting acoustic features from the full data;

[0125] The acoustic features are used to train the speech recognition model to be trained to obtain a pre-trained model of the speech recognition model.

[0126] Optionally, the dialect data set includes dialect data and dialect labels corresponding to the dialect data; and using the dialect data set to train the expert routing network in the pre-trained model includes:

[0127] Using the dialect data in the dialect data set and the dialect label to train the expert routing network in the pre-trained model for the first stage of training;

[0128] The expert routing network in the pre-trained model is trained in the second stage using the dialect data in the dialect data set.

[0129] Optionally, the first stage of training is performed by using the dialect data in the dialect data set and the dialect label to train the expert routing network in the pre-trained model, including:

[0130] Using the dialect data in the dialect data set and the dialect label to train the expert routing network in the pre-trained model;

[0131] When the number of iterations of training the expert routing network in the pre-trained model using the dialect data in the dialect data set and the dialect label reaches a preset number of iterations, it is determined that the first stage training of the pre-trained model is completed.

[0132] Optionally, the second stage training of the expert routing network in the pre-trained model is performed using the dialect data in the dialect data set, including:

[0133] Input the dialect data in the dialect data set into the routing gating in the pre-trained model and the expert network to obtain the output result of the pre-trained model;

[0134] Calculating the loss value of the pre-trained model according to the output result and the dialect label of the dialect data set; the loss value is used to adjust the parameters of the routing gating and the expert network in the pre-trained model;

[0135] When the loss value satisfies a preset convergence condition, it is determined that the second stage training of the pre-trained model is completed.

[0136] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0137] The communication interface is used for communication between the above terminal and other devices.

[0138] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0139] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0140] In another embodiment provided by the present invention, a computer-readable storage medium is also provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the multi-dialect speech recognition model training method described in any of the above embodiments.

[0141] In another embodiment provided by the present invention, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the multi-dialect speech recognition model training method described in any one of the above embodiments.

[0142] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0143] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or still includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0144] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0145] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A multi-dialect speech recognition model training method, characterized in that: The method comprises: Obtain a pre-trained model of a speech recognition model; wherein the pre-trained model includes multiple network layers, a specified linear layer in the network layer is replaced with an expert routing network, and the expert routing network includes routing gating and an expert network; Obtain a dialect dataset after vectorization of at least one dialect; The expert routing network in the pre-trained model is trained using the dialect data set, and when the training model reaches a preset convergence condition, a trained speech recognition model is obtained; the speech recognition model performs speech recognition on dialect speech of at least one dialect.

2. The method according to claim 1, characterized in that: The network layer of the pre-trained model includes a feedforward layer and a multi-head self-attention layer, and the specified linear layer includes at least a linear layer in the feedforward layer and a linear layer in the multi-head self-attention layer.

3. The method according to claim 1, characterized in that Before obtaining the pre-trained model of the speech recognition model, the method further includes: Acquiring full data, and extracting acoustic features from the full data; The acoustic features are used to train the speech recognition model to be trained to obtain a pre-trained model of the speech recognition model.

4. The method according to claim 1, characterized in that: The dialect data set includes dialect data and dialect labels corresponding to the dialect data; using the dialect data set to train the expert routing network in the pre-trained model includes: Using the dialect data in the dialect data set and the dialect label to train the expert routing network in the pre-trained model for the first stage of training; The expert routing network in the pre-trained model is trained in the second stage using the dialect data in the dialect data set.

5. The method according to claim 4, characterized in that The first stage of training is to train the expert routing network in the pre-trained model using the dialect data in the dialect data set and the dialect label, including: Using the dialect data in the dialect data set and the dialect label to train the expert routing network in the pre-trained model; When the number of iterations of training the expert routing network in the pre-trained model using the dialect data in the dialect data set and the dialect label reaches a preset number of iterations, it is determined that the first stage training of the pre-trained model is completed.

6. The method according to claim 4, characterized in that The expert routing network in the pre-trained model is trained in the second stage using the dialect data in the dialect data set, including: Input the dialect data in the dialect data set into the routing gating in the pre-trained model and the expert network to obtain the output result of the pre-trained model; Calculating the loss value of the pre-trained model according to the output result and the dialect label of the dialect data set; the loss value is used to adjust the parameters of the routing gating and the expert network in the pre-trained model; When the loss value meets the preset convergence condition, it is determined that the second stage training of the pre-trained model is completed.

7. The method according to any one of claim 6, characterized in that: The output result is calculated using the following formula: G(x)=Softmax(W g *x) Wherein, o is the output result of the pre-trained model, G(x) is the routing score output by the routing gating, x is the dialect dataset, W0 is the original weight matrix, α and r are constant hyperparameters, B and A are learnable low-quality matrices, and W g is the weight network of the routing gating.

8. A multi-dialect speech recognition model training device, characterized in that: The device comprises: A pre-trained model acquisition module, used to acquire a pre-trained model of a speech recognition model; wherein the pre-trained model includes a plurality of network layers, a specified linear layer in the network layer is replaced with an expert routing network, and the expert routing network includes routing gating and an expert network; A dialect data set acquisition module, used to acquire a vectorized dialect data set of at least one dialect; The speech recognition model completion module is used to use the dialect data set to train the expert routing network in the pre-trained model, and when the training model reaches a preset convergence condition, obtain a trained speech recognition model; the speech recognition model performs speech recognition on dialect speech of at least one dialect.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images