Multi-dialect speech recognition method and device based on ensemble learning, equipment and medium

Abstract

The invention discloses a multi-dialect speech recognition method based on ensemble learning, relates to the technical field of speech recognition, and is used for solving the problem that an existing model cannot realize recognition of different dialects with relatively large difference degrees. The method comprises the following steps: acquiring sample data which is a multi-language data set, the information comprises voice recording, transcriptional text and speaker information; and inputting the sample data into a multi-dialect speech recognition model for recognition, the multi-dialect speech recognition model being an end-to-end speech recognition model based on Transform and comprising an expert mixture and a general expert, the expert mixture being used for processing specific accent data, and the general expert being used for processing common data in different accent. The invention also discloses a multi-dialect speech recognition device based on ensemble learning, electronic equipment and a computer storage medium. Through combination of the expert mixture and the general experts, multi-dialect recognition is quickly completed.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a multi-dialect speech recognition method, device, equipment and medium for integrated learning. Background Art

[0002] Traditional single-dialect automatic speech recognition (ASR) systems are usually difficult to adapt to the speech recognition needs of different accents and dialects. With the rise of end-to-end ASR models, researchers have begun to study multi-dialect ASR models based on this model, hoping to improve the recognition performance of low-resource dialects by learning shared features between different dialects.

[0003] However, most of the existing multi-dialect ASR model research is aimed at low-resource or similar dialects, and cannot achieve recognition of different dialects with large differences. In order to make the multi-dialect ASR model widely applicable to various dialects and to maintain high performance when the amount of training data of different dialects varies greatly, more in-depth model research is needed. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, one of the objectives of the present invention is to provide a multi-dialect speech recognition method based on ensemble learning, which introduces a task-specific expert mixture architecture and combines general experts to perform multi-dialect recognition.

[0005] One of the purposes of the present invention is achieved by the following technical solution:

[0006] A multi-dialect speech recognition method based on ensemble learning comprises the following steps:

[0007] Acquire sample data, where the sample data is a multilingual data set, which includes voice recordings, transcribed texts, and speaker information;

[0008] The sample data is input into a multi-dialect speech recognition model for recognition. The multi-dialect speech model is an end-to-end speech recognition model based on Transformer, which includes an expert mixture and a general expert. The expert mixture is used to process specific accent data, and the general expert is used to process common data in different accents.

[0009] Furthermore, the sample data is a multilingual Common Voice dataset hosted by Mozilla

[0010] Furthermore, the RNN networks of the encoder and decoder of the multi-dialect speech recognition model are replaced by the attention mechanism. Each layer of the encoder includes: Self Attention, Layer Norm and Feed Forward, and each layer of the decoder includes Self Attention, Layer Norm, Encoder-Decoder Attention and Feed Forward. Each Attention layer is connected through a Residual residual, and Layer Norm normalizes the activation value of the layer.

[0011] Furthermore, the multi-dialect speech recognition model is trained using the CTC-attention framework, and the loss function is calculated to satisfy: L jca =λ ctc L ctc +(1-λ ctc )L s2s , where L ctc and L s2s are the target loss functions of CTC and sequence-to-sequence S2S structures, λ ctc are adjustable parameters used to control the contribution of each loss.

[0012] Furthermore, in the forward propagation of the expert mixture, each expert receives the same input, and the output is the weighted sum of each expert; in the backward propagation of the expert mixture, the expert weights are updated by the pre-assigned accent sentences; the integrated calculation of each expert satisfies:

[0013] l out =∑ i α i ·e i ,

[0014] Among them, l out represents the output from the MTE layer, and e i represents the output of the i-th expert, α i is the weight.

[0015] Furthermore, the weight α i The calculation satisfies: α=softmax(G(Concat(e c ;o l-1 ))), where G is the gating network and n is the number of experts.

[0016] Furthermore, the model is evaluated, including the following steps:

[0017] Build multiple benchmark models, including a benchmark model for each single dialect and a mixed dialect model;

[0018] Construct variants of ensemble learning models, including a basic ensemble learning model that requires a given dialect type for recognition, an ensemble learning model-linear version that uses linear layers to calculate expert weights, and an ensemble learning-long short-term memory (LSTM) version that uses LSTM layers to calculate expert weights.

[0019] The model was evaluated by comparing word error rates on different dialect data test sets;

[0020] Extract the weights of each expert in the last integrated learning layer of the multi-dialect speech recognition model, and observe the necessity of each dialect expert and common experts through the weight size;

[0021] Delete some modules of the multi-dialect speech recognition model, and continue to observe the model to verify the role of the deleted modules, including: delete the common expert module to verify whether the model performance decreases; delete any dialect expert to verify whether the remaining modules can achieve similar performance in the dialect corresponding to the deleted dialect expert; delete the entire integrated learning layer to observe whether the model performance in the dialect data decreases significantly. The second purpose of the present invention is to provide a multi-dialect speech recognition device based on integrated learning.

[0022] The second object of the present invention is achieved by the following technical solution:

[0023] A multi-dialect speech recognition device based on ensemble learning, comprising:

[0024] An acquisition module, used to acquire sample data, wherein the sample data is a multilingual data set, which includes voice recordings, transcribed texts, and speaker information;

[0025] The recognition module is used to input the sample data into a multi-dialect speech recognition model for recognition. The multi-dialect speech model is an end-to-end speech recognition model based on Transformer, which includes an expert mixture and a general expert. The expert mixture is used to process specific accent data, and the general expert is used to process common data in different accents.

[0026] The third object of the present invention is to provide an electronic device for performing one of the objects of the invention, which includes a processor, a storage medium and a computer program, wherein the computer program is stored in the storage medium, and when the computer program is executed by the processor, it implements the above-mentioned multi-dialect speech recognition method based on ensemble learning.

[0027] A fourth object of the present invention is to provide a computer-readable storage medium storing one of the objects of the invention, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned multi-dialect speech recognition method based on ensemble learning is implemented.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The present invention proposes an innovative expert hybrid method, which combines expert hybrids with general experts. Different dialects can be recognized through a model, and common points in dialects can be accurately extracted and recognized by a general expert, thereby improving recognition efficiency. It not only significantly reduces resource utilization for low-resource accents, but also improves overall performance, and can maintain high efficiency even when the amount of training data for different dialects varies greatly. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a flow chart of a multi-dialect speech recognition method based on ensemble learning according to Embodiment 1;

[0031] Figure 2 is an overall architecture diagram of the multi-dialect speech recognition method based on ensemble learning in Example 1;

[0032] Figure 3 is a schematic diagram of the Transformer model structure of the first embodiment;

[0033] Figure 4 is a schematic diagram of the Self Attention structure of the first embodiment;

[0034] Figure 5 is a schematic diagram of the integrated learning layer architecture of the first embodiment;

[0035] Figure 6 is the weight histogram of different experts in the ensemble learning layer of the second embodiment;

[0036] Figure 7 is a structural block diagram of a multi-dialect speech recognition device based on ensemble learning according to Embodiment 3;

[0037] Figure 8 It is a structural block diagram of an electronic device of the fourth embodiment. DETAILED DESCRIPTION

[0038] The present invention will be described in more detail below in conjunction with the accompanying drawings. It should be noted that the following description of the present invention with reference to the accompanying drawings is only illustrative and not restrictive. The various embodiments can be combined with each other to form other embodiments not shown in the following description.

[0039] Embodiment 1

[0040] Embodiment 1 provides a multi-dialect speech recognition method based on ensemble learning, which aims to distinguish and recognize the similarities and differences of different dialects through expert mixtures and general experts, thereby realizing rapid recognition of different dialects.

[0041] The purpose of this embodiment is to solve some key problems in multi-dialect ASR through an innovative mixture of experts (MOE) method. The multi-dialect ASR model of this embodiment not only does not require prior knowledge of dialect identity during reasoning, but also shows significant performance improvement in experiments.

[0042] Please refer to Figure 1 and Figure 2 As shown, a multi-dialect speech recognition method based on ensemble learning is characterized by comprising the following steps:

[0043] S1. Obtain sample data, where the sample data is a multilingual data set, which includes voice recordings, transcribed texts, and speaker information;

[0044] The sample data in S1 is the multilingual Common Voice dataset hosted by Mozilla. This dataset includes more than 4,200 hours of voice recordings, accompanied by transcriptions and detailed speaker information, such as accent type, age, and gender. The research focuses on the English part of the dataset, especially those voice data with accent labels. This subset of English data covers English accents from 16 different countries. After classifying the data according to these accent labels, analysis of the dataset shows significant imbalance; some countries have a large amount of training data, while other countries have relatively little, which highlights the challenges in training speech recognition models with specific accents.

[0045] S2. Input the sample data into a multi-dialect speech recognition model for recognition. The multi-dialect speech model is an end-to-end speech recognition model based on Transformer, which includes an expert mixture and a general expert. The expert mixture is used to process specific accent data, and the general expert is used to process common data in different accents.

[0046] Please refer to Figure 3 As shown, the RNN networks of the encoder and decoder of the multi-dialect speech recognition model of this embodiment are replaced by the attention mechanism. Each layer of the encoder includes: a self-attention layer, a layer normalization LayerNorm, and a feedforward layer FeedForward. Each layer of the decoder includes a self-attention layer, a layer normalization Layer Norm, an encoder-decoder attention module Encoder-Decoder Attention, and a feedforward layer FeedForward. Each Attention layer is connected through a Residual residual, and Layer Norm normalizes the activation value of the layer.

[0047] Specifically, the architecture of the attention-based Transformer uses the attention mechanism in each decoder and encoder, completely replacing the traditional RNN with attention, and has achieved better results in machine translation and speech recognition. Since both belong to the seq2seq problem, after it was introduced into the field of speech recognition, the Transformer, as a model based on attention, has good sequence-to-sequence modeling capabilities. It does not need to make any prior assumptions about the alignment of input and output sequences, but can learn encoding, decoding and how to align at the same time. Therefore, it has achieved good results, and the Transformer model has gradually become the mainstream model in speech recognition.

[0048] Transformer establishes a sequence relationship between speech features and text symbols. Its essence is still a seq2seq structure, which includes multiple encoders and decoders. Each layer of the encoder has three operations, namely Self Attention, Layer Norm, and Feed Forward; and each layer of the decoder has four operations, namely Self Attention, Layer Norm, Encoder-Decoder Attention, and Feed Forward. There are residual connections in the attention layer, so that the information of the previous layer can be passed to the next layer. Layer Norm can further avoid gradient problems by normalizing the activation values ​​of the layer, and accelerate the training process of the model, making it converge faster.

[0049] The main problem that Transformer solves is to improve the parallelism of the Encoder. The key points are the two mechanisms of Self-Attention and Multi-Head Attention. Self-Attention is that each word must calculate the attention with all the words, which can capture long-distance dependencies. In Multi-Head Attention, different heads learn different subspace dependencies and focus on different parts of the encoder output.

[0050] Please refer to Figure 4In the Self Attention structure shown in the figure, the input X is embedded in the word and then added to the position encoding matrix to obtain A, which is then multiplied by the three matrices WQ, WK, and WV representing Query, Key, and Value to obtain Q, K, and V respectively. Q and K are respectively calculated through inner products to calculate their relevance. In order to prevent numerical imbalance, the calculation result should be divided by the square root of the feature dimension, and then the relevance is output as a weighted value of 0-1 through softmax, and then the weighted sum of V is calculated to obtain the result. The complete calculation formula of Self Attention satisfies: where d k is the number of columns of matrix K.

[0051] Therefore, the output result of the current node contains the information of the entire sentence context, that is, it not only focuses on the current frame, but also obtains the information of other frames before and after. The importance of this information is adjusted by Attention. Currently, due to the good effect of Transformer, end-to-end speech recognition is fully realized. It can be recognized with high accuracy without using a language model, making it one of the mainstream models for speech recognition.

[0052] This example uses a Transformer-based network architecture to implement speech recognition, and on this basis introduces the CTC loss function to help better train the model. The multi-dialect speech recognition model is trained using the CTC-attention framework, and the loss function calculation satisfies: L jca =λ ctc L ctc +

[0053] (1-λ ctc )L s2s , where L ctc and L s2s They are the target loss functions of CTC and sequence-to-sequence S2S structures, and the adjustable parameters (λ ctc ∈[0,1]) is used to control the contribution of each loss. The Transformer encoder takes acoustic features as input and maps them into a high-level representation h. The decoder network uses the encoded representation h and the attention mechanism to output the predicted token in an autoregressive manner. At each decoding step, the decoder emits the posterior probability of the next token based on the previous output. The Transformer model is trained using a joint CTC-attention framework to fully leverage the advantages of CTC and attention-based models.

[0054] In the forward propagation of the expert mixture, each expert receives the same input, and the output is the weighted sum of each expert; in the backward propagation of the expert mixture, the expert weights are updated by the pre-assigned accent sentences; the integrated calculation of each expert satisfies:

[0055] lout =∑ i α i ·e i ,

[0056] Among them, l out represents the output from the MTE layer, and e i represents the output of the i-th expert, α i is the weight, the weight α i The calculation satisfies:

[0057] α=softmax(G(Concat(e c ;o l-1 ))), where G is the gating network and n is the number of experts; G is an affine transformation function, and the LSTM gating network can also be used as the gating function.

[0058] Specifically, in the mixed expert layer training, this embodiment introduces two concepts: "integration" and "division of labor" to simulate the cooperation mode of multiple experts in integrated learning. Figure 5 As shown in Figure 1, during the forward propagation, all experts in a layer receive the same input, and the output is the weighted sum of each expert's output. This is similar to how MOE processes data in the forward propagation.

[0059] The specialized “division of labor” of different experts is achieved by modifying the gradients during back-propagation: in back-propagation, the weights of the experts are updated only by the sentences of their pre-assigned accents. This can be achieved by setting the gradients corresponding to sentences of irrelevant dialects to zero.

[0060] In addition to the experts for each accent, this embodiment also introduces a commonality expert. The introduction of commonality experts in the MTE framework has a dual purpose: it enables individual experts to focus on generating features for a specific accent while capturing common features of multiple dialects through a single commonality expert component. The proportional weights assigned to each expert in different layers can be referred to Figure 6 . It is noteworthy that when the commonality expert is present, it accounts for approximately half of the total weight, which provides strong evidence for its inclusion.

[0061] This embodiment also provides a model evaluation method. In order to evaluate the effectiveness of the model, this embodiment establishes a baseline model by training a model for a single accent. Each model is trained using only the data of its corresponding accent; then a FT-Full model is trained using the same architecture as the baseline model, but the model combines the data of all accents and is fine-tuned using the corresponding accent dataset. In order to ensure fair comparison, the capacity of the model needs to be increased accordingly, which can be achieved by doubling the hidden units in each encoder layer, while doubling the batch size of the training.

[0062] Evaluating the model involves the following steps:

[0063] Model building: Build multiple benchmark models, including: a) a benchmark model for each single dialect, i.e. a model trained on the corresponding dialect and accent; b) a mixed dialect model, i.e. a model trained on a dataset that mixes multiple dialects;

[0064] Construct variants of the ensemble learning model, including a) the basic ensemble learning model, which requires a given dialect type for recognition, b) the ensemble learning model-linear version, which uses the linear layer to calculate the expert weights in the ensemble learning layer, and c) the ensemble learning-long short-term memory (LSTM) version, which uses LSTM to calculate the expert weights in the ensemble learning layer.

[0065] Model performance evaluation: Evaluate the above models on different dialect data test sets and compare their word error rates;

[0066] Ensemble learning expert layer weight evaluation: Extract the weights of each expert in the last ensemble learning layer in the speech recognition model, and observe the necessity of each dialect expert and common expert through the weight size.

[0067] Ablation experiment: For the model proposed in this invention, delete some of its modules and observe the model effect to verify the role of the module. The main operations are: delete the common expert module to verify whether the model performance has dropped; delete a dialect expert to verify whether similar performance can be achieved on the dialect through the remaining modules; delete the entire integrated learning layer to observe whether the model performance on the dialect data has dropped significantly.

[0068] Embodiment 2

[0069] Embodiment 2 is an experiment performed by the method of embodiment 1.

[0070] The model used in the experiments of this embodiment receives acoustic input in the form of 80-dimensional logarithmic Mel spectrograms, with each feature vector containing three consecutive frames. The present invention adjusts the Transformer-based architecture and modifies the encoder and decoder components. Table 1 provides a detailed breakdown of the model configurations used in the experiments. To ensure a balanced comparison, the number of parameters of the multi-accent baseline was adjusted to align with the capacity of the MTE architecture. In each MTE layer, individual experts are implemented as LSTM units containing 2048 hidden nodes. One exception is the setting without common experts, which increases the number of hidden units by 25% to 2560 to be consistent with the model containing common experts. For experiments using LSTM-based gating, 2048 hidden units are uniformly used. The target vocabulary includes letters of all accents under consideration. In the initial training steps, this embodiment applies uniform averaging to the expert outputs and abandons specialization, allowing gradients of all experts to flow. Most models are trained on an 8x8 tensor processing unit configuration with a fairly large batch size of 16,384. In comparison, the baseline uses 4x4 TPUs and runs with a more efficient batch size of 4,096.

[0071] The experiments in this example were conducted using the ESPnet toolkit. The baseline model used in the study was pre-trained on the Librispeech dataset. For subword segmentation, the SentencePiece tool was used and the number of subwords was set to 150. The bottleneck size of the adapter was configured to be 16 and 8 for the source and target accents, respectively. During AccentFusion training, a linear annealing strategy was applied to the learning rate, which was reduced to zero as the training progressed. An early stopping mechanism was introduced with a patience parameter set to 10 cycles. The batch size was fixed to 64 and the learning rate was set to 0.0001.

[0072] Table 1 Model parameter settings

[0073]

[0074]

[0075] Table 2 Test results

[0076] Method Malaysia Afghanistan the Philippines Singapore Hong Kong Mono(Baseline) 45.82 20.41 43.54 41.62 31.54 FT-Full 32.15 13.86 35.85 33.23 23.58 MTE 16.12 12.25 22.57 24.25 14.65 MTE-linear 17.21 14.02 23.88 26.03 15.69 MTE-LSTM 15.92 13.75 23.12 25.32 15.42

[0077] The MTE layer architecture consists of five different experts: four specialized for a specific accent, each processing data for its corresponding accent, and a "common expert" whose weights are updated using data from all accents. A language gate, described in detail in Section 2, is used to merge the outputs of these experts. To enhance the input features, they are augmented with a one-bit efficient vector representing the accent ID. The findings presented in the "MTE" row of Table 2 show that the performance of the MTE model either matches or exceeds the FT-Full and monolingual baselines in all considered languages.

[0078] While MTE performs well, using language gating requires providing accent labels at inference time. To overcome this limitation, language gating is replaced with LSTM gating or linear layer gating as described previously. Although accent labels are required to train the correct experts during training, they are not used during inference. LSTM gating takes the shared encoder output as input and produces a score for each expert, which is consistent across all MTE layers. The results show that although the performance of linear layer gating decreases compared to the language gate model, LSTM gating almost matches the performance of the MTE model. This shows that combining LSTM gating with different experts can learn a fairly accurate implicit accent model without changing the loss function. Table 2 shows these results.

[0079] There are two main purposes for introducing common experts into the MTE framework: on the one hand, it enables individual experts to focus on producing features with specific accents; on the other hand, through a unified common expert component, it can capture common features. Figure 6 As shown in the figure, the horizontal axis is the name of each expert in different layers (comm represents common experts, MA, AF, PH, SG, HK represent a dialect expert respectively: for example, MA represents Malaysian dialect, AF represents Afghan dialect), and the vertical axis is the weight ratio given to the expert. It shows that common experts account for about half of the total weight, which strongly supports the decision to introduce common experts in this study. In order to evaluate its importance, a control experiment was further conducted, in which common experts were not included. In order to maintain the consistency of parameters, compensation was made by increasing the number of hidden units in the LSTM layer of each expert by 25%.

[0080] In addition, check Figure 6 The "no commonality" output of reveals a noteworthy point: the weights previously assigned to the commonality experts were primarily redistributed to the experts corresponding to the correct accents. This redistribution of weights suggests that in the absence of commonality experts, the system adapts the expert weights by routing relevant information through the most appropriate accent-specific path.

[0081] Experimental results show that this method has significant innovation and practicality in the field of multi-dialect ASR, and provides an effective technical solution for speech recognition in different dialects.

[0082] Embodiment 3

[0083] Embodiment 3 discloses a device corresponding to the multi-dialect speech recognition method based on ensemble learning in the above embodiment, which is a virtual device structure of the above embodiment. Please refer to Figure 7 As shown, including:

[0084] An acquisition module 310 is used to acquire sample data, where the sample data is a multilingual data set including voice recordings, transcribed texts, and speaker information;

[0085] The recognition module 320 is used to input the sample data into a multi-dialect speech recognition model for recognition. The multi-dialect speech model is an end-to-end speech recognition model based on Transformer, which includes an expert mixture and a general expert. The expert mixture is used to process specific accent data, and the general expert is used to process common data in different accents.

[0086] Embodiment 4

[0087] Figure 8 A schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention is shown in FIG. Figure 8 As shown, the electronic device includes a processor 410, a memory 420, an input device 430, and an output device 440; the number of processors 410 in the computer device can be one or more. Figure 8 A processor 410 is taken as an example; the processor 410, the memory 420, the input device 430 and the output device 440 in the electronic device can be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0088] The memory 420, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the multi-dialect speech recognition method based on ensemble learning in the embodiment of the present invention. The processor 410 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the memory 420, that is, implements the multi-dialect speech recognition method based on ensemble learning in the above-mentioned embodiments 1 to 2.

[0089] The memory 420 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 420 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 420 may further include a memory remotely arranged relative to the processor 410, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0090] The input device 430 may be used to receive input user identity information, sample data, etc. The output device 440 may include a display device such as a display screen.

[0091] Preferably, the sample data is a multilingual Common Voice dataset hosted by Mozilla.

[0092] Preferably, the RNN networks of the encoder and decoder of the multi-dialect speech recognition model are replaced by the attention mechanism, each layer of the encoder includes: a self-attention layer, a layer normalization Layer Norm and a feedforward layer Feed Forward, and each layer of the decoder includes a self-attention layer, a layer normalization Layer Norm, an encoder-decoder attention Encoder-Decoder Attention and a feedforward layer Feed Forward, each Attention layer is connected through a Residual residual, and Layer Norm normalizes the activation value of the layer.

[0093] Preferably, the multi-dialect speech recognition model is trained using a CTC-attention framework, and the loss function is calculated to satisfy: L jca =λ ctc L ctc +(1-λ ctc )L s2s , where L ctc and L s2s are the target loss functions of CTC and sequence-to-sequence S2S structures, λ ctc are adjustable parameters used to control the contribution of each loss.

[0094] Preferably, in the forward propagation of the expert mixture, each expert receives the same input, and the output is the weighted sum of each expert; in the backward propagation of the expert mixture, the expert weights are updated by the pre-assigned accent sentences; the integrated calculation of each expert satisfies:

[0095] l out =∑ i α i ·e i ,

[0096] Among them, l out represents the output from the MTE layer, and e i represents the output of the i-th expert, α i is the weight.

[0097] Preferably, the weight α i The calculation satisfies: α=softmax(G(Concat(e c ;o l-1 ))), where G is the gating network and n is the number of experts.

[0098] Preferably, the method further comprises evaluating the model, comprising the following steps:

[0099] Build multiple benchmark models, including a benchmark model for each single dialect and a mixed dialect model;

[0100] Construct variants of ensemble learning models, including a basic ensemble learning model that requires a given dialect type for recognition, an ensemble learning model-linear version that uses linear layers to calculate expert weights, and an ensemble learning-long short-term memory (LSTM) version that uses LSTM layers to calculate expert weights.

[0101] The model was evaluated by comparing word error rates on different dialect data test sets;

[0102] Extract the weights of each expert in the last integrated learning layer of the multi-dialect speech recognition model, and observe the necessity of each dialect expert and common experts through the weight size;

[0103] Delete some modules of the multi-dialect speech recognition model and continue to observe the model to verify the role of the deleted modules, including: delete the common expert module to verify whether the model performance decreases; delete any dialect expert to verify whether the remaining modules can achieve similar performance in the dialect corresponding to the deleted dialect expert; delete the entire integrated learning layer to observe whether the model performance on dialect data decreases significantly.

[0104] Embodiment 5

[0105] Embodiment 5 of the present invention further provides a storage medium containing computer executable instructions, which can be used for a computer to execute a multi-dialect speech recognition method based on ensemble learning, the method comprising:

[0106] Acquire sample data, where the sample data is a multilingual data set, which includes voice recordings, transcribed texts, and speaker information;

[0107] The sample data is input into a multi-dialect speech recognition model for recognition. The multi-dialect speech model is an end-to-end speech recognition model based on Transformer, which includes an expert mixture and a general expert. The expert mixture is used to process specific accent data, and the general expert is used to process common data in different accents.

[0108] Of course, the storage medium containing computer executable instructions provided in an embodiment of the present invention is not limited to the method operations described above, and can also execute related operations in the multi-dialect speech recognition method based on ensemble learning provided in any embodiment of the present invention.

[0109] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for an electronic device (which can be a mobile phone, a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0110] It is worth noting that in the above-mentioned embodiment of the multi-dialect speech recognition device based on ensemble learning, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0111] For those skilled in the art, various other corresponding changes and deformations can be made according to the technical solutions and concepts described above, and all of these changes and deformations should fall within the protection scope of the claims of the present invention.

Claims

1. A multi-dialect speech recognition method based on ensemble learning, characterized in that: The following steps are involved: Acquire sample data, where the sample data is a multilingual data set, which includes voice recordings, transcribed texts, and speaker information; The sample data is input into a multi-dialect speech recognition model for recognition. The multi-dialect speech model is an end-to-end speech recognition model based on Transformer, which includes an expert mixture and a general expert. The expert mixture is used to process specific accent data, and the general expert is used to process common data in different accents.

2. The multi-dialect speech recognition method based on ensemble learning as claimed in claim 1, characterized in that: The sample data is the multilingual Common Voice dataset hosted by Mozilla.

3. The multi-dialect speech recognition method based on ensemble learning as claimed in claim 1, characterized in that: The RNN networks of the encoder and decoder of the multi-dialect speech recognition model are replaced by the attention mechanism. Each layer of the encoder includes: a self-attention layer, a layer normalization layer norm and a feed forward layer. Each layer of the decoder includes a self-attention layer, a layer normalization layer norm, an encoder-decoder attention Encoder-Decoder Attention and a feed forward layer. Each Attention layer is connected through a residual residual, and the Layer Norm layer normalizes the activation value of the layer.

4. The multi-dialect speech recognition method based on ensemble learning as claimed in claim 3, characterized in that: The multi-dialect speech recognition model is trained using the CTC-attention framework, and the loss function is calculated to satisfy: L jca =λ ctc L ctc +(1-λ ctc )L s2s , where L ctc and L s2s are the target loss functions of CTC and sequence-to-sequence S2S structures, λ ctc are adjustable parameters used to control the contribution of each loss.

5. The multi-dialect speech recognition method based on ensemble learning as claimed in claim 1, characterized in that: In the forward propagation of the expert mixture, each expert receives the same input, and the output is the weighted sum of each expert; in the backward propagation of the expert mixture, the expert weights are updated by the pre-assigned accent sentences; the integrated calculation of each expert satisfies: l out ＝∑ i α i ·e i ， Among them, l out represents the output from the MTE layer, and e i represents the output of the i-th expert, α i is the weight.

6. The multi-dialect speech recognition method based on ensemble learning as claimed in claim 5, characterized in that: The weight α i The calculation satisfies: α=softmax(G(Concat(e c ;o l-1 ))), where G is the gating network and n is the number of experts.

7. The multi-dialect speech recognition method based on ensemble learning as claimed in claim 1, characterized in that: It also includes the evaluation of the model, which includes the following steps: Build multiple benchmark models, including a benchmark model for each single dialect and a mixed dialect model; Construct variants of ensemble learning models, including a basic ensemble learning model that requires a given dialect type for recognition, an ensemble learning model-linear version that uses linear layers to calculate expert weights, and an ensemble learning-long short-term memory (LSTM) version that uses LSTM layers to calculate expert weights. The model was evaluated by comparing word error rates on different dialect data test sets; Extract the weights of each expert in the last integrated learning layer of the multi-dialect speech recognition model, and observe the necessity of each dialect expert and common experts through the weight size; Delete some modules of the multi-dialect speech recognition model and continue to observe the model to verify the role of the deleted modules, including: delete the common expert module to verify whether the model performance decreases; delete any dialect expert to verify whether the remaining modules can achieve similar performance in the dialect corresponding to the deleted dialect expert; delete the entire integrated learning layer to observe whether the model performance on dialect data decreases significantly.

8. A multi-dialect speech recognition device based on ensemble learning, characterized in that: It includes: An acquisition module is used to acquire sample data, wherein the sample data is a multilingual data set, which includes voice recordings, transcribed texts, and speaker information; The recognition module is used to input the sample data into a multi-dialect speech recognition model for recognition. The multi-dialect speech model is an end-to-end speech recognition model based on Transformer, which includes an expert mixture and a general expert. The expert mixture is used to process specific accent data, and the general expert is used to process common data in different accents.

9. An electronic device comprising a processor, a storage medium and a computer program, wherein the computer program is stored in the storage medium, wherein: When the computer program is executed by a processor, the multi-dialect speech recognition method based on ensemble learning as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-dialect speech recognition method based on ensemble learning as described in any one of claims 1 to 7 is implemented.

Images