Multi-dialect speech recognition method, device, equipment and medium

Abstract

The invention discloses a multi-dialect speech recognition method, which relates to the technical field of language processing and is used for solving the problem of difficulty in multi-dialect recognition in the prior art. Extracting features of the dialect data, and representing the features as voice representation; inputting the voice representation into a Transform encoder based on a sparse bias calculation mode to extract a dialect representation; performing dialect classification according to the dialect representation information to obtain a dialect classification result; and inputting the classification result and the voice pointer into a multi-expert integrated system for prediction to obtain a voice recognition result. The invention further discloses a multi-dialect speech recognition device, electronic equipment and a computer storage medium. According to the method, accurate and efficient identification of multiple dialects is realized by combining a multi-expert integrated system based on a sparse bias calculation mode encoder.

Description

Technical Field

[0001] The present invention relates to the field of language processing technology, and in particular to a multi-dialect speech recognition method, device, equipment and medium based on dialect type recognition and multi-expert integration system. Background Art

[0002] Automatic speech recognition technology refers to the use of machine learning to convert audio and spoken input into text form. It is widely used in voice search, voice assistants, automatic generation of movie subtitles, etc.

[0003] However, dialects and accents in spoken language can lead to inaccurate speech recognition. Specifically, due to the heterogeneity of language and differences in data quality between different dialect types, the distribution of speech recognition model training data is often uneven. This data imbalance will weaken the performance of the model and prevent the trained speech recognition model from achieving the best recognition effect. To this end, the existing technology conducts separate model training for each dialect or realizes text conversion of dialects, but this method consumes more computing resources and requires switching different models for different dialects. In summary, the speech recognition models currently studied are insufficient in the ability to detect acoustic changes such as dialects or accents, and lack a method that can recognize multiple dialects or accents. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, one of the objects of the present invention is to provide a multi-dialect speech recognition method, which extracts dialect representations through an encoder based on a sparsity bias calculation mode, and then combines a multi-expert integrated system to perform accurate dialect recognition.

[0005] One of the purposes of the present invention is achieved by the following technical solution:

[0006] A multi-dialect speech recognition method comprises the following steps:

[0007] Get dialect data;

[0008] Extracting features of the dialect data and representing the features as speech representations;

[0009] Inputting the speech representation into a Transformer encoder based on a sparsity bias calculation mode to extract the dialect representation;

[0010] Performing dialect classification according to the dialect representation information to obtain a dialect classification result;

[0011] The classification result and the speech pattern are input into a multi-expert integrated system for prediction to obtain a speech recognition result.

[0012] Further, extracting features of the dialect data through a feature extractor and a shared encoder, and representing the features as speech representations, comprises the following steps:

[0013] The feature extractor extracts the feature representation of the dialect data, satisfying: t =Extract(x t ), where f t is the characteristic representation, x t The original speech data is the dialect data;

[0014] The feature representation is encoded into a speech representation S by the shared encoder t , satisfy: S t =Encoder(f t ).

[0015] Furthermore, the Transformer encoder based on the sparsity bias calculation mode includes: a context-based sparse self-attention mechanism block, a layer normalization layer, a feedforward neural network and a layer normalization layer.

[0016] Furthermore, each layer of the Transformer encoder based on the sparsity bias calculation mode includes three stages, each stage including:

[0017] The speech representation is input into the self-attention mechanism block, which includes a query set Q key and a key value set K. The key value set K is used to generate a matrix M by scaling the dot product, and the top k largest scores and corresponding position information of each row in M ​​are extracted into the matrix L, P, satisfying M = QK T ; L,P = top k(M,k);

[0018] Traverse the matrix P and matrix L, fill the elements in the matrix L at the same position into the matrix M according to the position information stored in the matrix P, obtain the matrix M′ after coefficient processing, and calculate the attention score to satisfy:

[0019]

[0020] Among them, V represents the value set in the attention mechanism, d k is the dimension of the key-value set;

[0021] The attention mechanism is input into the subsequent layer normalization layer, feedforward neural network and layer normalization layer to obtain the dialect representation Where n represents the number of encoder layers.

[0022] Furthermore, dialect classification is performed according to the dialect representation information to obtain a dialect classification result, including:

[0023] aggregating the frame-level representations of the dialect representation into a sentence-level representation;

[0024] The sentence-level representation is input into a fully connected network and output to obtain a dialect classification result.

[0025] The above-mentioned fully connected network is a common fully connected network, which is a common structure at the end of the classification task and will not be described in detail in this embodiment.

[0026] Furthermore, the classification result and the speech pattern are input into a multi-expert integrated system for prediction to obtain a speech recognition result, including:

[0027] Input the dialect classification result into the expert selection network to obtain the distribution probability, wherein the expert selection network is a single-layer linear layer network, and map the distribution space corresponding to the dialect classification result to the distribution space of the expert selection network to obtain the distribution probability, wherein the distribution probability is the weight of each decoder in the expert selection network during decoding prediction;

[0028] According to the distribution probability and the corresponding weight, the expert mixed recognition result is weighted to be used as the speech recognition result.

[0029] Furthermore, the training of the multi-expert integrated system includes CTC learning, ATTENTION learning and classification learning on the dialect speech data, including the following steps:

[0030] Calculating dialect classification loss satisfy: Where C is the number of dialect categories, y i,j Indicates the label, p i,j represents the predicted probability, and B is the batch size;

[0031] Calculating CTC loss satisfy: Among them, Fun -1 is the inverse function, which is used to give all valid alignment paths between the input sequence and the output sequence, a t With S t Represent the output text sequence and the input speech representation sequence respectively;

[0032] Calculating ATTENTION loss satisfy: in, is a true transcript, y t is predictive text;

[0033] Calculate the total loss to satisfy: Among them, α1, α1 are weight parameters.

[0034] A second object of the present invention is to provide a multi-dialect speech recognition device.

[0035] The second object of the present invention is achieved by adopting the following technical solution:

[0036] A multi-dialect speech recognition device, comprising:

[0037] A feature extraction module is used to obtain dialect data; extract features of the dialect data and represent the features as speech representations; input the speech representations into a Transformer encoder based on a sparsity bias calculation mode to extract dialect representations;

[0038] The speech recognition module is used to classify the dialect according to the dialect representation information to obtain the dialect classification result; the classification result and the speech representation information are input into the multi-expert integrated system for prediction to obtain the speech recognition result.

[0039] The third object of the present invention is to provide an electronic device for performing one of the objects of the invention, which includes a processor, a storage medium and a computer program, wherein the computer program is stored in the storage medium, and when the computer program is executed by the processor, the above-mentioned multi-dialect speech recognition method is implemented.

[0040] A fourth object of the present invention is to provide a computer-readable storage medium storing one of the objects of the invention, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned multi-dialect speech recognition method is implemented.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention introduces a context-based sparse self-attention mechanism to make the extracted dialect representation more accurate; on this basis, combining the advantages of ensemble learning and a unified framework, provides multi-dialect speech recognition based on dialect type recognition and a multi-expert integrated system, so that the method can accurately recognize various dialects, overcoming the problem that existing methods cannot correctly recognize multi-dialect speech, and there is no need to switch models, reducing the computing resources required for dialect recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of the multi-dialect speech recognition method of embodiment 1;

[0044] Figure 2 1 is a schematic diagram of the overall architecture of the multi-dialect speech recognition method of embodiment 1;

[0045] Figure 3 is a structural block diagram of a multi-dialect speech recognition device of Embodiment 3;

[0046] Figure 4It is a structural block diagram of an electronic device of the fourth embodiment. DETAILED DESCRIPTION

[0047] The present invention will be described in more detail below in conjunction with the accompanying drawings. It should be noted that the following description of the present invention with reference to the accompanying drawings is only illustrative and not restrictive. The various embodiments can be combined with each other to form other embodiments not shown in the following description.

[0048] Embodiment 1

[0049] Embodiment 1 provides a multi-dialect speech recognition method, which aims to extract features and encode dialect data through a feature extractor and a shared encoder, and input the encoding results into a dialect type recognition model based on a sparsity bias calculation mode to complete dialect classification, thereby replacing the traditional model switching method.

[0050] Please refer to Figure 1 and Figure 2 As shown, a multi-dialect speech recognition method comprises the following steps:

[0051] S1. Obtain dialect data;

[0052] The dialect data of S1 can obtain the corresponding data of different dialects according to the needs, and of course, it can also obtain one dialect data at a time for training.

[0053] S2, extracting features of the dialect data, and representing the features as speech representations;

[0054] S2 includes extracting features of the dialect data through a feature extractor and a shared encoder, and representing the features as speech representations, including the following steps:

[0055] The feature extractor extracts the feature representation of the dialect data, satisfying: t =Extract(x t ), where f t is the characteristic representation, x t The original speech data is the dialect data;

[0056] The feature representation is encoded into a speech representation S by the shared encoder t , satisfy: S t =Encoder(f t ).

[0057] S3, inputting the speech representation into a Transformer encoder based on a sparsity bias calculation mode to extract the dialect representation;

[0058] Each layer of the Transformer encoder based on the sparsity bias calculation mode mentioned above includes: context-based sparse self-attention mechanism block, layer normalization layer, feedforward neural network and layer normalization layer. Each layer has three different stages, including:

[0059] The speech representation is input into the self-attention mechanism block, which includes a query set Q key and a key value set K. The key value set K is used to generate a matrix M by scaling the dot product, and the top k largest scores and corresponding position information of each row in M ​​are extracted into the matrix L, P, satisfying M = QK T ; L,P = topk(M,k);

[0060] Perform Softmax normalization on each row in matrix L. Traverse matrix P and matrix L, fill the elements in matrix L at the same position into matrix M according to the position information stored in matrix P, obtain the coefficient-processed matrix M′, and calculate the attention score to satisfy:

[0061]

[0062] Among them, V represents the value set in the attention mechanism, d k is the dimension of the key-value set;

[0063] The attention mechanism is input into the subsequent layer normalization layer, feedforward neural network and layer normalization layer to obtain the dialect representation Where n represents the number of encoder layers.

[0064] S4, performing dialect classification according to the dialect representation information to obtain a dialect classification result;

[0065] S4 includes:

[0066] Aggregate the frame-level representation of the dialect representation into a sentence-level representation, satisfying: Among them, ASP (Attentive Statistic Pooling) represents the attention aggregation layer;

[0067] The sentence-level representation is input into the fully connected network output to obtain the dialect classification result, satisfying Among them, FCN (Fully Connected Layer) represents the fully connected network, and M is the classification probability distribution.

[0068] The above aggregation is a simple concatenation, which concatenates frame-level features into sentence-level features in chronological order.

[0069] S5. Input the classification result and the speech pattern into a multi-expert integrated system for prediction to obtain a speech recognition result.

[0070] S5 specifically includes:

[0071] The classification result and the speech pattern are input into a multi-expert integrated system for prediction to obtain a speech recognition result, including:

[0072] Input the dialect classification result into the expert selection network (Router) to obtain the distribution probability. The expert selection network is a single-layer linear layer network. The distribution space corresponding to the dialect classification result is mapped to the distribution space of the expert selection network to obtain the distribution probability. The distribution probability is the weight of each decoder in the expert selection network during decoding prediction;

[0073] According to the distribution probability and the corresponding weight, the expert mixed recognition result is weighted to be used as the speech recognition result.

[0074] The purpose of weighting is to superimpose the weight vectors of the results of each expert decoder for optimization during training.

[0075] The training of S5's multi-expert integration system includes CTC learning, ATTENTION learning, and classification learning for dialect speech data, including the following steps:

[0076] Calculating dialect classification loss satisfy: Where C is the number of dialect categories, y i,j Indicates the label, p i,j represents the predicted probability, and B is the batch size;

[0077] Calculating CTC loss satisfy: Among them, Fun -1 is the inverse function, which is used to give all valid alignment paths between the input sequence and the output sequence, a t With S t Represent the output text sequence and the input speech representation sequence respectively;

[0078] Calculating ATTENTION loss satisfy: in, is a true transcript, y t is predictive text;

[0079] Calculate the total loss to satisfy: Among them, α1, α1 are weight parameters.

[0080] Embodiment 2

[0081] Embodiment 2 is a practical application description based on Embodiment 1.

[0082] This embodiment uses KeSpeech as a data set. KeSpeech is the first public data set that includes both Mandarin dialects and Mandarin Chinese. It involves a total of 1,542 hours of speech data from 27,237 contributors, including 8 different types of Mandarin Chinese and Mandarin dialects. Including Beijing Mandarin, Southwestern Mandarin, Central Plains Mandarin, Northeastern Mandarin, Lanyin Mandarin, Jianghuai Mandarin, Ji-Lu Mandarin, Jiao-Liao Mandarin, hereinafter referred to as BJ, SW, ZY, NE, LY, JH, JH, Ji-Lu, Jiao-Liao. The audio data is collected and produced based on real-life scenarios through the Shell Company platform. The audio format is mono and the sampling rate is 16kHz.

[0083] This example uses a pre-trained HuBERT model as a feature extractor and a Conformer architecture as a shared encoder. The Conformer encoder has 12 layers, each with 2048 hidden units and 8 attention heads. The dimension of the self-supervised learning speech representation generated by the pre-trained model is 1024. In terms of experimental settings, the experiment was trained on a multi-GPU machine with a batch size of 64 and an optimizer using Adam. The learning rate was adjusted using a warm-up strategy: it gradually increased to 1e-4 during the first 10% of iterations, remained unchanged for the next 40% of iterations, and then decreased linearly. The training process was performed for a total of 10 rounds.

[0084] This embodiment uses a quantitative method to evaluate the performance of the method and compares it with the dialect recognition baseline methods: KeSpeechBaseline, DIMNet, and Qifusion-Net. The present invention compares the dialect classification effect, the individual dialect recognition effect, and the multi-dialect mixed effect.

[0085] This embodiment uses word error rate (WER) to evaluate the performance of different models for dialect recognition, and uses accuracy (ACC) to evaluate the performance for dialect classification. Table 1 lists the quantitative results of the model of this embodiment on the test set KeSpeech.

[0086] Table 1 Quantitative evaluation results of accuracy on the KeSpeech test set

[0087]

[0088]

[0089] From the evaluation results in the table, it can be seen that the model proposed in this embodiment can achieve the best results in dialect classification and multi-dialect mixed scenarios of the KeSpeech test set, and achieves results that are better than the existing most advanced models in multiple dialect recognition effects. This proves the superiority of this method in terms of versatility and accuracy, and has better application value.

[0090] Embodiment 3

[0091] Embodiment 3 discloses a device corresponding to the multi-dialect speech recognition method of the above embodiment, which is a virtual device structure of the above embodiment. Please refer to Figure 3 As shown, including:

[0092] The feature extraction module 310 is used to obtain dialect data; extract features of the dialect data and represent the features as speech representations; input the speech representations into a Transformer encoder based on a sparsity bias calculation mode to extract dialect representations;

[0093] The speech recognition module 320 is used to classify the dialect according to the dialect representation information to obtain the dialect classification result; input the classification result and the speech representation information into the multi-expert integrated system for prediction to obtain the speech recognition result.

[0094] Preferably, extracting the features of the dialect data by a feature extractor and a shared encoder, and representing the features as speech representations, comprises the following steps:

[0095] The feature extractor extracts the feature representation of the dialect data, satisfying: t =Extract(x t ), where f t is the characteristic representation, x t The original speech data is the dialect data;

[0096] The feature representation is encoded into a speech representation S by the shared encoder t , satisfy: S t =Encoder(f t ).

[0097] Preferably, the Transformer encoder based on the sparsity bias calculation mode includes: a context-based sparse self-attention mechanism block, a layer normalization layer, a feedforward neural network and a layer normalization layer.

[0098] Preferably, each layer of the Transformer encoder based on the sparsity bias calculation mode includes three stages, each stage including:

[0099] The speech representation is input into the self-attention mechanism block, which includes a query set Q key and a key value set K. The key value set K is used to generate a matrix M by scaling the dot product, and the top k largest scores and corresponding position information of each row in M ​​are extracted into the matrix L, P, satisfying M = QK T ; L,P = top k(M,k);

[0100] Traverse the matrix P and matrix L, fill the elements in the matrix L at the same position into the matrix M according to the position information stored in the matrix P, obtain the matrix M′ after coefficient processing, and calculate the attention score to satisfy:

[0101]

[0102] Among them, V represents the value set in the attention mechanism, d k is the dimension of the key-value set;

[0103] The attention mechanism is input into the subsequent layer normalization layer, feedforward neural network and layer normalization layer to obtain the dialect representation Where n represents the number of encoder layers.

[0104] Preferably, performing dialect classification according to the dialect representation information to obtain a dialect classification result includes:

[0105] aggregating the frame-level representations of the dialect representation into a sentence-level representation;

[0106] The sentence-level representation is input into a fully connected network and output to obtain a dialect classification result.

[0107] Preferably, the classification result and the speech pattern are input into a multi-expert integrated system for prediction to obtain a speech recognition result, including:

[0108] Input the dialect classification result into the expert selection network to obtain the distribution probability, wherein the expert selection network is a single-layer linear layer network, and map the distribution space corresponding to the dialect classification result to the distribution space of the expert selection network to obtain the distribution probability, wherein the distribution probability is the weight of each decoder in the expert selection network during decoding prediction;

[0109] According to the distribution probability and the corresponding weight, the expert mixed recognition result is weighted to be used as the speech recognition result.

[0110] Preferably, the training of the multi-expert integrated system includes CTC learning, ATTENTION learning and classification learning on the dialect speech data, including the following steps:

[0111] Calculating dialect classification loss satisfy: Where C is the number of dialect categories, y i,j Indicates the label, p i,j represents the predicted probability, and B is the batch size;

[0112] Calculating CTC loss satisfy: Among them, Fun -1 is the inverse function, which is used to give all valid alignment paths between the input sequence and the output sequence, a t With S t Represent the output text sequence and the input speech representation sequence respectively;

[0113] Calculating ATTENTION loss satisfy: in, is a true transcript, y t is predictive text;

[0114] Calculate the total loss to satisfy: Among them, α1, α1 are weight parameters. The weight parameters are continuously optimized through training, and the specific values ​​of the weights are not limited here.

[0115] Embodiment 4

[0116] Figure 4 A schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention is shown in FIG. Figure 4 As shown, the electronic device includes a processor 410, a memory 420, an input device 430, and an output device 440; the number of processors 410 in the computer device can be one or more. Figure 4 A processor 410 is taken as an example; the processor 410, the memory 420, the input device 430 and the output device 440 in the electronic device can be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0117] The memory 420 is a computer-readable storage medium that can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the multi-dialect speech recognition method in the embodiment of the present invention. The processor 410 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the memory 420, that is, implements the multi-dialect speech recognition method in the above-mentioned embodiments 1 to 2.

[0118] The memory 420 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 420 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 420 may further include a memory remotely arranged relative to the processor 410, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0119] The input device 430 may be used to receive input user identity information, dialect data, etc. The output device 440 may include a display device such as a display screen.

[0120] Embodiment 5

[0121] Embodiment 5 of the present invention further provides a storage medium containing computer executable instructions, which can be used for a computer to execute a multi-dialect speech recognition method, the method comprising:

[0122] Get dialect data;

[0123] Extracting features of the dialect data and representing the features as speech representations;

[0124] Inputting the speech representation into a Transformer encoder based on a sparsity bias calculation mode to extract the dialect representation;

[0125] Performing dialect classification according to the dialect representation information to obtain a dialect classification result;

[0126] The classification result and the speech pattern are input into a multi-expert integrated system for prediction to obtain a speech recognition result.

[0127] Of course, the storage medium containing computer executable instructions provided by an embodiment of the present invention is not limited to the method operations described above, and the computer executable instructions can also execute related operations in the multi-dialect speech recognition method provided by any embodiment of the present invention.

[0128] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for an electronic device (which can be a mobile phone, a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0129] It is worth noting that in the above-mentioned embodiment of the multi-dialect speech recognition method device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0130] For those skilled in the art, various other corresponding changes and deformations can be made according to the technical solutions and concepts described above, and all of these changes and deformations should fall within the protection scope of the claims of the present invention.

Claims

1. A multi-dialect speech recognition method, characterized in that: The following steps are involved: Get dialect data; Extracting features of the dialect data and representing the features as speech representations; Inputting the speech representation into a Transformer encoder based on a sparsity bias calculation mode to extract the dialect representation; Performing dialect classification according to the dialect representation information to obtain a dialect classification result; The classification result and the speech pattern are input into a multi-expert integrated system for prediction to obtain a speech recognition result.

2. The multi-dialect speech recognition method according to claim 1, characterized in that: The features of the dialect data are extracted by a feature extractor and a shared encoder, and the features are represented as speech representations, including the following steps: The feature extractor extracts the feature representation of the dialect data, satisfying: t =Extract(x t ), where f t is the characteristic representation, x t The original speech data is the dialect data; The feature representation is encoded into a speech representation S by the shared encoder t , satisfy: S t =Encoder(f t ).

3. The multi-dialect speech recognition method according to claim 1 or 2, characterized in that: The Transformer encoder based on the sparsity bias calculation mode includes: a context-based sparse self-attention mechanism block, a layer normalization layer, a feedforward neural network and a layer normalization layer.

4. The multi-dialect speech recognition method according to claim 3, characterized in that: Each layer of the Transformer encoder based on the sparsity bias calculation mode includes three stages, each stage includes: The speech representation is input into the self-attention mechanism block, which includes a query set Q key and a key value set K. The key value set K is used to generate a matrix M by scaling the dot product, and the top k largest scores and corresponding position information of each row in M ​​are extracted into the matrix L, P, satisfying M=QK T ; L, P = topk(M, k); Traverse the matrix P and matrix L, fill the elements in the matrix L at the same position into the matrix M according to the position information stored in the matrix P, obtain the matrix M′ after coefficient processing, and calculate the attention score to satisfy: Among them, V represents the value set in the attention mechanism, d k is the dimension of the key-value set; The attention mechanism is input into the subsequent layer normalization layer, feedforward neural network and layer normalization layer to obtain the dialect representation Where n represents the number of encoder layers.

5. The multi-dialect speech recognition method according to claim 1, characterized in that: Performing dialect classification according to the dialect representation information to obtain a dialect classification result includes: aggregating the frame-level representations of the dialect representation into a sentence-level representation; The sentence-level representation is input into a fully connected network and output to obtain a dialect classification result.

6. The multi-dialect speech recognition method according to claim 1, characterized in that: The classification result and the speech pattern are input into a multi-expert integrated system for prediction to obtain a speech recognition result, including: Input the dialect classification result into the expert selection network to obtain the distribution probability, wherein the expert selection network is a single-layer linear layer network, and map the distribution space corresponding to the dialect classification result to the distribution space of the expert selection network to obtain the distribution probability, wherein the distribution probability is the weight of each decoder in the expert selection network during decoding prediction; According to the distribution probability and the corresponding weight, the expert mixed recognition result is weighted to be used as the speech recognition result.

7. The multi-dialect speech recognition method according to claim 1 or 6, characterized in that: The training of the multi-expert integrated system includes CTC learning, ATTENTION learning and classification learning for the dialect speech data, including the following steps: Calculating dialect classification loss satisfy: Among them, C is the number of dialect categories, B is the number of speech to be classified, and y i，j Indicates the label that the i-th speech belongs to dialect j, p i，j represents the probability of the corresponding prediction; Calculating CTC loss satisfy: Among them, Fun -1 is the inverse function, which is used to give all valid alignment paths between the input sequence and the output sequence, a t With S t Represent the output text sequence and the input speech representation sequence respectively, f is the total duration sequence, and t is the time; Calculating ATTENTION loss satisfy: in, is a true transcript, t is predictive text; Calculate the total loss to satisfy: Among them, α1, α1 are weight parameters.

8. A multi-dialect speech recognition device, characterized in that: It includes: Feature extraction module, used to obtain dialect data; Extracting features of the dialect data and representing the features as speech representations; Inputting the speech representation into a Transformer encoder based on a sparsity bias calculation mode to extract the dialect representation; A speech recognition module, used for performing dialect classification according to the dialect representation information to obtain a dialect classification result; The classification result and the speech pattern are input into a multi-expert integrated system for prediction to obtain a speech recognition result.

9. An electronic device comprising a processor, a storage medium and a computer program, wherein the computer program is stored in the storage medium, wherein: When the computer program is executed by a processor, the multi-dialect speech recognition method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-dialect speech recognition method according to any one of claims 1 to 7 is implemented.

Images