Dialect recognition method, device, equipment and computer program product

Abstract

The invention provides a dialect recognition method, device and equipment and a computer program product, and the method comprises the steps: carrying out the coding and decoding of a to-be-recognized dialect voice, and obtaining a voice decoding feature; decoding the voice decoding features through a pre-trained language model to obtain a dialect recognition result; wherein the dialect text in the decoding network of the language model and the mandarin text with the same semantics as the dialect text correspond to the same language model probability. The dialect recognition method can improve the dialect recognition effect.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a dialect recognition method, device, equipment and computer program product. Background Art

[0002] In speech recognition scenarios, the recognition of dialect speech has always been a difficult problem. Due to my country's vast territory, large population, rich dialect systems and dialect types, the amount of data for different dialects is uneven. Some dialects have less data, making it difficult to obtain sufficient parallel data for training speech recognition models, which makes it difficult for speech recognition models to achieve ideal recognition results when recognizing speech in these dialects. Summary of the invention

[0003] In view of the above-mentioned technical status, the present application proposes a dialect recognition method, device, equipment and computer program product, which can improve the dialect recognition effect.

[0004] The first aspect of the present application provides a dialect recognition method, comprising:

[0005] Perform encoding and decoding processing on the dialect speech to be recognized to obtain speech decoding features;

[0006] Decoding the speech decoding features through a pre-trained language model to obtain a dialect recognition result;

[0007] The dialect text in the decoding network of the language model and the Mandarin text with the same semantics as the dialect text correspond to the same language model probability.

[0008] In some implementations, the dialect speech to be recognized is subjected to encoding and decoding processing to obtain speech decoding features, including:

[0009] Encoding the dialect speech to be recognized to obtain a first encoding result;

[0010] Generating a specific type of phoneme label corresponding to the dialect speech to be recognized based on the first encoding result;

[0011] Performing language modeling encoding on the first encoding result based on the phoneme label to obtain a second encoding result;

[0012] The second encoding result is decoded to obtain a speech decoding feature.

[0013] In some implementations, encoding the dialect speech to be recognized to obtain a first encoding result; generating a specific type of phoneme label corresponding to the dialect speech to be recognized based on the first encoding result; and performing language modeling encoding on the first encoding result based on the phoneme label to obtain a second encoding result, including:

[0014] Inputting the dialect speech to be recognized into a pre-trained speech encoder so that the speech encoder encodes the dialect speech to be recognized to obtain a first encoding result, generating a specific type of phoneme label corresponding to the dialect speech to be recognized based on the first encoding result, and performing language modeling encoding on the first encoding result based on the phoneme label to obtain a second encoding result;

[0015] The speech encoder is obtained by encoding speech samples of multiple dialects through training, and the speech encoder generates the specific type of phoneme labels of the speech samples when encoding the speech samples of the multiple dialects during the training process.

[0016] In some implementations, the speech encoder shares model parameters with a pre-trained unified recognition model;

[0017] The unified recognition model learns the encoding features of various dialects through a meta-learning training method.

[0018] In some implementations, the training process of the unified recognition model includes:

[0019] Get the initial codes of various dialects;

[0020] Performing meta-learning training on the unified recognition model using training audio in a support set to update language classification-related parameters of the unified recognition model; the support set includes training audio in a specific dialect;

[0021] The unified recognition model is meta-learned and trained using the test audio in the query set to update parameters of the unified recognition model that are not related to language classification; the query set includes the test audio in the specific dialect.

[0022] In some implementations, obtaining initial codes of various dialects includes:

[0023] Get the initial code of the dialect language family;

[0024] Based on the initial code of the dialect system, initial codes of various dialect languages ​​belonging to the dialect system are obtained.

[0025] In some implementations, encoding and decoding the dialect speech to be recognized to obtain speech decoding features; decoding the speech decoding features using a pre-trained language model to obtain a dialect recognition result includes:

[0026] Input the dialect speech to be recognized into a pre-trained dialect speech recognition model, so that the dialect speech recognition model performs encoding and decoding processing on the dialect speech to be recognized to obtain speech decoding features, and decodes the speech decoding features through a pre-trained language model to obtain a dialect recognition result; wherein the dialect text in the decoding network of the language model and the Mandarin text with the same semantics as the dialect text correspond to the same language model probability;

[0027] The dialect speech recognition model is obtained by performing dialect speech recognition training based on dialect speech samples and performing dialect text prediction training based on dialect text pronunciation samples, wherein the dialect text pronunciation samples include Mandarin pronunciation corresponding to the dialect text.

[0028] The second aspect of the present application provides a dialect recognition device, comprising:

[0029] A speech encoding and decoding unit, used for encoding and decoding the dialect speech to be recognized to obtain speech decoding features;

[0030] A language decoding unit, used to decode the speech decoding features through a pre-trained language model to obtain a dialect recognition result;

[0031] The dialect text in the decoding network of the language model and the Mandarin text with the same semantics as the dialect text correspond to the same language model probability.

[0032] A third aspect of the present application provides an electronic device, including a memory and a processor;

[0033] The memory is connected to the processor and is used to store programs;

[0034] The processor is used to implement the above-mentioned dialect recognition method by running the program in the memory.

[0035] A fourth aspect of the present application provides a computer program product, comprising computer program instructions, which, when executed by a processor, enable the processor to execute the above-mentioned dialect recognition method.

[0036] The dialect recognition method proposed in the present application optimizes the decoding network in the language model of the dialect speech recognition model, so that the dialect text in the decoding network and the Mandarin text with the same semantics as the dialect text correspond to the same language model probability. In this way, the dialect text of the low-resource dialect language shares the language model probability of the text of the high-resource language, thereby improving the decoding accuracy of the low-resource dialect language. When the dialect speech is recognized, the above-mentioned optimized language model is used for decoding, which can improve the dialect speech recognition effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0038] Figure 1 A schematic diagram of the speech recognition process provided in an embodiment of the present application.

[0039] Figure 2 A flowchart of a dialect recognition method provided in an embodiment of the present application.

[0040] Figure 3 A schematic diagram of the process of encoding and decoding the dialect speech to be recognized provided in an embodiment of the present application.

[0041] Figure 4 A schematic diagram of the training of the unified recognition model provided in an embodiment of the present application.

[0042] Figure 5 A schematic diagram of the training process of a dialect speech recognition model provided in an embodiment of the present application.

[0043] Figure 6 A schematic diagram of the training process of another dialect speech recognition model provided in an embodiment of the present application.

[0044] Figure 7 A schematic diagram of the training process of another dialect speech recognition model provided in an embodiment of the present application.

[0045] Figure 8 A schematic diagram of the structure of a dialect recognition device provided in an embodiment of the present application.

[0046] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In speech recognition scenarios, the recognition of dialect speech has always been a difficult problem. Due to my country's vast territory, large population, rich dialect systems and dialect types, the amount of data for different dialects is uneven. Some dialects have less data, making it difficult to obtain sufficient parallel data for training speech recognition models, which makes it difficult for speech recognition models to achieve ideal recognition results when recognizing speech in these dialects.

[0048] Speech recognition technology refers to the technology that automatically converts the content of human speech into text, also known as Automatic Speech Recognition, that is, ASR technology. Since the 1980s, speech recognition has adopted the basic framework of pattern recognition, which is divided into four steps: data preparation, feature extraction, model training, and test application. After the model is trained, a speech to be tested needs to go through signal processing and feature extraction, and then use the trained acoustic model and language model to obtain the result of speech recognition.

[0049] See Figure 1 As shown, the process of speech recognition by a speech recognition model mainly includes four parts: signal processing (feature extraction), acoustic model, language model, and dictionary and decoding. Among them, in order to extract features more effectively, it is often necessary to perform preprocessing operations such as filtering and framing on the collected sound signals to extract the signal to be analyzed from the original signal; after that, the signal processing (feature extraction) work converts the sound signal from the time domain to the frequency domain to provide a suitable feature vector for the acoustic model; in the acoustic model, the score of each feature vector on the acoustic feature is calculated according to the acoustic features; while the language model calculates the probability of the possible phrase sequence corresponding to the sound signal according to the relevant linguistic theories; finally, according to the existing dictionary, the phrase sequence is decoded to obtain the final possible text representation.

[0050] In the above speech recognition process, the acoustic model is used to connect the observed features of the speech signal with the speech modeling unit of the sentence, while the language model is related to text processing. The physical meaning of the language model is to reflect the prior probability of the occurrence of words. For example, the two characters "郝" and "好" have the same pronunciation, but the probability of "郝" appearing is lower than that of "好", and it generally appears in surnames; another example is that for the pinyin "nihao", the candidate words decoded by the language model will be "你好" instead of "尼毫", and the arrangement of the candidate words refers to the order of the scores (probabilities) of the language model from high to low. In addition, the physical meaning of the language model also lies in reflecting whether the word order conforms to language habits and reflecting the semantic information of words. In addition, the language model in speech recognition is also used to process text sequences, and it combines the output of the acoustic model to give the text sequence with the highest probability as the speech recognition result.

[0051] When training a speech recognition model, there is a phenomenon that the speech modeling units of low-resource languages are not fully trained. For phrase collocations that do not appear in the training set, the model generally cannot recognize them, resulting in poor recognition effects of the model on some low-resource languages.

[0052] In response to the above technical problems, the embodiments of the present application propose a new speech recognition scheme and a speech recognition model training scheme. The above scheme unifies the decoding spaces of different languages ​​and realizes unified modeling of different languages. In this way, the decoding and modeling of high-resource languages ​​can be utilized to assist the decoding and modeling of low-resource languages, thereby improving the decoding and modeling effects of low-resource languages ​​and improving recognition accuracy.

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] This application embodiment proposes a dialect recognition method, see Figure 2 As shown, the method includes:

[0055] S101, performing encoding and decoding processing on the dialect speech to be recognized to obtain speech decoding features.

[0056] Among them, encoding and decoding the dialect speech to be recognized refers to inputting the dialect speech to be recognized into the dialect speech recognition model, and the dialect speech recognition model performs encoding and decoding processing on the speech to be recognized to obtain speech decoding features of the decoded output. The above-mentioned inputting the dialect speech to be recognized into the dialect speech recognition model can be inputting the audio signal of the dialect speech to be recognized into the dialect speech recognition model, and the dialect speech recognition model performs feature extraction, encoding and decoding processing on the dialect speech to be recognized, or it can be inputting the extracted audio features of the dialect speech to be recognized into the dialect speech recognition model, and the dialect speech recognition model performs encoding and decoding processing on the audio features of the dialect speech to be recognized.

[0057] The dialect speech recognition model encodes and decodes the dialect speech to be recognized, which can be achieved through the encoder and decoder inside the model respectively.

[0058] The speech decoding features obtained through the above processing can be used to combine the language model and the dictionary to decode the recognition text of the dialect speech to be recognized.

[0059] S102, decoding the speech decoding feature through a pre-trained language model to obtain a dialect recognition result;

[0060] The dialect text in the decoding network of the language model and the Mandarin text with the same semantics as the dialect text correspond to the same language model probability.

[0061] The above-mentioned language model probability refers to the decoding probability set for various texts (including text sequences, text characters) in the decoding network, that is, for a given voice input, when searching for the optimal decoding text path, the selection probability of various decoding text paths. In some embodiments, the voice model probability in the decoding network can also be expressed in the form of a language model score. For a given voice input, when searching for the optimal decoding text path, if the language model probability or language model score of a certain text path is the highest, the text on the text path will be selected as the current decoding text; on the contrary, if the language model probability or language model score of a certain text path is low, the text on the text path will not be selected as the current decoding text.

[0062] The language model probability of each text in the decoding network is determined during the model training process by accumulating a large amount of text decoding experience and text expression habits. Therefore, for some low-resource dialects, due to the small amount of training data that can be collected and insufficient model training, the language model probability of these dialect voices corresponding to the text in the language model decoding network is low, resulting in poor recognition of these dialect voices.

[0063] For example, in Mandarin, "catch dragonflies" is a common expression. Based on a large amount of Mandarin data training, in the various decoded texts corresponding to the speech "zhuo qing ting" in the language model's decoding network, the language model probability score of "catch dragonflies" is the highest, so the final output recognition result is "catch dragonflies". Specifically, when the speech recognition model recognizes the speech "zhuo qing ting", after recognizing "zhuo" as "catch", when decoding "qing ting", the language model probability of each decoded text path corresponding to "qing ting" is queried in the language model's decoding network. For example, "dragonfly" is 90, "listen" is 8, and "green court" is 2. Since the language model probability of "dragonfly" is the highest, "qing ting" is decoded as "dragonfly", and the final recognition result is "catch dragonflies".

[0064] When recognizing dialect speech, the recognition is also performed according to the above recognition process. The difference is that due to the small amount of sample data of low-resource dialects, when jointly trained with other high-resource languages ​​(because speech models are usually used to recognize speech in various languages, sample speech in multiple languages ​​is required to train the model), the language model probability of the decoding text path corresponding to the low-resource dialect will be lowered. Therefore, when the language recognition model performs speech recognition on the low-resource dialect, due to the low language model probability of the corresponding decoding text path, the dialect speech will be decoded into text with a higher probability of other language models, resulting in decoding errors.

[0065] For example, there is a word "Malang" in the dialect, and there is a word "Malang" in Mandarin. The corresponding pronunciations of these two words are the same, "ma lang". Since the sample data volume of the dialect is much smaller than that of Mandarin, after training the speech recognition model, in the decoding network of the language model, the language model probability of "Malang" is smaller than that of "Malang", for example, the language model probability of "Malang" is 10, and the language model probability of "Malang" is 20. In this case, when recognizing the dialect speech "zhuo ma lang", after "zhuo" is decoded as "zhuo", when decoding "ma lang", the language model probability of "Malang" is smaller than that of "Malang", so "malang" is likely to be recognized as "Malang" instead of "Malang", and finally the wrong recognition result of "zhuo Malang" is obtained.

[0066] Through the analysis of the above problems, the embodiment of the present application optimizes the language model probability of the dialect text in the decoding network of the language model during the training of the dialect speech recognition model. Specifically, the language model probability of the dialect text is set to the language model probability corresponding to the Mandarin text with the same semantics as the dialect text.

[0067] For example, the "Malang" in the dialect is actually the "Dragonfly" in Mandarin. The two are different names for the same creature, that is, the "Dragonfly" in Mandarin and the "Malang" in the dialect have the same meaning. Therefore, in the embodiment of the present application, when training the dialect speech recognition model, the language model probability of the dialect text "Malang" in the decoding network of the language model is set to the language model probability corresponding to the Mandarin text "Dragonfly" with the same semantics as the dialect text "Malang". Assuming that the language model probability of "Malang" in the decoding network of the original language model is 10, the language model probability of "Malang" is 20, and the language model probability of "Dragonfly" is 90, then during the training process, the language model probability of "Malang" in the decoding network of the language model is set to 90, that is, the same as the language model probability of "Dragonfly", while "Malang" and "Dragonfly" have different semantics, so its language model probability remains at 20.

[0068] After setting the language model probability as above, when recognizing the dialect speech "zhuo ma lang", after decoding "zhuo" as "catch", when decoding "ma lang", since the language model probability of "ants" is greater than the language model probability of "horse wolf", "ma lang" will be recognized as "ants" instead of "horse wolf", and finally the correct recognition result of "catch ants" is obtained.

[0069] Based on the above optimization of the decoding network of the language model, after obtaining the speech decoding features of the dialect speech to be recognized, the speech decoding features are decoded by the above optimized language model to obtain a more accurate dialect recognition result.

[0070] From the above introduction, it can be seen that the dialect recognition method proposed in the embodiment of the present application optimizes the decoding network in the language model of the dialect speech recognition model, so that the dialect text in the decoding network and the Mandarin text with the same semantics as the dialect text correspond to the same language model probability. In this way, the dialect text of the low-resource dialect language shares the language model probability of the text of the high-resource language, thereby improving the decoding accuracy of the low-resource dialect language. When the dialect speech is recognized, the above-mentioned optimized language model is used for decoding, which can improve the dialect speech recognition effect.

[0071] In some embodiments, when encoding and decoding the dialect speech to be recognized, see Figure 3 As shown, the dialect speech to be recognized is first encoded by the shallow encoding layer encoder1 to obtain a first encoding result, and then a specific type of phoneme label corresponding to the dialect speech to be recognized is generated based on the obtained first encoding result, that is, the first encoding result is annotated with a phoneme label of a specific type of phoneme. Then, the first encoding result is encoded by the deep encoding layer encoder2 based on the annotated phoneme label to obtain a second encoding result. Finally, the second encoding result is decoded to obtain the speech decoding features of the dialect speech to be recognized.

[0072] In some embodiments, the dialect speech to be recognized is input into a pre-trained speech encoder, so that the speech encoder encodes the dialect speech to be recognized to obtain a first encoding result, generates a specific type of phoneme label corresponding to the dialect speech to be recognized based on the first encoding result, and performs language modeling encoding on the first encoding result based on the phoneme label to obtain a second encoding result;

[0073] The speech encoder is obtained by encoding speech samples of multiple dialects (including Mandarin) through training, and the speech encoder generates the specific type of phoneme labels for the speech samples when encoding the speech samples of the multiple dialects during the training process.

[0074] In the above processing, the first encoding result is annotated with a phoneme label of a specific type of phoneme. Specifically, the international phonetic alphabet (IPA) may be used to annotate the first encoding result with a phoneme label.

[0075] During the training process, when encoding language samples of any dialect or Mandarin, the first encoding results of the speech samples all generate phoneme labels of the same specific type of phonemes mentioned above, that is, the speech is acoustically modeled in the same acoustic space to construct a unified acoustic decoding space.

[0076] Generally speaking, due to insufficient acoustic training data for low-resource languages, end-to-end training cannot model robust semantic dependencies. For some dialect words, even though their homophones have been fully trained in Chinese, the model may not be able to recognize them due to the statistical sparsity of the language model probability.

[0077] The embodiment of the present application unifies the acoustic modeling space of different dialects, that is, in the encoding process of acoustic modeling of different dialects, all dialect encodings generate unified phoneme labels, such as Figure 3 As shown in the figure, after the middle layer of the model recognizes and generates phoneme labels (IPA results), they are input into the subsequent layers through gating, which is equivalent to using the shallow layer of the network as an acoustic model to learn the unified acoustic knowledge of various dialects, and the deep layer of the network performs language modeling of specific languages. In this way, the acoustic information of high-resource languages ​​can directly assist the modeling of acoustic information of low-resource languages, reducing the dependence of low-resource languages ​​on acoustic training data. Due to the unification of the acoustic level, different languages ​​can share the parameters of pronunciation feature extraction, and the deep layer of the network can also use plain text data for training and learning language information, further reducing the dependence of low-resource languages ​​on acoustic data.

[0078] In the process of training the above-mentioned speech encoder, the process of generating the first encoding result and the phoneme label, and the process of performing language modeling encoding on the first encoding result based on the phoneme label to obtain the second encoding result, can both be optimized and trained by the CTC loss function.

[0079] In some embodiments, the speech encoder in the dialect speech recognition model shares model parameters with a pre-trained unified recognition model.

[0080] The unified recognition model mentioned above refers to a model for encoding speech in various languages, and the model learns the encoding features of speech in various dialects by means of meta-learning.

[0081] Among them, the meaning of meta-learning is learning to learn. The birth of meta-learning is to enable the model to acquire a "learning to learn" ability, so that it can quickly learn new tasks based on the existing knowledge. In the embodiment of the present application, the above-mentioned unified recognition model is trained by meta-learning, so that the unified recognition model can quickly learn the speech encoding of the new language under the condition of the speech encoding of the previously learned dialect. Therefore, for low-resource dialects, although the amount of data is small, the unified recognition model can still quickly learn the speech encoding of the new dialect based on the speech encoding of other languages ​​that have been learned through meta-learning. Therefore, the above-mentioned unified recognition model training method can reduce the dependence on training data of low-resource languages, so that accurate speech encoding can be obtained for any dialect.

[0082] The training of the above unified recognition model is as follows Figure 4 shown.

[0083] During the training process, the unified recognition model first obtains the initial codes of various dialects. The initial codes of various dialects obtained by the unified recognition model based on meta-learning are generated using the dialect classification model, that is, the initial codes of various dialects are obtained by performing the dialect classification task.

[0084] In addition, due to the correlation between dialects, for example, different schemes in some regions constitute a dialect system, and different dialects within a language system may have many similarities, so when obtaining the encoding of different dialects under the same language system, if the common encoding information of the language system can be used, it will effectively improve the encoding efficiency of various dialects within the language system.

[0085] Therefore, in the embodiment of the present application, when obtaining the initial codes of various dialects, the unified recognition model first obtains the initial code of the dialect system, and then obtains the initial codes of various dialects belonging to the dialect system based on the initial code of the dialect system.

[0086] Specifically, during the training process, when the audio features of the dialect speech enter the speech encoder, language information is generated through pooling, the classification layer is used for language prediction, and the classification layer weights are used as language encoding.

[0087] At the same time, in order to increase the discrimination of language encoding, the embodiment of the present application uses AM-Softmax as the loss function. AM-Softmax reduces the score of the target language, so that the model needs to assign a higher score to the target language. Because the score gap between the target language and the non-target language increases, the discrimination of different language encodings is higher.

[0088] The meta-learning training process of the unified recognition model first fixes the language-independent parameters of the model, and uses the training audio in the support set to perform meta-learning training on the unified recognition model, so that the model classifies the training audio in the support set by language, and obtains the language encoding at the same time. The training process updates the parameters of the unified recognition model related to language classification. The above-mentioned support set includes training audio of a specific dialect, such as training audio of any dialect, or training audio of all dialects.

[0089] Then, the language encoding of the unified recognition model is fixed, and the unified recognition model is meta-learned and trained using the test audio in the query set, and the parameters of the unified recognition model that are not related to language classification are updated. The query set includes the test audio of the specific dialect. That is, the support set and query set are obtained by dividing all the audio data collected from a specific dialect. For example, if there are 500 hours of audio data collected from a specific dialect, 400 hours of audio data can be used as the support set, and the remaining 100 hours of audio data can be used as the query set.

[0090] The meta-learning training method can quickly learn the encoding of new dialects based on the encoding information of various dialects that have been acquired by the unified recognition model. Therefore, the dialect encoding generated based on meta-learning training has good generalization for new languages ​​and can reduce the demand for training data.

[0091] The unified recognition model obtained through the above training shares model parameters with the speech encoder in the dialect speech recognition model, which can enable the speech encoder to fully realize data sharing between dialect speeches and improve the encoding effect on low-resource dialects, which in turn is conducive to improving the recognition effect of the dialect speech recognition model on low-resource dialects.

[0092] In other embodiments, the dialect recognition method proposed in the embodiments of the present application can be executed by a dialect speech recognition model, that is, the recognized dialect speech is input into a pre-trained dialect speech recognition model, so that the dialect speech recognition model performs encoding and decoding processing on the dialect speech to be recognized to obtain speech decoding features, and the speech decoding features are decoded by a pre-trained language model to obtain a dialect recognition result; wherein the dialect text in the decoding network of the language model and the Mandarin text with the same semantics as the dialect text correspond to the same language model probability.

[0093] The dialect speech recognition model is obtained by performing dialect speech recognition training based on dialect speech samples and performing dialect text prediction training based on dialect text pronunciation samples, wherein the dialect text pronunciation samples include Mandarin pronunciation corresponding to the dialect text.

[0094] The above-mentioned dialect text pronunciation sample is a pronunciation sequence sample consisting of the Mandarin pronunciation corresponding to the dialect text. For example, the Mandarin pronunciation "m a3 lang2" corresponding to the dialect word "ants" can be used as a dialect text pronunciation sample.

[0095] In the above-mentioned dialect speech recognition model training process, performing dialect speech recognition training on the dialect speech recognition model based on the dialect speech sample means inputting the dialect speech audio into the dialect speech recognition model, and the dialect speech recognition model recognizes the input dialect speech to obtain a text recognition result.

[0096] The dialect text prediction training of the dialect speech recognition model based on the dialect text pronunciation samples is to input the dialect text pronunciation samples into the dialect speech recognition model, and the dialect speech recognition model performs autoregressive text prediction based on the input dialect text pronunciation samples, such as predicting the next recognized character based on the recognized text, or making the dialect speech recognition model perform a character mask prediction task based on the dialect text pronunciation samples.

[0097] The above-mentioned dialect speech samples can be obtained by collecting dialect speech data of a specific language, and the collected dialect speech data are annotated with corresponding text labels, which can be used to perform dialect speech recognition training on the dialect speech recognition model.

[0098] The above-mentioned dialect text pronunciation samples can be obtained by first generating the dialect text and then generating the Mandarin pronunciation corresponding to the dialect text.

[0099] In one embodiment of the present application, an implementation method for obtaining a dialect text pronunciation sample and a speech recognition tag corresponding to the dialect speech sample is disclosed.

[0100] Specifically, in order to obtain sufficient dialect text, in this embodiment, Mandarin text is used to generate dialect text, that is, Mandarin text is converted into dialect text.

[0101] For example, Mandarin text can be input into a dialect translation engine or a large language model, and based on the translation capability of the dialect translation engine or the language conversion capability of the large language model, the Mandarin text can be converted into a dialect text. For example, the Mandarin word "dragonfly" can be converted into the dialect word "ant roach", and the Mandarin word "handsome guy and beautiful girl" can be converted into the dialect word "handsome boy and beautiful girl", and so on.

[0102] Through the above-mentioned text conversion method, we can take advantage of the quantity of Mandarin text to generate a large amount of dialect text, thereby providing a data basis for the training of dialect speech recognition models.

[0103] After obtaining the dialect text in the above manner, the embodiment of the present application further generates the dialect pronunciation corresponding to each dialect text based on the dialect text to dialect pronunciation mapping table, that is, determines the dialect pronunciation sequence corresponding to each dialect text.

[0104] Then, the dialect pronunciation is converted into Mandarin pronunciation, for example, by manual conversion or by using a pronunciation prediction model, or by combining manual conversion with a pronunciation prediction model.

[0105] Finally, based on the above conversion process, a mapping table from dialect text to Mandarin pronunciation can be compiled. In the mapping table, the Mandarin pronunciation corresponding to each dialect text is recorded, that is, each dialect text corresponds to a Mandarin pronunciation sequence.

[0106] Based on the above mapping table, a dialect text pronunciation sample corresponding to the dialect text can be generated. That is, for any dialect text, by querying the above mapping table, the Mandarin pronunciation sequence corresponding to the dialect text can be obtained, and the Mandarin pronunciation sequence can be used as the dialect text speech sample corresponding to the dialect text.

[0107] In addition, for dialect speech, it can be used to construct dialect speech samples to perform speech recognition training on the dialect speech recognition model. For minority dialects, due to the small number of speech samples, in order to better use the collected dialect speech for the training of the dialect speech recognition model and improve the training effect, the embodiment of the present application trains a phoneme forced alignment model to generate speech recognition labels corresponding to the dialect speech samples.

[0108] Specifically, the embodiment of the present application trains the Kaldi model as a phoneme forced alignment model. Pre-collect dialect speech samples and corresponding speech tags, which include text tags and phoneme tags, and in some cases only phoneme tags. In addition, the mapping table from dialect text to Mandarin pronunciation obtained by the processing method introduced in the above embodiment is also used for model training.

[0109] When the Kaldi model is trained, the dialect speech sample and the speech label parallel data, as well as the above-mentioned mapping table, are used as training inputs, and the Kaldi model performs the speech recognition task. In the process of the Kaldi model performing the speech recognition task, the speech and phoneme alignment process is automatically performed. Therefore, the embodiment of the present application performs dialect speech phoneme annotation training on the Kaldi model, and the trained Kaldi model can be used to perform the phoneme alignment and annotation tasks of the speech, that is, to obtain a phoneme forced alignment model.

[0110] Then, the dialect speech samples without phoneme labels are input into the above-mentioned phoneme forced alignment model, so that the phonemes of the dialect speech samples can be labeled and the phoneme labels of the dialect speech samples can be obtained. Through the above-mentioned processing, the collected dialect speech samples can all be labeled with their phoneme labels, and then the dialect speech samples labeled with phoneme labels can be used to train the dialect speech recognition model, which increases the number of model training samples, thereby further improving the model training effect.

[0111] Through the processing of the above-mentioned embodiments, dialect text samples can be generated based on Mandarin text, and further through the mapping table of dialect text to Mandarin pronunciation, dialect text pronunciation samples can be obtained. And, through the phoneme forced alignment model constructed in the above manner, dialect speech samples carrying phoneme labels can be obtained. The above-mentioned dialect text pronunciation samples and dialect speech samples carrying phoneme labels can both be used as training data for training the dialect speech recognition model. The embodiments of the present application use them for training the dialect speech recognition model, which can increase the amount of model training data, thereby improving the model training effect and improving the model's recognition performance for dialect speech.

[0112] In other embodiments, it is disclosed that when the dialect speech recognition model is trained using the above-mentioned dialect text pronunciation samples and dialect speech samples carrying phoneme labels, the dialect speech recognition model is trained for dialect speech recognition based on the dialect speech samples, and the dialect text prediction training is performed on the dialect speech recognition model based on the dialect text pronunciation samples.

[0113] Specifically, Figure 5 A schematic diagram of the process of training a dialect speech recognition model is shown. Figure 5 As shown, when the dialect speech recognition model is trained, the dialect speech samples and the dialect text pronunciation samples are input into the dialect speech recognition model so that the dialect speech recognition model recognizes the dialect speech samples and performs dialect text prediction on the dialect text pronunciation samples to obtain speech recognition results and text prediction results.

[0114] When training the dialect speech recognition model, features can be extracted from the dialect speech sample and the dialect text pronunciation sample respectively, and then the dialect speech sample features and the dialect text pronunciation sample features can be input into the dialect speech recognition model. Exemplarily, the dialect speech sample features and the dialect text pronunciation sample features can be simultaneously input into the dialect speech recognition model, so that the dialect speech recognition model performs the recognition task of the dialect speech sample and the dialect text prediction task of the dialect text pronunciation sample; or, the dialect speech sample features and the dialect text pronunciation sample features can be successively input into the dialect speech recognition model, so that the dialect speech recognition model performs the recognition task of the dialect speech sample and the dialect text prediction task of the dialect text pronunciation sample.

[0115] By making the dialect speech recognition model perform the dialect speech recognition task and the text prediction task, the speech recognition result and the text prediction result can be obtained respectively.

[0116] Then, the speech recognition result corresponding to the dialect speech sample is compared with the speech recognition label corresponding to the dialect speech sample, and the dialect recognition loss function is calculated; and the text prediction result is compared with the dialect text pronunciation sample, specifically, the text characters in the text prediction result are compared with the text characters corresponding to the dialect text pronunciation sample, and the text prediction loss function is calculated.

[0117] Finally, with the goal of reducing dialect recognition loss and text prediction loss, the parameters of the dialect speech recognition model are optimized.

[0118] Repeat the above training process and continue to let the dialect speech recognition model perform dialect speech recognition and dialect text prediction tasks. During the training process, the dialect speech recognition model can learn dialect text pronunciation information while performing dialect speech recognition, so that the dialect speech recognition model can more fully learn and explore the potential relationship between dialect speech and its pronunciation and text, thereby improving the training effect of the dialect speech recognition model and improving its recognition performance of dialect speech.

[0119] In other embodiments, the speech recognition labels of the dialect speech samples used to train the dialect speech recognition model include word-level labels and phoneme-level labels, wherein the word-level labels include text characters corresponding to the dialect speech samples, and the phoneme-level labels include phonemes in the pronunciation of the dialect speech samples.

[0120] Based on the above speech recognition tags, see Figure 6 As shown, during the training of the dialect speech recognition model, after the dialect speech sample features are input into the dialect speech recognition model to obtain the speech recognition result, when calculating the dialect recognition loss, on the one hand, the word-level dialect recognition loss is determined based on the speech recognition result and the word-level label in the speech recognition label, and on the other hand, the phoneme-level dialect recognition loss is determined based on the speech recognition result and the phoneme-level label in the speech recognition label. The above-mentioned phoneme-level label is obtained by phoneme-annotating the dialect language sample through the phoneme forced alignment model in the above-mentioned embodiment.

[0121] Among them, the character-level dialect recognition loss is determined based on the speech recognition result and the character-level label in the speech recognition label. Specifically, the characters in the speech recognition result output by the dialect speech recognition model are compared with the character labels in the character-level label to determine whether the recognized characters and character labels at the same position are consistent, and then the character recognition loss function is calculated. The above-mentioned character recognition loss function can be obtained by calculating the loss function corresponding to each recognized character and then calculating the sum of the loss functions corresponding to each character, or it can be obtained by synchronously comparing the recognized characters and character labels at each character position and calculating the loss function.

[0122] In some embodiments, the above-mentioned word-level dialect recognition loss and phoneme-level dialect recognition loss can be calculated at the same time and used to optimize the dialect speech recognition model. In some cases, only one of the above-mentioned word-level dialect recognition loss and phoneme-level dialect recognition loss can be calculated. In this case, the calculated word-level dialect recognition loss or phoneme-level dialect recognition loss can be used to optimize the dialect speech recognition model.

[0123] It can be understood that the dialect speech recognition model training scheme proposed in the embodiment of the present application sets a variety of speech recognition labels, so that different types of dialect recognition losses can be calculated for the same dialect speech sample, thereby expanding the scope of application of the model training data. In particular, the embodiment of the present application can annotate the phoneme labels of the dialect speech samples through the phoneme forced alignment model, and then when the dialect speech recognition model is trained, the dialect speech recognition model can be optimized by calculating the phoneme-level dialect recognition loss. This training scheme makes up for the disadvantage that the dialect speech samples cannot be used for model training due to the inability to obtain parallel data of the dialect speech samples (dialect speech samples and corresponding text labels), thereby advantageously expanding the amount of model training data.

[0124] In other embodiments, the dialect speech recognition model performs dialect text prediction on the dialect text pronunciation sample, specifically, the dialect speech recognition model performs autoregressive text prediction and / or mask prediction based on the input dialect text pronunciation sample to obtain autoregressive prediction results and / or mask prediction results.

[0125] Among them, the dialect speech recognition model performs autoregressive text prediction based on the input dialect text pronunciation samples, which means that the dialect speech recognition model predicts the text characters corresponding to the dialect text pronunciation samples in sequence based on the input dialect text pronunciation samples, such as predicting the next character based on the predicted character sequence.

[0126] The dialect speech recognition model performs mask prediction based on the input dialect text pronunciation sample, which means masking the character speech at a certain position in the dialect text pronunciation sample, and then making the dialect speech recognition model predict the masked character based on the text pronunciation at other positions.

[0127] Based on the above dialect text prediction task, see Figure 6 As shown, when calculating the text prediction loss based on the text prediction results output by the dialect speech recognition model, the first text prediction loss is determined based on the autoregressive prediction results and the dialect text pronunciation samples, and / or the second text prediction loss is determined based on the mask prediction results and the dialect text pronunciation samples.

[0128] Among them, calculating and determining the first text prediction loss based on the autoregressive prediction result and the dialect text speech sample refers to comparing the autoregressive prediction result with the dialect text corresponding to the dialect text pronunciation sample to calculate the text prediction loss.

[0129] Calculating and determining the second text prediction loss based on the mask prediction result and the dialect text speech sample refers to comparing the mask prediction result with the dialect text corresponding to the dialect text pronunciation sample to calculate the text prediction loss.

[0130] It can be understood that the model training scheme of the embodiment of the present application can enable the dialect speech recognition model to perform different text prediction tasks based on the input dialect text pronunciation samples, and then calculate different text prediction losses for parameter optimization of the dialect speech recognition model. This model training method improves the diversity of model training tasks, allowing the dialect speech recognition model to learn and mine the relationship between dialect text and dialect pronunciation in different ways, which is conducive to improving the model training effect.

[0131] In other embodiments, Figure 7 As shown, the dialect speech samples and dialect text pronunciation samples input when training the dialect speech prediction model can be parallel sample data corresponding to each other, that is, the input dialect speech samples and dialect text pronunciation samples correspond to the same dialect text. The difference is that the dialect speech samples are the speech corresponding to the dialect text, and the dialect text pronunciation samples are the pronunciation sequences corresponding to the dialect text.

[0132] Based on the above-mentioned training data characteristics, when training the dialect speech recognition model, the dialect speech features obtained by the dialect speech recognition model processing the dialect speech samples are obtained, and the dialect text features obtained by the dialect speech recognition model processing the dialect text pronunciation samples are obtained.

[0133] Then, the dialect speech features and dialect text features obtained through the dialect speech recognition model are compared to determine the alignment loss between the dialect speech features and the dialect text features.

[0134] The above alignment loss can be used to measure whether the dialect speech features and dialect text features processed by the dialect speech recognition model are consistent in feature content and aligned in time sequence. In theory, since the dialect speech samples and dialect text pronunciation samples input into the dialect speech recognition model are parallel data, the dialect speech features processed by the dialect speech recognition model for the dialect speech samples and the dialect text features processed for the dialect text pronunciation samples should be aligned. If the two are not aligned, it means that there is still room for optimization in the dialect speech recognition model's feature extraction process for the two modal data.

[0135] Based on the above ideas, in the process of training the dialect speech recognition model, the model training scheme proposed in the embodiment of the present application, if the dialect speech samples and dialect text pronunciation samples input into the dialect speech recognition model are parallel sample data corresponding to each other, then in addition to calculating the dialect recognition loss and the text prediction loss, the alignment loss between the dialect speech features and the dialect text features processed by the dialect speech recognition model is also calculated. And, when optimizing the dialect speech recognition model, the above alignment loss, dialect recognition loss and text prediction loss are used together to optimize the dialect speech recognition model, that is, the parameters of the dialect speech recognition model are optimized with the goal of reducing the above alignment loss, dialect recognition loss and text prediction loss.

[0136] It can be understood that the model training scheme applies a variety of different loss functions, so that the dialect speech recognition model can perceive the relationship between dialect speech, dialect text and dialect pronunciation through multiple dimensions during the training process, thereby improving the model training effect and accelerating the model training efficiency.

[0137] Corresponding to the above-mentioned dialect recognition method, the present application embodiment also provides a dialect recognition device, see Figure 8 As shown, the device comprises:

[0138] The speech encoding and decoding unit 100 is used to perform encoding and decoding processing on the dialect speech to be recognized to obtain speech decoding features;

[0139] A language decoding unit 110 is used to decode the speech decoding feature through a pre-trained language model to obtain a dialect recognition result;

[0140] The dialect text in the decoding network of the language model and the Mandarin text with the same semantics as the dialect text correspond to the same language model probability.

[0141] In some implementations, the speech encoding and decoding unit 100 performs encoding and decoding processing on the dialect speech to be recognized to obtain speech decoding features, including:

[0142] Encoding the dialect speech to be recognized to obtain a first encoding result;

[0143] Generating a specific type of phoneme label corresponding to the dialect speech to be recognized based on the first encoding result;

[0144] Performing language modeling encoding on the first encoding result based on the phoneme label to obtain a second encoding result;

[0145] The second encoding result is decoded to obtain a speech decoding feature.

[0146] In some implementations, the speech encoding and decoding unit 100 encodes the dialect speech to be recognized to obtain a first encoding result; generates a specific type of phoneme label corresponding to the dialect speech to be recognized based on the first encoding result; and performs language modeling encoding on the first encoding result based on the phoneme label to obtain a second encoding result, including:

[0147] Inputting the dialect speech to be recognized into a pre-trained speech encoder so that the speech encoder encodes the dialect speech to be recognized to obtain a first encoding result, generating a specific type of phoneme label corresponding to the dialect speech to be recognized based on the first encoding result, and performing language modeling encoding on the first encoding result based on the phoneme label to obtain a second encoding result;

[0148] The speech encoder is obtained by encoding speech samples of multiple dialects through training, and the speech encoder generates the specific type of phoneme labels of the speech samples when encoding the speech samples of the multiple dialects during the training process.

[0149] In some implementations, the speech encoder shares model parameters with a pre-trained unified recognition model;

[0150] The unified recognition model learns the encoding features of various dialects through a meta-learning training method.

[0151] In some implementations, the training process of the unified recognition model includes:

[0152] Get the initial codes of various dialects;

[0153] Performing meta-learning training on the unified recognition model using training audio in a support set to update language classification-related parameters of the unified recognition model; the support set includes training audio in a specific dialect;

[0154] The unified recognition model is meta-learned and trained using the test audio in the query set to update parameters of the unified recognition model that are not related to language classification; the query set includes the test audio in the specific dialect.

[0155] In some implementations, obtaining initial codes of various dialects includes:

[0156] Get the initial code of the dialect language family;

[0157] Based on the initial code of the dialect system, initial codes of various dialect languages ​​belonging to the dialect system are obtained.

[0158] In some implementations, the speech encoding and decoding unit 100 and the language decoding unit 110 are arranged in a dialect speech recognition model, the speech encoding and decoding unit 100 performs encoding and decoding processing on the dialect speech to be recognized to obtain speech decoding features; and the language decoding unit 110 decodes the speech decoding features through a pre-trained language model to obtain a dialect recognition result, including:

[0159] Input the dialect speech to be recognized into a pre-trained dialect speech recognition model, so that the dialect speech recognition model performs encoding and decoding processing on the dialect speech to be recognized to obtain speech decoding features, and decodes the speech decoding features through a pre-trained language model to obtain a dialect recognition result; wherein the dialect text in the decoding network of the language model and the Mandarin text with the same semantics as the dialect text correspond to the same language model probability;

[0160] The dialect speech recognition model is obtained by performing dialect speech recognition training based on dialect speech samples and performing dialect text prediction training based on dialect text pronunciation samples, wherein the dialect text pronunciation samples include Mandarin pronunciation corresponding to the dialect text.

[0161] The dialect recognition device provided in this embodiment belongs to the same application concept as the dialect recognition method provided in the above embodiments of this application, and can execute the dialect recognition method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not fully described in this embodiment, please refer to the specific processing content of the dialect recognition method provided in the above embodiments of this application, and will not be repeated here.

[0162] The functions implemented by the above speech encoding and decoding unit 100 and language decoding unit 110 can be implemented by the same or different processors respectively, and the embodiments of the present application are not limited thereto.

[0163] It should be understood that the units in the above devices can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits, and the functions of some or all units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by the configuration file, so as to realize the functions of some or all of the above units. All units of the above devices can be implemented in the form of a processor calling software, or in the form of hardware circuits, or in part by a processor calling software, and the remaining part is implemented in the form of hardware circuits.

[0164] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0165] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0166] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0167] Another embodiment of the present application also provides an electronic device, see Fig. 9 As shown, the device includes:

[0168] Memory 200 and processor 210;

[0169] The memory 200 is connected to the processor 210 and is used to store programs;

[0170] The processor 210 is used to implement the dialect recognition method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0171] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0172] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.

[0173] A bus may include a pathway that transfers information between components of a computer system.

[0174] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0175] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0176] The memory 200 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0177] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0178] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0179] The communication interface 220 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0180] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement the various steps of any dialect recognition method provided in the above embodiments of the present application.

[0181] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the dialect recognition method described in any of the above-mentioned embodiments of this specification.

[0182] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0183] In addition, an embodiment of the present application may also be a storage medium on which a computer program is stored, and the computer program is executed by a processor to execute the steps of the dialect recognition method described in any of the above embodiments of this specification.

[0184] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0185] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0186] The steps in the methods of each embodiment of the present application can be adjusted in sequence, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0187] The modules and sub-modules in the devices and terminals of the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0188] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0189] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0190] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0191] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0192] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0193] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0194] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A dialect recognition method, characterized in that: include: Perform encoding and decoding processing on the dialect speech to be recognized to obtain speech decoding features; Decoding the speech decoding features through a pre-trained language model to obtain a dialect recognition result; The dialect text in the decoding network of the language model and the Mandarin text with the same semantics as the dialect text correspond to the same language model probability.

2. The method according to claim 1, characterized in that Perform encoding and decoding on the dialect speech to be recognized to obtain speech decoding features, including: Encoding the dialect speech to be recognized to obtain a first encoding result; Generating a specific type of phoneme label corresponding to the dialect speech to be recognized based on the first encoding result; Performing language modeling encoding on the first encoding result based on the phoneme label to obtain a second encoding result; The second encoding result is decoded to obtain a speech decoding feature.

3. The method according to claim 2, characterized in that Encoding the dialect speech to be recognized to obtain a first encoding result; generating a specific type of phoneme label corresponding to the dialect speech to be recognized based on the first encoding result; Performing language modeling encoding on the first encoding result based on the phoneme label to obtain a second encoding result includes: Inputting the dialect speech to be recognized into a pre-trained speech encoder so that the speech encoder encodes the dialect speech to be recognized to obtain a first encoding result, generating a specific type of phoneme label corresponding to the dialect speech to be recognized based on the first encoding result, and performing language modeling encoding on the first encoding result based on the phoneme label to obtain a second encoding result; The speech encoder is obtained by encoding speech samples of multiple dialects through training, and the speech encoder generates the specific type of phoneme labels of the speech samples when encoding the speech samples of the multiple dialects during the training process.

4. The method according to claim 3, characterized in that The speech encoder shares model parameters with a pre-trained unified recognition model; The unified recognition model learns the encoding features of various dialects through a meta-learning training method.

5. The method according to claim 4, characterized in that The training process of the unified recognition model includes: Get the initial codes of various dialects; Performing meta-learning training on the unified recognition model using training audio in a support set to update language classification-related parameters of the unified recognition model; the support set includes training audio in a specific dialect; The unified recognition model is meta-learned and trained using the test audio in the query set to update parameters of the unified recognition model that are not related to language classification; the query set includes the test audio in the specific dialect.

6. The method according to claim 5, characterized in that Get the initial encoding of various dialects, including: Get the initial code of the dialect language family; Based on the initial code of the dialect system, initial codes of various dialect languages ​​belonging to the dialect system are obtained.

7. The method according to claim 1, characterized in that Perform encoding and decoding processing on the dialect speech to be recognized to obtain speech decoding features; The speech decoding feature is decoded by a pre-trained language model to obtain a dialect recognition result, including: Input the dialect speech to be recognized into a pre-trained dialect speech recognition model, so that the dialect speech recognition model performs encoding and decoding processing on the dialect speech to be recognized to obtain speech decoding features, and decodes the speech decoding features through a pre-trained language model to obtain a dialect recognition result; wherein the dialect text in the decoding network of the language model and the Mandarin text with the same semantics as the dialect text correspond to the same language model probability; The dialect speech recognition model is obtained by performing dialect speech recognition training based on dialect speech samples and performing dialect text prediction training based on dialect text pronunciation samples, wherein the dialect text pronunciation samples include Mandarin pronunciation corresponding to the dialect text.

8. A dialect recognition device, characterized in that: include: A speech encoding and decoding unit, used for encoding and decoding the dialect speech to be recognized to obtain speech decoding features; A language decoding unit, used to decode the speech decoding features through a pre-trained language model to obtain a dialect recognition result; The dialect text in the decoding network of the language model and the Mandarin text with the same semantics as the dialect text correspond to the same language model probability.

9. An electronic device, characterized in that: including memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the dialect recognition method as described in any one of claims 1 to 7 by running the program in the memory.

10. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, enable the processor to execute the dialect recognition method as claimed in any one of claims 1 to 7.

Images