Dialect recognition method, apparatus, device, and computer program product

Abstract

The application provides a dialect recognition method, device, equipment and computer program product. The method comprises the following steps: performing coding and decoding processing on a dialect speech to be recognized to obtain speech decoding features; and performing decoding on the speech decoding features by using a pre-trained language model to obtain a dialect recognition result; wherein dialect texts in a decoding network of the language model and common language texts with the same semantics as the dialect texts correspond to the same language model probability. The dialect recognition method can improve the dialect recognition effect.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a dialect recognition method, apparatus, device, and computer program product. Background Art

[0002] In speech recognition scenarios, recognizing speech in dialects has always been a challenge. Due to my country's vast territory, large population, and rich variety of dialects and languages, the amount of data available for different dialects is uneven. Some dialects have relatively small amounts of data, making it difficult to obtain sufficient parallel data for training speech recognition models. As a result, speech recognition models struggle to achieve ideal recognition results for speech in these dialects. Summary of the Invention

[0003] In response to the above-mentioned technical status quo, the present application proposes a dialect recognition method, device, equipment and computer program product, which can improve the dialect recognition effect.

[0004] In a first aspect, the present application proposes a dialect recognition method, comprising:

[0005] Perform encoding and decoding on the dialect speech to be recognized to obtain speech decoding features;

[0006] Decoding the speech decoding features using a pre-trained language model to obtain a dialect recognition result;

[0007] The dialect text in the decoding network of the language model and the Mandarin text with the same semantics as the dialect text correspond to the same language model probability.

[0008] In some implementations, encoding and decoding the dialect speech to be recognized to obtain speech decoding features includes:

[0009] Encoding the dialect speech to be recognized to obtain a first encoding result;

[0010] generating a specific type of phoneme label corresponding to the dialect speech to be recognized based on the first encoding result;

[0011] Performing language modeling encoding on the first encoding result based on the phoneme label to obtain a second encoding result;

[0012] The second encoding result is decoded to obtain a speech decoding feature.

[0013] In some implementations, encoding the dialect speech to be recognized to obtain a first encoding result; generating a specific type of phoneme label corresponding to the dialect speech to be recognized based on the first encoding result; and performing language modeling encoding on the first encoding result based on the phoneme label to obtain a second encoding result includes:

[0014] Inputting the dialect speech to be recognized into a pre-trained speech encoder so that the speech encoder encodes the dialect speech to be recognized to obtain a first encoding result, generating a specific type of phoneme label corresponding to the dialect speech to be recognized based on the first encoding result, and performing language modeling encoding on the first encoding result based on the phoneme label to obtain a second encoding result;

[0015] The speech encoder is obtained by encoding speech samples of multiple dialects through training, and the speech encoder generates the specific type of phoneme labels for the speech samples when encoding the speech samples of the multiple dialects during the training process.

[0016] In some implementations, the speech encoder shares model parameters with a pre-trained unified recognition model;

[0017] The unified recognition model learns the encoding features of various dialects through a meta-learning training method.

[0018] In some implementations, the training process of the unified recognition model includes:

[0019] Obtain the initial codes of various dialects;

[0020] Performing meta-learning training on the unified recognition model using training audio in a support set, thereby updating language classification-related parameters of the unified recognition model; the support set includes training audio in a specific dialect;

[0021] Meta-learning training is performed on the unified recognition model using test audio in a query set, so as to update parameters of the unified recognition model that are not related to language classification; the query set includes the test audio in the specific dialect.

[0022] In some implementations, obtaining initial codes for various dialects includes:

[0023] Obtain the initial code of the dialect language family;

[0024] Based on the initial code of the dialect language family, initial codes of various dialect languages ​​belonging to the dialect language family are acquired.

[0025] In some implementations, encoding and decoding the dialect speech to be recognized to obtain speech decoding features; and decoding the speech decoding features using a pre-trained language model to obtain a dialect recognition result, including:

[0026] Inputting the dialect speech to be recognized into a pre-trained dialect speech recognition model, so that the dialect speech recognition model performs encoding and decoding processing on the dialect speech to be recognized to obtain speech decoding features, and decoding the speech decoding features using a pre-trained language model to obtain a dialect recognition result; wherein the dialect text in the decoding network of the language model and the Mandarin text with the same semantics as the dialect text correspond to the same language model probability;

[0027] The dialect speech recognition model is obtained by performing dialect speech recognition training based on dialect speech samples and performing dialect text prediction training based on dialect text pronunciation samples, wherein the dialect text pronunciation samples include Mandarin pronunciation corresponding to the dialect text.

[0028] A second aspect of the present application provides a dialect recognition device, comprising:

[0029] The speech encoding and decoding unit is used to perform encoding and decoding processing on the dialect speech to be recognized to obtain speech decoding features;

[0030] A language decoding unit, configured to decode the speech decoding features using a pre-trained language model to obtain a dialect recognition result;

[0031] The dialect text in the decoding network of the language model and the Mandarin text with the same semantics as the dialect text correspond to the same language model probability.

[0032] A third aspect of the present application provides an electronic device, including a memory and a processor;

[0033] The memory is connected to the processor and is used to store programs;

[0034] The processor is used to implement the above-mentioned dialect recognition method by running the program in the memory.

[0035] In a fourth aspect, the present application proposes a computer program product, comprising computer program instructions, which, when executed by a processor, enable the processor to perform the above-mentioned dialect recognition method.

[0036] The dialect recognition method proposed in this application optimizes the decoding network in the language model of the dialect speech recognition model, so that the dialect text in the decoding network and the Mandarin text with the same semantics as the dialect text correspond to the same language model probability. In this way, the dialect text of the low-resource dialect language shares the language model probability of the text of the high-resource language, thereby improving the decoding accuracy of the low-resource dialect language. When the dialect speech is recognized, the above-mentioned optimized language model is used for decoding, which can improve the dialect speech recognition effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required by the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained on the basis of the provided drawings without creative labor.

[0038] Figure 1 The voice recognition process schematic diagram provided for the embodiments of the present application.

[0039] Figure 2 The flowchart of a dialect recognition method provided for the embodiments of the present application.

[0040] Figure 3 The processing process schematic diagram of the dialect speech to be recognized for the embodiments of the present application.

[0041] Figure 4 The training schematic diagram of a unified recognition model provided for the embodiments of the present application.

[0042] Figure 5 The training process schematic diagram of a dialect speech recognition model provided for the embodiments of the present application.

[0043] Figure 6 The training process schematic diagram of another dialect speech recognition model provided for the embodiments of the present application.

[0044] Figure 7 The training process schematic diagram of still another dialect speech recognition model provided for the embodiments of the present application.

[0045] Figure 8 The structural schematic diagram of a dialect recognition device provided for the embodiments of the present application.

[0046] Figure 9 The structural schematic diagram of an electronic device provided for the embodiments of the present application. DETAILED DESCRIPTION

[0047] In the voice recognition scene, the recognition of dialect speech has always been a difficulty. Because China is vast in territory and populous, and the dialects are rich in dialects, the data volume of different dialects is not balanced, and the data volume of some dialects is small, so it is difficult to obtain enough parallel data for training the voice recognition model, so that the voice recognition model is difficult to achieve ideal recognition effect when recognizing the voice of these dialects.

[0048] Speech recognition technology, also known as Automatic Speech Recognition (ASR), refers to the automatic conversion of human speech into text. Since the 1980s, speech recognition has adopted the basic framework of pattern recognition, consisting of four steps: data preparation, feature extraction, model training, and testing and application. After the model is trained, the test speech undergoes signal processing and feature extraction. The trained acoustic and language models are then used to generate speech recognition results.

[0049] See also Figure 1 As shown in the figure, the speech recognition process of the speech recognition model mainly includes four parts: signal processing (feature extraction), acoustic model, language model, dictionary and decoding. In order to extract features more effectively, it is often necessary to perform preprocessing work such as filtering and framing on the collected sound signal to extract the signal to be analyzed from the original signal; then, the signal processing (feature extraction) work converts the sound signal from the time domain to the frequency domain to provide a suitable feature vector for the acoustic model; the acoustic model then calculates the score of each feature vector on the acoustic feature based on the acoustic feature; and the language model calculates the probability that the sound signal corresponds to a possible phrase sequence based on relevant linguistic theories; finally, the phrase sequence is decoded according to the existing dictionary to obtain the final possible text representation.

[0050] In the aforementioned speech recognition process, the acoustic model is used to link the observed features of the speech signal with the speech modeling units of the sentence, while the language model is related to text processing. The physical meaning of the language model is to reflect the prior probability of word occurrence. For example, the two characters "Hao" and "Good" have the same pronunciation, but "Hao" is less likely to occur than "Good" and generally appears in surnames. For another example, for the pinyin "nihao", the language model decodes the candidate word "Hello" instead of "Nihao". The candidate words are arranged in order of language model score (probability). In addition, the physical meaning of the language model is to reflect whether the word order conforms to linguistic conventions and reflect the semantic information of the words. In addition, the language model in speech recognition is also used to process text sequences. It combines the output of the acoustic model and outputs the text sequence with the highest probability as the speech recognition result.

[0051] When training speech recognition models, the speech modeling units of low-resource dialects are often insufficiently trained. The model is generally unable to recognize phrase combinations that do not appear in the training set, resulting in poor recognition performance for certain low-resource dialects.

[0052] In response to the above technical problems, the embodiments of the present application propose a new speech recognition solution and speech recognition model training solution. The above solution unifies the decoding space of different languages ​​and realizes unified modeling of different languages. In this way, the decoding and modeling of high-resource languages ​​can be used to assist the decoding and modeling of low-resource languages, thereby improving the decoding and modeling effects of low-resource languages ​​and improving recognition accuracy.

[0053] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0054] This application embodiment proposes a dialect recognition method, see Figure 2 As shown, the method includes:

[0055] S101 , performing encoding and decoding processing on the dialect speech to be recognized to obtain speech decoding features.

[0056] The encoding and decoding of the dialect speech to be recognized refers to inputting the dialect speech to be recognized into the dialect speech recognition model, and having the dialect speech recognition model perform encoding and decoding processing on the dialect speech to be recognized, thereby obtaining speech decoding features as a decoded output. The aforementioned inputting of the dialect speech to be recognized into the dialect speech recognition model may mean inputting the audio signal of the dialect speech to be recognized into the dialect speech recognition model, and having the dialect speech recognition model perform feature extraction, encoding, and decoding processing on the dialect speech to be recognized, or may mean inputting the extracted audio features of the dialect speech to be recognized into the dialect speech recognition model, and having the dialect speech recognition model perform encoding and decoding processing on the audio features of the dialect speech to be recognized.

[0057] The dialect speech recognition model encodes and decodes the dialect speech to be recognized, which can be achieved through the encoder and decoder inside the model respectively.

[0058] The speech decoding features obtained through the above processing can be used to combine the language model and dictionary to decode the recognition text of the dialect speech to be recognized.

[0059] S102, decoding the speech decoding features using a pre-trained language model to obtain a dialect recognition result;

[0060] The dialect text in the decoding network of the language model and the Mandarin text with the same semantics as the dialect text correspond to the same language model probability.

[0061] The above-mentioned language model probability refers to the decoding probability set for various texts (including text sequences, text characters) in the decoding network, that is, for a given voice input, when searching for the optimal decoding text path, the selection probability of various decoding text paths. In some embodiments, the voice model probability in the decoding network can also be expressed in the form of a language model score. For a given voice input, when searching for the optimal decoding text path, if the language model probability or language model score of a certain text path is the highest, the text on the text path will be selected as the current decoding text; on the contrary, if the language model probability or language model score of a certain text path is low, the text on the text path will not be selected as the current decoding text.

[0062] The language model probabilities for each text in the decoding network are determined during model training, accumulating extensive text decoding experience and text expression habits. Therefore, for certain low-resource dialects, due to limited available training data and insufficient model training, the language model probabilities for these dialects' corresponding texts in the language model's decoding network are low, resulting in poor recognition performance for these dialects.

[0063] For example, in Mandarin, "catch a dragonfly" is a common expression. Based on extensive Mandarin data training, the language model's decoding network has the highest language model probability score for the various decoded texts corresponding to the speech sound "zhuo qing ting," resulting in the final recognition output of "catch a dragonfly." Specifically, when the speech recognition model recognizes "zhuo qing ting" as "catch," and then decodes "qing ting," it queries the language model's decoding network for the language model probabilities of the various decoded text paths corresponding to "qing ting." For example, if "dragonfly" has a probability of 90, "listen" has a probability of 8, and "green court" has a probability of 2, then because "dragonfly" has the highest language model probability, "qing ting" is decoded as "dragonfly," resulting in the final recognition result of "catch a dragonfly."

[0064] When recognizing dialect speech, the recognition process described above is also followed. However, due to the small amount of sample data for low-resource dialects, when jointly trained with other high-resource languages ​​(since speech models are typically used to recognize speech in various languages, training the models requires sample speech from multiple languages), the language model probability of the decoding text path corresponding to the low-resource dialect will be lowered. Consequently, when the language recognition model performs speech recognition on the low-resource dialect, the dialect speech will be decoded as text with a higher language model probability due to the lower language model probability of the corresponding decoding text path, resulting in decoding errors.

[0065] For example, the word "Malang" exists in a dialect, and "Malang" exists in Mandarin. These two words have the same pronunciation, "ma lang." Because the amount of sample data for dialects is much smaller than that for Mandarin, after training the speech recognition model, the language model probability of "Malang" is lower than that of "Malang" in the language model's decoding network. For example, the language model probability of "Malang" is 10, while the language model probability of "Malang" is 20. In this case, when recognizing the dialect speech "zhuo ma lang," after decoding "zhuo" as "zhuo," the language model probability of "Malang" is lower than that of "Malang" when decoding "ma lang." Therefore, "Malang" is more likely to be recognized as "Malang" instead of "Malang," resulting in an incorrect recognition result of "Zhua Malang."

[0066] Through the analysis of the above problems, the embodiment of the present application optimizes the language model probability of the dialect text in the decoding network of the language model during the training of the dialect speech recognition model. Specifically, the language model probability of the dialect text is set to the language model probability corresponding to the Mandarin text with the same semantics as the dialect text.

[0067] For example, the word "Malang" in the dialect is actually the word "Qingting" in Mandarin. The two are different names for the same creature. That is to say, "Qingting" in Mandarin and "Malang" in the dialect have the same meaning. Therefore, in an embodiment of the present application, when training the dialect speech recognition model, the language model probability of the dialect text "Malang" in the decoding network of the language model is set to the language model probability corresponding to the Mandarin text "Qingting" with the same semantics as the dialect text "Malang". Assuming that the language model probability of "Malang" in the decoding network of the original language model is 10, the language model probability of "Malang" is 20, and the language model probability of "Qingting" is 90, then during the training process, the language model probability of "Malang" in the decoding network of the language model is set to 90, that is, the same as the language model probability of "Qingting", while "Malang" and "Qingting" have different semantics, so its language model probability remains at 20.

[0068] After setting the language model probability as described above, when recognizing the dialect speech "zhuo ma lang", after decoding "zhuo" as "zhua", when decoding "ma lang", since the language model probability of "ma lang" is greater than the language model probability of "ma lang", "ma lang" will be recognized as "ma lang" instead of "ma lang", and finally the correct recognition result of "zhua ma lang" is obtained.

[0069] Based on the above optimization of the decoding network of the language model, after obtaining the speech decoding features of the dialect speech to be recognized, the speech decoding features are decoded by the above optimized language model to obtain a more accurate dialect recognition result.

[0070] From the above introduction, it can be seen that the dialect recognition method proposed in the embodiment of the present application optimizes the decoding network in the language model of the dialect speech recognition model, so that the dialect text in the decoding network and the Mandarin text with the same semantics as the dialect text correspond to the same language model probability. In this way, the dialect text of the low-resource dialect language shares the language model probability of the text of the high-resource language, thereby improving the decoding accuracy of the low-resource dialect language. When the dialect speech is recognized, the above-mentioned optimized language model is used for decoding, which can improve the dialect speech recognition effect.

[0071] In some embodiments, when encoding and decoding the dialect speech to be recognized, see Figure 3 As shown, the shallow encoding layer, encoder 1, first encodes the dialect speech to be recognized to obtain a first encoding result. Based on the first encoding result, a specific type of phoneme label corresponding to the dialect speech to be recognized is generated. In other words, the first encoding result is annotated with the phoneme label of the specific type of phoneme. Then, based on the annotated phoneme label, encoder 2, the deep encoding layer, performs language modeling encoding on the first encoding result to obtain a second encoding result. Finally, the second encoding result is decoded to obtain the speech decoding features of the dialect speech to be recognized.

[0072] In some embodiments, a dialect speech to be recognized is input into a pre-trained speech encoder, so that the speech encoder encodes the dialect speech to be recognized to obtain a first encoding result, generates a specific type of phoneme label corresponding to the dialect speech to be recognized based on the first encoding result, and performs language modeling encoding on the first encoding result based on the phoneme label to obtain a second encoding result;

[0073] The speech encoder is obtained by encoding speech samples of multiple dialects (including Mandarin) through training, and when encoding the speech samples of the multiple dialects during the training process, the speech encoder generates the specific type of phoneme labels for the speech samples.

[0074] In the above processing, the first encoding result is annotated with a phoneme label of a specific type of phoneme. Specifically, the International Phonetic Alphabet (IPA) may be used to annotate the first encoding result with a phoneme label.

[0075] During the training process, when encoding language samples of any dialect or Mandarin, the first encoding results of the speech samples all generate phoneme labels of the same specific type of phonemes mentioned above, that is, the speech is acoustically modeled in the same acoustic space and a unified acoustic decoding space is constructed.

[0076] Generally speaking, due to insufficient acoustic training data for low-resource languages, end-to-end training cannot model robust semantic dependencies. Even though the homophones of some dialects have been fully trained in Chinese, the model may not be able to recognize them due to the statistical sparsity of the language model probability.

[0077] The embodiment of the present application unifies the acoustic modeling space of different dialects, that is, in the process of encoding acoustic modeling of different dialects, all dialect encodings generate unified phoneme labels, such as Figure 3 As shown, after the model's intermediate layers recognize and generate phoneme labels (IPA results), they are then fed into subsequent layers through gating. This effectively uses the shallow layers of the network as acoustic models, learning the unified acoustic knowledge across dialects, while the deeper layers of the network perform language modeling for specific languages. This allows acoustic information from high-resource languages ​​to directly assist in modeling the acoustic information of low-resource languages, reducing the reliance of low-resource languages ​​on acoustic training data. Due to this unified acoustic layer, different languages ​​can share parameters for pronunciation feature extraction, and the deeper layers of the network can also be trained using plain text data to learn language information, further reducing the reliance of low-resource languages ​​on acoustic data.

[0078] In the process of training the above-mentioned speech encoder, the process of generating the first encoding result and the phoneme label, and the process of performing language modeling encoding on the first encoding result based on the phoneme label to obtain the second encoding result, can both be optimized and trained using the CTC loss function.

[0079] In some embodiments, the speech encoder in the dialect speech recognition model shares model parameters with a pre-trained unified recognition model.

[0080] The unified recognition model mentioned above refers to a model for encoding speech in various languages. The model learns the encoding features of speech in various dialects through meta-learning.

[0081] Among them, the meaning of meta-learning is learning to learn. The birth of meta-learning is to enable the model to acquire the ability to "learn to learn" so that it can quickly learn new tasks based on the existing knowledge. In an embodiment of the present application, the above-mentioned unified recognition model is trained by meta-learning, which allows the unified recognition model to quickly learn the speech encoding of the new language under the previously learned speech encoding of the dialect. Therefore, for low-resource dialects, although the amount of data is small, the unified recognition model can still quickly learn the speech encoding of the new dialect based on the speech encoding of other languages ​​that have been learned through meta-learning. Thus, the above-mentioned unified recognition model training method can reduce the dependence on training data for low-resource languages, so that accurate speech encoding can be obtained for any dialect.

[0082] The training of the above unified recognition model is as follows Figure 4 shown.

[0083] During training, the unified recognition model first obtains initial codes for each dialect. The meta-learning-based unified recognition model uses the dialect classification model to obtain these initial codes, that is, by performing dialect classification tasks.

[0084] Moreover, due to the correlation between dialects, for example, different schemes in some regions constitute dialect families, and different dialects within a language family may have many similarities, so when obtaining the encoding of different dialects under the same language family, if the common encoding information of the language family can be utilized, the encoding efficiency of various dialects within the language family will be effectively improved.

[0085] Therefore, in the embodiment of the present application, when obtaining the initial codes of various dialects, the unified recognition model first obtains the initial code of the dialect system, and then obtains the initial codes of various dialects belonging to the dialect system based on the initial code of the dialect system.

[0086] Specifically, during the training process, after the audio features of the dialect speech enter the speech encoder, language information is generated through pooling, the classification layer is used for language prediction, and the classification layer weights are used as language encoding.

[0087] To improve the discrimination between language encodings, this embodiment uses AM-Softmax as the loss function. AM-Softmax reduces the target language score, forcing the model to assign a higher score to the target language. Because the score gap between the target and non-target languages ​​increases, the discrimination between different language encodings is improved.

[0088] The meta-learning training process for the unified recognition model first fixes the model's language-independent parameters. The model is then meta-trained using training audio from a support set, enabling the model to classify the training audio in the support set by language, while simultaneously obtaining language codes. The training process then updates the language-related parameters of the unified recognition model. The support set can include training audio from a specific dialect, such as any dialect, or all dialects.

[0089] Then, the language encoding of the unified recognition model is fixed, and the unified recognition model is meta-learned and trained using the test audio in the query set, updating the parameters of the unified recognition model that are not related to language classification. The query set includes the test audio in the specific dialect. That is, the support set and query set are obtained by partitioning all the audio data collected in a specific dialect. For example, if there are 500 hours of audio data collected in a specific dialect, 400 hours of audio data can be used as the support set, and the remaining 100 hours of audio data can be used as the query set.

[0090] The meta-learning training method can quickly learn the encoding of new dialects based on the encoding information of various dialects already acquired by the unified recognition model. Therefore, the dialect encoding generated based on meta-learning training has good generalization for new languages ​​and can reduce the demand for training data.

[0091] The unified recognition model obtained through the above training shares model parameters with the speech encoder in the dialect speech recognition model, which enables the speech encoder to fully realize data sharing between dialect speeches, improve the encoding effect of low-resource dialects, and thus help improve the recognition effect of the dialect speech recognition model on low-resource dialect languages.

[0092] In other embodiments, the dialect recognition method proposed in the embodiments of the present application can be executed through a dialect speech recognition model, that is, the recognized dialect speech is input into a pre-trained dialect speech recognition model, so that the dialect speech recognition model performs encoding and decoding processing on the dialect speech to be recognized to obtain speech decoding features, and the speech decoding features are decoded by a pre-trained language model to obtain a dialect recognition result; wherein, the dialect text in the decoding network of the language model and the Mandarin text with the same semantics as the dialect text correspond to the same language model probability.

[0093] The dialect speech recognition model is obtained by performing dialect speech recognition training based on dialect speech samples and performing dialect text prediction training based on dialect text pronunciation samples, wherein the dialect text pronunciation samples include Mandarin pronunciation corresponding to the dialect text.

[0094] The above-mentioned dialect text pronunciation samples are pronunciation sequence samples consisting of the Mandarin pronunciation corresponding to the dialect text. For example, the Mandarin pronunciation of the dialect word "Malang" (a roach) "m a3 lang2" can be used as a dialect text pronunciation sample.

[0095] In the above-mentioned dialect speech recognition model training process, performing dialect speech recognition training on the dialect speech recognition model based on the dialect speech sample means inputting the dialect speech audio into the dialect speech recognition model, and having the dialect speech recognition model recognize the input dialect speech to obtain a text recognition result.

[0096] Dialect text prediction training of a dialect speech recognition model based on dialect text pronunciation samples involves inputting the dialect text pronunciation samples into the dialect speech recognition model, and having the dialect speech recognition model perform autoregressive text prediction based on the input dialect text pronunciation samples, such as predicting the next recognized character based on the recognized text, or enabling the dialect speech recognition model to perform a character mask prediction task based on the dialect text pronunciation samples.

[0097] The above-mentioned dialect speech samples can be obtained by collecting dialect speech data of a specific language. The collected dialect speech data is annotated with corresponding text labels, which can be used to perform dialect speech recognition training on the dialect speech recognition model.

[0098] The above-mentioned dialect text pronunciation samples can be obtained by first generating the dialect text and then generating the Mandarin pronunciation corresponding to the dialect text.

[0099] In one embodiment of the present application, an implementation method for obtaining a dialect text pronunciation sample and a speech recognition tag corresponding to the dialect speech sample is disclosed.

[0100] Specifically, in order to obtain sufficient dialect text, in this embodiment, Mandarin text is used to generate dialect text, that is, the Mandarin text is converted into dialect text.

[0101] For example, Mandarin text can be input into a dialect translation engine or a large language model. Based on the translation capabilities of the dialect translation engine or the language conversion capabilities of the large language model, the Mandarin text can be converted into the dialect text. For example, the Mandarin word "dragonfly" can be converted into the dialect word "antang" (a roach), and the Mandarin word "handsome boy and beautiful girl" can be converted into the dialect word "handsome boy and beautiful girl."

[0102] Through the above-mentioned text conversion method, we can take advantage of the quantity advantage of Mandarin text to generate a large amount of dialect text, thereby providing a data basis for the training of dialect speech recognition models.

[0103] After obtaining the dialect text in the above manner, the embodiment of the application further generates the dialect pronunciation corresponding to each dialect text based on the dialect text to dialect pronunciation mapping table, i.e., determines the dialect pronunciation sequence corresponding to each dialect text.

[0104] Then, the dialect pronunciation is converted into Mandarin pronunciation. For example, the conversion of the dialect pronunciation into the Mandarin pronunciation can be achieved by manual conversion or with the help of a pronunciation prediction model, or the conversion of the dialect pronunciation into the Mandarin pronunciation can be achieved by manual conversion combined with the pronunciation prediction model.

[0105] Finally, based on the conversion process described above, a mapping table of dialect text to Mandarin pronunciation can be arranged. In the mapping table, the Mandarin pronunciation corresponding to each dialect text is recorded, i.e., each dialect text corresponds to a Mandarin pronunciation sequence.

[0106] Based on the mapping table described above, a dialect text pronunciation sample corresponding to the dialect text can be generated. That is, for any dialect text, the Mandarin pronunciation sequence corresponding to the dialect text can be obtained by querying the mapping table described above, and the Mandarin pronunciation sequence can be used as the dialect text pronunciation sample corresponding to the dialect text.

[0107] In addition, for dialect speech, it can be used to construct a dialect speech sample for speech recognition training of a dialect speech recognition model. For a small dialect language, due to the small amount of speech samples, in order to better use the collected dialect speech for training of the dialect speech recognition model and improve the training effect, the embodiment of the application trains a phoneme forced alignment model for generating speech recognition labels corresponding to the dialect speech sample.

[0108] Specifically, the embodiment of the application trains a kaldi model as a phoneme forced alignment model. Dialect speech samples and corresponding speech labels are collected in advance. The speech labels include text labels and phoneme labels, and in some cases, there can be only phoneme labels. In addition, the mapping table of dialect text to Mandarin pronunciation obtained by the processing method introduced in the above embodiment is also used for model training.

[0109] During the training of the kaldi model, the dialect speech sample and speech label parallel data, and the mapping table described above are used as training input, and the kaldi model performs a speech recognition task. During the process of the kaldi model performing the speech recognition task, the speech and phoneme alignment processing is automatically performed. Therefore, the kaldi model is trained for dialect speech phoneme labeling in the embodiment of the application, and the trained kaldi model can be used to perform the phoneme alignment and labeling tasks of the speech, i.e., to obtain a phoneme forced alignment model.

[0110] Further, the dialect speech sample without annotated phonemes is input into the phoneme forced alignment model, so as to annotate the phonemes of the dialect speech sample, and obtain the phoneme labels of the dialect speech sample. Through the above processing, the phoneme labels of the collected dialect speech samples can be annotated, and further, the dialect speech samples with annotated phoneme labels can be used for training the dialect speech recognition model, so as to increase the model training sample amount, and further improve the model training effect.

[0111] Through the processing of the above embodiments, the dialect text sample can be generated based on the Mandarin text, and further, the dialect text pronunciation sample can be obtained through the mapping table of the dialect text to the Mandarin pronunciation. Further, the phoneme forced alignment model constructed through the above manner can obtain the dialect speech sample carrying phoneme labels. The above dialect text pronunciation sample and the dialect speech sample carrying phoneme labels can be used as the training data for training the dialect speech recognition model, and the embodiments of the present application use them for training the dialect speech recognition model, so as to increase the model training data amount, and further improve the model training effect and the recognition performance of the model to the dialect speech.

[0112] In some other embodiments, when the dialect speech recognition model is trained by using the above dialect text pronunciation sample and the dialect speech sample carrying phoneme labels, the dialect speech recognition model is trained based on the dialect speech sample, and the dialect text prediction model is trained based on the dialect text pronunciation sample.

[0113] Specifically, Figure 5 A process diagram for training the dialect speech recognition model is shown. Referring to FIG. 6, when the dialect speech recognition model is trained, the dialect speech sample and the dialect text pronunciation sample are input into the dialect speech recognition model, so as to make the dialect speech recognition model recognize the dialect speech sample, and predict the dialect text of the dialect text pronunciation sample, and obtain the speech recognition result and the text prediction result. Figure 5 As shown in FIG. 6, when the dialect speech recognition model is trained, the dialect speech sample and the dialect text pronunciation sample are input into the dialect speech recognition model, so as to make the dialect speech recognition model recognize the dialect speech sample, and predict the dialect text of the dialect text pronunciation sample, and obtain the speech recognition result and the text prediction result.

[0114] When the dialect speech recognition model is trained, the features of the dialect speech sample and the dialect text pronunciation sample can be extracted respectively, and then the dialect speech sample features and the dialect text pronunciation sample features are input into the dialect speech recognition model. For example, the dialect speech sample features and the dialect text pronunciation sample features can be simultaneously input into the dialect speech recognition model, so as to make the dialect speech recognition model perform the recognition task of the dialect speech sample and the dialect text prediction task of the dialect text pronunciation sample; or, the dialect speech sample features and the dialect text pronunciation sample features can be sequentially input into the dialect speech recognition model, so as to make the dialect speech recognition model sequentially perform the recognition task of the dialect speech sample and the dialect text prediction task of the dialect text pronunciation sample.

[0115] By enabling the dialect speech recognition model to perform the dialect speech recognition task and the text prediction task, speech recognition results and text prediction results can be obtained respectively.

[0116] Then, the speech recognition result corresponding to the dialect speech sample is compared with the speech recognition label corresponding to the dialect speech sample, and the dialect recognition loss function is calculated; and the text prediction result is compared with the dialect text pronunciation sample, specifically, the text characters in the text prediction result are compared with the text characters corresponding to the dialect text pronunciation sample, and the text prediction loss function is calculated.

[0117] Finally, with the goal of reducing dialect recognition loss and text prediction loss, the parameters of the dialect speech recognition model are optimized.

[0118] Repeat the above training process, allowing the dialect speech recognition model to continuously perform dialect speech recognition and dialect text prediction tasks. During this training process, the dialect speech recognition model can simultaneously learn dialect text pronunciation information while performing dialect speech recognition. This allows the dialect speech recognition model to more fully learn and explore the potential relationships between dialect speech, its pronunciation, and text. This can improve the training effect of the dialect speech recognition model and enhance its dialect speech recognition performance.

[0119] In other embodiments, the speech recognition labels of the dialect speech samples used to train the dialect speech recognition model include word-level labels and phoneme-level labels, wherein the word-level labels include the text characters corresponding to the dialect speech samples, and the phoneme-level labels include the phonemes in the pronunciation of the dialect speech samples.

[0120] Based on the above speech recognition tags, see Figure 6 As shown, during the training of the dialect speech recognition model, after inputting the dialect speech sample features into the dialect speech recognition model to obtain the speech recognition results, when calculating the dialect recognition loss, on the one hand, the word-level dialect recognition loss is determined based on the speech recognition results and the word-level labels in the speech recognition labels, and on the other hand, the phoneme-level dialect recognition loss is determined based on the speech recognition results and the phoneme-level labels in the speech recognition labels. The above-mentioned phoneme-level labels are obtained by phoneme-annotating the dialect language samples using the phoneme-forced alignment model in the above embodiment.

[0121] The character-level dialect recognition loss is determined based on the speech recognition results and the character-level labels in the speech recognition labels. Specifically, the characters in the speech recognition results output by the dialect speech recognition model are compared with the character labels in the character-level labels to determine whether the recognized characters and character labels at the same position are consistent, and then the character recognition loss function is calculated. The above-mentioned character recognition loss function can be obtained by calculating the loss function corresponding to each recognized character and then calculating the sum of the loss functions corresponding to each character, or it can be obtained by synchronously comparing the recognized characters and character labels at each character position and calculating the loss function.

[0122] In some embodiments, the aforementioned character-level dialect recognition loss and phoneme-level dialect recognition loss can be calculated simultaneously and used to optimize the dialect speech recognition model. In some cases, only one of the character-level dialect recognition loss and phoneme-level dialect recognition loss can be calculated. In this case, the calculated character-level dialect recognition loss or phoneme-level dialect recognition loss can be used to optimize the dialect speech recognition model.

[0123] It can be understood that the dialect speech recognition model training scheme proposed in the embodiment of the present application sets a variety of speech recognition labels, so that different types of dialect recognition losses can be calculated for the same dialect speech sample, thereby expanding the scope of application of the model training data. In particular, the embodiment of the present application can annotate the phoneme labels of the dialect speech samples through the phoneme forced alignment model, and then when training the dialect speech recognition model, the dialect speech recognition model can be optimized by calculating the phoneme-level dialect recognition loss. This training scheme makes up for the disadvantage that the dialect speech samples cannot be used for model training due to the inability to obtain parallel data of the dialect speech samples (dialect speech samples and corresponding text labels), thereby advantageously expanding the amount of model training data.

[0124] In other embodiments, the dialect speech recognition model performs dialect text prediction on the dialect text pronunciation sample. Specifically, the dialect speech recognition model performs autoregressive text prediction and / or mask prediction based on the input dialect text pronunciation sample to obtain an autoregressive prediction result and / or a mask prediction result.

[0125] Among them, the dialect speech recognition model performs autoregressive text prediction based on the input dialect text pronunciation samples, which means that the dialect speech recognition model predicts the text characters corresponding to the dialect text pronunciation samples in sequence based on the input dialect text pronunciation samples, such as predicting the next character based on the predicted character sequence.

[0126] The dialect speech recognition model performs mask prediction based on the input dialect text pronunciation sample, which means masking the character speech at a certain position in the dialect text pronunciation sample, and then making the dialect speech recognition model predict the masked character based on the text pronunciation at other positions.

[0127] Based on the above dialect text prediction task, see Figure 6 As shown, when calculating the text prediction loss based on the text prediction results output by the dialect speech recognition model, the first text prediction loss is determined based on the autoregressive prediction results and the dialect text pronunciation samples, and / or the second text prediction loss is determined based on the mask prediction results and the dialect text pronunciation samples.

[0128] Among them, calculating and determining the first text prediction loss based on the autoregressive prediction result and the dialect text speech sample refers to comparing the autoregressive prediction result with the dialect text corresponding to the dialect text pronunciation sample to calculate the text prediction loss.

[0129] Determining the second text prediction loss based on the mask prediction result and the dialect text speech sample refers to comparing the mask prediction result with the dialect text corresponding to the dialect text pronunciation sample to calculate the text prediction loss.

[0130] It is understood that the model training scheme of the embodiments of the present application enables the dialect speech recognition model to perform different text prediction tasks based on the input dialect text pronunciation samples, thereby calculating different text prediction losses for parameter optimization of the dialect speech recognition model. This model training method increases the diversity of model training tasks, allowing the dialect speech recognition model to learn and explore the relationship between dialect text and dialect pronunciation in different ways, which is conducive to improving model training results.

[0131] In other embodiments, Figure 7 As shown, the dialect speech samples and dialect text pronunciation samples input when training the dialect speech prediction model can be parallel sample data corresponding to each other, that is, the input dialect speech samples and dialect text pronunciation samples correspond to the same dialect text. The difference is that the dialect speech samples are the speech corresponding to the dialect text, and the dialect text pronunciation samples are the pronunciation sequences corresponding to the dialect text.

[0132] Based on the above-mentioned training data characteristics, when training the dialect speech recognition model, the dialect speech features obtained by the dialect speech recognition model from processing the dialect speech samples are obtained, and the dialect text features obtained by the dialect speech recognition model from processing the dialect text pronunciation samples are obtained.

[0133] Then, the dialect speech features and dialect text features obtained through the dialect speech recognition model are compared to determine the alignment loss between the dialect speech features and the dialect text features.

[0134] The aforementioned alignment loss can be used to measure whether the dialect speech features and dialect text features processed by the dialect speech recognition model are consistent in feature content and aligned in time. In theory, since the dialect speech samples and dialect text pronunciation samples input to the dialect speech recognition model are parallel data, the dialect speech features processed by the dialect speech recognition model for the dialect speech samples and the dialect text features processed for the dialect text pronunciation samples should be aligned. If the two are not aligned, it indicates that there is room for improvement in the dialect speech recognition model's feature extraction process for the two modal data.

[0135] Based on the above ideas, the model training scheme proposed in the embodiment of this application, during the training of the dialect speech recognition model, if the dialect speech samples and dialect text pronunciation samples input to the dialect speech recognition model are corresponding parallel sample data, then in addition to calculating the dialect recognition loss and text prediction loss, the alignment loss between the dialect speech features and dialect text features processed by the dialect speech recognition model is also calculated. Furthermore, when optimizing the dialect speech recognition model, the above-mentioned alignment loss, dialect recognition loss, and text prediction loss are jointly used to optimize the dialect speech recognition model. That is, the parameters of the dialect speech recognition model are optimized with the goal of reducing the above-mentioned alignment loss, dialect recognition loss, and text prediction loss.

[0136] It can be understood that this model training scheme applies a variety of different loss functions, which enables the dialect speech recognition model to perceive the relationship between dialect speech, dialect text and dialect pronunciation through multiple dimensions during the training process, thereby improving the model training effect and accelerating the model training efficiency.

[0137] Corresponding to the above-mentioned dialect recognition method, the present application embodiment also provides a dialect recognition device, see Figure 8 As shown, the device includes:

[0138] The speech encoding and decoding unit 100 is used to perform encoding and decoding processing on the dialect speech to be recognized to obtain speech decoding features;

[0139] A language decoding unit 110 is configured to decode the speech decoding features using a pre-trained language model to obtain a dialect recognition result;

[0140] The dialect text in the decoding network of the language model and the Mandarin text with the same semantics as the dialect text correspond to the same language model probability.

[0141] In some implementations, the speech codec unit 100 performs codec processing on the dialect speech to be recognized to obtain speech decoding features, including:

[0142] Encoding the dialect speech to be recognized to obtain a first encoding result;

[0143] generating a specific type of phoneme label corresponding to the dialect speech to be recognized based on the first encoding result;

[0144] Performing language modeling encoding on the first encoding result based on the phoneme label to obtain a second encoding result;

[0145] The second encoding result is decoded to obtain a speech decoding feature.

[0146] In some implementations, the speech encoding and decoding unit 100 encodes the dialect speech to be recognized to obtain a first encoding result; generates a specific type of phoneme label corresponding to the dialect speech to be recognized based on the first encoding result; and performs language modeling encoding on the first encoding result based on the phoneme label to obtain a second encoding result, including:

[0147] Inputting the dialect speech to be recognized into a pre-trained speech encoder so that the speech encoder encodes the dialect speech to be recognized to obtain a first encoding result, generating a specific type of phoneme label corresponding to the dialect speech to be recognized based on the first encoding result, and performing language modeling encoding on the first encoding result based on the phoneme label to obtain a second encoding result;

[0148] The speech encoder is obtained by encoding speech samples of multiple dialects through training, and the speech encoder generates the specific type of phoneme labels for the speech samples when encoding the speech samples of the multiple dialects during the training process.

[0149] In some implementations, the speech encoder shares model parameters with a pre-trained unified recognition model;

[0150] The unified recognition model learns the encoding features of various dialects through a meta-learning training method.

[0151] In some implementations, the training process of the unified recognition model includes:

[0152] Obtain the initial codes of various dialects;

[0153] Performing meta-learning training on the unified recognition model using training audio in a support set, thereby updating language classification-related parameters of the unified recognition model; the support set includes training audio in a specific dialect;

[0154] Meta-learning training is performed on the unified recognition model using test audio in a query set, so as to update parameters of the unified recognition model that are not related to language classification; the query set includes the test audio in the specific dialect.

[0155] In some implementations, obtaining initial codes for various dialects includes:

[0156] Obtain the initial code of the dialect language family;

[0157] Based on the initial code of the dialect language family, initial codes of various dialect languages ​​belonging to the dialect language family are acquired.

[0158] In some implementations, the speech encoding and decoding unit 100 and the language decoding unit 110 are provided in a dialect speech recognition model. The speech encoding and decoding unit 100 performs encoding and decoding processing on the dialect speech to be recognized to obtain speech decoding features; and the language decoding unit 110 decodes the speech decoding features using a pre-trained language model to obtain a dialect recognition result, including:

[0159] Inputting the dialect speech to be recognized into a pre-trained dialect speech recognition model, so that the dialect speech recognition model performs encoding and decoding processing on the dialect speech to be recognized to obtain speech decoding features, and decoding the speech decoding features using a pre-trained language model to obtain a dialect recognition result; wherein the dialect text in the decoding network of the language model and the Mandarin text with the same semantics as the dialect text correspond to the same language model probability;

[0160] The dialect speech recognition model is obtained by performing dialect speech recognition training based on dialect speech samples and performing dialect text prediction training based on dialect text pronunciation samples, wherein the dialect text pronunciation samples include Mandarin pronunciation corresponding to the dialect text.

[0161] The dialect recognition device provided in this embodiment is based on the same concept as the dialect recognition method provided in the aforementioned embodiments of this application. It can execute the dialect recognition method provided in any of the aforementioned embodiments of this application and has the corresponding functional modules and beneficial effects. For technical details not fully described in this embodiment, please refer to the specific processing content of the dialect recognition method provided in the aforementioned embodiments of this application and will not be repeated here.

[0162] The functions implemented by the above-mentioned speech encoding and decoding unit 100 and language decoding unit 110 can be implemented by the same or different processors respectively, and the embodiment of the present application is not limited thereto.

[0163] It should be understood that the units in the above devices can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and the memory stores instructions. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device. The processor can be a general-purpose processor, such as a CPU or a microprocessor, and the memory can be a memory within the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits. The functions of some or all units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units can be realized by designing the logical relationships between the components within the circuit. For another example, in another implementation, the hardware circuit can be implemented by a PLD. For example, an FPGA can include a large number of logic gate circuits. The connection relationships between the logic gate circuits are configured through a configuration file to realize the functions of some or all of the above units. All units of the above devices can be implemented entirely in the form of a processor calling software, or entirely in the form of hardware circuits, or partially in the form of a processor calling software, with the remaining parts implemented in the form of hardware circuits.

[0164] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and execute instructions, such as a CPU, a microprocessor, a GPU, or a DSP. In another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit may be fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the configuration of the hardware circuit can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0165] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0166] In addition, the various units in the above apparatus may be fully or partially integrated together, or may be implemented independently. In one implementation, these units are integrated together and implemented in the form of a system-on-chip (SOC). The SOC may include at least one processor for implementing any of the above methods or implementing the functions of the various units of the apparatus. The at least one processor may be of different types, such as a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0167] Another embodiment of the present application further provides an electronic device, see Figure 9 As shown, the device includes:

[0168] Memory 200 and processor 210;

[0169] The memory 200 is connected to the processor 210 and is used to store programs;

[0170] The processor 210 is configured to implement the dialect recognition method disclosed in any of the above embodiments by running the program stored in the memory 200 .

[0171] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0172] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are interconnected via a bus.

[0173] A bus may include a pathway that transfers information between components of a computer system.

[0174] Processor 210 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, or the like, or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present invention. Alternatively, it can be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware components.

[0175] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0176] The memory 200 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include program code, which includes computer operating instructions. More specifically, the memory 200 may include read-only memory (ROM), other types of static storage devices that can store static information and instructions, random access memory (RAM), other types of dynamic storage devices that can store information and instructions, disk storage, flash memory, etc.

[0177] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0178] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speakers, etc.

[0179] The communication interface 220 may include any device such as a transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0180] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement the various steps of any dialect recognition method provided in the above embodiments of the present application.

[0181] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the dialect recognition method described in any of the above-mentioned embodiments of this specification.

[0182] The computer program product may be written in any combination of one or more programming languages ​​to implement the program code for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0183] In addition, an embodiment of the present application may also be a storage medium on which a computer program is stored, and the computer program is used by a processor to execute the steps of the dialect recognition method described in any of the above embodiments of this specification.

[0184] For the sake of simplicity, the aforementioned method embodiments are described as a series of action combinations. However, those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0185] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similarities between the various embodiments can be referred to in conjunction with each other. For device embodiments, since they are generally similar to method embodiments, their description is relatively simple, and for relevant details, reference can be made to the description of the method embodiments.

[0186] The steps in the methods of each embodiment of the present application can be adjusted in sequence, merged, and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0187] The modules and sub-modules in the devices and terminals of the various embodiments of the present application can be merged, divided, and deleted according to actual needs.

[0188] In the several embodiments provided in this application, it should be understood that the disclosed terminals, devices, and methods can be implemented in other ways. For example, the terminal embodiments described above are merely illustrative. For example, the division of modules or submodules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or module, which can be electrical, mechanical or other forms.

[0189] The modules or submodules described as separate components may or may not be physically separate, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules may be selected to achieve the purpose of this embodiment according to actual needs.

[0190] In addition, each functional module or submodule in each embodiment of the present application may be integrated into a processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into a single module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or software functional modules or submodules.

[0191] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0192] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, software units executed by a processor, or a combination of the two. The software units may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0193] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0194] The above description of the disclosed embodiments will enable those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is to be construed in the widest manner consistent with the principles and novel features disclosed herein.

Claims

1. A dialect recognition method, characterized in that: include: Perform encoding and decoding on the dialect speech to be recognized to obtain speech decoding features; Decoding the speech decoding features using a pre-trained language model to obtain a dialect recognition result; The dialect text in the decoding network of the language model and the Mandarin text with the same semantics as the dialect text correspond to the same language model probability.

2. The method according to claim 1, characterized in that Perform encoding and decoding on the dialect speech to be recognized to obtain speech decoding features, including: Encoding the dialect speech to be recognized to obtain a first encoding result; Generating a specific type of phoneme label corresponding to the dialect speech to be recognized based on the first encoding result; wherein the specific type of phoneme label includes a phoneme label marked according to the International Phonetic Alphabet; Performing language modeling encoding on the first encoding result based on the phoneme label to obtain a second encoding result; The second encoding result is decoded to obtain a speech decoding feature.

3. The method according to claim 2, characterized in that Encoding the dialect speech to be recognized to obtain a first encoding result; generating a specific type of phoneme label corresponding to the dialect speech to be recognized based on the first encoding result; Performing language modeling encoding on the first encoding result based on the phoneme label to obtain a second encoding result, including: Inputting the dialect speech to be recognized into a pre-trained speech encoder so that the speech encoder encodes the dialect speech to be recognized to obtain a first encoding result, generating a specific type of phoneme label corresponding to the dialect speech to be recognized based on the first encoding result, and performing language modeling encoding on the first encoding result based on the phoneme label to obtain a second encoding result; The speech encoder is obtained by encoding speech samples of multiple dialects through training, and the speech encoder generates the specific type of phoneme labels for the speech samples when encoding the speech samples of the multiple dialects during the training process.

4. The method according to claim 3, characterized in that The speech encoder shares model parameters with a pre-trained unified recognition model; The unified recognition model learns the encoding features of various dialects through a meta-learning training method.

5. The method according to claim 4, characterized in that The training process of the unified recognition model includes: Obtain the initial codes of various dialects; Performing meta-learning training on the unified recognition model using training audio in a support set to update language classification-related parameters of the unified recognition model; the support set includes training audio in a specific dialect; the specific dialect includes any dialect or all dialects; Meta-learning training is performed on the unified recognition model using test audio in a query set, so as to update parameters of the unified recognition model that are not related to language classification; the query set includes the test audio in the specific dialect.

6. The method according to claim 5, characterized in that Get the initial encoding of various dialects, including: Obtain the initial code of the dialect language family; Based on the initial code of the dialect language family, initial codes of various dialect languages ​​belonging to the dialect language family are acquired.

7. The method according to claim 1, characterized in that Perform encoding and decoding on the dialect speech to be recognized to obtain speech decoding features; The speech decoding features are decoded using a pre-trained language model to obtain a dialect recognition result, including: Inputting the dialect speech to be recognized into a pre-trained dialect speech recognition model, so that the dialect speech recognition model performs encoding and decoding processing on the dialect speech to be recognized to obtain speech decoding features, and decoding the speech decoding features using a pre-trained language model to obtain a dialect recognition result; wherein the dialect text in the decoding network of the language model and the Mandarin text with the same semantics as the dialect text correspond to the same language model probability; The dialect speech recognition model is obtained by performing dialect speech recognition training based on dialect speech samples and performing dialect text prediction training based on dialect text pronunciation samples, wherein the dialect text pronunciation samples include Mandarin pronunciation corresponding to the dialect text.

8. A dialect recognition device, characterized in that: include: The speech encoding and decoding unit is used to perform encoding and decoding processing on the dialect speech to be recognized to obtain speech decoding features; A language decoding unit, configured to decode the speech decoding features using a pre-trained language model to obtain a dialect recognition result; The dialect text in the decoding network of the language model and the Mandarin text with the same semantics as the dialect text correspond to the same language model probability.

9. An electronic device, characterized in that: including memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the dialect recognition method according to any one of claims 1 to 7 by running the program in the memory.

10. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, enable the processor to perform the dialect identification method according to any one of claims 1 to 7.

Images