Dialect recognition method, device, equipment and program product

Abstract

The application provides a dialect recognition method, device, equipment and program product. The method comprises the following steps: inputting dialect speech into a dialect speech recognition model to obtain a dialect recognition result; wherein the dialect speech recognition model is obtained based on dialect speech sample dialect speech recognition training and dialect text prediction training based on dialect text pronunciation sample, and the dialect text pronunciation sample comprises Mandarin pronunciation corresponding to dialect text. The above scheme trains the dialect speech recognition model based on a new model training scheme and applies the dialect speech recognition model to dialect speech recognition, so that the dialect speech recognition effect can be improved.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a dialect recognition method, apparatus, device and program product. Background Art

[0002] Dialect speech recognition has always been a challenge in speech recognition tasks. This is because some dialects have a small sample size, making it difficult to obtain sufficient parallel data to train speech recognition models. As a result, the recognition accuracy of speech recognition models in these dialects is low.

[0003] Therefore, it is necessary to study effective technical solutions to improve the effect of dialect speech recognition. Summary of the Invention

[0004] Based on the above technical status, this application proposes a dialect recognition method, device, equipment and program product. The above scheme trains a dialect speech recognition model based on a new model training scheme and uses it for dialect speech recognition, which can improve the dialect speech recognition effect.

[0005] In a first aspect, the present application proposes a dialect recognition method, comprising:

[0006] Input the dialect speech into the dialect speech recognition model to obtain the dialect recognition result;

[0007] The dialect speech recognition model is obtained by performing dialect speech recognition training based on dialect speech samples and performing dialect text prediction training based on dialect text pronunciation samples, wherein the dialect text pronunciation samples include the Mandarin pronunciation corresponding to the dialect text.

[0008] In some implementations, the training process of the dialect speech recognition model includes:

[0009] Inputting the dialect speech sample and the dialect text pronunciation sample into the dialect speech recognition model, so that the dialect speech recognition model recognizes the dialect speech sample and performs dialect text prediction on the dialect text pronunciation sample, thereby obtaining a speech recognition result and a text prediction result;

[0010] Determining a dialect recognition loss based on the speech recognition result and the speech recognition label corresponding to the dialect speech sample, and determining a text prediction loss based on the text prediction result and the dialect text pronunciation sample;

[0011] The dialect speech recognition model is optimized based on the dialect recognition loss and the text prediction loss.

[0012] In some implementations, the speech recognition labels include word-level labels and / or phoneme-level labels;

[0013] The determining of the dialect recognition loss according to the speech recognition result and the speech recognition label corresponding to the dialect speech sample includes:

[0014] A word-level dialect recognition loss is determined based on the speech recognition result and the word-level label, and / or a phoneme-level dialect recognition loss is determined based on the speech recognition result and the phoneme-level label.

[0015] In some implementations, the dialect speech recognition model performs dialect text prediction on the dialect text pronunciation sample, including:

[0016] The dialect speech recognition model performs autoregressive text prediction and / or mask prediction based on the dialect text pronunciation sample to obtain an autoregressive prediction result and / or a mask prediction result;

[0017] Determining a text prediction loss according to the text prediction result and the dialect text pronunciation sample includes:

[0018] A first text prediction loss is determined based on the autoregressive prediction result and the dialect text pronunciation sample, and / or a second text prediction loss is determined based on the mask prediction result and the dialect text pronunciation sample.

[0019] In some implementations, the dialect speech sample and the dialect text pronunciation sample are parallel sample data corresponding to each other;

[0020] The method further comprises:

[0021] Acquire dialect speech features obtained by processing the dialect speech sample by the dialect speech recognition model, and obtain dialect text features obtained by processing the dialect text pronunciation sample by the dialect speech recognition model;

[0022] determining an alignment loss between the dialect speech feature and the dialect text feature;

[0023] Optimizing the dialect speech recognition model based on the dialect recognition loss and the text prediction loss includes:

[0024] The dialect speech recognition model is optimized based on the alignment loss, the dialect recognition loss, and the text prediction loss.

[0025] In some implementations, the method further includes:

[0026] Generate dialect text using Mandarin text, and generate a mapping table from dialect text to Mandarin pronunciation;

[0027] Generating a dialect text pronunciation sample corresponding to the dialect text based on the mapping table;

[0028] The phoneme labels corresponding to the dialect speech samples are generated by a phoneme forced alignment model, wherein the phoneme forced alignment model is obtained by performing dialect speech phoneme annotation training based on the dialect speech sample parallel data and the mapping table.

[0029] In some implementations, generating dialect text using Mandarin text includes:

[0030] The Mandarin text is input into the dialect translation engine and / or the large language model to obtain the dialect text corresponding to the input Mandarin text.

[0031] A second aspect of the present application provides a dialect recognition device, comprising:

[0032] The dialect recognition unit is used to input the dialect speech into the dialect speech recognition model to obtain the dialect recognition result;

[0033] The dialect speech recognition model is obtained by performing dialect speech recognition training based on dialect speech samples and performing dialect text prediction training based on dialect text pronunciation samples, wherein the dialect text pronunciation samples include the Mandarin pronunciation corresponding to the dialect text.

[0034] In some implementations, the training process of the dialect speech recognition model includes:

[0035] Inputting the dialect speech sample and the dialect text pronunciation sample into the dialect speech recognition model, so that the dialect speech recognition model recognizes the dialect speech sample and performs dialect text prediction on the dialect text pronunciation sample, thereby obtaining a speech recognition result and a text prediction result;

[0036] Determining a dialect recognition loss based on the speech recognition result and the speech recognition label corresponding to the dialect speech sample, and determining a text prediction loss based on the text prediction result and the dialect text pronunciation sample;

[0037] The dialect speech recognition model is optimized based on the dialect recognition loss and the text prediction loss.

[0038] In some implementations, the speech recognition labels include word-level labels and / or phoneme-level labels;

[0039] The determining of the dialect recognition loss according to the speech recognition result and the speech recognition label corresponding to the dialect speech sample includes:

[0040] A word-level dialect recognition loss is determined based on the speech recognition result and the word-level label, and / or a phoneme-level dialect recognition loss is determined based on the speech recognition result and the phoneme-level label.

[0041] In some implementations, the dialect speech recognition model performs dialect text prediction on the dialect text pronunciation sample, including:

[0042] The dialect speech recognition model performs autoregressive text prediction and / or mask prediction based on the dialect text pronunciation sample to obtain an autoregressive prediction result and / or a mask prediction result;

[0043] Determining a text prediction loss according to the text prediction result and the dialect text pronunciation sample includes:

[0044] A first text prediction loss is determined based on the autoregressive prediction result and the dialect text pronunciation sample, and / or a second text prediction loss is determined based on the mask prediction result and the dialect text pronunciation sample.

[0045] In some implementations, the dialect speech sample and the dialect text pronunciation sample are parallel sample data corresponding to each other;

[0046] The method further comprises:

[0047] Acquire dialect speech features obtained by processing the dialect speech sample by the dialect speech recognition model, and obtain dialect text features obtained by processing the dialect text pronunciation sample by the dialect speech recognition model;

[0048] determining an alignment loss between the dialect speech feature and the dialect text feature;

[0049] Optimizing the dialect speech recognition model based on the dialect recognition loss and the text prediction loss includes:

[0050] The dialect speech recognition model is optimized based on the alignment loss, the dialect recognition loss, and the text prediction loss.

[0051] In some implementations, the training process of the dialect speech recognition model further includes:

[0052] Generate dialect text using Mandarin text, and generate a mapping table from dialect text to Mandarin pronunciation;

[0053] Generating a dialect text pronunciation sample corresponding to the dialect text based on the mapping table;

[0054] The phoneme labels corresponding to the dialect speech samples are generated by a phoneme forced alignment model, wherein the phoneme forced alignment model is obtained by performing dialect speech phoneme annotation training based on the dialect speech sample parallel data and the mapping table.

[0055] In some implementations, generating dialect text using Mandarin text includes:

[0056] The Mandarin text is input into the dialect translation engine and / or the large language model to obtain the dialect text corresponding to the input Mandarin text.

[0057] A third aspect of the present application provides an electronic device, including a memory and a processor;

[0058] The memory is connected to the processor and is used to store programs;

[0059] The processor is used to implement the above-mentioned dialect recognition method by running the program in the memory.

[0060] In a fourth aspect, the present application proposes a computer program product, comprising computer program instructions, which, when executed by a processor, enable the processor to perform the above-mentioned dialect recognition method.

[0061] The dialect speech recognition model used in the dialect recognition method proposed in this application is obtained by performing dialect speech recognition training based on dialect speech samples and performing dialect text prediction training based on dialect text pronunciation samples, wherein the dialect text pronunciation samples include the Mandarin pronunciation corresponding to the dialect text. Through the above-mentioned training method, on the one hand, dialect text pronunciation samples constructed by the Mandarin pronunciation corresponding to the dialect text are added to the conventional dialect speech samples, thereby increasing the amount of training sample data. On the other hand, during the dialect speech recognition model training process, the model is made to perform two tasks, dialect speech recognition and dialect text prediction, so that the model can fully explore and learn the logical relationship between dialect speech, pronunciation, and text, thereby improving the model training effect and improving the model training efficiency. Through the above-mentioned training method, the model's recognition effect on dialect speech can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0063] Figure 1 A schematic diagram of the training process of a dialect speech recognition model provided in an embodiment of the present application.

[0064] Figure 2 A schematic diagram of the training process of another dialect speech recognition model provided in an embodiment of the present application.

[0065] Figure 3 A schematic diagram of the training process of another dialect speech recognition model provided in an embodiment of the present application.

[0066] Figure 4 A flowchart of a dialect recognition method provided in an embodiment of the present application.

[0067] Figure 5 A schematic diagram of the structure of a dialect recognition device provided in an embodiment of the present application.

[0068] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] In speech recognition scenarios, recognizing speech in dialects has always been a challenge. Due to my country's vast territory, large population, and rich variety of dialects and languages, the amount of data available for different dialects is uneven. Some dialects have relatively small amounts of data, making it difficult to obtain sufficient parallel data for training speech recognition models. As a result, speech recognition models struggle to achieve ideal recognition results for speech in these dialects.

[0070] To address the above technical issues, the embodiments of the present application propose a new dialect speech recognition model training scheme, and a dialect recognition method using the dialect speech recognition model trained using the above training scheme. This training scheme can improve the dialect speech recognition model's recognition effect on dialect speech, especially for small sample sizes. The dialect recognition method using the dialect speech recognition model trained using the above model training scheme can more accurately recognize dialect speech.

[0071] The conventional speech recognition model training scheme is to collect speech recognition training parallel data in advance, that is, training data including speech audio and corresponding text labels, and then use this parallel data to train the speech recognition model.

[0072] The dialect speech recognition model training scheme proposed in the embodiment of the present application is a new model training scheme proposed in addition to the conventional model training scheme mentioned above. The dialect speech recognition model training scheme proposed in the embodiment of the present application can make up for the defect of poor dialect recognition effect caused by insufficient training sample size in the conventional model training scheme mentioned above, thereby further improving the dialect recognition effect of the dialect speech recognition model.

[0073] In the subsequent embodiments of this application, the new model training scheme proposed in this application will be introduced in detail, and the conventional model training scheme mentioned above is also applicable to the dialect speech recognition model trained in the embodiments of this application. That is, in addition to using the model training scheme proposed in the embodiments of this application to train the dialect speech recognition model, the conventional model training scheme mentioned above can also be used to train the dialect speech recognition model using parallel data of dialect speech and corresponding text labels. The execution process of this conventional model training scheme will not be introduced in detail in the embodiments of this application.

[0074] It should also be noted that in the subsequent embodiments of this application, the dialect speech recognition model training scheme and dialect recognition scheme proposed in this application are introduced by taking the recognition of a certain specific dialect as an example. The embodiments of this application can improve the recognition effect of this specific dialect through an innovative dialect speech recognition model training method. When the dialect speech recognition model is required to recognize other dialects, sample data of other dialects can be used to train the dialect speech recognition model according to the training method introduced in the embodiments of this application, thereby improving the recognition effect of other dialects.

[0075] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0076] The training method of the dialect speech recognition model proposed in the embodiment of the present application simultaneously trains the dialect speech recognition model based on dialect speech samples and dialect text pronunciation samples. Specifically, the dialect speech recognition model is trained based on dialect speech samples, and the dialect text prediction training is performed on the dialect speech recognition model based on dialect text pronunciation samples.

[0077] The above-mentioned dialect text pronunciation samples are pronunciation sequence samples consisting of the Mandarin pronunciation corresponding to the dialect text. For example, the Mandarin pronunciation of the dialect word "Malang" (a roach) "m a3 lang2" can be used as a dialect text pronunciation sample.

[0078] In the above-mentioned dialect speech recognition model training process, performing dialect speech recognition training on the dialect speech recognition model based on the dialect speech sample means inputting the dialect speech audio into the dialect speech recognition model, and having the dialect speech recognition model recognize the input dialect speech to obtain a text recognition result.

[0079] And the dialect text prediction training based on the dialect text pronunciation sample and the dialect speech recognition model is that the dialect text pronunciation sample is input into the dialect speech recognition model, and the dialect speech recognition model performs autoregressive text prediction based on the input dialect text pronunciation sample, such as predicting the next recognized character based on the recognized text, or making the dialect speech recognition model perform a character mask prediction task based on the dialect text pronunciation sample.

[0080] The above-mentioned dialect speech sample can be obtained by collecting dialect speech data of a specific language. The collected dialect speech data is labeled with a corresponding text label, which can be used for dialect speech recognition training of the dialect speech recognition model.

[0081] The above-mentioned dialect text pronunciation sample can be obtained by first generating dialect text and then generating the corresponding Mandarin pronunciation of the dialect text.

[0082] In an embodiment of the present application, an implementation of obtaining the speech recognition label corresponding to the dialect text pronunciation sample and the dialect speech sample is disclosed.

[0083] Specifically, in order to obtain sufficient dialect text, the Mandarin text is used to generate dialect text in this embodiment, that is, the Mandarin text is converted into dialect text.

[0084] For example, the Mandarin text can be input into a dialect translation engine or a large language model, and the Mandarin text can be converted into dialect text based on the translation capability of the dialect translation engine or the language conversion capability of the large language model. For example, "dragonfly" in Mandarin is converted into "mole cricket" in dialect, "handsome men and beautiful women" in Mandarin is converted into "beautiful boys and girls" in dialect, and so on.

[0085] Through the above-mentioned text conversion method, a large amount of dialect text can be generated with the advantage of the quantity of Mandarin text, thereby providing a data basis for the training of the dialect speech recognition model.

[0086] After obtaining the dialect text through the above-mentioned method, the present embodiment further generates the corresponding dialect pronunciation of each dialect text based on the dialect text to dialect pronunciation mapping table, that is, determines the respective dialect pronunciation sequence corresponding to each dialect text.

[0087] Then, the dialect pronunciation is converted into Mandarin pronunciation. For example, the conversion of dialect pronunciation into Mandarin pronunciation can be realized by artificial conversion or with the help of a pronunciation prediction model, or artificial conversion combined with a pronunciation prediction model can be used to convert dialect pronunciation into Mandarin pronunciation.

[0088] Finally, based on the above conversion process, a mapping table of dialect texts to Mandarin pronunciations can be compiled. In this mapping table, the Mandarin pronunciation corresponding to each dialect text is recorded, that is, each dialect text corresponds to a Mandarin pronunciation sequence.

[0089] Based on the above mapping table, a dialect text pronunciation sample corresponding to the dialect text can be generated. That is, for any dialect text, by querying the above mapping table, the Mandarin pronunciation sequence corresponding to the dialect text can be obtained, and the Mandarin pronunciation sequence can be used as the dialect text speech sample corresponding to the dialect text.

[0090] Furthermore, for dialect speech, it can be used to construct dialect speech samples for speech recognition training of dialect speech recognition models. For niche dialects, due to the limited number of speech samples, in order to better utilize the collected dialect speech for training the dialect speech recognition model and improve the training effect, the embodiment of the present application trains a phoneme forced alignment model to generate speech recognition labels corresponding to the dialect speech samples.

[0091] Specifically, the present embodiment trains a Kaldi model as a phoneme-forced alignment model. Pre-collect dialect speech samples and their corresponding speech tags, which include both text and phoneme tags, or in some cases, only phoneme tags. Furthermore, a mapping table from dialect text to Mandarin pronunciation, obtained through the processing described in the above embodiment, is also used for model training.

[0092] When training the Kaldi model, the dialect speech samples and speech label parallel data, as well as the above-mentioned mapping table, are used as training inputs, and the Kaldi model performs the speech recognition task. During the process of the Kaldi model performing the speech recognition task, the speech and phoneme alignment process is automatically performed. Therefore, in the embodiment of the present application, the Kaldi model is trained for dialect speech phoneme annotation. The trained Kaldi model can be used to perform speech phoneme alignment and annotation tasks, that is, a phoneme forced alignment model is obtained.

[0093] Furthermore, by inputting unlabeled dialect speech samples into the aforementioned phoneme forced alignment model, the phonemes can be labeled for the dialect speech samples, resulting in their phoneme labels. Through this process, all collected dialect speech samples can be labeled with their phoneme labels. These labeled dialect speech samples can then be used to train the dialect speech recognition model, increasing the number of training samples and further improving the model training results.

[0094] Through the processing of the above embodiment, a dialect text sample can be generated based on the Mandarin text, and further, a dialect text pronunciation sample can be obtained through a mapping table from the dialect text to the Mandarin pronunciation. In addition, the phoneme forced alignment model constructed in the above manner can obtain a dialect speech sample with phoneme labels. The above-mentioned dialect text pronunciation samples and dialect speech samples with phoneme labels can both be used as training data for training the dialect speech recognition model. The embodiment of the present application uses them to train the dialect speech recognition model, which can increase the amount of model training data, thereby improving the model training effect and improving the model's recognition performance for dialect speech.

[0095] In other embodiments, it is disclosed that when the dialect speech recognition model is trained using the above-mentioned dialect text pronunciation samples and dialect speech samples carrying phoneme labels, the dialect speech recognition model is trained for dialect speech recognition based on the dialect speech samples, and the dialect text prediction training is performed on the dialect speech recognition model based on the dialect text pronunciation samples.

[0096] Specifically, Figure 1 A diagram showing the process of training a dialect speech recognition model is shown. Figure 1 As shown, when the dialect speech recognition model is trained, the dialect speech samples and the dialect text pronunciation samples are input into the dialect speech recognition model so that the dialect speech recognition model recognizes the dialect speech samples and performs dialect text prediction on the dialect text pronunciation samples to obtain speech recognition results and text prediction results.

[0097] When training a dialect speech recognition model, features can be extracted from the dialect speech sample and the dialect text pronunciation sample respectively, and then the dialect speech sample features and the dialect text pronunciation sample features can be input into the dialect speech recognition model. For example, the dialect speech sample features and the dialect text pronunciation sample features can be input into the dialect speech recognition model simultaneously, so that the dialect speech recognition model performs the recognition task of the dialect speech sample and the dialect text prediction task of the dialect text pronunciation sample; alternatively, the dialect speech sample features and the dialect text pronunciation sample features can be input into the dialect speech recognition model successively, so that the dialect speech recognition model performs the recognition task of the dialect speech sample and the dialect text prediction task of the dialect text pronunciation sample.

[0098] By enabling the dialect speech recognition model to perform the dialect speech recognition task and the text prediction task, speech recognition results and text prediction results can be obtained respectively.

[0099] Then, the speech recognition result corresponding to the dialect speech sample is compared with the speech recognition label corresponding to the dialect speech sample, and the dialect recognition loss function is calculated; and the text prediction result is compared with the dialect text pronunciation sample, specifically, the text characters in the text prediction result are compared with the text characters corresponding to the dialect text pronunciation sample, and the text prediction loss function is calculated.

[0100] Finally, with the goal of reducing dialect recognition loss and text prediction loss, the parameters of the dialect speech recognition model are optimized.

[0101] Repeat the above training process, allowing the dialect speech recognition model to continuously perform dialect speech recognition and dialect text prediction tasks. During this training process, the dialect speech recognition model can simultaneously learn dialect text pronunciation information while performing dialect speech recognition. This allows the dialect speech recognition model to more fully learn and explore the potential relationships between dialect speech, its pronunciation, and text. This can improve the training effect of the dialect speech recognition model and enhance its dialect speech recognition performance.

[0102] In other embodiments, the speech recognition labels of the dialect speech samples used to train the dialect speech recognition model include word-level labels and phoneme-level labels, wherein the word-level labels include the text characters corresponding to the dialect speech samples, and the phoneme-level labels include the phonemes in the pronunciation of the dialect speech samples.

[0103] Based on the above speech recognition tags, see Figure 2 As shown, during the training of the dialect speech recognition model, after inputting the dialect speech sample features into the dialect speech recognition model to obtain the speech recognition results, when calculating the dialect recognition loss, on the one hand, the word-level dialect recognition loss is determined based on the speech recognition results and the word-level labels in the speech recognition labels, and on the other hand, the phoneme-level dialect recognition loss is determined based on the speech recognition results and the phoneme-level labels in the speech recognition labels. The above-mentioned phoneme-level labels are obtained by phoneme-annotating the dialect language samples using the phoneme-forced alignment model in the above embodiment.

[0104] The character-level dialect recognition loss is determined based on the speech recognition results and the character-level labels in the speech recognition labels. Specifically, the characters in the speech recognition results output by the dialect speech recognition model are compared with the character labels in the character-level labels to determine whether the recognized characters and character labels at the same position are consistent, and then the character recognition loss function is calculated. The above-mentioned character recognition loss function can be obtained by calculating the loss function corresponding to each recognized character and then calculating the sum of the loss functions corresponding to each character, or it can be obtained by synchronously comparing the recognized characters and character labels at each character position and calculating the loss function.

[0105] In some embodiments, the aforementioned character-level dialect recognition loss and phoneme-level dialect recognition loss can be calculated simultaneously and used to optimize the dialect speech recognition model. In some cases, only one of the character-level dialect recognition loss and phoneme-level dialect recognition loss can be calculated. In this case, the calculated character-level dialect recognition loss or phoneme-level dialect recognition loss can be used to optimize the dialect speech recognition model.

[0106] For example, in some cases, the speech recognition labels corresponding to dialect speech samples may include only word-level labels, only phoneme-level labels, or both word-level labels and phoneme-level labels.

[0107] Based on the above different situations, when calculating the dialect recognition loss, if the speech recognition label includes word-level labels, the word-level dialect recognition loss can be calculated; if the speech recognition label includes phoneme-level labels, the phoneme-level dialect recognition loss can be calculated; if the speech recognition label includes both word-level labels and phoneme-level labels, both word-level dialect recognition loss and phoneme-level dialect recognition loss can be calculated. Finally, based on the actual dialect recognition loss obtained, the parameters of the dialect speech recognition model are optimized with the goal of minimizing the loss function.

[0108] It can be understood that the dialect speech recognition model training scheme proposed in the embodiment of the present application sets a variety of speech recognition labels, so that different types of dialect recognition losses can be calculated for the same dialect speech sample, thereby expanding the scope of application of the model training data. In particular, the embodiment of the present application can annotate the phoneme labels of the dialect speech samples through the phoneme forced alignment model, and then when training the dialect speech recognition model, the dialect speech recognition model can be optimized by calculating the phoneme-level dialect recognition loss. This training scheme makes up for the disadvantage that the dialect speech samples cannot be used for model training due to the inability to obtain parallel data of the dialect speech samples (dialect speech samples and corresponding text labels), thereby advantageously expanding the amount of model training data.

[0109] In other embodiments, the dialect speech recognition model performs dialect text prediction on the dialect text pronunciation sample. Specifically, the dialect speech recognition model performs autoregressive text prediction and / or mask prediction based on the input dialect text pronunciation sample to obtain an autoregressive prediction result and / or a mask prediction result.

[0110] Among them, the dialect speech recognition model performs autoregressive text prediction based on the input dialect text pronunciation samples, which means that the dialect speech recognition model predicts the text characters corresponding to the dialect text pronunciation samples in sequence based on the input dialect text pronunciation samples, such as predicting the next character based on the predicted character sequence.

[0111] The dialect speech recognition model performs mask prediction based on the input dialect text pronunciation sample, which means masking the character speech at a certain position in the dialect text pronunciation sample, and then making the dialect speech recognition model predict the masked character based on the text pronunciation at other positions.

[0112] Based on the above dialect text prediction task, see Figure 2 As shown, when calculating the text prediction loss based on the text prediction results output by the dialect speech recognition model, the first text prediction loss is determined based on the autoregressive prediction results and the dialect text pronunciation samples, and / or the second text prediction loss is determined based on the mask prediction results and the dialect text pronunciation samples.

[0113] Among them, calculating and determining the first text prediction loss based on the autoregressive prediction result and the dialect text speech sample refers to comparing the autoregressive prediction result with the dialect text corresponding to the dialect text pronunciation sample to calculate the text prediction loss.

[0114] Determining the second text prediction loss based on the mask prediction result and the dialect text speech sample refers to comparing the mask prediction result with the dialect text corresponding to the dialect text pronunciation sample to calculate the text prediction loss.

[0115] It is understood that the model training scheme of the embodiments of the present application enables the dialect speech recognition model to perform different text prediction tasks based on the input dialect text pronunciation samples, thereby calculating different text prediction losses for parameter optimization of the dialect speech recognition model. This model training method increases the diversity of model training tasks, allowing the dialect speech recognition model to learn and explore the relationship between dialect text and dialect pronunciation in different ways, which is conducive to improving model training results.

[0116] In some embodiments, the dialect speech recognition model can be made to simultaneously perform an autoregressive text prediction task and a mask prediction task based on the input dialect text pronunciation sample, and then calculate the first text prediction loss and the second text prediction loss according to the autoregressive prediction result and the mask prediction result, and use the first text prediction loss and the second text prediction loss to optimize the dialect speech recognition model. Alternatively, in some cases, the dialect speech recognition model can also be made to perform any one of the autoregressive text prediction task and the mask prediction task based on the input dialect text pronunciation sample, and then calculate the first text prediction loss or the second text prediction loss according to the model output result, and use the calculated loss to optimize the dialect speech recognition model. Similarly, the model training scheme proposed in the embodiment of the present application has more diverse ways of obtaining text prediction losses, which can improve the scope of application of model training. At the same time, by optimizing the model through different text prediction losses, the model optimization effect can be improved and the model training efficiency can be improved.

[0117] In other embodiments, Figure 3 As shown, the dialect speech samples and dialect text pronunciation samples input when training the dialect speech prediction model can be parallel sample data corresponding to each other, that is, the input dialect speech samples and dialect text pronunciation samples correspond to the same dialect text. The difference is that the dialect speech samples are the speech corresponding to the dialect text, and the dialect text pronunciation samples are the pronunciation sequences corresponding to the dialect text.

[0118] Based on the above-mentioned training data characteristics, when training the dialect speech recognition model, the dialect speech features obtained by the dialect speech recognition model from processing the dialect speech samples are obtained, and the dialect text features obtained by the dialect speech recognition model from processing the dialect text pronunciation samples are obtained.

[0119] Then, the dialect speech features and dialect text features obtained through the dialect speech recognition model are compared to determine the alignment loss between the dialect speech features and the dialect text features.

[0120] The aforementioned alignment loss can be used to measure whether the dialect speech features and dialect text features processed by the dialect speech recognition model are consistent in feature content and aligned in time. In theory, since the dialect speech samples and dialect text pronunciation samples input to the dialect speech recognition model are parallel data, the dialect speech features processed by the dialect speech recognition model for the dialect speech samples and the dialect text features processed for the dialect text pronunciation samples should be aligned. If the two are not aligned, it indicates that there is room for improvement in the dialect speech recognition model's feature extraction process for the two modal data.

[0121] Based on the above ideas, the model training scheme proposed in the embodiment of this application, during the training of the dialect speech recognition model, if the dialect speech samples and dialect text pronunciation samples input to the dialect speech recognition model are corresponding parallel sample data, then in addition to calculating the dialect recognition loss and text prediction loss, the alignment loss between the dialect speech features and dialect text features processed by the dialect speech recognition model is also calculated. Furthermore, when optimizing the dialect speech recognition model, the above-mentioned alignment loss, dialect recognition loss, and text prediction loss are jointly used to optimize the dialect speech recognition model. That is, the parameters of the dialect speech recognition model are optimized with the goal of reducing the above-mentioned alignment loss, dialect recognition loss, and text prediction loss.

[0122] It can be understood that this model training scheme applies a variety of different loss functions, which enables the dialect speech recognition model to perceive the relationship between dialect speech, dialect text and dialect pronunciation through multiple dimensions during the training process, thereby improving the model training effect and accelerating the model training efficiency.

[0123] From the above embodiments, it can be seen that the dialect speech recognition model used in the dialect recognition method proposed in this application is obtained by performing dialect speech recognition training based on dialect speech samples and performing dialect text prediction training based on dialect text pronunciation samples, and the dialect text pronunciation samples include the Mandarin pronunciation corresponding to the dialect text. Through the above-mentioned training method, on the one hand, dialect text pronunciation samples constructed by the Mandarin pronunciation corresponding to the dialect text are added on the basis of conventional dialect speech samples, thereby increasing the amount of training sample data. On the other hand, in the process of training the dialect speech recognition model, the model is made to perform two tasks, dialect speech recognition and dialect text prediction, so that the model can fully explore and learn the logical relationship between dialect speech, pronunciation, and text, thereby improving the model training effect and improving the model training efficiency. Through the above-mentioned training method, the model's recognition effect on dialect speech can be improved.

[0124] The dialect speech recognition model trained based on the above model training scheme can be used to recognize dialect speech, such as Figure 4 As shown, when the dialect speech is recognized, step S101 is executed:

[0125] S101: Input the dialect speech into the dialect speech recognition model to obtain the dialect recognition result.

[0126] Since the above-mentioned dialect speech recognition model is obtained by performing dialect speech recognition training based on dialect speech samples and performing dialect text prediction training based on dialect text pronunciation samples in accordance with the introduction of the above-mentioned embodiments of the present application, it has better training effect and higher recognition accuracy of dialect speech. Therefore, the dialect recognition method using this dialect speech recognition model can obtain more accurate dialect recognition results and improve the dialect recognition effect.

[0127] Corresponding to the above-mentioned dialect recognition method, the present application embodiment also provides a dialect recognition device, see Figure 5 As shown, the device includes:

[0128] The dialect recognition unit 100 is used to input the dialect speech into the dialect speech recognition model to obtain the dialect recognition result;

[0129] The dialect speech recognition model is obtained by performing dialect speech recognition training based on dialect speech samples and performing dialect text prediction training based on dialect text pronunciation samples, wherein the dialect text pronunciation samples include the Mandarin pronunciation corresponding to the dialect text.

[0130] In some implementations, the training process of the dialect speech recognition model includes:

[0131] Inputting the dialect speech sample and the dialect text pronunciation sample into the dialect speech recognition model, so that the dialect speech recognition model recognizes the dialect speech sample and performs dialect text prediction on the dialect text pronunciation sample, thereby obtaining a speech recognition result and a text prediction result;

[0132] Determining a dialect recognition loss based on the speech recognition result and the speech recognition label corresponding to the dialect speech sample, and determining a text prediction loss based on the text prediction result and the dialect text pronunciation sample;

[0133] The dialect speech recognition model is optimized based on the dialect recognition loss and the text prediction loss.

[0134] In some implementations, the speech recognition labels include word-level labels and / or phoneme-level labels;

[0135] The determining of the dialect recognition loss according to the speech recognition result and the speech recognition label corresponding to the dialect speech sample includes:

[0136] A word-level dialect recognition loss is determined based on the speech recognition result and the word-level label, and / or a phoneme-level dialect recognition loss is determined based on the speech recognition result and the phoneme-level label.

[0137] In some implementations, the dialect speech recognition model performs dialect text prediction on the dialect text pronunciation sample, including:

[0138] The dialect speech recognition model performs autoregressive text prediction and / or mask prediction based on the dialect text pronunciation sample to obtain an autoregressive prediction result and / or a mask prediction result;

[0139] Determining a text prediction loss according to the text prediction result and the dialect text pronunciation sample includes:

[0140] A first text prediction loss is determined based on the autoregressive prediction result and the dialect text pronunciation sample, and / or a second text prediction loss is determined based on the mask prediction result and the dialect text pronunciation sample.

[0141] In some implementations, the dialect speech sample and the dialect text pronunciation sample are parallel sample data corresponding to each other;

[0142] The method further comprises:

[0143] Acquire dialect speech features obtained by processing the dialect speech sample by the dialect speech recognition model, and obtain dialect text features obtained by processing the dialect text pronunciation sample by the dialect speech recognition model;

[0144] determine an alignment loss between the dialect speech features and the dialect text features;

[0145] optimize the dialect speech recognition model based on the dialect recognition loss and the text prediction loss, including:

[0146] optimize the dialect speech recognition model based on the alignment loss, the dialect recognition loss and the text prediction loss.

[0147] In some implementations, the training process of the dialect speech recognition model further includes:

[0148] generating dialect text from Mandarin text, and generating a mapping table of dialect text to Mandarin pronunciation;

[0149] generating dialect text pronunciation samples corresponding to the dialect text based on the mapping table;

[0150] generating phoneme labels corresponding to the dialect speech samples through a phoneme forced alignment model, wherein the phoneme forced alignment model is obtained based on dialect speech sample parallel data and the mapping table.

[0151] In some implementations, generating dialect text from Mandarin text includes:

[0152] inputting the Mandarin text into a dialect translation engine and / or a large language model to obtain dialect text corresponding to the input Mandarin text.

[0153] The dialect recognition device provided by the embodiment belongs to the same application concept as the dialect recognition method provided by the above-mentioned embodiments of the present application, can execute the dialect recognition method provided by any of the above-mentioned embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method. Technical details not described in detail in the embodiment can be referred to the specific processing content of the dialect recognition method provided by the above-mentioned embodiments of the present application, which will not be described here.

[0154] The functions implemented by the dialect recognition unit 100 above can be implemented by a processor, and the embodiments of the present application are not limited.

[0155] It should be understood that the units in the above devices can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and the memory stores instructions. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of the device. The processor can be a general-purpose processor, such as a CPU or a microprocessor, and the memory can be a memory within the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits. The functions of some or all units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units can be realized by designing the logical relationships between the components within the circuit. For another example, the hardware circuit can be implemented by a PLD. For example, an FPGA can include a large number of logic gate circuits. The connection relationships between the logic gate circuits are configured through a configuration file to realize the functions of some or all of the above units. All units of the above devices can be implemented entirely in the form of a processor calling software, or entirely in the form of hardware circuits, or partially in the form of a processor calling software, with the remaining parts implemented in the form of hardware circuits.

[0156] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and execute instructions, such as a CPU, a microprocessor, a GPU, or a DSP. In another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit may be fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the configuration of the hardware circuit can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0157] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0158] In addition, the various units in the above apparatus may be fully or partially integrated together, or may be implemented independently. In one implementation, these units are integrated together and implemented in the form of a system-on-chip (SOC). The SOC may include at least one processor for implementing any of the above methods or implementing the functions of the various units of the apparatus. The at least one processor may be of different types, such as a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0159] Another embodiment of the present application further provides an electronic device, see Figure 6 As shown, the device includes:

[0160] Memory 200 and processor 210;

[0161] The memory 200 is connected to the processor 210 and is used to store programs;

[0162] The processor 210 is configured to implement the dialect recognition method disclosed in any of the above embodiments by running the program stored in the memory 200 .

[0163] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0164] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are interconnected via a bus.

[0165] A bus may include a pathway that transfers information between components of a computer system.

[0166] Processor 210 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, or the like, or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present invention. Alternatively, it can be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware components.

[0167] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0168] The memory 200 stores programs for implementing the technical solutions of the present application, and can also store operating systems and other key services. Specifically, the programs can include program codes, and the program codes include computer operation instructions. More specifically, the memory 200 can include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk memory, a flash, and the like.

[0169] The input device 230 can include devices that receive data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor, and the like.

[0170] The output device 240 can include devices that allow information to be output to a user, such as a display screen, a printer, a speaker, and the like.

[0171] The communication interface 220 can include devices using any transceiver to communicate with other devices or communication networks, such as an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), and the like.

[0172] The processor 210 executes programs stored in the memory 200 and calls other devices, which can be used to implement each step of any dialect recognition method provided by the above-described embodiments of the present application.

[0173] In addition to the above-described methods and devices, the embodiments of the present application can also be computer program products that include computer program instructions that, when executed by a processor, cause the processor to perform the steps of the dialect recognition method described in any of the above-described embodiments of the present application.

[0174] The computer program product can be written in any combination of one or more programming languages to perform the operations of the embodiments of the present application, including an object-oriented programming language, such as Java, C++, and the like, and a conventional procedural programming language, such as the "C" language or a similar programming language. The program code can be executed entirely on a user computing device, partially on a user device, as a separate software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0175] In addition, the embodiments of the present application can also be storage media that store computer programs, and the computer programs are executed by a processor to perform the steps of the dialect recognition method described in any of the above-described embodiments of the present application.

[0176] For the sake of simplicity, the aforementioned method embodiments are described as a series of action combinations. However, those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0177] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similarities between the various embodiments can be referred to in conjunction with each other. For device embodiments, since they are generally similar to method embodiments, their description is relatively simple, and for relevant details, reference can be made to the description of the method embodiments.

[0178] The steps in the methods of each embodiment of the present application can be adjusted in sequence, merged, and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0179] The modules and sub-modules in the devices and terminals of the various embodiments of the present application can be merged, divided, and deleted according to actual needs.

[0180] In the several embodiments provided in this application, it should be understood that the disclosed terminals, devices, and methods can be implemented in other ways. For example, the terminal embodiments described above are merely illustrative. For example, the division of modules or submodules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or module, which can be electrical, mechanical or other forms.

[0181] The modules or submodules described as separate components may or may not be physically separate, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules may be selected to achieve the purpose of this embodiment according to actual needs.

[0182] In addition, each functional module or submodule in each embodiment of the present application may be integrated into a processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into a single module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or software functional modules or submodules.

[0183] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0184] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, software units executed by a processor, or a combination of the two. The software units may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0185] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0186] The above description of the disclosed embodiments will enable those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is to be construed in the widest manner consistent with the principles and novel features disclosed herein.

Claims

1. A dialect recognition method, characterized in that: include: Input the dialect speech into the dialect speech recognition model to obtain the dialect recognition result; Among them, the dialect speech recognition model is obtained by performing dialect speech recognition training based on dialect speech samples and performing dialect text prediction training based on dialect text pronunciation samples, and the dialect text pronunciation samples include the Mandarin pronunciation corresponding to the dialect text; the dialect speech samples and the dialect text pronunciation samples are parallel sample data corresponding to each other, and the training process of the dialect speech recognition model includes: obtaining the dialect speech features obtained by the dialect speech recognition model processing the dialect speech samples, and the dialect text features obtained by the dialect speech recognition model processing the dialect text pronunciation samples; determining the alignment loss between the dialect speech features and the dialect text features.

2. The method according to claim 1, characterized in that The training process of the dialect speech recognition model further includes: Inputting the dialect speech sample and the dialect text pronunciation sample into the dialect speech recognition model, so that the dialect speech recognition model recognizes the dialect speech sample and performs dialect text prediction on the dialect text pronunciation sample, thereby obtaining a speech recognition result and a text prediction result; Determining a dialect recognition loss based on the speech recognition result and the speech recognition label corresponding to the dialect speech sample, and determining a text prediction loss based on the text prediction result and the dialect text pronunciation sample; The dialect speech recognition model is optimized based on the alignment loss, the dialect recognition loss, and the text prediction loss.

3. The method according to claim 2, characterized in that The speech recognition labels include word-level labels and / or phoneme-level labels; The determining of the dialect recognition loss according to the speech recognition result and the speech recognition label corresponding to the dialect speech sample includes: A word-level dialect recognition loss is determined based on the speech recognition result and the word-level label, and / or a phoneme-level dialect recognition loss is determined based on the speech recognition result and the phoneme-level label.

4. The method according to claim 2, characterized in that The dialect speech recognition model performs dialect text prediction on the dialect text pronunciation sample, including: The dialect speech recognition model performs autoregressive text prediction and / or mask prediction based on the dialect text pronunciation sample to obtain an autoregressive prediction result and / or a mask prediction result; Determining a text prediction loss according to the text prediction result and the dialect text pronunciation sample includes: A first text prediction loss is determined based on the autoregressive prediction result and the dialect text pronunciation sample, and / or a second text prediction loss is determined based on the mask prediction result and the dialect text pronunciation sample.

5. The method according to any one of claims 2 to 4, characterized in that The method further comprises: Generate dialect text using Mandarin text, and generate a mapping table from dialect text to Mandarin pronunciation; Generating a dialect text pronunciation sample corresponding to the dialect text based on the mapping table; The phoneme labels corresponding to the dialect speech samples are generated by a phoneme forced alignment model, wherein the phoneme forced alignment model is obtained by performing dialect speech phoneme annotation training based on the dialect speech sample parallel data and the mapping table.

6. The method according to claim 5, characterized in that Generate dialect text using Mandarin text, including: The Mandarin text is input into the dialect translation engine and / or the large language model to obtain the dialect text corresponding to the input Mandarin text.

7. A dialect recognition device, characterized in that: include: The dialect recognition unit is used to input the dialect speech into the dialect speech recognition model to obtain the dialect recognition result; Among them, the dialect speech recognition model is obtained by performing dialect speech recognition training based on dialect speech samples and performing dialect text prediction training based on dialect text pronunciation samples, and the dialect text pronunciation samples include the Mandarin pronunciation corresponding to the dialect text; the dialect speech samples and the dialect text pronunciation samples are parallel sample data corresponding to each other, and the training process of the dialect speech recognition model includes: obtaining the dialect speech features obtained by the dialect speech recognition model processing the dialect speech samples, and the dialect text features obtained by the dialect speech recognition model processing the dialect text pronunciation samples; determining the alignment loss between the dialect speech features and the dialect text features.

8. An electronic device, characterized in that: including memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the dialect recognition method according to any one of claims 1 to 6 by running the program in the memory.

9. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, enable the processor to perform the dialect recognition method according to any one of claims 1 to 6.

Images