Method for constructing pronunciation dictionary, speech recognition method and related device

Abstract

The present application provides a pronunciation dictionary construction method, a speech recognition method and related devices. The pronunciation dictionary construction method includes: extracting phonemes from target audio data to obtain a corresponding phoneme set; the target audio data is audio data covering all phonemes; according to the recognition probability of each phoneme in the extracted phoneme set and the phoneme label corresponding to the target audio data, determining similar phonemes of the phonemes in the phoneme label from the phoneme set; constructing a multi-pronunciation dictionary according to the rule of storing words corresponding to their standard pronunciations and similar pronunciations; wherein, the standard pronunciation is composed of the phonemes in the phoneme label, and the similar pronunciation is composed of the similar phonemes of the phonemes in the phoneme label. The multi-pronunciation dictionary constructed by the above solution can improve the fault tolerance and robustness of speech recognition, and thus can improve the speech recognition effect.

Description

Technical Field

[0001] The present application relates to the technical field of speech recognition, and particularly to a method for constructing a pronunciation dictionary, a speech recognition method, apparatus, device, and storage medium. Background Art

[0002] Currently, the mainstream commercial speech recognition systems still adopt a framework of joint decoding based on an acoustic model and a language model. Among them, the acoustic model is mainly responsible for mapping speech features to phonemes, and the language model combines a pronunciation dictionary to transform a phoneme string into a corresponding character string.

[0003] In the pronunciation dictionary, the corresponding relationship between the pronunciation composed of phonemes and the text is recorded. When the pronunciation of the phoneme string recognized by the acoustic model matches a certain pronunciation in the pronunciation dictionary, the text corresponding to the phoneme string can be determined according to the corresponding relationship between the pronunciation and the text in the pronunciation dictionary, that is, the conversion from phonemes to text is realized.

[0004] The existing speech recognition solutions have poor fault tolerance. Specifically, only the phoneme strings that strictly match the pronunciations in the pronunciation dictionary will be recognized as the texts corresponding to the pronunciations in the pronunciation dictionary. Although this can ensure the absolute accuracy of speech recognition, it reduces the robustness of speech recognition and is not conducive to improving the speech recognition effect. Summary of the Invention

[0005] Based on the above technical status quo, the present application proposes a method for constructing a pronunciation dictionary, a speech recognition method, apparatus, device, and storage medium, which can improve the fault tolerance and robustness of speech recognition and enhance the speech recognition effect.

[0006] A method for constructing a pronunciation dictionary includes:

[0007] Performing phoneme extraction on target audio data to obtain a corresponding phoneme set; the target audio data is audio data covering all phonemes;

[0008] Determining similar phonemes of the phonemes in the phoneme label from the phoneme set according to the recognition probability of each phoneme in the extracted phoneme set and the phoneme label corresponding to the target audio data; wherein, the similar phonemes of the phonemes in the phoneme label refer to a set number of phonemes selected from the phoneme set that correspond to the phonemes in the phoneme label and have the highest recognition probability;

[0009] Constructing a multi-pronunciation dictionary according to the rule of storing words corresponding to their standard pronunciations and similar pronunciations; wherein, the similar pronunciation is composed of the similar phonemes of the phonemes in the phoneme label.

[0010] Optionally, a multi - pronunciation dictionary is constructed according to the rule of storing words corresponding to their standard pronunciations and similar pronunciations, including:

[0011] Determine the standard pronunciation corresponding to the word in the pronunciation dictionary, and determine the similar pronunciation corresponding to the word in the pronunciation dictionary according to the similar phonemes of the phonemes in the phoneme label;

[0012] By storing the word corresponding to its standard pronunciation and similar pronunciation correspondingly, a multi - pronunciation dictionary is obtained.

[0013] Optionally, determining the standard pronunciation corresponding to the word in the pronunciation dictionary, and determining the similar pronunciation corresponding to the word in the pronunciation dictionary according to the similar phonemes of the phonemes in the phoneme label, includes:

[0014] Respectively determine the standard pronunciation corresponding to the high - frequency error - prone words in the pronunciation dictionary, and respectively determine the similar pronunciation corresponding to the high - frequency error - prone words in the pronunciation dictionary according to the similar phonemes of the phonemes in the phoneme label;

[0015] By storing the word corresponding to its standard pronunciation and similar pronunciation correspondingly, a multi - pronunciation dictionary is obtained, including:

[0016] By storing the high - frequency error - prone words in the pronunciation dictionary corresponding to their standard pronunciations and similar pronunciations correspondingly, a multi - pronunciation dictionary is obtained.

[0017] Optionally, by storing the high - frequency error - prone words in the pronunciation dictionary corresponding to their standard pronunciations and similar pronunciations correspondingly, a multi - pronunciation dictionary is obtained, including:

[0018] According to the similar pronunciation corresponding to the high - frequency error - prone words in the pronunciation dictionary and the recognition probability of each phoneme in the phoneme set, calculate the score of the similar pronunciation corresponding to the high - frequency error - prone words;

[0019] According to the score of the similar pronunciation corresponding to the high - frequency error - prone words, select the similar pronunciation with a score higher than the set score threshold from the similar pronunciations corresponding to each high - frequency error - prone word as the target similar pronunciation;

[0020] Store the high - frequency error - prone words in the pronunciation dictionary corresponding to their standard pronunciations and target similar pronunciations correspondingly, to obtain a multi - pronunciation dictionary.

[0021] Optionally, the extracting phonemes from the target audio data to obtain the corresponding phoneme set includes:

[0022] Input the target audio data into a pre - trained acoustic model for phoneme extraction to obtain the phoneme set corresponding to the target audio data.

[0023] A speech recognition method, comprising:

[0024] Obtaining a phoneme sequence of the speech to be recognized;

[0025] Determining a speech recognition result of the speech to be recognized according to the phoneme sequence of the speech to be recognized and a pre-constructed multiple pronunciation dictionary;

[0026] Wherein, words in the multiple pronunciation dictionary are stored corresponding to the standard pronunciation and similar pronunciations of the words; the similar pronunciations corresponding to the words are constructed from the similar phonemes of the phonemes in the phoneme labels corresponding to the target audio data; the target audio data includes the audio data corresponding to the words.

[0027] Optionally, the multiple pronunciation dictionary is constructed according to the above pronunciation dictionary construction method.

[0028] Optionally, obtaining the phoneme sequence of the speech to be recognized includes:

[0029] Inputting the speech to be recognized into a pre-trained acoustic model for phoneme extraction to obtain the phoneme sequence of the speech to be recognized.

[0030] Optionally, the acoustic model is trained as follows:

[0031] Inputting the audio features of the training speech into the acoustic model to obtain the phoneme information of the training speech;

[0032] Inputting the phoneme information of the training speech and the non-semantic information extracted according to the audio features of the training speech into an audio synthesis model to obtain an audio synthesis result;

[0033] Correcting the parameters of the acoustic model according to the speech recognition loss of the acoustic model and the audio synthesis loss of the audio synthesis model.

[0034] Optionally, inputting the phoneme information of the training speech and the non-semantic information extracted according to the audio features of the training speech into an audio synthesis model to obtain an audio synthesis result, including:

[0035] Performing downsampling of the same scale on the phoneme information of the training speech and the audio features of the training speech;

[0036] Extracting the non-semantic information of the training speech according to the downsampled audio features of the training speech;

[0037] Fusing the downsampled phoneme information and the non-semantic information to obtain basic audio synthesis information;

[0038] Inputting the basic audio synthesis information into the audio synthesis model to obtain an audio synthesis result.

[0039] Optionally, the speech recognition loss of the acoustic model is determined by a cross-entropy loss function, and the audio synthesis loss of the audio synthesis model is determined by a mean squared error loss function.

[0040] A pronunciation dictionary construction device includes:

[0041] A phoneme extraction unit, configured to extract phonemes from target audio data to obtain a corresponding phoneme set; the target audio data is audio data covering all phonemes;

[0042] A phoneme screening unit, configured to determine similar phonemes of the phonemes in the phoneme label from the phoneme set according to the recognition probability of each phoneme in the extracted phoneme set and the phoneme label corresponding to the target audio data; wherein, the similar phonemes of the phonemes in the phoneme label refer to a set number of phonemes with the highest recognition probability selected from the phoneme set and corresponding to the phonemes in the phoneme label;

[0043] A dictionary construction unit, configured to construct a multi-pronunciation dictionary according to the rule of storing words corresponding to the standard pronunciation and similar pronunciations of the words; wherein, the similar pronunciation is composed of the similar phonemes of the phonemes in the phoneme label.

[0044] A speech recognition device includes:

[0045] A speech processing unit, configured to obtain a phoneme sequence of the speech to be recognized;

[0046] A speech recognition unit, configured to determine a speech recognition result of the speech to be recognized according to the phoneme sequence of the speech to be recognized and a pre-constructed multi-pronunciation dictionary;

[0047] Wherein, the words in the multi-pronunciation dictionary are stored corresponding to the standard pronunciation and similar pronunciations of the words; the similar pronunciation corresponding to a word is constructed from the similar phonemes of the phonemes in the phoneme label corresponding to the target audio data; the target audio data includes the audio data corresponding to the word.

[0048] An electronic device, characterized by including:

[0049] A memory and a processor;

[0050] The memory is connected to the processor and is configured to store programs;

[0051] The processor is configured to implement the above pronunciation dictionary construction method or the above speech recognition method by running the programs in the memory.

[0052] A storage medium stores a computer program thereon. When the computer program is run by a processor, the pronunciation dictionary construction method described above is implemented, or the speech recognition method described above is implemented.

[0053] The multiple pronunciation dictionary constructed based on the pronunciation dictionary construction method provided in this application, where one word corresponds to multiple pronunciations, specifically the standard pronunciation of the word and the similar pronunciations of the word. When performing speech recognition based on this pronunciation dictionary, even if the phoneme extraction of the speech to be recognized is inaccurate, for example, the pronunciation of the speech to be recognized is predicted as a similar pronunciation rather than the accurate pronunciation, the correct text can still be recognized based on this multiple pronunciation dictionary. Therefore, the multiple pronunciation dictionary constructed by the technical solution of this application can improve the fault tolerance and robustness of speech recognition, and thus can improve the speech recognition effect, and can achieve a better recognition effect even in complex scenarios. Description of the Drawings

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on the provided drawings without creative efforts.

[0055] Figure 1 It is a schematic flowchart of a pronunciation dictionary construction method provided by an embodiment of the present application;

[0056] Figure 2 It is a schematic flowchart of a speech recognition method provided by an embodiment of the present application;

[0057] Figure 3 It is a schematic diagram of the acoustic model training process provided by an embodiment of the present application;

[0058] Figure 4 It is a schematic structural diagram of a pronunciation dictionary construction device provided by an embodiment of the present application;

[0059] Figure 5 It is a schematic structural diagram of a speech recognition device provided by an embodiment of the present application;

[0060] Figure 6 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed Embodiments

[0061] The technical solution of the embodiment of the present application is applicable to the speech recognition application scenario. By adopting the technical solution of the embodiment of the present application, the robustness of speech recognition can be improved, and thus the speech recognition effect can be improved.

[0062] Speech recognition technology, also known as automatic speech recognition (ASR), focuses on speech and aims to convert human voice signals into text or commands. With the rapid development of artificial intelligence, speech recognition technology is the first step in enabling machines to "understand" human language.

[0063] Currently, mainstream commercial speech recognition systems are still based on a framework of joint decoding of acoustic models and language models. The acoustic model is mainly responsible for mapping speech features to phonemes, and the language model combines with the pronunciation dictionary to convert phoneme strings into corresponding text strings.

[0064] The pronunciation dictionary records the correspondence between the pronunciations of phonemes and text. When the pronunciation of a phoneme string identified by the acoustic model matches a pronunciation in the pronunciation dictionary, the corresponding text of the phoneme string can be determined based on the correspondence between the pronunciation in the pronunciation dictionary and the text, thus achieving phoneme-to-text conversion.

[0065] Existing speech recognition solutions have poor fault tolerance. Specifically, only phoneme strings that strictly match the pronunciation in the pronunciation dictionary will be recognized as text corresponding to the pronunciation in the pronunciation dictionary. Although this can ensure the absolute accuracy of speech recognition, it reduces the robustness of speech recognition and is not conducive to improving speech recognition effects.

[0066] For example, the correct pronunciation of the word "ready" in the pronunciation dictionary is "zhun3 bei4". However, in complex scenarios, due to the influence of speaking speed or environment, the acoustic model may predict the phoneme string corresponding to the voice of "ready" as "zun3 bei4". At this time, according to the matching rules, since "zun3 bei4" does not match "zhun3 bei4" in the pronunciation dictionary, the phoneme string "zun3 bei4" cannot be recognized as "ready", that is, correct recognition cannot be achieved.

[0067] It can be seen that the existing speech recognition solutions have poor fault tolerance and poor recognition effect in complex scenarios.

[0068] Based on the above-mentioned technical status, the embodiment of the present application proposes a method for constructing a pronunciation dictionary. The pronunciation dictionary constructed with the help of the technical solution of the embodiment of the present application can improve the fault tolerance and robustness of speech recognition, improve the speech recognition effect, especially improve the speech recognition effect in complex scenarios.

[0069] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0070] An embodiment of the present application proposes a method for constructing a pronunciation dictionary. As shown in Figure 1 the following, the method includes:

[0071] S101. Extract phonemes from the target audio data to obtain a corresponding phoneme set.

[0072] Specifically, the above-mentioned target audio data is audio data covering all phonemes. The all phonemes are all phonemes included in a certain language, such as all phonemes in the Chinese language, all phonemes in the English language, etc., and are specifically determined according to the language field to which the pronunciation dictionary applies.

[0073] The above-mentioned target audio data is specifically a certain number of audio data. The fact that the target audio data covers all phonemes is specifically reflected in that any one of the all phonemes appears in the pronunciation of any one or more words in the target audio data. For example, for the phoneme "a" in the Chinese language, as long as the phoneme "a" is included in the pronunciation of at least one word in the target audio data, it is considered that the target audio data covers the phoneme "a".

[0074] Among them, the above-mentioned words vary according to the language. For example, in the Chinese language, the above-mentioned words can specifically be individual Chinese characters or words composed of multiple Chinese characters; while in the English language, the above-mentioned words can specifically be English words or English letters.

[0075] As an exemplary implementation, inputting the above-mentioned target audio data or the acoustic features of the target audio data into a pre-trained acoustic model for phoneme extraction respectively can obtain phonemes corresponding to the input target audio data. Since the target audio data is composed of a certain amount of audio data, the phonemes obtained by extracting phonemes from the target audio data are also a large number of phonemes, and the large number of phonemes can form a phoneme set.

[0076] The above-mentioned acoustic model is a pre-trained model for extracting phonemes from audio data, and its specific training process can be referred to the introduction of subsequent embodiments.

[0077] S102. Determine similar phonemes of the phonemes in the phoneme label from the phoneme set according to the recognition probability of each phoneme in the extracted phoneme set and the phoneme label corresponding to the target audio data.

[0078] Among them, the recognition probability of each phoneme in the above phoneme set refers to the probability that a certain audio frame is recognized as this phoneme when extracting phonemes from the above target audio data. Exemplarily, when extracting phonemes from the target audio data through the above acoustic model, the extracted phonemes and the recognition probability of this phoneme can be output. For example, assume that the probability that a certain audio frame is recognized as the phoneme "a" is 0.8, then the recognition probability of this phoneme "a" is 0.8.

[0079] The recognition probability of the phonemes in the phoneme set reflects the probability that this phoneme is correctly recognized. The higher the recognition probability that a certain audio frame is recognized as a certain phoneme, the higher the correct rate of recognizing this audio frame as this phoneme. Therefore, when the same audio frame is simultaneously recognized as multiple different phonemes, the correct phoneme recognition result can be selected from them according to the recognition probabilities of each phoneme. Usually, this correct phoneme recognition result is the phoneme with the highest recognition probability among the multiple different phonemes recognized. For example, assume that the probability that a certain audio frame is recognized as "a" is 0.8 and the probability that it is recognized as "b" is 0.4, then it can be determined that the phoneme corresponding to this audio frame is "a" because the recognition probability of the phoneme "a" is higher.

[0080] Specific content regarding phoneme extraction and the recognition probability of phonemes can also refer to the content regarding phoneme extraction and the phoneme scores during phoneme extraction in the conventional technical solutions.

[0081] The above phoneme labels are phoneme labels obtained by manually performing phoneme annotation on the above target audio data. The phoneme labels of the target audio data contain the correct phonemes corresponding to each audio frame in the target audio data.

[0082] The similar phonemes of the phonemes in the phoneme label refer to the set number of phonemes with the highest recognition probability selected from the phoneme set corresponding to the phonemes in the phoneme label.

[0083] Exemplarily, by comparing the phonemes in the phoneme set and the phonemes in the above phoneme label corresponding to the same audio frame, the phonemes with the TOPN recognition probabilities corresponding to the phonemes in the phoneme label in the phoneme set for the same audio frame are selected, and thus the similar phonemes of the phonemes in the phoneme label can be obtained. Among them, N is the above set number. In the embodiments of the present application, N = 3, that is, the similar phonemes of the phonemes in the phoneme label are specifically 3 phonemes with the highest recognition probability selected from the phoneme set corresponding to the phonemes in the phoneme label.

[0084] Specifically, for example, assume that for a certain audio frame A in the target audio data, its phoneme label is "a". When the target audio data is input into a pre-trained acoustic model for phoneme extraction, there may be multiple audio frames A in multiple different audio sentences in the target audio data. During phoneme extraction, these multiple audio frames A are respectively recognized as phonemes "a1", "a2", "a3", "a4", and the recognition probabilities of "a1", "a2", "a3", "a4" are 0.8, 0.6, 0.7, 0.3 respectively. Then, "a1", "a2", "a3" are the 3 phonemes with the highest recognition probabilities. Therefore, "a1", "a2", "a3" can be used as the similar phonemes of the phoneme "a" in the phoneme label respectively.

[0085] Furthermore, in order to facilitate the comparison of the similarity between each similar phoneme and the phoneme label, the embodiments of the present application normalize the recognition probabilities of the similar phonemes of the phoneme in the phoneme label as the final recognition probabilities of each similar phoneme. Then, the normalized recognition probability of the similar phoneme of the phoneme in the phoneme label can represent the similarity between the phoneme in the phoneme label and this similar phoneme.

[0086] In addition, due to the data diversity of the target audio data, the same audio frame may appear multiple times in the target audio data. And due to different contexts, the same audio frame may be recognized as the same phoneme or different phonemes. Therefore, the phonemes corresponding to the same audio frame in the above phoneme set may be the same or different. Moreover, even for the same phoneme, its recognition probability will be different.

[0087] For example, for the above audio frame A, when performing phoneme extraction on it, the 3 phonemes with the highest recognition rates may be: phoneme "a2" with a recognition probability of 0.5; phoneme "a2" with a recognition probability of 0.8; phoneme "a4" with a recognition probability of 0.3. It can be seen that although the 3 phonemes with the highest recognition probabilities are selected as the similar phonemes of the phoneme label "a" of audio frame A, there are actually only 2 different phonemes. At this time, when normalizing the recognition probabilities of each similar phoneme, for the same phoneme, the recognition probabilities are merged and normalized.

[0088] Specifically, for the above phoneme "a2", its normalized recognition probability is (0.5 + 0.8) / (0.5 + 0.8 + 0.3) = 0.81, and the normalized recognition probability of phoneme "a4" is 0.3 / (0.5 + 0.8 + 0.3) = 0.19.

[0089] Further, in order to more clearly record the relationship and similarity between the phonemes in the phoneme label and the similar phonemes, the embodiment of the present application constructs a phoneme confusion matrix for the phonemes and their similar phonemes in the phoneme label of the above-mentioned target audio data. In this phoneme confusion matrix, each phoneme is used as the row and column of the matrix, and the similarity between phonemes is used as the matrix element.

[0090] The specific phoneme confusion matrix can be seen in Table 1 below:

[0091] Table 1

[0092] a b c d a 1.0 0.3 0.4 0.3 b 0.3 1.0 0.5 0.2 c 0.4 0.5 1.0 0.1 d 0.3 0.2 0.1 1.0

[0093] In Table 1 above, taking phonemes a, b, c, and d as examples, their similarities to each other are shown. For example, when phoneme a is the phoneme in the above-mentioned phoneme label, it can be determined that the similar phonemes to it are phonemes b, c, and d, and it is known that the similarities between phoneme a and phonemes b, c, and d are 0.3, 0.4, and 0.3 respectively.

[0094] Referring to the above introduction, for each phoneme in the phoneme label of the above-mentioned target audio data, a phoneme confusion matrix as shown in Table 1 can be constructed to record the similar phonemes of the phoneme and the similarity between the phoneme and its similar phonemes.

[0095] S103. Construct a multi-pronunciation dictionary according to the rule that the word is stored corresponding to its standard pronunciation and similar pronunciation; wherein, the standard pronunciation is composed of the phonemes in the phoneme label, and the similar pronunciation is composed of the similar phonemes of the phonemes in the phoneme label.

[0096] Specifically, the above-mentioned word refers to each word in the pronunciation dictionary, and the specific form of the word in this pronunciation dictionary varies according to the language to which this pronunciation dictionary belongs. For example, in the pronunciation dictionary of the Chinese language, the above-mentioned word can specifically be a single Chinese character or a word composed of multiple Chinese characters; while in the pronunciation dictionary of the English language, the above-mentioned word can specifically be an English word or an English letter.

[0097] In a conventional pronunciation dictionary, a word corresponds to only one pronunciation. Then, during speech recognition, only when the pronunciation x composed of the phonemes of the speech to be recognized strictly matches the pronunciation x in the pronunciation dictionary, can it be confirmed that the text of the speech to be recognized is the text corresponding to the pronunciation x in the pronunciation dictionary; otherwise, the speech to be recognized cannot be recognized as the text corresponding to the pronunciation x in the pronunciation dictionary.

[0098] Unlike the conventional pronunciation dictionary described above, the embodiment of the present application expands the pronunciation dictionary and constructs a multi-pronunciation dictionary according to the rules for storing the corresponding standard pronunciation and similar pronunciations of words. It can be understood that in this multi-pronunciation dictionary, a word can correspond to two or more pronunciations. Among the two or more pronunciations, there are both standard pronunciations corresponding to the word and similar pronunciations corresponding to the word. The number of similar pronunciations corresponding to a word can be one or more.

[0099] The standard pronunciation is the pronunciation composed of the phonemes in the phoneme labels of the target audio data. Since the target audio data contains all phonemes, the correct phonemes corresponding to any word in the pronunciation dictionary can be found from the phoneme labels of the target audio data, thereby constructing the standard pronunciation of the word from the phonemes in the phoneme labels.

[0100] The similar pronunciations are pronunciations composed of similar phonemes of the phonemes in the phoneme labels. The similar phonemes of the phonemes in the phoneme labels can be described with reference to the above embodiments.

[0101] For example, for the word "ready" in the pronunciation dictionary, according to the phoneme labels of the target speech data and the phonemes in the phoneme labels, the pronunciation of the word is "zhun3bei4", which contains the phonemes "zh", "un", "b", and "ei" in that order. Therefore, "zhun3bei4" is the standard pronunciation corresponding to the word "ready".

[0102] Referring to the above content, assuming that the phoneme "zh" has a similar phoneme "z", then the similar phoneme "z" can be used to form the similar pronunciation "zun3 bei4" of the word "prepare"; at the same time, assuming that the phoneme "ei" has a similar phoneme "en", then the similar phoneme "en" can be used to form the similar pronunciation "zhun3 ben4" of the word "prepare".

[0103] On this basis, the word "prepare" is stored correspondingly with its corresponding standard pronunciation "zhun3 bei4" and similar pronunciations "zun3bei4" and "zhun3 ben4". Then, in this multi-pronunciation dictionary, the word "prepare" corresponds to three pronunciations, namely "zhun3 bei4", "zun3 bei4" and "zhun3 ben4".

[0104] As a preferred embodiment, in the above-mentioned multi-pronunciation dictionary, the similar pronunciations corresponding to a word can specifically be the similar pronunciations whose similarity to the standard pronunciation corresponding to the word is greater than a set threshold. For example, for a certain word X, the word X may contain multiple phonemes, and each phoneme has multiple similar phonemes respectively. In this way, multiple phonemes and the similar phonemes of multiple phonemes can form multiple similar pronunciations corresponding to the standard pronunciation of the word X. Moreover, the combination of the similar phonemes of multiple phonemes can derive a larger number of pronunciations than the number of phonemes. However, due to the difference in similarity between phonemes, the similarity between the pronunciations composed of phonemes will also vary. Some similar pronunciations have a very low similarity to the standard pronunciation. For example, each phoneme of the similar pronunciation has a relatively low similarity to each phoneme of the standard pronunciation, which will result in a low similarity between the similar pronunciation and the standard pronunciation.

[0105] If a large number of similar pronunciations with a low similarity to the marked pronunciation are recorded in the multi-pronunciation dictionary, it will lead to a decline in the speech recognition effect. For example, for the word X, if the user casually says a word similar in pronunciation to the word X, such as the user says the word Y, and if the pronunciation of the word Y is the same as a pronunciation with a low similarity to the standard pronunciation of the word X, then at this time, the word Y will be recognized as the word X, which is obviously incorrect.

[0106] Therefore, in order to ensure the speech recognition effect, for the similar pronunciations corresponding to a word in the multi-pronunciation dictionary, screening conditions can be set. The specific screening conditions can be flexibly set according to actual needs. For example, they can be set according to the speech recognition accuracy, recognition rate, etc. Specifically, it can be set that only the similar pronunciations whose similarity exceeds the set similarity threshold can be recorded in the multi-pronunciation dictionary, or for a word, only select the several similar pronunciations with the highest similarity to its standard pronunciation and record them in the multi-pronunciation dictionary, and so on. The specific screening of similar pronunciations can be flexibly implemented under the above technical concept.

[0107] After constructing the above-mentioned multi-pronunciation dictionary, a decoding network is reconstructed based on this multi-pronunciation dictionary for speech recognition, and then speech decoding can be realized using the above-mentioned multi-pronunciation dictionary.

[0108] It can be understood that based on the above-mentioned multi-pronunciation dictionary, when the phoneme extraction of the speech to be recognized is inaccurate, or due to the speaker's accent or due to the complex scene resulting in the speech to be recognized being not clear and accurate enough to affect the accuracy of phoneme extraction, even if the phoneme extraction result of the speech to be recognized is different from the standard pronunciation of the speech to be recognized, for example, the extracted phonemes form a similar pronunciation of the speech to be recognized, the correct text can still be recognized.

[0109] For example, suppose that the word "prepare" is included in a sentence of the user's speech, but due to the complexity of the scene or the limited performance of the acoustic model, the extracted phoneme corresponding to the word "prepare" is "zun b ei", and the pronunciation of the speech corresponding to the word "prepare" based on the tone is "zun3 bei4". At this time, although the phoneme extraction of the word "prepare" is not accurate enough, based on the above-mentioned multi-pronunciation dictionary, since the pronunciation "zun3 bei4" corresponding to the word "prepare" is stored in the multi-pronunciation dictionary, "zun3 bei4" can be recognized as the word "prepare", and the correct recognition result is obtained.

[0110] From the above introduction, it can be seen that in the multi-pronunciation dictionary constructed based on the pronunciation dictionary construction method proposed in the embodiment of the present application, one word corresponds to multiple pronunciations, specifically the standard pronunciation of the word, and the similar pronunciation of the word. When performing speech recognition based on the pronunciation dictionary, even if the phoneme extraction of the speech to be recognized is inaccurate, such as predicting the pronunciation of the speech to be recognized as a similar pronunciation rather than an accurate pronunciation, the correct text can be recognized based on the multi-pronunciation dictionary. Therefore, the multi-pronunciation dictionary constructed by the technical solution of the embodiment of the present application can improve the fault tolerance and robustness of speech recognition, thereby improving the speech recognition effect, and can achieve better recognition effects even in complex scenarios.

[0111] As an optional implementation, the above-mentioned "building a multi-pronunciation dictionary according to the rules of storing the corresponding standard pronunciation and similar pronunciations of the word" can be achieved by performing the following steps A1-A2:

[0112] A1. Determine the standard pronunciation corresponding to the word in the pronunciation dictionary, and determine the similar pronunciation corresponding to the word in the pronunciation dictionary based on the similar phonemes of the phonemes in the phoneme tags.

[0113] Specifically, for a word in the pronunciation dictionary, its standard pronunciation is determined, specifically, the various phonemes included in its pronunciation are determined. Then, for the phoneme in the standard pronunciation corresponding to the word, a similar phoneme is selected from similar phonemes to replace the phoneme, thereby obtaining a similar pronunciation of the standard pronunciation, that is, a similar pronunciation corresponding to the word.

[0114] For example, for the word "ready," we first determine its standard pronunciation, namely "zhun3bei4." Then, based on the similar phonemes in the phoneme labels described above, we replace "zhun3bei4" with similar phonemes to obtain a similar pronunciation. Specifically, based on the similar phonemes in the phoneme labels described above, we determine similar phonemes in the standard pronunciation of the word "ready," and then replace them with similar phonemes to obtain a similar pronunciation of the word "ready." For example, replacing the phoneme "zh" in "zhun3bei4" with its similar phoneme "z" yields a similar pronunciation of "zun3bei4."

[0115] As a preferred implementation, in order to minimize the negative impact of modifying the existing pronunciation dictionary, similar pronunciation expansion may be performed only on high-frequency and easily misinterpreted words in the pronunciation dictionary.

[0116] That is, when executing the above step A1, the standard pronunciation corresponding to the high-frequency and easily mistaken words in the pronunciation dictionary is determined respectively, and, based on the similar phonemes of the phonemes in the phoneme labels, the similar pronunciation corresponding to the high-frequency and easily mistaken words in the pronunciation dictionary is determined respectively.

[0117] Specifically, the speech recognition baseline system was tested using the development dataset to identify words with high recognition error rates, often referred to as frequently misinterpreted words. Then, for these frequently misinterpreted words, the standard pronunciation and similar pronunciations were determined, referring to the above description.

[0118] As another optional implementation, when determining the similar pronunciation of a high-frequency error-prone word, the pronunciation of the incorrect recognition result of the high-frequency error-prone word can be directly used as the similar pronunciation of the high-frequency error-prone word.

[0119] Specifically, the speech recognition baseline system was tested using the development dataset to identify frequently misidentified word pairs with high recognition error rates. These frequently misidentified word pairs specifically consisted of misidentified words and the actual recognition results for those words when they were misidentified.

[0120] For example, if the speech recognition baseline system frequently misidentifies the word "ready" as "quasi-ben," then it can be determined that "ready" and "quasi-ben" are a frequently misidentified word pair. For the word "ready," when determining its similar pronunciation, the pronunciation of "quasi-ben" (zhun3ben4) can be directly identified as a similar pronunciation of "ready."

[0121] A2. A multi-pronunciation dictionary is obtained by storing the word in correspondence with the standard pronunciation and similar pronunciation corresponding to the word.

[0122] Specifically, for the words in the pronunciation dictionary, storing the word, as well as the corresponding standard pronunciation and similar pronunciations of the word can achieve the expansion of the pronunciation dictionary and obtain a multi-pronunciation dictionary.

[0123] When step A1 determines the specific standard pronunciation and similar pronunciations corresponding to the words in the pronunciation dictionary as the standard pronunciation and similar pronunciations corresponding to the frequently mispronounced and error-prone words in the pronunciation dictionary, step A2 specifically stores the frequently mispronounced and error-prone words in the pronunciation dictionary and their corresponding standard pronunciation and similar pronunciations to obtain a multi-pronunciation dictionary, that is, only the frequently mispronounced and error-prone words in the pronunciation dictionary are expanded in terms of pronunciation.

[0124] Expanding only the pronunciation of the frequently mispronounced and error-prone words in the pronunciation dictionary can specifically improve the recognition effect of speech recognition for frequently mispronounced and error-prone words and avoid the negative impact on words with relatively high original recognition accuracy due to multi-pronunciation expansion.

[0125] As a more preferred implementation, when constructing a multi-pronunciation dictionary by storing the frequently mispronounced and error-prone words in the pronunciation dictionary and their corresponding standard pronunciation and similar pronunciations, the similar pronunciations of the frequently mispronounced and error-prone words can also be screened. The specific processing content is shown in steps B1 - B3 as follows:

[0126] B1. Calculate the score of the similar pronunciation corresponding to the frequently mispronounced and error-prone word according to the similar pronunciation corresponding to the frequently mispronounced and error-prone word in the pronunciation dictionary and the recognition probability of each phoneme in the phoneme set.

[0127] Specifically, as described above, when extracting phonemes from the target audio data, not only can the phoneme set be obtained, but also the recognition probability of each phoneme in the phoneme set can be determined. Furthermore, according to the corresponding relationship between each phoneme in the phoneme set and the phonemes in the phoneme label of the target audio data, and the recognition probability of each phoneme in the phoneme set, the similar phonemes corresponding to the phonemes in the phoneme label can be determined, and the similarity between the phonemes in the phoneme label and their similar phonemes can be determined.

[0128] Based on the above processing, for the similar pronunciation corresponding to the frequently mispronounced and error-prone word in the pronunciation dictionary, the score of the similar pronunciation corresponding to the frequently mispronounced and error-prone word can be calculated according to the similarity between the phonemes in the similar pronunciation and the phonemes in the standard pronunciation.

[0129] Exemplarily, calculating the arithmetic mean of the similarity between each phoneme in the similar pronunciation of the frequently mispronounced and error-prone word and the corresponding phoneme in the standard pronunciation of the frequently mispronounced and error-prone word can be used as the score of the similar pronunciation of the frequently mispronounced and error-prone word.

[0130] Taking the word "prepare" as an example, its standard pronunciation is "zhun3bei4". For its similar pronunciation "zun3bei4", assuming that the similarity between the phoneme "z" and the phoneme "zh" is 0.8, and it can be determined that the other phonemes of "zhun3bei4" and "zun3bei4" are the same, and the similarity is 1, then the score of the similar pronunciation "zun3bei4" can be determined to be (0.8+1+1+1) / 4=0.95.

[0131] According to the above method, the score of each similar pronunciation of the high-frequency error-prone words can be calculated.

[0132] B2. Based on the scores of the similar pronunciations corresponding to the high-frequency and easily mistaken words, a similar pronunciation with a score higher than a set score threshold is selected from the similar pronunciations corresponding to each high-frequency and easily mistaken word as the target similar pronunciation.

[0133] Specifically, from the similar pronunciations corresponding to the high-frequency error-prone words, a similar pronunciation with a score higher than a set score threshold is selected as the target similar pronunciation corresponding to the high-frequency error-prone words.

[0134] B3. Correspondingly storing the high-frequency error-prone words in the pronunciation dictionary, the standard pronunciation corresponding to the high-frequency error-prone words, and the target similar pronunciation to obtain a multi-pronunciation dictionary.

[0135] Specifically, when the pronunciation of a high-frequency error-prone word in the pronunciation dictionary is expanded, the standard pronunciation and the target similar pronunciation of the high-frequency error-prone word are stored corresponding to the high-frequency error-prone word.

[0136] According to the processing described in B1-B3 above, target similar pronunciations are selected for each high-frequency and easily mispronounced word in the pronunciation dictionary, and pronunciation expansion is performed to obtain the final multi-pronunciation dictionary.

[0137] The processing of the above steps B1-B3 realizes the screening of similar pronunciations of high-frequency and easily mistaken words. This operation can ensure that the similar pronunciations of high-frequency and easily mistaken words in the multi-pronunciation dictionary are pronunciations with a high degree of similarity to the standard pronunciations of high-frequency and easily mistaken words, thereby avoiding the influence of a large number of similar pronunciations on the recognition effect of other non-high-frequency and easily mistaken words.

[0138] Based on the above-mentioned pronunciation dictionary construction method, another embodiment of the present application also proposes a speech recognition method, see Figure 2 As shown, the method includes:

[0139] S201: Obtain a phoneme sequence of a speech to be recognized.

[0140] Specifically, a phoneme extraction process is performed on the speech to be recognized to obtain a phoneme sequence of the speech to be recognized.

[0141] Exemplarily, the speech to be recognized or the acoustic features of the speech to be recognized are input into a pre-trained acoustic model for phoneme extraction, and a phoneme sequence of the speech to be recognized output by the acoustic model can be obtained.

[0142] The above-mentioned acoustic model is a pre-trained model for phoneme extraction of audio data, and the specific training process can be referred to the introduction of the following embodiments.

[0143] S202. Determine the speech recognition result of the speech to be recognized according to the phoneme sequence of the speech to be recognized and a pre-constructed multiple pronunciation dictionary.

[0144] Wherein, the words in the multiple pronunciation dictionary are stored corresponding to the standard pronunciation and similar pronunciations of the words; the standard pronunciation corresponding to a word is constructed by the phonemes in the phoneme label corresponding to the target audio data, and the similar pronunciation corresponding to a word is constructed by the similar phonemes of the phonemes in the phoneme label; the target audio data includes the audio data corresponding to the word.

[0145] Specifically, the above-mentioned multiple pronunciation dictionary is a multiple pronunciation dictionary constructed by the pronunciation dictionary construction method introduced in any of the above embodiments. The specific construction process and the specific content of the multiple pronunciation dictionary can both be referred to the introduction of the embodiments of the pronunciation dictionary construction method above, and will not be repeated here.

[0146] Based on the above multiple pronunciation dictionary, when the phoneme sequence of the speech to be recognized is obtained, the phoneme sequence is decoded based on the multiple pronunciation dictionary to obtain the corresponding text. Exemplarily, by matching the phoneme sequence with the pronunciations in the multiple pronunciation dictionary, the speech recognition result of the speech to be recognized is determined.

[0147] For example, when a certain phoneme string in the phoneme sequence of the speech to be recognized matches a certain pronunciation in the multiple pronunciation dictionary, it can be determined that the text corresponding to the phoneme string is the text corresponding to the pronunciation in the multiple pronunciation dictionary.

[0148] Since the words in the multiple pronunciation dictionary not only correspond to their standard pronunciations but also correspond to their similar pronunciations, therefore, when the phoneme extraction of the speech to be recognized is inaccurate, or when the speech to be recognized is not clear and accurate enough due to the speaker having an accent or due to a complex scene, which affects the accuracy of phoneme extraction, even if the phoneme extraction result of the speech to be recognized is different from the standard pronunciation of the speech to be recognized, for example, the extracted phonemes form a similar pronunciation of the speech to be recognized, the correct text can still be recognized.

[0149] As can be seen from the above introduction, the speech recognition method proposed in the embodiments of the present application decodes the phoneme sequence of the speech to be recognized based on a pre-constructed multi-pronunciation dictionary to obtain a speech recognition result. The words in the pronunciation dictionary are stored corresponding to the standard pronunciation and similar pronunciations of the word. When decoding the phoneme sequence of the speech to be recognized based on this pronunciation dictionary, even if the phoneme extraction of the speech to be recognized is inaccurate, for example, the pronunciation of the speech to be recognized is predicted as a similar pronunciation rather than the accurate pronunciation, based on this multi-pronunciation dictionary, the correct text can still be recognized. Therefore, the speech recognition method based on a multi-pronunciation dictionary proposed in the embodiments of the present application can improve the fault tolerance and robustness of speech recognition, and thus can improve the speech recognition effect, and can achieve good recognition results even in complex scenarios.

[0150] Next, the training process of the above acoustic model will be introduced.

[0151] First of all, it should be noted that the acoustic model training scheme proposed in the embodiments of the present application is applicable to the acoustic model involved in any of the above embodiments, especially applicable to the acoustic model used when the pronunciation dictionary construction method proposed in the above embodiments inputs the target audio data into a pre-trained acoustic model for phoneme extraction, and at the same time, applicable to the acoustic model used when the speech recognition method proposed in the above embodiments inputs the speech to be recognized into a pre-trained acoustic model for phoneme extraction.

[0152] As a preferred implementation manner, an acoustic model can be trained according to the model training method introduced in the embodiments of the present application, and then the acoustic model can be used respectively in the pronunciation dictionary construction method introduced in the above embodiments and the speech recognition method introduced in the above embodiments. Specifically, the acoustic model is used in the pronunciation dictionary construction method introduced in the above embodiments to perform phoneme extraction on the target audio data to achieve the purpose of constructing a multi-pronunciation dictionary. Then, based on the constructed multi-pronunciation dictionary, the acoustic model is used again in the speech recognition method introduced in the above embodiments to perform phoneme extraction on the speech to be recognized, and thus the recognition of the speech to be recognized is realized.

[0153] Before introducing the acoustic model training scheme proposed in the embodiments of the present application, a conventional acoustic model training scheme will be briefly introduced:

[0154] An acoustic model generally uses a deep neural network for modeling, and the loss function used in its training is a cross-entropy loss function, and the specific form is as follows:

[0155]

[0156] Among them, x represents the spectral feature vector of the speech (for example: FilterBank, MFCC, etc.), and y t represents the phoneme label corresponding to the t-th moment.

[0157] To solve the accuracy problem of acoustic model classification in complex scenarios, the current mainstream solution is to conduct research on training data, mainly by collecting a large number of real audio in complex scenarios, or generating a large number of data that match the target scenario distribution through machine simulation, and then adding them to the training set for mixed training.

[0158] In addition, there are certain limitations in the cross-entropy training criterion for acoustic model training. Because the cross-entropy criterion only focuses on whether the current predicted value is consistent with the true target value, it is a maximum likelihood training method; when the model prediction output is inconsistent with the true target, the training cost is the same. This training scheme has limitations for acoustic model training, especially in terms of improving the robustness of the model. For example, in Chinese acoustic model modeling, when the true phoneme label is a1, and the model prediction outputs a2 and z1, the training costs are the same. However, when the target phoneme a1 is recognized as a2, users generally do not have much difficulty in understanding the sentence meaning, while when the target phoneme a1 is recognized as z1, it generally leads to a large deviation in the sentence meaning. Therefore, the acoustic model training scheme based solely on the cross-entropy loss function will lead to a significant decline in the robustness of the model.

[0159] Combined Figure 3 As shown, the acoustic model training scheme proposed in the embodiments of the present application mainly includes the following steps C1-C3:

[0160] C1. Input the audio features of the training speech into the acoustic model to obtain the phoneme information of the training speech.

[0161] Specifically, as shown in Figure 3 , input the audio features x t＝1,…T of the training speech into the acoustic model ASR-Net. This acoustic model ASR-Net can be any type of acoustic model, such as LSTM, TDNN, and DFSMN, etc.

[0162] Based on the input acoustic features, this acoustic model extracts the phoneme information H txt corresponding to the audio. This phoneme information is the information content output by the last hidden layer of the acoustic model. Most of this information content is audio content information, and information such as the speaker and channel in the audio has been weakened. Therefore, the phoneme information extracted by the acoustic model can generally be understood as the semantic information of the audio. Therefore, the phoneme information H txt extracted by the acoustic model ASR-Net can be equivalent to the semantic feature information of the training speech.

[0163] C2. Input the phoneme information of the training speech and the non-semantic information extracted according to the audio features of the training speech into the audio synthesis model to obtain the audio synthesis result.

[0164] Specifically, the above audio synthesis model is specifically a Text To Speech (TTS) network, TTS-Net, which can synthesize speech based on semantic information corresponding to the speech, speaker information, environmental information, etc. For the specific functions and structures of the TTS network, reference can be made to the introduction of conventional TTS networks.

[0165] Based on the above audio synthesis model, in the embodiment of the present application, the phoneme information of the extracted training speech and the non-semantic information extracted according to the audio features of the training speech are input into the audio synthesis model to obtain the audio synthesis result output by the audio synthesis model.

[0166] The above non-semantic information extracted according to the audio features of the training speech refers to the non-semantic information such as the speaker and environmental factors included in the training speech extracted through the audio features of the training speech.

[0167] As an exemplary implementation, see Figure 3 As shown, inputting the audio features of the training speech into a Long Short-Term Memory (LSTM) network can extract the non-semantic information of the training speech from it. In actual implementation, a Convolutional Neural Network (CNN) or other neural networks based on the attention mechanism can also be used to extract the non-semantic information of the training speech.

[0168] Then, the phoneme information of the training speech and the non-semantic information of the training speech are input into the audio synthesis model TTS-Net to obtain the audio synthesis result output by the audio synthesis model.

[0169] It should be noted that the above audio synthesis model is an audio synthesis model with ideal effects, that is, in this ideal state, inputting the semantic information of the training speech, that is, phoneme information, and the non-semantic information of the training speech into the audio synthesis model, the audio synthesis model can accurately synthesize the training speech, that is, the training speech can be accurately restored based on the semantic information and non-semantic information of the training speech.

[0170] As a preferred implementation manner, in order to prevent the audio synthesis model TTS-Net from directly copying the input semantic information and non-semantic information, in the embodiment of the present application, downsampling is performed on the phoneme information of the training speech and the audio features of the training speech, and then the audio synthesis model performs audio synthesis based on the downsampled information.

[0171] Specifically, see Figure 3As shown, the phoneme information of the training speech and the audio features of the training speech are downsampled at the same scale. In an exemplary embodiment of the present application, the phoneme information and the audio features of the training speech are respectively downsampled by R frames, that is, the average value of every R-frame data is used as the value after sampling. For example, an average value is taken every 4 frames as the sampling value.

[0172] If it is single-frame prediction, when R = 1, refer to Figure 3 As shown, noise needs to be added to the audio features of the training speech input to the LSTM network.

[0173] Taking the phoneme information H of the training speech txt as an example, it is downsampled by R frames. Specifically, the average value of every R-frame phoneme information h txt,1 , …, h txt,R is calculated to obtain an average value AVG(h txt,1 , …, h txt,R ), where where K represents taking topK, and α k represents the posterior probability corresponding to the top-k label of the current frame, and W k represents the embedding or connection weight of the top-k node.

[0174] Then, according to the downsampled audio features of the training speech, the non-semantic information of the training speech is extracted.

[0175] As an exemplary implementation method, refer to Figure 3 As shown, the downsampled audio features of the training speech are input into a long short-term memory (LSTM) network, and the non-semantic information of the training speech can be extracted therefrom. In actual implementation, a convolutional neural network CNN or other neural networks based on the attention mechanism can also be used to extract the non-semantic information of the training speech.

[0176] Finally, the downsampled phoneme information and the non-semantic information are fused to obtain the basic information for audio synthesis; the basic information for audio synthesis is input into an audio synthesis model to obtain an audio synthesis result.

[0177] Exemplarily, refer to Figure 3 As shown, after the non-semantic information of the training speech is encoded by code, it is concatenated with the downsampled phoneme information of the training speech, and the concatenated information obtained is used as the basic information for audio synthesis. Then, the basic information for audio synthesis is input into an audio synthesis model TTS-Net for audio synthesis to obtain an audio synthesis result.

[0178] As an alternative implementation, for the above-mentioned fusion of the downsampled phoneme information and non-semantic information, in addition to directly concatenating using the concat method, it can also be achieved through network fusion or by means of an attention mechanism.

[0179] C3. Correct the parameters of the acoustic model according to the speech recognition loss of the acoustic model and the audio synthesis loss of the audio synthesis model.

[0180] Specifically, the speech recognition loss of the acoustic model can be the loss between the phonemes extracted by the acoustic model from the training speech and the phoneme labels of the training speech, or the loss between the speech recognition result determined based on the phoneme extraction result of the acoustic model and the text labels of the training speech. This loss can be determined by an exemplary cross-entropy loss function, that is, the speech recognition loss of the acoustic model is the cross-entropy loss CE-loss.

[0181] The above-mentioned audio synthesis loss of the audio synthesis model is specifically the loss between the audio synthesis result output by the audio synthesis model and the training speech. This loss can be determined by an exemplary mean squared error loss function, that is, the audio synthesis loss of the audio synthesis model is the mean squared error loss MSE-loss. In addition, the audio synthesis loss of the audio synthesis model can also be the MDN-loss.

[0182] Assume that the speech recognition loss of the acoustic model is L CE , and the audio synthesis loss of the audio synthesis model is L TTS , then the final training loss L SUM of the acoustic model is the sum of the two, that is: L SUM = L CE + αL TTS , where α is a hyperparameter.

[0183] Then, based on the above-mentioned training loss L SUM of the acoustic model, correct the parameters of the acoustic model.

[0184] It can be understood that when training the acoustic model according to the above scheme, it not only includes the loss of the acoustic model itself but also considers the loss of the audio synthesis model.

[0185] Based on the characteristics of the audio synthesis model TTS-Net network, it can be known that when the TTS network synthesizes audio, it is through the text features of the speech (corresponding to H on the left in the above figure txt) and the voice code information (including information such as the speaker, environment, etc.) for audio synthesis. Since information such as the speaker and environmental factors are steady-state information in a piece of voice, the information from the previous moment or the previous N moments can be directly reused for the synthesis of the current voice frame, while the corresponding text information is time-varying information and needs to correspond to the current moment one by one. From Figure 3 it can be seen that the semantic features come from the predicted output of the acoustic model ASR-Net network. In order to make the synthesis more accurate, the output of the ASR-Net network needs to be more accurate.

[0186] At the same time, the TTS-Net network is trained in a way of probability likelihood. The accuracy of its output is linearly related to the accuracy of the input content. That is, the greater the difference between the input content and the standard content, the greater the difference between its output voice and the standard voice, that is, the greater its loss; the smaller the difference between the input content and the standard content, the smaller the difference between its output voice and the standard voice, that is, the smaller its loss.

[0187] Therefore, for the TTS-Net network, different types of errors result in different costs calculated by the corresponding training criteria, that is: the greater the difference between the predicted output of the ASR-Net network and the target label, the greater the loss cost of the corresponding TTS-Net network, and correspondingly, the greater the training loss of the acoustic model; conversely, the smaller the difference between the predicted output of the ASR-Net network and the target label, the smaller the loss cost of the corresponding TTS-Net network, and correspondingly, the smaller the training loss of the acoustic model. This achieves the purpose of differential error punishment for the predicted output of the ASR-Net network. After convergence through multiple iterative trainings, when the acoustic model ASR-Net makes a mistake in phoneme prediction, it will only make a mistake in phonemes similar to the current true label.

[0188] It can be seen that based on the addition of the above audio synthesis model and the above audio synthesis loss, the acoustic model training scheme proposed in the embodiments of this application can perform differential punishment on model classification errors, that is: the closer the classification error is to the target phoneme in pronunciation, the smaller the punishment, and the farther the classification error is from the target phoneme in pronunciation, the greater the punishment; through the audio synthesis model to assist in training the acoustic model, when performing complex-scene audio tests, even if the acoustic model prediction makes a mistake, it will predict the same phoneme but there may be a difference in its corresponding pitch. That is, the acoustic model prediction obtained by training through the training scheme proposed in the embodiments of this application basically will not make outrageous mistakes, thus improving the speech recognition effect.

[0189] The acoustic model trained based on the above method, combined with the multi-pronunciation dictionary constructed by the pronunciation dictionary construction method proposed in the above embodiments of the present application, and the corresponding speech recognition method, can systematically and significantly improve the fault tolerance and robustness of speech recognition, and significantly improve the speech recognition effect.

[0190] Corresponding to the above pronunciation dictionary construction method, another embodiment of the present application also proposes a pronunciation dictionary construction device, as shown in Figure 4 shown, the device includes:

[0191] A phoneme extraction unit 001, configured to extract phonemes from target audio data to obtain a corresponding phoneme set; the target audio data is audio data covering all phonemes;

[0192] A phoneme screening unit 002, configured to determine similar phonemes of the phonemes in the phoneme label from the phoneme set according to the recognition probability of each phoneme in the extracted phoneme set and the phoneme label corresponding to the target audio data; wherein, the similar phonemes of the phonemes in the phoneme label refer to a set number of phonemes with the highest recognition probability selected from the phoneme set corresponding to the phonemes in the phoneme label;

[0193] A dictionary construction unit 003, configured to construct a multi-pronunciation dictionary according to the rule of storing words corresponding to the standard pronunciation and similar pronunciation of the words; wherein, the standard pronunciation is composed of the phonemes in the phoneme label, and the similar pronunciation is composed of the similar phonemes of the phonemes in the phoneme label.

[0194] As an optional implementation manner, constructing a multi-pronunciation dictionary according to the rule of storing words corresponding to the standard pronunciation and similar pronunciation of the words includes:

[0195] Determining the standard pronunciation corresponding to the words in the pronunciation dictionary, and determining the similar pronunciation corresponding to the words in the pronunciation dictionary according to the similar phonemes of the phonemes in the phoneme label;

[0196] Storing the words corresponding to the standard pronunciation and similar pronunciation of the words to obtain a multi-pronunciation dictionary.

[0197] As an optional implementation manner, determining the standard pronunciation corresponding to the words in the pronunciation dictionary, and determining the similar pronunciation corresponding to the words in the pronunciation dictionary according to the similar phonemes of the phonemes in the phoneme label includes:

[0198] Respectively determining the standard pronunciation corresponding to the high-frequency error-prone words in the pronunciation dictionary, and respectively determining the similar pronunciation corresponding to the high-frequency error-prone words in the pronunciation dictionary according to the similar phonemes of the phonemes in the phoneme label;

[0199] By storing the corresponding standard pronunciations and similar pronunciations of words, a multi - pronunciation dictionary is obtained, including:

[0200] By storing the corresponding standard pronunciations and similar pronunciations of high - frequency error - prone words in the pronunciation dictionary, a multi - pronunciation dictionary is obtained.

[0201] As an alternative implementation, by storing the corresponding standard pronunciations and similar pronunciations of high - frequency error - prone words in the pronunciation dictionary, a multi - pronunciation dictionary is obtained, including:

[0202] According to the similar pronunciations corresponding to the high - frequency error - prone words in the pronunciation dictionary and the recognition probabilities of each phoneme in the phoneme set, the scores of the similar pronunciations corresponding to the high - frequency error - prone words are calculated;

[0203] According to the scores of the similar pronunciations corresponding to the high - frequency error - prone words, the similar pronunciations with scores higher than the set score threshold are selected from the similar pronunciations corresponding to each high - frequency error - prone word as the target similar pronunciations;

[0204] The high - frequency error - prone words in the pronunciation dictionary are stored corresponding to their standard pronunciations and target similar pronunciations to obtain a multi - pronunciation dictionary.

[0205] As an alternative implementation, the extracting phonemes from the target audio data to obtain the corresponding phoneme set includes:

[0206] Inputting the target audio data into a pre - trained acoustic model for phoneme extraction to obtain a phoneme set corresponding to the target audio data.

[0207] As an alternative implementation, the acoustic model is trained as follows:

[0208] Inputting the audio features of the training speech into the acoustic model to obtain the phoneme information of the training speech;

[0209] Inputting the phoneme information of the training speech and the non - semantic information extracted according to the audio features of the training speech into an audio synthesis model to obtain an audio synthesis result;

[0210] According to the speech recognition loss of the acoustic model and the audio synthesis loss of the audio synthesis model, the parameters of the acoustic model are corrected.

[0211] As an alternative implementation, inputting the phoneme information of the training speech and the non - semantic information extracted according to the audio features of the training speech into an audio synthesis model to obtain an audio synthesis result includes:

[0212] Downsample the phoneme information of the training speech and the audio features of the training speech at the same scale;

[0213] Extract the non-semantic information of the training speech based on the audio features of the downsampled training speech;

[0214] Fuse the downsampled phoneme information and the non-semantic information to obtain the basic audio synthesis information;

[0215] Input the basic audio synthesis information into an audio synthesis model to obtain an audio synthesis result.

[0216] As an alternative implementation, the speech recognition loss of the acoustic model is determined by a cross-entropy loss function, and the audio synthesis loss of the audio synthesis model is determined by a mean squared error loss function.

[0217] Specifically, for the specific working content of each unit of the above pronunciation dictionary construction device, please refer to the processing content of the corresponding processing steps of the above pronunciation dictionary construction method, which will not be repeated here.

[0218] Corresponding to the above speech recognition method, another embodiment of the present application also proposes a speech recognition device, as shown in Figure 5 The device includes:

[0219] A speech processing unit 010, configured to obtain a phoneme sequence of a speech to be recognized;

[0220] A speech recognition unit 011, configured to determine a speech recognition result of the speech to be recognized according to the phoneme sequence of the speech to be recognized and a pre-constructed multiple pronunciation dictionary;

[0221] Wherein, words in the multiple pronunciation dictionary are stored corresponding to the standard pronunciation and similar pronunciations of the word; the standard pronunciation corresponding to the word is constructed from phonemes in the phoneme label corresponding to the target audio data, and the similar pronunciation corresponding to the word is constructed from similar phonemes of the phonemes in the phoneme label; the target audio data includes the audio data corresponding to the word.

[0222] As an alternative implementation, the multiple pronunciation dictionary is constructed according to the above pronunciation dictionary construction method.

[0223] As an alternative implementation, the obtaining of the phoneme sequence of the speech to be recognized includes:

[0224] Input the speech to be recognized into a pre-trained acoustic model for phoneme extraction to obtain the phoneme sequence of the speech to be recognized.

[0225] As an alternative implementation, the acoustic model is trained as follows:

[0226] Input the audio features of the training speech into the acoustic model to obtain the phoneme information of the training speech;

[0227] Input the phoneme information of the training speech and the non-semantic information extracted according to the audio features of the training speech into the audio synthesis model to obtain an audio synthesis result;

[0228] Perform parameter correction on the acoustic model according to the speech recognition loss of the acoustic model and the audio synthesis loss of the audio synthesis model.

[0229] As an alternative implementation, inputting the phoneme information of the training speech and the non-semantic information extracted according to the audio features of the training speech into the audio synthesis model to obtain an audio synthesis result includes:

[0230] Perform downsampling of the same scale on the phoneme information of the training speech and the audio features of the training speech;

[0231] Extract the non-semantic information of the training speech according to the downsampled audio features of the training speech;

[0232] Fuse the downsampled phoneme information and the non-semantic information to obtain basic audio synthesis information;

[0233] Input the basic audio synthesis information into the audio synthesis model to obtain an audio synthesis result.

[0234] As an alternative implementation, the speech recognition loss of the acoustic model is determined by a cross-entropy loss function, and the audio synthesis loss of the audio synthesis model is determined by a mean squared error loss function.

[0235] Specifically, for the specific working content of each unit of the above speech recognition device and the specific content of the above acoustic model training process, please refer to the processing content of the corresponding processing steps of the above speech recognition method, which will not be repeated here.

[0236] Another embodiment of the present application also proposes an electronic device. Refer to Figure 6 as shown. The device includes:

[0237] A memory 200 and a processor 210;

[0238] Among them, the memory 200 is connected to the processor 210 and is used to store programs;

[0239] The processor 210 is configured to implement the pronunciation dictionary building method disclosed in any of the above embodiments or the speech recognition method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0240] Specifically, the above electronic device may further include: a bus, a communication interface 220, an input device 230, and an output device 240.

[0241] The processor 210, the memory 200, the communication interface 220, the input device 230, and the output device 240 are interconnected via the bus. Among them:

[0242] The bus may include a path for transmitting information between various components of the computer system.

[0243] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0244] The processor 210 may include a main processor, and may also include a baseband chip, a modem, etc.

[0245] The memory 200 stores a program for implementing the technical solution of the present invention, and may also store an operating system and other critical services. Specifically, the program may include program code, and the program code includes computer operation instructions. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk memory, a flash memory, etc.

[0246] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor, etc.

[0247] The output device 240 may include a device for allowing information to be output to the user, such as a display screen, a printer, a speaker, etc.

[0248] The communication interface 220 may include a device such as any transceiver to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc.

[0249] The processor 210 executes the programs stored in the memory 200 and calls other devices, which can be used to implement the steps of the pronunciation dictionary construction method provided in the above embodiments of the present application, or to implement the steps of the speech recognition method provided in the above embodiments of the present application.

[0250] Another embodiment of the present application also provides a storage medium, on which a computer program is stored. When the computer program is run by a processor, it implements the steps of the pronunciation dictionary construction method provided in the above embodiments of the present application, or is used to implement the steps of the speech recognition method provided in the above embodiments of the present application.

[0251] Specifically, for the specific working content of each part of the above electronic device, and the specific processing content when the computer program on the above storage medium is run by a processor, reference can be made to the content of each embodiment of the above pronunciation dictionary construction method, or reference can be made to the content of each embodiment of the above speech recognition method, which will not be elaborated here.

[0252] For the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be in other sequences or carried out simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0253] It should be noted that each embodiment in this specification is described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other. For device embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiments.

[0254] The steps in the methods of the embodiments of the present application can be adjusted, combined, and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0255] The modules and sub-modules in the devices and terminals in the embodiments of the present application can be combined, divided, and deleted according to actual needs.

[0256] In several embodiments provided by the present application, it should be understood that the disclosed terminals, devices, and methods can be implemented in other ways. For example, the terminal embodiments described above are merely illustrative. For example, the division of modules or sub-modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple sub-modules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of devices or modules can be in electrical, mechanical, or other forms.

[0257] The modules or sub-modules described as separate components may or may not be physically separated. The components as modules or sub-modules may or may not be physical modules or sub-modules, that is, they can be located in one place, or can be distributed to multiple network modules or sub-modules. Some or all of the modules or sub-modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0258] In addition, each functional module or sub-module in various embodiments of the present application can be integrated in a processing module, or each module or sub-module can exist physically alone, or two or more modules or sub-modules can be integrated in one module. The above-mentioned integrated modules or sub-modules can be implemented in the form of hardware, or in the form of software functional modules or sub-modules.

[0259] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in this article can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0260] The steps of the method or algorithm described in combination with the embodiments disclosed in this article can be directly implemented by hardware, software units executed by a processor, or a combination of the two. The software units can be placed in a random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.

[0261] Finally, it should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising said element.

[0262] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for constructing a pronunciation dictionary, characterized in that, Including: Input the target audio data into a pre-trained acoustic model for phoneme extraction to obtain a phoneme set corresponding to the target audio data; The target audio data is audio data covering all phonemes; According to the recognition probability of each phoneme in the extracted phoneme set and the phoneme label corresponding to the target audio data, determine the similar phonemes of the phonemes in the phoneme label from the phoneme set; wherein, the similar phonemes of the phonemes in the phoneme label refer to a set number of phonemes selected from the phoneme set that correspond to the phonemes in the phoneme label and have the highest recognition probability; Construct a multi-pronunciation dictionary according to the rule of storing words corresponding to their standard pronunciations and similar pronunciations; wherein, the similar pronunciation is composed of the similar phonemes of the phonemes in the phoneme label; The acoustic model is trained in the following manner: Input the audio features of the training speech into the acoustic model to obtain the phoneme information of the training speech; Input the phoneme information of the training speech and the non-semantic information extracted according to the audio features of the training speech into the audio synthesis model to obtain an audio synthesis result; Perform parameter correction on the acoustic model according to the speech recognition loss of the acoustic model and the audio synthesis loss of the audio synthesis model.

2. The method according to claim 1, characterized in that, Construct a multi-pronunciation dictionary according to the rule of storing words corresponding to their standard pronunciations and similar pronunciations, including: Determine the standard pronunciation corresponding to the word in the pronunciation dictionary, and determine the similar pronunciation corresponding to the word in the pronunciation dictionary according to the similar phonemes of the phonemes in the phoneme label; Obtain a multi-pronunciation dictionary by storing the word corresponding to its standard pronunciation and similar pronunciation.

3. The method according to claim 2, wherein Determine the standard pronunciation corresponding to the word in the pronunciation dictionary, and determine the similar pronunciation corresponding to the word in the pronunciation dictionary according to the similar phonemes of the phonemes in the phoneme label, including: Respectively determine the standard pronunciation corresponding to the high-frequency error-prone words in the pronunciation dictionary, and respectively determine the similar pronunciations corresponding to the high-frequency error-prone words in the pronunciation dictionary according to the similar phonemes of the phonemes in the phoneme label; Obtain a multi-pronunciation dictionary by storing the word corresponding to its standard pronunciation and similar pronunciation, including: Obtain a multi-pronunciation dictionary by storing the high-frequency error-prone words in the pronunciation dictionary corresponding to their standard pronunciations and similar pronunciations.

4. The method according to claim 3, wherein Obtain a multi-pronunciation dictionary by storing the high-frequency error-prone words in the pronunciation dictionary corresponding to their standard pronunciations and similar pronunciations, including: Calculate the score of the similar pronunciation corresponding to the high-frequency error-prone word according to the similar pronunciation corresponding to the high-frequency error-prone word in the pronunciation dictionary and the recognition probability of each phoneme in the phoneme set; Select the similar pronunciations with scores higher than the set score threshold from the similar pronunciations corresponding to each high-frequency error-prone word as the target similar pronunciations; Store the high-frequency error-prone words in the pronunciation dictionary corresponding to their standard pronunciations and target similar pronunciations to obtain a multi-pronunciation dictionary.

5. A speech recognition method, characterized in that, Including: Input the speech to be recognized into a pre-trained acoustic model for phoneme extraction to obtain the phoneme sequence of the speech to be recognized; Determine the speech recognition result of the speech to be recognized according to the phoneme sequence of the speech to be recognized and a pre-constructed multi-pronunciation dictionary; Wherein, the words in the multi-pronunciation dictionary are stored corresponding to the standard pronunciation and similar pronunciations of the words; the similar pronunciations corresponding to the words are constructed from the similar phonemes of the phonemes in the phoneme labels corresponding to the target audio data; the target audio data includes the audio data corresponding to the words; The acoustic model is trained in the following manner: Input the audio features of the training speech into the acoustic model to obtain the phoneme information of the training speech; Input the phoneme information of the training speech and the non-semantic information extracted according to the audio features of the training speech into the audio synthesis model to obtain an audio synthesis result; Correct the parameters of the acoustic model according to the speech recognition loss of the acoustic model and the audio synthesis loss of the audio synthesis model.

6. The method according to claim 5, wherein The multi-pronunciation dictionary is constructed according to the pronunciation dictionary construction method described in any one of claims 1 to 4.

7. The method according to claim 1 or 5, characterized in that Inputting the phoneme information of the training speech and the non-semantic information extracted according to the audio features of the training speech into the audio synthesis model to obtain an audio synthesis result includes: Performing downsampling of the same scale on the phoneme information of the training speech and the audio features of the training speech; Extracting the non-semantic information of the training speech according to the downsampled audio features of the training speech; Fusing the downsampled phoneme information and the non-semantic information to obtain basic audio synthesis information; Inputting the basic audio synthesis information into the audio synthesis model to obtain an audio synthesis result.

8. The method according to claim 1 or 5, characterized in that, The speech recognition loss of the acoustic model is determined by a cross-entropy loss function, and the audio synthesis loss of the audio synthesis model is determined by a mean squared error loss function.

9. A pronunciation dictionary construction device, characterized in that Including: A phoneme extraction unit for inputting target audio data into a pre-trained acoustic model for phoneme extraction to obtain a phoneme set corresponding to the target audio data; The target audio data is audio data covering all phonemes; A phoneme screening unit for determining the similar phonemes of the phonemes in the phoneme label from the phoneme set according to the recognition probability of each phoneme in the extracted phoneme set and the phoneme label corresponding to the target audio data; wherein, the similar phonemes of the phonemes in the phoneme label refer to a set number of phonemes selected from the phoneme set with the highest recognition probability corresponding to the phonemes in the phoneme label; A dictionary construction unit for constructing a multi-pronunciation dictionary according to the rule of storing words corresponding to the standard pronunciation and similar pronunciations of the words; wherein, the similar pronunciations are composed of the similar phonemes of the phonemes in the phoneme label; The acoustic model is trained in the following manner: Input the audio features of the training speech into the acoustic model to obtain the phoneme information of the training speech; Input the phoneme information of the training speech and the non-semantic information extracted from the audio features of the training speech into an audio synthesis model to obtain an audio synthesis result; Perform parameter correction on the acoustic model according to the speech recognition loss of the acoustic model and the audio synthesis loss of the audio synthesis model.

10. A voice recognition device, characterized in that, It includes: A speech processing unit for inputting a speech to be recognized into a pre-trained acoustic model for phoneme extraction to obtain a phoneme sequence of the speech to be recognized; A speech recognition unit for determining a speech recognition result of the speech to be recognized according to the phoneme sequence of the speech to be recognized and a pre-constructed multiple pronunciation dictionary; Wherein, the words in the multiple pronunciation dictionary are stored corresponding to the standard pronunciation and similar pronunciations of the words; the similar pronunciations corresponding to the words are constructed from the similar phonemes of the phonemes in the phoneme labels corresponding to the target audio data; the target audio data includes the audio data corresponding to the words; The acoustic model is trained in the following manner: Input the audio features of the training speech into the acoustic model to obtain the phoneme information of the training speech; Input the phoneme information of the training speech and the non-semantic information extracted from the audio features of the training speech into an audio synthesis model to obtain an audio synthesis result; Perform parameter correction on the acoustic model according to the speech recognition loss of the acoustic model and the audio synthesis loss of the audio synthesis model.

11. An electronic device, characterized in that, It includes: A memory and a processor; The memory is connected to the processor for storing programs; The processor is configured to implement the pronunciation dictionary construction method according to any one of claims 1 to 4 or the speech recognition method according to any one of claims 5 to 8 by running the programs in the memory.

12. A storage medium, characterized in that, A computer program is stored on the storage medium, and when the computer program is run by the processor, it implements the pronunciation dictionary construction method according to any one of claims 1 to 4 or the speech recognition method according to any one of claims 5 to 8.

Images