Speech recognition method and related apparatus, electronic device, and storage medium

Abstract

The application discloses a speech recognition method and related device, electronic equipment and storage medium, wherein the speech recognition method comprises: extracting acoustic features of each audio frame of the speech to be recognized; then predicting based on the acoustic features to obtain a state sequence, decoding the state sequence based on a finite state converter to obtain a first candidate recognition text and a first recognition score thereof; wherein the state sequence comprises a pronunciation state corresponding to the audio frame; and performing beam search decoding based on the acoustic features to obtain a second candidate recognition text and a second recognition score thereof; and determining a target recognition text of the speech to be recognized based on the first candidate recognition text and the first recognition score thereof and the second candidate recognition text and the second recognition score thereof. The above scheme can improve the accuracy of speech recognition.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and in particular to a speech recognition method and related devices, electronic devices, and storage media. Background Technology

[0002] In recent years, with the development and implementation of artificial intelligence technology, human-computer interaction has become more and more frequent. Voice-based interaction has gradually become the mainstream form in the field of human-computer interaction, and the importance of voice recognition technology has become increasingly prominent.

[0003] Currently, while speech recognition methods can accurately identify human voices in quiet, simple scenarios, their widespread application means that unacceptable errors can still occur in noisy, complex, or even extreme environments. Therefore, improving the accuracy of speech recognition has become a pressing issue. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a speech recognition method and related devices, electronic devices, and storage media that can improve the accuracy of speech recognition.

[0005] To address the aforementioned technical problems, the first aspect of this application provides a speech recognition method, comprising: extracting acoustic features from each audio frame of the speech to be recognized; performing prediction based on the acoustic features to obtain a state sequence, and decoding the state sequence based on a finite state converter to obtain a first candidate text for recognition and its first recognition score; wherein the state sequence includes the pronunciation state corresponding to the audio frame; performing beam search decoding based on the acoustic features to obtain a second candidate text for recognition and its second recognition score; and determining the target text for recognition of the speech to be recognized based on the first candidate text for recognition and its first recognition score, and the second candidate text for recognition and its second recognition score.

[0006] To address the aforementioned technical problems, a second aspect of this application provides a speech recognition device, comprising: a feature extraction module, a state prediction module, a first decoding module, a second decoding module, and a text determination module. The feature extraction module extracts acoustic features from each audio frame of the speech to be recognized; the state prediction module predicts based on the acoustic features to obtain a state sequence, wherein the state sequence includes the pronunciation state corresponding to each audio frame; the first decoding module decodes the state sequence using a finite state converter to obtain a first candidate text for recognition and its first recognition score; the second decoding module performs beam search decoding based on the acoustic features to obtain a second candidate text for recognition and its second recognition score; and the text determination module determines the target text for recognition based on the first candidate text for recognition and its first recognition score, and the second candidate text for recognition and its second recognition score.

[0007] To address the aforementioned technical problems, a third aspect of this application provides an electronic device including a memory and a processor coupled to each other. The memory stores program instructions, and the processor executes the program instructions to implement the speech recognition method of the first aspect described above.

[0008] To address the aforementioned technical problems, a fourth aspect of this application provides a computer-readable storage medium storing program instructions executable by a processor, the program instructions being used to implement the speech recognition method of the first aspect described above.

[0009] The above scheme extracts acoustic features from each audio frame of the speech to be recognized; then predicts based on the acoustic features to obtain a state sequence, and decodes the state sequence using a finite state converter to obtain a first candidate text and its first recognition score; wherein the state sequence includes the pronunciation state corresponding to the audio frame; and performs beam search decoding based on the acoustic features to obtain a second candidate text and its second recognition score; based on the first candidate text and its first recognition score and the second candidate text and its second recognition score, the target text of the speech to be recognized is determined. On the one hand, decoding the state sequence using a finite state converter to obtain the first candidate text and its first recognition score helps improve the accuracy of the first candidate text and its first recognition score; on the other hand, performing beam search decoding based on the acoustic features to obtain the second candidate text and its second recognition score helps improve the stability of the acoustic feature decoding process and improves the efficiency of obtaining the second candidate text and its second recognition score. On this basis, the target text of the speech to be recognized is determined based on the first candidate text and its first recognition score and the second candidate text and its second recognition score. Compared with determining the target text based on a single candidate text, this method helps improve the accuracy of the target text. Therefore, it can improve the accuracy of speech recognition. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating an embodiment of the speech recognition method of this application;

[0011] Figure 2 This is a schematic diagram of an embodiment of the first candidate text to be identified;

[0012] Figure 3 This is a schematic diagram of an embodiment of the second candidate text recognition;

[0013] Figure 4 This is a flowchart illustrating an embodiment of acquiring an audio segment;

[0014] Figure 5 This is a schematic diagram of the framework of an embodiment of the speech recognition method of this application;

[0015] Figure 6 This is a schematic diagram of the framework of an embodiment of the speech recognition device of this application;

[0016] Figure 7 This is a schematic diagram of the framework of an embodiment of the electronic device of this application;

[0017] Figure 8 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0018] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0019] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0020] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0021] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the speech recognition method of this application.

[0022] Specifically, this may include the following steps:

[0023] Step S11: Extract the acoustic features of each audio frame of the speech to be recognized.

[0024] In one implementation scenario, acoustic features of each audio frame of the speech to be recognized can be extracted. This can be done using MFCC (Mel Frequency Cepstral Coefficients) feature extraction or deep learning feature extraction. The method for extracting acoustic features can be determined based on the actual situation and is not specifically limited here.

[0025] Step S12: Based on acoustic features, make predictions to obtain a state sequence, and decode the state sequence based on a finite state converter to obtain the first candidate text for recognition and its first recognition score.

[0026] In this disclosed implementation scenario, the state sequence includes the pronunciation state corresponding to the audio frame. Specifically, the pronunciation state corresponding to the audio frame can include phonemes aligned with each audio frame. Each audio frame can be aligned with a single phoneme, meaning it only contains the phoneme corresponding to that audio frame itself; or it can be aligned with three phonemes, meaning it not only contains the phoneme corresponding to itself but also the phonemes corresponding to the previous and next audio frames. Further, the pronunciation state is used to characterize states such as start, continuation, termination, and stop during the pronunciation process, and may include, but is not limited to, phoneme start state, phoneme intermediate state, phoneme end state, blank silence state, etc.

[0027] In one implementation scenario, acoustic features can be directly predicted to obtain a state sequence.

[0028] In another implementation scenario, unlike the aforementioned implementation, the acoustic features can be encoded first using an encoder to obtain encoded features, and then prediction can be performed using these encoded features to obtain a state sequence. The method for obtaining the state sequence can be determined based on the actual situation and is not specifically limited here.

[0029] In one implementation scenario, the state sequence can be converted into a phoneme sequence, and the first candidate recognition text and its first recognition score can be synthesized based on the phoneme sequence.

[0030] Please see Figure 2 , Figure 2 This is a schematic diagram of an embodiment of the first candidate text recognition. First, the state sequence is converted into a phoneme sequence. Then, a finite-state converter based on dictionary packaging is used to transform the phoneme sequence, obtaining candidate words corresponding to each phoneme in the phoneme sequence. Next, a finite-state converter based on a language model is used to process the candidate sentences composed of any candidate words corresponding to each phoneme in the phoneme sequence, obtaining the first candidate text and its first recognition score. Specifically, by using a finite-state converter based on a language model to process the candidate sentences composed of any candidate words corresponding to each phoneme in the phoneme sequence, obviously erroneous content in the candidate words can be eliminated, thus obtaining the first candidate text and its first recognition score. It is understood that any first candidate text has a corresponding first recognition score, and the first recognition scores can be the same or different. The first recognition score can be determined according to the actual situation and is not specifically limited here. The above method, using a finite-state converter based on a language model to process the candidate sentences composed of any candidate words corresponding to each phoneme in the phoneme sequence, helps to improve the accuracy of the first candidate text recognition, thereby improving the accuracy of speech recognition.

[0031] It is worth noting that the finite state converters packaged with dictionaries and the finite state converters packaged with language models are matched with the domain of the speech to be recognized, so as to make the first candidate text to be recognized and its first recognition score more accurate.

[0032] Step S13: Perform beam search decoding based on acoustic features to obtain the second candidate text for recognition and its second recognition score.

[0033] In one implementation scenario, as a possible approach, acoustic features can be encoded first to obtain encoded features, and then beam search decoding can be performed on the encoded features to obtain the second candidate recognition text and its second recognition score.

[0034] Please see Figure 3 , Figure 3 This is a schematic diagram of an embodiment of the second candidate text recognition. First, based on the silence state in the state sequence, the audio frames in the speech to be recognized are segmented to obtain several audio segments. For each audio segment, the acoustic features of the audio frames in the audio segment are encoded to obtain the encoded features of the audio segment. Based on the encoded features of several audio segments, beam search decoding is performed to obtain the second candidate text and its second recognition score. It is understood that any second candidate text has a corresponding second recognition score, and the second recognition scores can be the same or different. The second recognition score can be determined according to the actual situation and is not specifically limited here.

[0035] In another implementation scenario, unlike the aforementioned implementation, the audio frames in the speech to be recognized are first segmented based on the silence state in the state sequence to obtain several audio segments. After obtaining several audio segments, in response to containing one audio segment, beam search decoding is performed based on the audio segment to obtain a second candidate recognition text and its second recognition score; in response to containing multiple audio segments, for each audio segment, the acoustic features of the audio frames in the audio segment are encoded to obtain the encoded features of the audio segment; then, beam search decoding is performed based on the encoded features of several audio segments to obtain the second candidate recognition text and its second recognition score. This method, by encoding several audio segments to obtain the encoded features of the audio segments, and then performing beam search decoding based on the encoded features of several audio segments to obtain the second candidate recognition text and its second recognition score, helps to improve the accuracy of the second candidate recognition text and its second recognition score, thereby improving the accuracy of speech recognition.

[0036] Further, please refer to Figure 4 , Figure 4 This is a flowchart illustrating an embodiment of acquiring an audio segment. Specifically, it may include the following steps:

[0037] Step S41: According to the audio frames corresponding to the silence state in the state sequence, the audio frames in the speech to be recognized are segmented to obtain several candidate segments.

[0038] Specifically, the silence state in the state sequence can be obtained, and the audio frames in the speech to be recognized can be segmented according to the audio frames corresponding to the silence state, thereby obtaining several candidate segments. The several candidate segments may include one segment or multiple segments, and the number of candidate segments can be determined according to the actual situation, without specific limitations here.

[0039] Step S42: Is the duration of the candidate segment shorter than the duration threshold? If yes, proceed to step S43; otherwise, proceed to step S44.

[0040] In one implementation scenario, after obtaining several candidate segments, it is possible to further determine whether the duration of the candidate segments is shorter than a duration threshold. This helps to minimize poor decoding performance and numerous deletion errors when the speech to be recognized is long. The duration threshold can be set to 3S, 4S, 5S, etc., and the threshold can be set according to the actual situation. No specific limitation is made here.

[0041] Step S43: The candidate fragment is spliced ​​with its adjacent candidate fragments to obtain a new candidate fragment.

[0042] In one implementation scenario, if the duration of a candidate segment is shorter than a duration threshold, the candidate segment with a duration shorter than the duration threshold is concatenated with its adjacent candidate segments until the duration of the concatenated segment is not shorter than the duration threshold. Specifically, a candidate segment can be concatenated with its adjacent candidate segments to obtain a new candidate segment, and it is determined whether the duration of the new candidate segment is shorter than the duration threshold until the duration of the concatenated segment is not shorter than the duration threshold.

[0043] Step S44: Encode each audio segment and proceed with subsequent steps.

[0044] Specifically, if the duration of a candidate segment is not shorter than a duration threshold, each audio segment is encoded and subsequent steps are performed.

[0045] It should be noted that the duration of the audio segment is not shorter than the duration threshold. This is to avoid poor decoding and numerous deletion errors when the speech to be recognized is long, thereby improving the accuracy of speech recognition.

[0046] In one implementation scenario, during the execution of the beam search decoding process, for each decoding path, it can be determined whether the decoding path has ended. In response to the total decoding score as of the current decoding time being lower than a preset threshold, a first weighting coefficient is obtained based on the path length of the decoding path as of the current decoding time. This first weighting coefficient is then used to weight the prediction probability value of the ending character at the current decoding time, thereby increasing the prediction probability value of the ending character. The preset threshold value can be set to 0.5, 0.4, 0.3, etc., and can be determined according to the actual situation; no specific limitation is made here. Specifically, a first weighting coefficient can be obtained based on the path length of the decoding path up to the current decoding time, and this first weighting coefficient is not less than 1. The predicted probability value of the end character at the current decoding time is then weighted based on this first weighting coefficient to obtain the predicted probability value of the end character. Alternatively, the first weighting coefficient can be obtained using the path length of the decoding path at the current decoding time. This first weighting coefficient is a value not greater than 1. Then, the logarithm of the predicted probability of the end character at the current decoding time is taken based on this first weighting coefficient to obtain a weighted score. Finally, the antilogarithm of this weighted score is taken to obtain the predicted probability value of the end character. The specific formula is as follows:

[0047]

[0048] Among them, Score[ <os>L represents the logarithm of the probability of predicting the end character at the current decoding moment. beam Here, T is the path length of the decoding path at the current decoding moment, and T is a hyperparameter that can be adjusted based on the results on the validation set. Based on this, and using the predicted probabilities of each preset character in the preset dictionary at the current decoding moment and the weighted predicted probability of the ending character, it is determined whether the decoding path ends at the current decoding moment. For example, if the predicted probability of the weighted ending character is higher than the predicted probabilities of each preset character in the preset dictionary at the current decoding moment, then the decoding path ends at the current decoding moment; otherwise, the decoding path continues decoding. The method for determining whether the decoding path ends at the current decoding moment can be determined according to the actual situation and is not specifically limited here. This method, when the total decoding score at the current decoding moment is lower than a preset threshold, increases the predicted probability of the ending character by weighting the predicted probability value at the current decoding moment, which helps to suppress the situation of erroneous triggering of random characters during speech recognition, thereby improving the accuracy of speech recognition.

[0049] Furthermore, the second recognition score may also include at least one of a coverage penalty score and a decoding score for each decoded character in the second candidate recognition text. For example, when the second recognition score includes a coverage penalty score, to obtain the coverage penalty score in the second recognition score of the second candidate recognition text, one can first obtain the attention weights of each decoded character in the second candidate recognition text with each audio frame at the decoding time corresponding to the decoded character. Then, for each audio frame, one can calculate the sum of the attention weights of each audio frame with each decoded character, and obtain the sub-penalty score of the audio frame based on the smaller value between the sum of the weights and a preset value. On this basis, the sub-penalty scores of each audio frame are fused to obtain the coverage penalty score in the second recognition score of the second candidate recognition text. The preset value can be 0.4, 0.5, 0.6, etc., and is not specifically limited here. The specific formula for the coverage penalty score is as follows:

[0050]

[0051] Where β is the penalty weight, ranging from 0 to 1; X and Y are the total duration of acoustic features and the total duration of the second candidate text, respectively; p i,j Let be the attention weight for the i-th audio frame and the j-th decoded character; 0.5 is a preset value. Specifically, if the second candidate recognized text does not overlap (i.e., no characters are missing), then in any i-th audio frame, at least one p will always appear. i,j If the sum of the attention weights of the i-th audio frame and each decoded character is large, it will inevitably be greater than the preset value. Therefore, the sub-penalty scores of each audio frame are fused to obtain the coverage penalty score in the second recognition score of the second candidate text. If coverage occurs in the second candidate text, i.e., there are missing characters, then for any i-th audio frame, for any p... i,j The sum of the attention weights of the i-th audio frame and each decoded character may be smaller than a preset value. Therefore, the coverage penalty score is fused based on the sub-penalty scores of each audio frame to obtain the second recognition score of the second candidate text. According to the function's characteristics, the coverage penalty score will inevitably be greater when there are missing characters than when there are no missing characters. Further, during the beam search decoding process, the initial recognition score of each decoding path can be obtained first. This can be based on the sum of the decoding probabilities of each decoded character in each decoding path; alternatively, the sum of the decoding probabilities of each decoded character in each decoding path can be obtained first, and then weighted to obtain the initial recognition score. The method for obtaining the initial recognition score can be determined according to the actual situation and is not specifically limited here. After obtaining the initial recognition score, the difference between the initial recognition score and the coverage penalty score in each decoding path is used as the second recognition score of the second candidate text for that decoding path. Understandably, during the decoding process, an attention mechanism can be used to establish a mapping relationship between acoustic features and the second candidate text, thereby generating attention weight curves with the acoustic time axis and the second candidate text time axis as the horizontal and vertical axes, respectively. Furthermore, since the relationship between acoustic features and the second candidate text is monotonic during speech recognition, and the second candidate text should completely cover the text content implied by the acoustic features, the attention weight curve should satisfy both monotonicity and completeness. In this approach, the second recognition score includes a coverage penalty score, which can be combined with coverage issues that occur during speech recognition, i.e., the problem of missing words. When coverage occurs in speech recognition, the attention weight of the acoustic feature corresponding to the covered portion has a lower second recognition score in the second candidate text, helping to suppress the score of that path and thus improve the accuracy of the second candidate text and its second recognition score.

[0052] For example, when the second recognition score includes the decoding scores of each decoded character in the second candidate recognition text, in order to obtain the decoding score, in response to the repetition of the decoded character at the current decoding time with the decoded character at a historical decoding time, a second weighting coefficient can be determined based on the cumulative number of character repetitions on the decoding path of the second candidate recognition text. The cumulative number of repetitions is positively correlated with the second weighting coefficient. Then, the initial decoding score of the decoded character at the current decoding time is weighted based on the second weighting coefficient to obtain the final decoding score of the decoded character at the current decoding time. The initial decoding score can be the predicted probability of the model or it can be preset. The initial decoding score can be determined according to the actual situation, and no specific limitation is made here. The specific decoding score formula is as follows:

[0053]

[0054] Where Score is the final decoding score of the decoded character at the current decoding moment, and penalty is... total The second weighting coefficient is denoted by T, which is a hyperparameter that can be adjusted based on the validation set performance. Specifically, during the decoding process for each decoding path, the system first checks whether the decoded character at the current decoding moment is repeated with the decoded character at a previous decoding moment. If not, the system checks again at the next decoding moment, continuing this process until the decoding path is complete. If so, the system obtains the final decoding score for the decoded character at the current decoding moment. This is determined by the second weighting coefficient, based on the cumulative number of character repetitions along the decoding path containing the second candidate text. The initial decoding score of the decoded character at the current decoding moment is then weighted by this second weighting coefficient to obtain the final decoding score. The higher the cumulative number of character repetitions along the decoding path containing the second candidate text, the lower the final decoding score at the current decoding moment. Finally, the final decoding scores of the decoded characters at each decoding moment along the decoding path are summed to obtain the second recognition score for the second candidate text along that decoding path. In the above method, the second recognition score includes the decoding score of each decoded character in the second candidate recognition text. It can be combined with the decoding situation that occurs during the speech recognition process, that is, the situation where the decoding result is repeated. The attention weight of the acoustic feature corresponding to the part of the decoding result that is repeated is lower in the second recognition score of the second candidate recognition text, which helps to suppress the score of the path and thus improve the accuracy of the second candidate recognition text and its second recognition score.

[0055] In addition, the second recognition score may also include a coverage penalty score and the decoding score of each decoded character in the second candidate recognition text. Specifically, the final decoding score of the decoded character at each decoding moment in each decoding path can be used as the initial recognition score in each decoding path, and the difference between the initial recognition score and the coverage penalty score in the corresponding decoding path can be used as the second recognition score of the second candidate recognition text in that decoding path.

[0056] Step S14: Based on the first candidate recognition text and its first recognition score, and the second candidate recognition text and its second recognition score, determine the target recognition text of the speech to be recognized.

[0057] In one implementation scenario, as a possible approach, based on a first candidate recognition text and its first recognition score, and a second candidate recognition text and its second recognition score, the candidate recognition text corresponding to the higher of the first recognition score and the second recognition score is selected as the target recognition text for the speech to be recognized.

[0058] In another implementation scenario, as another possible implementation, the first candidate text with the highest first recognition score is designated as the first preferred recognition text, and the second candidate text with the highest second recognition score is designated as the second preferred recognition text. In response to the first and second sets of recognized texts containing identical text content, the identical text is designated as the target recognition text for the speech to be recognized; in response to the first and second sets of recognized texts not containing identical text content, either the first preferred recognition text or the second preferred recognition text is designated as the target recognition text for the speech to be recognized.

[0059] In another implementation scenario, the first candidate text with the highest first recognition score is taken as the first preferred recognition text, and the second candidate text with the highest second recognition score is taken as the second preferred recognition text. A first decision result is obtained based on the first and second preferred recognition texts. Specifically, it can first detect whether the first and second preferred recognition texts are the same; if so, the first decision result includes deciding either the first or second preferred recognition text as the target recognition text; if not, it can detect whether the first and second preferred recognition texts have repeated characters. If only one of the first and second preferred recognition texts has repeated characters, the first decision result includes deciding the other as the target recognition text; if both the first and second preferred recognition texts have repeated characters or neither has repeated characters, the first decision result includes not being able to decide either the first or second preferred recognition text as the target recognition text. This method, by detecting whether the first and second preferred recognition texts are the same, helps improve the accuracy of the target recognition text and further improves the accuracy of speech recognition.

[0060] Furthermore, in response to a first decision result including the inability to decide either the first preferred identification text or the second preferred identification text as the target identification text, a second decision result can be obtained based on whether the first preferred identification text is included in the first identification text set and whether the second preferred identification text is included in the second identification text set. Specifically, in response to the first preferred identification text being included in the first identification text set, the second decision result can be determined to include deciding the first preferred identification text as the target identification text; in response to the second preferred identification text being included in the second identification text set, the second decision result can be determined to include deciding the second preferred identification text as the target identification text; in response to the first preferred identification text not being included in the first identification text set and the second preferred identification text not being included in the second identification text set, the second decision result can be determined to include the inability to decide either the first preferred identification text or the second preferred identification text as the target identification text. It should be noted that the first identification text set includes second candidate identification texts ranked in the first order after being sorted from high to low according to the second identification score, and the second identification text set includes first candidate identification texts ranked in the second order after being sorted from high to low according to the first identification score. The values ​​of the first and second positions can be the same, for example, any one of 3, 4, or 5. The values ​​of the first and second positions can also be different, for example, the first position can be set to 3 and the second position to 4. The first and second positions can be determined according to the actual situation, and no specific limitation is made here. The above method, by determining whether the first preferred identification text is contained in the first identification text set and whether the second preferred identification text is contained in the second identification text set, helps to improve the accuracy of target identification text.

[0061] Furthermore, it checks whether the first and second preferred identification texts have the same number of words. If so, the third decision result is determined by selecting the text with the larger identification score from the first and second preferred identification texts as the target identification text. If not, the third decision result is determined based on the first identification score of the first preferred identification text, the second identification score of the second preferred identification text, and the word count difference between the two texts. For example, when the word count difference between the first and second preferred identification texts meets a preset word count difference, the text with the larger first identification score or the second identification score can be selected as the target identification text. The preset word count difference can be set to 1, 2, 3, etc., without specific limitations. Alternatively, when the first identification score of the first preferred identification text and the second identification score of the second preferred identification text meet a preset score, either text can be selected as the target identification text. The preset score can be set to 0.5, 0.6, 0.8, etc., without specific limitations. The method for determining the third decision result can be determined according to the actual situation and is not specifically limited here. The above method determines the target text by detecting whether the first and second preferred recognition texts have the same number of words, thereby improving the accuracy of both target text recognition and speech recognition.

[0062] Furthermore, after determining the third decision result, in response to the third decision result including the inability to decide the first preferred identification text or the second preferred identification text as the target identification text, the fourth decision result can be directly determined to include deciding the second preferred identification text as the target identification text.

[0063] Table 1. Schematic diagram of an embodiment of target recognition text.

[0064]

[0065]

[0066] Please refer to Table 1, which is a schematic table of an embodiment of target recognition text. ED-top1 is the second candidate recognition text with the highest second recognition score, CD-top1 is the first candidate recognition text with the highest first recognition score, ED-top5 is the first recognition text set, which includes the second candidate recognition texts (ED-top1, ED-top2, ED-top3, ED-top4, ED-top5) ranked in the top 5 after the second recognition scores are sorted from high to low, and CD-top5 is the second recognition text set, which includes the first candidate recognition texts (CD-top1, CD-top2, CD-top3, CD-top4, CD-top5) ranked in the top 5 after the first recognition scores are sorted from high to low. Specifically, we can first check if ED-top1 and CD-top1 are the same. If they are the same, either the first or second preferred recognition text is selected as the target recognition text. If they are different, we check if ED-top1 and CD-top1 produce repeated characters. If ED-top1 produces repeated characters, CD-top1 is selected; if CD-top1 produces repeated characters, ED-top1 is selected; if both ED-top1 and CD-top1 produce repeated characters or neither produces repeated characters, no selection is made. Then, we compare ED / CD-top1 and ED / CD-top5. If CD-top1 is included in ED-top5, CD-top1 is selected; if ED-top1 is included in ED-top5, CD-top1 is selected. If a word is included in CD-top5, then ED-top1 is selected. If neither CD-top1 nor ED-top1 is included in CD-top5, then no word is selected. Furthermore, it checks if ED-top1 and CD-top1 have the same number of words. If they do, the one with the higher recognition score is selected as the target text. If they do not, the selection can be based on the difference in word count and recognition score between ED-top1 and CD-top1. For example, if the difference in word count is no greater than 2, the one with the higher recognition score is selected as the target text. Alternatively, ED-top1 can be directly selected as the target text. This method, by comparing the first candidate text with the first recognition score and the second candidate text with the second recognition score, improves the controllability and stability of the target text result, and reduces the likelihood of large-scale word loss and repetition, thus helping to improve the accuracy of speech recognition. It is understood that the method shown is only one possible method in actual application, and does not limit the specific method used in actual application. The specific method can be determined according to the actual situation, and no specific limitation is made here.

[0067] In one implementation scenario, the target text is identified based on a speech recognition model. This model includes an encoding network, a prediction network, and a decoding network. The encoding network encodes acoustic features to obtain encoded features, the prediction network predicts state sequences based on these encoded features, and the decoding network performs beam search decoding based on the encoded features. These three networks are jointly trained. This approach, through joint training of the encoding, prediction, and decoding networks, helps improve the recognition results of the speech recognition model, thereby increasing the accuracy of the speech recognition.

[0068] The above scheme extracts acoustic features from each audio frame of the speech to be recognized; then predicts based on the acoustic features to obtain a state sequence, and decodes the state sequence using a finite state converter to obtain a first candidate text and its first recognition score; wherein the state sequence includes the pronunciation state corresponding to the audio frame; and performs beam search decoding based on the acoustic features to obtain a second candidate text and its second recognition score; based on the first candidate text and its first recognition score and the second candidate text and its second recognition score, the target text of the speech to be recognized is determined. On the one hand, decoding the state sequence using a finite state converter to obtain the first candidate text and its first recognition score helps improve the accuracy of the first candidate text and its first recognition score; on the other hand, performing beam search decoding based on the acoustic features to obtain the second candidate text and its second recognition score helps improve the stability of the acoustic feature decoding process and improves the efficiency of obtaining the second candidate text and its second recognition score. On this basis, the target text of the speech to be recognized is determined based on the first candidate text and its first recognition score and the second candidate text and its second recognition score. Compared with determining the target text based on a single candidate text, this method helps improve the accuracy of the target text. Therefore, it can improve the accuracy of speech recognition.

[0069] Please see Figure 5 , Figure 5 This is a schematic diagram of the framework of an embodiment of the speech recognition method of this application. The method extracts acoustic features from each audio frame of the speech to be recognized. First, prediction is performed based on the acoustic features to obtain a state sequence. Then, the state sequence is decoded using a finite state converter to obtain a first candidate text for recognition and its first recognition score. Next, based on the silence states in the state sequence, the audio frames in the speech to be recognized are segmented to obtain several audio segments. For each audio segment, the acoustic features of the audio frames within the audio segment are encoded to obtain encoded features of the audio segment. Based on the encoded features of the several audio segments, a beam search decoding is performed to obtain a second candidate text for recognition and its second recognition score. The second recognition score includes a coverage penalty score and the decoding scores of each decoded character in the second candidate text. Furthermore, during the beam search decoding process, for each decoding path, it can be determined whether the decoding path ends at the current decoding time. Based on this, the target text for the speech to be recognized is determined based on the first candidate text and its first recognition score, and the second candidate text and its second recognition score.

[0070] The above scheme extracts acoustic features from each audio frame of the speech to be recognized; then predicts based on the acoustic features to obtain a state sequence, and decodes the state sequence using a finite state converter to obtain a first candidate text and its first recognition score; wherein the state sequence includes the pronunciation state corresponding to the audio frame; and performs beam search decoding based on the acoustic features to obtain a second candidate text and its second recognition score; based on the first candidate text and its first recognition score and the second candidate text and its second recognition score, the target text of the speech to be recognized is determined. On the one hand, decoding the state sequence using a finite state converter to obtain the first candidate text and its first recognition score helps improve the accuracy of the first candidate text and its first recognition score; on the other hand, performing beam search decoding based on the acoustic features to obtain the second candidate text and its second recognition score helps improve the stability of the acoustic feature decoding process and improves the efficiency of obtaining the second candidate text and its second recognition score. On this basis, the target text of the speech to be recognized is determined based on the first candidate text and its first recognition score and the second candidate text and its second recognition score. Compared with determining the target text based on a single candidate text, this method helps improve the accuracy of the target text. Therefore, it can improve the accuracy of speech recognition.

[0071] Please see Figure 6 , Figure 6 This is a schematic diagram of the framework of an embodiment of the speech recognition device of this application. The speech recognition device 60 includes: a feature extraction module 61, a state prediction module 62, a first decoding module 63, a second decoding module 64, and a text determination module 65. The feature extraction module 61 is used to extract acoustic features from each audio frame of the speech to be recognized; the state prediction module 62 is used to predict based on the acoustic features to obtain a state sequence; wherein the state sequence includes the pronunciation state corresponding to the audio frame; the first decoding module 63 is used to decode the state sequence based on a finite state converter to obtain a first candidate text for recognition and its first recognition score; the second decoding module 64 is used to perform beam search decoding based on the acoustic features to obtain a second candidate text for recognition and its second recognition score; the text determination module 65 is used to determine the target text for recognition of the speech to be recognized based on the first candidate text for recognition and its first recognition score, and the second candidate text for recognition and its second recognition score.

[0072] The above scheme, on the one hand, decodes the state sequence using a finite state converter to obtain the first candidate text and its first recognition score, which helps improve the accuracy of the first candidate text and its first recognition score. On the other hand, it performs beam search decoding based on acoustic features to obtain the second candidate text and its second recognition score, which helps improve the stability of the acoustic feature decoding process and also improves the efficiency of obtaining the second candidate text and its second recognition score. Based on this, the target text for the speech to be recognized is determined using the first candidate text and its first recognition score, and the second candidate text and its second recognition score. Compared to determining the target text based on a single candidate text, this method helps improve the accuracy of the target text. Therefore, it can improve the accuracy of speech recognition.

[0073] In some disclosed embodiments, the second decoding module 64 includes a segmentation module, which is used to segment the audio frames in the speech to be recognized based on the silence state in the state sequence to obtain several audio segments; the second decoding module 64 includes an encoding submodule, which is used to encode each audio segment based on the acoustic features of the audio frames in the audio segment to obtain the encoding features of the audio segment; the second decoding module 64 includes a decoding submodule, which is used to perform beam search decoding based on the encoding features of several audio segments to obtain the second candidate recognition text and its second recognition score.

[0074] Therefore, encoding several audio segments to obtain their encoding features, and then performing beam search decoding based on these features to obtain the second candidate recognition text and its second recognition score, helps to improve the accuracy of the second candidate recognition text and its second recognition score, thereby improving the accuracy of speech recognition.

[0075] In some disclosed embodiments, the duration of an audio segment is not shorter than a duration threshold. The segmentation module includes a segmentation unit, which is used to segment the audio frames in the speech to be recognized according to the audio frames corresponding to the silence state in the state sequence to obtain several candidate segments. The segmentation module also includes a splicing unit, which is used to splice candidate segments with durations shorter than the duration threshold with their adjacent candidate segments in response to the candidate segments having durations shorter than the duration threshold, until the duration after splicing is not shorter than the duration threshold.

[0076] Therefore, the duration of the audio segment should not be shorter than the duration threshold. This helps to avoid poor decoding and numerous deletion errors when the speech to be recognized is long, thereby improving the accuracy of speech recognition.

[0077] In some disclosed embodiments, the first decoding module 63 includes a first conversion submodule, which is used to convert the state sequence into a phoneme sequence; the first decoding module 63 includes a second conversion submodule, which is used to convert the phoneme sequence based on a finite state converter packed with a dictionary to obtain candidate words corresponding to each phoneme in the phoneme sequence; the first decoding module 63 includes a processing submodule, which is used to process the candidate sentence composed of any candidate words corresponding to each phoneme in the phoneme sequence based on a finite state converter packed with a language model to obtain a first candidate recognition text and its first recognition score.

[0078] Therefore, by using a finite-state converter packaged by a language model to process candidate sentences composed of any candidate words corresponding to each phoneme in the phoneme sequence, it helps to improve the accuracy of the first candidate text recognition, and thus improve the accuracy of speech recognition.

[0079] In some publicly available embodiments, both the dictionary-packaged finite-state converter and the language model-packaged finite-state converter are matched to the domain of the speech to be recognized.

[0080] In some disclosed embodiments, the second decoding module 64 includes a weighting submodule, which is used to, in response to the total decoding score up to the current decoding time being lower than a preset threshold, obtain a first weighting coefficient based on the path length of the decoding path up to the current decoding time, and weight the prediction probability value of the predicted end character at the current decoding time based on the first weighting coefficient, so as to increase the prediction probability value of the end character; the second decoding module 64 includes a determining submodule, which is used to determine whether the decoding path ends decoding at the current decoding time based on the prediction probability of each preset character in the preset dictionary at the current decoding time and the weighted prediction probability of the end character.

[0081] Therefore, when the total decoding score at the current decoding moment is lower than the preset threshold, the prediction probability of the end character at the current decoding moment is weighted to increase the prediction probability of the end character. This helps to suppress the situation of erroneous triggering of random characters during speech recognition, thereby improving the accuracy of speech recognition.

[0082] In some disclosed embodiments, the second decoding module 64 includes an acquisition submodule, which is used to acquire, for each decoded character in the second candidate recognition text, the attention weight of the decoded character with each audio frame at the decoding time corresponding to the decoded character; the second decoding module 64 includes a statistics submodule, which is used to calculate the sum of the attention weights of each audio frame with each decoded character for each audio frame, and obtain the sub-penalty score of the audio frame based on the smaller value between the sum of the weights and a preset value; the second decoding module 64 includes a fusion submodule, which is used to fuse based on the sub-penalty scores of each audio frame to obtain the coverage penalty score in the second recognition score of the second candidate recognition text.

[0083] Therefore, the second recognition score includes a coverage penalty score, which can be combined with the coverage situation that occurs in the speech recognition process, i.e. the problem of missing words. When coverage occurs in speech recognition, the attention weight of the acoustic features corresponding to the covered part is lower in the second recognition score of the second candidate text, which helps to suppress the score of this path and thus improve the accuracy of the second candidate text and its second recognition score.

[0084] In some disclosed embodiments, the second decoding module 64 includes a calculation submodule, which is used to determine a second weighting coefficient based on the cumulative number of character repetitions on the decoding path where the second candidate recognized text is located in response to the repetition of the decoded character at the current decoding time with the decoded character at a historical decoding time; the second decoding module 64 includes a processing submodule, which is used to weight the initial decoding score of the decoded character at the current decoding time based on the second weighting coefficient to obtain the final decoding score of the decoded character at the current decoding time.

[0085] Therefore, the second recognition score includes the decoding score of each decoded character in the second candidate recognition text. It can be combined with the decoding situation that occurs during the speech recognition process, that is, the situation where the decoding result is repeated. The attention weight of the acoustic feature corresponding to the part of the decoding result that is repeated is lower in the second recognition score of the second candidate recognition text, which helps to suppress the score of this path and thus improve the accuracy of the second candidate recognition text and its second recognition score.

[0086] In some disclosed embodiments, the text determination module 65 includes a first acquisition submodule and a second acquisition submodule; the first acquisition submodule is used to select the first candidate identification text with the highest first identification score as the first preferred identification text, and the second candidate identification text with the highest second identification score as the second preferred identification text, and to obtain a first decision result based on the first preferred identification text and the second preferred identification text; the text determination module 65 includes a second acquisition submodule, which is used in response to the first decision result including the inability to decide the first preferred identification text or the second preferred identification text as the target identification text, to obtain a second decision result based on whether the first preferred identification text is included in the first identification text set and whether the second preferred identification text is included in the second identification text set; wherein, the first identification text set includes the second candidate identification texts located in the first first order after being sorted from high to low according to the second identification score, and the second identification text set includes the first candidate identification texts located in the second second order after being sorted from high to low according to the first identification score.

[0087] In some disclosed embodiments, the first acquisition submodule includes a detection unit, which is used to detect whether the first preferred identification text and the second preferred identification text are the same; if so, the first decision result is determined to include deciding the first preferred identification text or the second preferred identification text as the target identification text; if not, the first preferred identification text and the second preferred identification text are detected to have repeated characters, and if only one of the first preferred identification text and the second preferred identification text has repeated characters, the first decision result is determined to include deciding the other one as the target identification text; and if both the first preferred identification text and the second preferred identification text have repeated characters or neither has repeated characters, the first decision result is determined to include not being able to decide the first preferred identification text or the second preferred identification text as the target identification text.

[0088] Therefore, by detecting whether the first and second preferred recognition texts are the same, and thus determining the method for identifying the target recognition text, the accuracy of the target recognition text can be improved, thereby further improving the accuracy of speech recognition.

[0089] In some disclosed embodiments, the second acquisition submodule includes a first response unit, a second response unit, and a third response unit; the first response unit is configured to determine a second decision result including deciding the first preferred identification text as the target identification text in response to the first preferred identification text being included in the first identification text set; the second response unit is configured to determine a second decision result including deciding the second preferred identification text as the target identification text in response to the second preferred identification text being included in the second identification text set; the third response unit is configured to determine a second decision result including being unable to decide the first preferred identification text or the second preferred identification text as the target identification text in response to the first preferred identification text not being included in the first identification text set and the second preferred identification text not being included in the second identification text set.

[0090] Therefore, by determining whether the first preferred identification text is contained in the first identification text set and whether the second preferred identification text is contained in the second identification text set, the method for determining the target identification text can be improved, which helps to improve the accuracy of the target identification text.

[0091] In some disclosed embodiments, the text determination module 65 includes a detection submodule, which is used to detect whether the first preferred identification text and the second preferred identification text have the same number of words; if so, the third decision result is determined by deciding that the text with the larger identification score between the first preferred identification text and the second preferred identification text is the target identification text; if not, the third decision result is determined based on the first identification score of the first preferred identification text, the second identification score of the second preferred identification text, and the difference in the number of words between the first preferred identification text and the second preferred identification text.

[0092] Therefore, by detecting whether the first and second preferred recognition texts have the same number of words, the method for determining the target recognition text can be determined, thereby improving the accuracy of both target recognition text and speech recognition.

[0093] In some disclosed embodiments, the detection submodule includes a determination unit, which is configured to directly determine a fourth decision result, including determining the second preferred identification text as the target identification text, in response to a third decision result including the inability to decide either the first preferred identification text or the second preferred identification text as the target identification text.

[0094] In some disclosed embodiments, the target recognition text is obtained based on a speech recognition model, which includes an encoding network, a prediction network, and a decoding network. The encoding network is used to encode acoustic features to obtain encoded features, the prediction network is used to predict the encoded features to obtain a state sequence, and the decoding network is used to perform beam search decoding based on the encoded features. The encoding network, prediction network, and decoding network are jointly trained.

[0095] Therefore, jointly training the encoding network, prediction network, and decoding network can help improve the recognition results of the speech recognition model, thereby improving the accuracy of the speech recognition results.

[0096] Please see Figure 7 , Figure 7 This is a schematic diagram of a framework of an embodiment of the electronic device 70 of this application. The electronic device 70 includes a memory 71 and a processor 72 coupled to each other. The memory 71 stores program instructions, and the processor 72 is used to execute the program instructions to implement the steps in any of the above-described embodiments of the speech recognition method. Specifically, the electronic device 70 may include, but is not limited to, desktop computers, laptops, servers, mobile phones, tablet computers, etc., and is not limited thereto.

[0097] Specifically, processor 72 controls itself and memory 71 to implement the steps in any of the above-described speech recognition method embodiments. Processor 72 can also be referred to as a CPU (Central Processing Unit). Processor 72 may be an integrated circuit chip with signal processing capabilities. Processor 72 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 72 can be implemented using integrated circuit chips.

[0098] In the above scheme, the electronic device 70 can implement the steps in any of the above speech recognition method embodiments. On the one hand, it decodes the state sequence based on a finite state converter to obtain a first candidate recognition text and its first recognition score, which helps improve the accuracy of the first candidate recognition text and its first recognition score. On the other hand, it performs beam search decoding based on acoustic features to obtain a second candidate recognition text and its second recognition score, which helps improve the stability of the acoustic feature decoding process and improves the acquisition efficiency of the second candidate recognition text and its second recognition score. On this basis, the target recognition text of the speech to be recognized is determined based on the first candidate recognition text and its first recognition score and the second candidate recognition text and its second recognition score. Compared with determining the target recognition text based on a single candidate recognition text, this helps improve the accuracy of the target recognition text. Therefore, the accuracy of speech recognition can be improved.

[0099] Please see Figure 8 , Figure 8 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium 80 of this application. The computer-readable storage medium 80 stores program instructions 81 that can be executed by a processor. The program instructions 81 are used to implement the steps in any of the above-described embodiments of the speech recognition method.

[0100] The above scheme, implemented by the computer-readable storage medium 80, allows for the execution of the steps in any of the above speech recognition method embodiments. On one hand, it decodes the state sequence using a finite state converter to obtain a first candidate text and its first recognition score, which helps improve the accuracy of the first candidate text and its first recognition score. On the other hand, it performs beam search decoding based on acoustic features to obtain a second candidate text and its second recognition score, which helps improve the stability of the acoustic feature decoding process and simultaneously improves the efficiency of acquiring the second candidate text and its second recognition score. Furthermore, based on the first candidate text and its first recognition score, and the second candidate text and its second recognition score, the target text for the speech to be recognized is determined. Compared to determining the target text based on a single candidate text, this method helps improve the accuracy of the target text. Therefore, it can improve the accuracy of speech recognition.

[0101] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0102] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0103] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0104] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0105] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0106] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0107] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, the personal information processing rules are clearly informed through signs / information, and authorization is obtained through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.< / os>

Claims

1. A speech recognition method, characterized in that, include: Extract the acoustic features of each audio frame of the speech to be recognized; Based on the acoustic features, a prediction is made to obtain a state sequence, and the state sequence is decoded using a finite state converter to obtain a first candidate text for recognition and its first recognition score; wherein, the state sequence includes the pronunciation state corresponding to the audio frame; Based on the acoustic features, beam search decoding is performed to obtain the second candidate text for recognition and its second recognition score. Determining the target recognition text of the speech to be recognized based on the first candidate recognition text and its first recognition score, and the second candidate recognition text and its second recognition score, includes: taking the first candidate recognition text with the highest first recognition score as the first preferred recognition text, and taking the second candidate recognition text with the highest second recognition score as the second preferred recognition text; and obtaining a first decision result based on the first preferred recognition text and the second preferred recognition text; in response to the first decision result including the inability to decide whether the first preferred recognition text or the second preferred recognition text is the target recognition text, obtaining a second decision result based on whether the first preferred recognition text is included in the first recognition text set and whether the second preferred recognition text is included in the second recognition text set, wherein the first recognition text set includes the second candidate recognition text located in the first first order after being sorted from high to low according to the second recognition score, and the second recognition text set includes the first candidate recognition text located in the second second order after being sorted from high to low according to the first recognition score; wherein, in the event that a final decision cannot be made, the second preferred recognition text is the target recognition text.

2. The method according to claim 1, characterized in that, The process of performing beam search decoding based on the acoustic features to obtain the second candidate text and its second recognition score includes: Based on the silence state in the state sequence, the audio frame in the speech to be identified is segmented to obtain several audio segments. For each audio segment, the audio segment is encoded based on the acoustic features of the audio frames in the audio segment to obtain the encoded features of the audio segment; Based on the encoding features of the aforementioned audio segments, beam search decoding is performed to obtain the second candidate recognition text and its second recognition score.

3. The method according to claim 2, characterized in that, The duration of the audio segment is not less than a duration threshold. Based on the silence state in the state sequence, the audio frames are segmented to obtain several audio segments, including: According to the audio frames corresponding to the silence state in the state sequence, the audio frames in the speech to be recognized are segmented to obtain several candidate segments; In response to a candidate segment having a duration shorter than the duration threshold, the candidate segment with a duration shorter than the duration threshold is spliced ​​with its adjacent candidate segments until the duration of the spliced ​​segment is not shorter than the duration threshold.

4. The method according to claim 1, characterized in that, The process of decoding the state sequence based on a finite state converter to obtain the first candidate recognition text and its first recognition score includes: Convert the state sequence into a phoneme sequence; A finite-state converter based on dictionary packaging is used to transform the phoneme sequence to obtain candidate words corresponding to each phoneme in the phoneme sequence. A finite-state converter based on language model packaging processes candidate sentences composed of any candidate words corresponding to each phoneme in the phoneme sequence to obtain the first candidate recognition text and its first recognition score.

5. The method according to claim 1, characterized in that, During the execution of the beam search decoding process, the steps for determining the end of decoding for each decoding path include: In response to the total decoding score being lower than a preset threshold as of the current decoding time, a first weighting coefficient is obtained based on the path length of the decoding path as of the current decoding time, and the prediction probability value of the end character is weighted based on the first weighting coefficient to improve the prediction probability value of the end character. Based on the predicted probabilities of each preset character in the preset dictionary at the current decoding time and the weighted predicted probability of the ending character, it is determined whether the decoding path ends decoding at the current decoding time.

6. The method according to claim 1, characterized in that, The second recognition score includes a coverage penalty score, and the steps for obtaining the coverage penalty score in the second recognition score of the second candidate recognition text include: For each decoded character in the second candidate recognition text, obtain the attention weights of the decoded character and each audio frame at the decoding time corresponding to the decoded character; For each audio frame, the sum of the attention weights of the audio frame and each decoded character is calculated, and the sub-penalty score of the audio frame is obtained based on the smaller value between the sum of the weights and a preset value. The coverage penalty score in the second recognition score of the second candidate recognition text is obtained by fusing the sub-penalty scores of each audio frame.

7. The method according to claim 1, characterized in that, The second recognition score includes the decoding score of each decoded character in the second candidate recognition text, and the steps for obtaining the decoding score include: In response to the duplication of the decoded character at the current decoding moment with the decoded character at a historical decoding moment, a second weighting coefficient is determined based on the cumulative number of character repetitions on the decoding path where the second candidate recognition text is located; The initial decoding score of the decoded character at the current decoding moment is weighted based on the second weighting coefficient to obtain the final decoding score of the decoded character at the current decoding moment.

8. The method according to claim 1, characterized in that, The first decision result obtained based on the first preferred identification text and the second preferred identification text includes: Detect whether the first preferred identification text and the second preferred identification text are the same; If so, then the first decision result is determined to include deciding the first preferred identification text or the second preferred identification text as the target identification text; If not, then it is detected whether the first preferred identification text and the second preferred identification text have repeated characters. If only one of the first preferred identification text and the second preferred identification text has repeated characters, the first decision result is determined to include deciding the other one as the target identification text. If both the first preferred identification text and the second preferred identification text have repeated characters or neither has repeated characters, the first decision result is determined to include not being able to decide the first preferred identification text or the second preferred identification text as the target identification text.

9. The method according to claim 1, characterized in that, The target recognition text is obtained based on a speech recognition model, which includes an encoding network, a prediction network, and a decoding network. The encoding network is used to encode the acoustic features to obtain encoded features. The prediction network is used to predict the state sequence based on the encoded features. The decoding network is used to perform the beam search decoding based on the encoded features. The encoding network, the prediction network, and the decoding network are jointly trained.

10. A voice recognition device, characterized in that, include: The feature extraction module is used to extract the acoustic features of each audio frame of the speech to be recognized; A state prediction module is used to predict based on the acoustic features to obtain a state sequence; wherein the state sequence includes the pronunciation state corresponding to the audio frame; The first decoding module is used to decode the state sequence based on a finite state converter to obtain the first candidate recognition text and its first recognition score; The second decoding module is used to perform beam search decoding based on the acoustic features to obtain the second candidate recognition text and its second recognition score. The text determination module is used to determine the target recognition text of the speech to be recognized based on the first candidate recognition text and its first recognition score, and the second candidate recognition text and its second recognition score. This includes: selecting the first candidate recognition text with the highest first recognition score as the first preferred recognition text, and selecting the second candidate recognition text with the highest second recognition score as the second preferred recognition text; and obtaining a first decision result based on the first preferred recognition text and the second preferred recognition text. In response to the first decision result including the inability to determine whether the first preferred recognition text or the second preferred recognition text is the target recognition text, a second decision result is obtained based on whether the first preferred recognition text is included in a first recognition text set and whether the second preferred recognition text is included in a second recognition text set. The first recognition text set includes second candidate recognition texts ranked in the first first position according to the second recognition score from high to low, and the second recognition text set includes first candidate recognition texts ranked in the second second position according to the first recognition score from high to low. In the case where no decision can be made, the second preferred recognition text is used as the target recognition text.

11. An electronic device, characterized in that, The device includes a memory and a processor, wherein the memory stores program instructions and the processor executes the program instructions to implement the speech recognition method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The device stores program instructions that can be executed by a processor, the program instructions being used to implement the speech recognition method according to any one of claims 1 to 9.

Images