Speech recognition method and related device

Abstract

The invention discloses a voice recognition method and a related device, and relates to the technical field of voice processing, and the method comprises the steps: converting a universal decoding text sequence into a generalizable text sequence with lower element granularity after the universal decoding text sequence of to-be-recognized voice data is determined; expanding the generalizable text sequence by using similar elements corresponding to each element in the generalizable text sequence to obtain an expanded text sequence with corresponding granularity; based on a preset long-tail vocabulary language model, performing score re-pasting operation on the generalizable text sequence and the extended text sequence to obtain a long-tail decoded text sequence; and finally, based on the general decoding text sequence and the long-tail decoding text sequence, determining a final speech recognition result. According to the scheme, the long-tail decoding text sequence can be obtained, so that the possibility that the final speech recognition result is the long-tail decoding text sequence can be improved, and the long-tail vocabulary recognition effect of the end-to-end speech recognition model can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech recognition method and related devices. Background Art

[0002] With the rapid development of artificial intelligence technology, speech recognition, as one of the important ways of human-computer interaction, has been widely used in mobile devices, customer service systems, smart homes and other fields. The end-to-end speech recognition model that jointly models the acoustic model, pronunciation model, and speech model is the most popular speech recognition framework at present. However, the current end-to-end speech recognition model has poor recognition effect on long-tail words (such as rare or proper nouns).

[0003] At present, the recognition effect of the end-to-end speech recognition model for long-tail words is generally improved by re-score, hot word incentive or external language model incentive. However, the improvement of the recognition effect of the long-tail words of the end-to-end speech recognition model by re-score, hot word incentive or external language model incentive is not obvious.

[0004] Therefore, how to provide a speech recognition method to effectively improve the long-tail vocabulary recognition effect of the end-to-end speech recognition model has become an urgent problem to be solved by technical personnel in this field. Summary of the invention

[0005] In view of the above problems, the present application provides a speech recognition method and related devices to achieve the purpose of effectively improving the long-tail vocabulary recognition effect of the end-to-end speech recognition model. The specific scheme is as follows:

[0006] The first aspect of the present application provides a speech recognition method, comprising:

[0007] Determine a universal decoding text sequence of the speech data to be recognized; each element in the universal decoding text sequence is a text unit of a first granularity;

[0008] Converting the universal decoded text sequence into a generalizable text sequence, wherein each element in the generalizable text sequence is a text unit of a second granularity, and the second granularity is smaller than the first granularity;

[0009] Expanding the generalizable text sequence using similar elements corresponding to each element in the generalizable text sequence to obtain an expanded text sequence, wherein each element in the expanded text sequence is a text unit of a second granularity;

[0010] Based on a preset long-tail vocabulary language model, the generalizable text sequence and the expanded text sequence are re-partitioned to obtain a long-tail decoded text sequence;

[0011] Based on the common decoded text sequence and the long-tail decoded text sequence, a final speech recognition result is determined.

[0012] In a possible implementation, the method for determining the similar elements corresponding to each element in the generalizable text sequence includes:

[0013] Determine a similarity matrix, wherein the similarity matrix includes similarities between text units of the second granularity;

[0014] Based on the similarity matrix, the similar elements corresponding to the elements in the generalizable text sequence are determined.

[0015] In a possible implementation, the step of expanding the generalizable text sequence using similar elements corresponding to each element in the generalizable text sequence to obtain an expanded text sequence includes:

[0016] The similar elements corresponding to the elements in the generalizable text sequence are used to replace the elements in the generalizable text sequence to obtain an expanded text sequence.

[0017] In a possible implementation, determining the similarity matrix includes:

[0018] Obtaining text sequence pairs, each text sequence pair comprising a decoded text sequence and a corresponding text sequence label; each element in the text sequence pair is a text unit of the first granularity;

[0019] Converting the text sequence pair into a generalizable text sequence pair, wherein each element in the generalizable text sequence pair is a text unit of a second granularity, and the second granularity is smaller than the first granularity;

[0020] Based on the similarity between the pairs of error elements in the generalizable text sequence pairs, the similarity between the text units of the second granularity is statistically obtained;

[0021] The similarity matrix is ​​constructed based on the similarities between the text units of the second granularity.

[0022] In a possible implementation, determining the similar elements corresponding to each element in the generalizable text sequence based on the similarity matrix includes:

[0023] For each element in the generalizable text sequence, determining whether there is a text unit of a second granularity with a similarity greater than a preset threshold in the similarity matrix;

[0024] If it exists, the text unit of the second granularity with a similarity greater than a preset threshold is taken as a similar element corresponding to the element.

[0025] In a possible implementation, determining a final speech recognition result based on the universal decoded text sequence and the long-tail decoded text sequence includes:

[0026] Obtaining the score of the universal decoding text sequence, the score of the long-tail decoding text sequence, a preset universal decoding weight, and a preset long-tail decoding weight;

[0027] Determining a final score of the universal decoding text sequence based on the score of the universal decoding text sequence and the preset universal decoding weight;

[0028] Determining a final score of the long-tail decoding text sequence based on the score of the long-tail decoding text sequence and the preset long-tail decoding weight;

[0029] The decoded text sequence with the highest final score is determined as the final speech recognition result.

[0030] A second aspect of the present application provides a speech recognition device, comprising:

[0031] A universal decoding text sequence determination unit, used to determine a universal decoding text sequence of the speech data to be recognized; each element in the universal decoding text sequence is a text unit of a first granularity;

[0032] A text sequence conversion unit, used for converting the universal decoding text sequence into a generalizable text sequence, each element in the generalizable text sequence is a text unit of a second granularity, and the second granularity is smaller than the first granularity;

[0033] A text sequence expansion unit, used to expand the generalizable text sequence using similar elements corresponding to each element in the generalizable text sequence to obtain an expanded text sequence, wherein each element in the expanded text sequence is a text unit of a second granularity;

[0034] A long-tail decoded text sequence determination unit, configured to perform a re-segmentation operation on the generalizable text sequence and the expanded text sequence based on a preset long-tail vocabulary language model to obtain a long-tail decoded text sequence;

[0035] The final speech recognition result determination unit is used to determine the final speech recognition result based on the common decoded text sequence and the long-tail decoded text sequence.

[0036] In a possible implementation, the text sequence expansion unit includes:

[0037] A similarity matrix determination unit, used to determine a similarity matrix, wherein the similarity matrix includes similarities between text units of the second granularity;

[0038] A similar element determination unit is used to determine the similar elements corresponding to each element in the generalizable text sequence based on the similarity matrix.

[0039] In a possible implementation, the text sequence expansion unit is specifically used to:

[0040] The similar elements corresponding to the elements in the generalizable text sequence are used to replace the elements in the generalizable text sequence to obtain an expanded text sequence.

[0041] In a possible implementation, the similarity matrix determination unit is specifically used to:

[0042] Obtaining text sequence pairs, each text sequence pair comprising a decoded text sequence and a corresponding text sequence label; each element in the text sequence pair is a text unit of the first granularity;

[0043] Converting the text sequence pair into a generalizable text sequence pair, each element in the generalizable text sequence pair being a text unit of a second granularity, wherein the second granularity is smaller than the first granularity;

[0044] Based on the similarity between the pairs of error elements in the generalizable text sequence pairs, the similarity between the text units of the second granularity is statistically obtained;

[0045] The similarity matrix is ​​constructed based on the similarities between the text units of the second granularity.

[0046] In a possible implementation, the similar element determining unit is specifically configured to:

[0047] For each element in the generalizable text sequence, determining whether there is a text unit of a second granularity with a similarity greater than a preset threshold in the similarity matrix;

[0048] If it exists, the text unit of the second granularity with a similarity greater than a preset threshold is taken as a similar element corresponding to the element.

[0049] In a possible implementation, the final speech recognition result determination unit is specifically used to:

[0050] Obtaining the score of the universal decoding text sequence, the score of the long-tail decoding text sequence, a preset universal decoding weight, and a preset long-tail decoding weight;

[0051] Determining a final score of the universal decoding text sequence based on the score of the universal decoding text sequence and the preset universal decoding weight;

[0052] Determining a final score of the long-tail decoding text sequence based on the score of the long-tail decoding text sequence and the preset long-tail decoding weight;

[0053] The decoded text sequence with the highest final score is determined as the final speech recognition result.

[0054] The third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the speech recognition method of the first aspect or any implementation of the first aspect.

[0055] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0056] The memory is used to store computer programs;

[0057] The processor is used to execute the computer program so that the electronic device can implement the speech recognition method of the above-mentioned first aspect or any implementation manner of the first aspect.

[0058] In a fifth aspect, the present application provides a computer-readable storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the speech recognition method of the above-mentioned first aspect or any implementation of the first aspect.

[0059] By means of the above technical scheme, the speech recognition method and related devices provided by the present application. After determining the universal decoding text sequence of the speech data to be recognized, first convert the universal decoding text sequence into a generalizable text sequence with a lower element granularity, and then use the similar elements corresponding to each element in the generalizable text sequence to expand the generalizable text sequence to obtain an expanded text sequence of the corresponding granularity; and based on the preset long-tail vocabulary language model, perform a re-distribution operation on the generalizable text sequence and the expanded text sequence to obtain a long-tail decoding text sequence; finally, based on the universal decoding text sequence and the long-tail decoding text sequence, determine the final speech recognition result. Since the scheme can obtain a long-tail decoding text sequence, it can increase the possibility that the final speech recognition result is a long-tail decoding text sequence, and therefore, can effectively improve the long-tail vocabulary recognition effect of the end-to-end speech recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0061] Figure 1 A flowchart of a speech recognition method provided in an embodiment of the present application;

[0062] Figure 2 A schematic diagram of a flow chart of a method for determining similar elements corresponding to each element in a generalized text sequence provided in an embodiment of the present application;

[0063] Figure 3 A flowchart of a method for determining a similarity matrix provided in an embodiment of the present application;

[0064] Figure 4 A flowchart of a method for determining a final speech recognition result based on a general decoded text sequence and a long-tail decoded text sequence provided in an embodiment of the present application;

[0065] Figure 5 A schematic diagram of the structure of a speech recognition device provided in an embodiment of the present application;

[0066] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0068] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0069] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0070] With the rapid development of artificial intelligence technology, speech recognition, as one of the important ways of human-computer interaction, has been widely used in mobile devices, customer service systems, smart homes and other fields.

[0071] Traditional speech recognition systems usually need to build a pronunciation system in advance according to the specific language to obtain accurate phoneme segmentation and pronunciation dictionaries. This step is often time-consuming and error-prone. In addition, since the system contains multiple independent modules, the errors of each module may be passed on to the next module, resulting in error accumulation.

[0072] Therefore, in the past few years, end-to-end models have attracted widespread attention in the field of speech recognition. In speech recognition tasks, end-to-end models replace the traditional combination of acoustic models, pronunciation models, and language models with a single neural network, and can be optimized through a unified goal, greatly simplifying the construction and maintenance of speech recognition systems. At the same time, combined with the unique advantages of Internet big data, the end-to-end speech recognition model that jointly models the acoustic model, pronunciation model, and speech model has defeated the traditional speech recognition system and has become the hottest speech recognition framework at present.

[0073] The best end-to-end speech recognition model currently available is mainly a neural network structure based on the attention model. This structure can convert speech sequences word by word into corresponding recognition result sequences. It consists of an encoder and a decoder, where the encoder can be a popular network structure such as DNN, CNN, RNN, LSTM, TRANSFORMER, etc., which is used to extract high-order latent features from speech features; the decoder can be a network structure such as RNN, LSTM, etc., and it also contains a set of attention modules inside, which converts the high-order latent features extracted by the encoder into a probability distribution of a set of character sequences, and then obtains the speech recognition sequence through this probability distribution.

[0074] The end-to-end speech recognition model avoids the early manual design process and can achieve good performance through large-scale data training. However, for long-tail words (such as uncommon or proper nouns), generally speaking, they appear less frequently in the standard training set, so the end-to-end speech recognition model may not be able to learn the appropriate spelling and contextual usage of long-tail words, making it difficult for the end-to-end speech recognition model to capture enough information to accurately identify long-tail words. As a result, the current end-to-end speech recognition model has poor recognition effect on long-tail words (such as uncommon or proper nouns).

[0075] At present, the recognition effect of the end-to-end speech recognition model on long-tail words is generally improved through re-score, hot word incentives or external language model incentives.

[0076] Re-scoring means retaining several of the most likely recognition results after model decoding is completed, and then re-scoring and sorting these recognition results through the language model. Hot word incentives are to give additional scoring incentives to some specified words during the model decoding process to ensure that they are not cut during the decoding process. External language model incentives are to constrain the decoding results through a language model generated by customized sentences that are highly related to long-tail words.

[0077] However, in the end-to-end speech recognition model, the long-tail vocabulary scores are extremely low during the speech recognition process and are easily trimmed during the decoding process. The re-score method cannot be used. As for the hot word excitation or external language model excitation methods, on the one hand, due to the low score of the acoustic model, it is difficult to balance the boundary between successful excitation and false triggering during the decoding process. On the other hand, there may be a situation where the long-tail vocabulary does not appear in all the candidates and has no chance of being excited at all.

[0078] It can be seen that the long-tail vocabulary recognition effect of the end-to-end speech recognition model is not significantly improved by re-scoring, hot word incentives, or external language model incentives.

[0079] In order to solve the above problems, the embodiment of the present application provides a speech recognition method, which can effectively improve the long-tail vocabulary recognition effect of the end-to-end speech recognition model.

[0080] The speech recognition method of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0081] Reference Figure 1 , Figure 1 A flow chart of a speech recognition method provided in an embodiment of the present application is shown as follows: Figure 1 As shown, a speech recognition method provided in an embodiment of the present application may include the following steps, and these steps are described in detail below.

[0082] S101: Determine a universal decoded text sequence of speech data to be recognized; each element in the universal decoded text sequence is a text unit of a first granularity;

[0083] In the present application, the speech data to be recognized may be speech data in any language and in any field. The decoded text sequence of the speech data to be recognized may be obtained after an end-to-end speech recognition model performs speech recognition on the speech data to be recognized.

[0084] In one possible implementation, determining the universal decoding text sequence of the speech data to be recognized includes: first extracting speech features from the speech data to be recognized, the speech features may be common speech features such as FilterBank, MFCC, etc., and FilterBank features may be used in the application. Then the speech features of the speech data to be recognized are input into an end-to-end speech recognition model, the end-to-end speech recognition model is combined with a universal language model for decoding, and a preset number of text sequences with the top scores obtained in the last decoding step are used as the universal decoding text sequences of the speech data to be recognized, and the scores of each universal decoding text sequence are also determined. The value of the preset number can be set based on the scenario requirements, and is not limited in this application.

[0085] In one possible implementation, the end-to-end speech recognition model, when combined with a general language model for decoding, can use a beam search decoding method, which is consistent with the standard end-to-end speech recognition model decoding process and is not described in detail in this application.

[0086] S102: converting the universal decoded text sequence into a generalizable text sequence, wherein each element in the generalizable text sequence is a text unit of a second granularity, and the second granularity is smaller than the first granularity;

[0087] In the present application, the number of universal decoding text sequences may be multiple, and for each of the universal decoding text sequences, the universal decoding text sequence needs to be converted into a generalizable text sequence.

[0088] In one possible implementation, taking Chinese as an example, the first granularity may be the word level, and the second granularity may be the character level, the syllable level, or the phoneme level; taking English as an example, the first granularity may be the word level, and the second granularity may be the subword level or the letter level.

[0089] In a possible implementation, the decoded text sequence can be converted into a generalizable text sequence by using a whole word splitting method or a pronunciation prediction method. It should be noted that the whole word splitting method and the pronunciation prediction method are existing mature technologies and will not be described in detail in this application.

[0090] S103: expanding the generalizable text sequence using similar elements corresponding to each element in the generalizable text sequence to obtain an expanded text sequence, wherein each element in the expanded text sequence is a text unit of a second granularity;

[0091] In the present application, the number of universal decoding text sequences can be multiple, and for each of the universal decoding text sequences, the universal decoding text sequence needs to be converted into a generalizable text sequence, so that multiple generalizable text sequences can be obtained. Therefore, in the present application, for each of the generalizable text sequences, the generalizable text sequence needs to be expanded using similar elements corresponding to each element in the generalizable text sequence to obtain an expanded text sequence.

[0092] In a possible implementation, the method of expanding the generalizable text sequence using similar elements corresponding to each element in the generalizable text sequence to obtain an expanded text sequence includes: replacing elements of the generalizable text sequence using similar elements corresponding to each element in the generalizable text sequence to obtain an expanded text sequence.

[0093] Specifically, for each element of each generalizable text sequence, determine whether the element has a similar element, and if the element has a similar element, replace the element with the similar element to obtain an extended text sequence. It should be noted that there may be multiple elements in the generalizable sequence that have similar elements, and the multiple elements can be replaced in whole or in part to obtain one or more extended text sequences.

[0094] S104: performing a re-segmentation operation on the generalizable text sequence and the expanded text sequence based on a preset long-tail vocabulary language model to obtain a long-tail decoded text sequence;

[0095] In the present application, the preset long-tail vocabulary language model is an existing mature technology and will not be described in detail in the present application. The re-score operation on the text sequence based on the preset long-tail vocabulary language model is also an existing mature technology and will not be described in detail in the present application. The re-score operation is performed on the generalizable text sequence and the extended text sequence based on the preset long-tail vocabulary language model, and while obtaining the long-tail decoded text sequence, the scores of each long-tail decoded text sequence can also be obtained.

[0096] S105: Determine a final speech recognition result based on the general decoded text sequence and the long-tail decoded text sequence.

[0097] In the present application, the final speech recognition result can be determined based on the score of the common decoded text sequence and the score of the long-tail decoded text sequence.

[0098] The speech recognition method provided in this embodiment, after determining the universal decoding text sequence of the speech data to be recognized, first converts the universal decoding text sequence into a generalizable text sequence with a lower element granularity, and then expands the generalizable text sequence using similar elements corresponding to each element in the generalizable text sequence to obtain an expanded text sequence of corresponding granularity; and based on a preset long-tail vocabulary language model, performs a re-distribution operation on the generalizable text sequence and the expanded text sequence to obtain a long-tail decoding text sequence; finally, based on the universal decoding text sequence and the long-tail decoding text sequence, determines the final speech recognition result. Since the scheme can obtain a long-tail decoding text sequence, it can increase the possibility that the final speech recognition result is a long-tail decoding text sequence, and therefore, can effectively improve the long-tail vocabulary recognition effect of the end-to-end speech recognition model.

[0099] In another embodiment of the present application, the method for determining the similar elements corresponding to each element in the generalizable text sequence is described in detail.

[0100] Reference Figure 2 , Figure 2 A flowchart of a method for determining similar elements corresponding to each element in a generalized text sequence provided in an embodiment of the present application, the method may include the following steps:

[0101] S201: Determine a similarity matrix, wherein the similarity matrix includes similarities between text units of the second granularity;

[0102] S202: Based on the similarity matrix, determine the similar elements corresponding to each element in the generalizable text sequence.

[0103] In a possible implementation, determining the similar elements corresponding to each element in the generalizable text sequence based on the similarity matrix includes:

[0104] For each element in the generalizable text sequence, determining whether there is a text unit of a second granularity with a similarity greater than a preset threshold in the similarity matrix;

[0105] If it exists, the text unit of the second granularity with a similarity greater than a preset threshold is taken as a similar element corresponding to the element.

[0106] In another embodiment of the present application, a specific implementation method of determining the similarity matrix is ​​described in detail.

[0107] Reference Figure 3 , Figure 3 A flowchart of a method for determining a similarity matrix provided in an embodiment of the present application, the method may include the following steps:

[0108] S301: Acquire text sequence pairs, each of which contains a decoded text sequence and a corresponding text sequence label; each element in the text sequence pair is a text unit of the first granularity;

[0109] In this application, text sequence pairs in different scenarios can be collected, and different collection strategies can be adopted for different scenarios. For languages ​​with large data volumes or general scenarios, multi-domain scenarios can be collected to improve the generalization of text sequence pairs; for specific scenarios, the TopN decoded text sequences and corresponding text sequence labels in the scenario can be collected to improve the specificity of the text sequence pairs.

[0110] S302: converting the text sequence pair into a generalizable text sequence pair, wherein each element in the generalizable text sequence pair is a text unit of a second granularity, and the second granularity is smaller than the first granularity;

[0111] In the present application, reference may be made to the related description of converting the general decoded text sequence into a generalizable text sequence, which will not be repeated here.

[0112] S303: based on the similarity between the pairs of erroneous elements in the generalizable text sequence pairs, statistically obtain the similarity between the text units of the second granularity;

[0113] In the present application, based on the edit distance of the generalizable text sequence pair, the wrong element pairing in the generalizable text sequence pair can be determined. The specific implementation method is an existing mature technology and will not be described in detail in the present application.

[0114] In a possible implementation, the similarity between each pair of error elements may be measured using the cosine distance.

[0115] It should be noted that in the present application, there may be multiple generalizable text sequence pairs containing the same erroneous element pairing, and the similarity of the erroneous element pairing in each generalizable text sequence pair may be different. In this case, the similarities that meet the requirements can be screened out by presetting a similarity threshold, and then the similarities that meet the requirements are averaged to obtain the final similarity of the erroneous element pairing.

[0116] S304: constructing the similarity matrix based on the similarities between the text units of the second granularity.

[0117] In another embodiment of the present application, a specific implementation method of determining the final speech recognition result based on the general decoded text sequence and the long-tail decoded text sequence is described.

[0118] In a possible implementation, the decoded text sequence with the highest score may be determined as the final speech recognition result.

[0119] Taking into account that different scenarios require different recognition effects for long-tail words, in another possible implementation, general decoding weights and preset long-tail decoding weights can be preset, and the scores of each decoded text sequence can be re-determined in combination with the preset general decoding weights and the preset long-tail decoding weights to further determine the final speech recognition result.

[0120] Reference Figure 4 , Figure 4 A flowchart of a method for determining a final speech recognition result based on a general decoded text sequence and a long-tail decoded text sequence provided in an embodiment of the present application, the method may include the following steps:

[0121] S401: Obtaining the score of the general decoding text sequence, the score of the long-tail decoding text sequence, a preset general decoding weight, and a preset long-tail decoding weight;

[0122] In the present application, specific values ​​of the preset general decoding weight and the preset long-tail decoding weight can be set based on scenario requirements and are not limited in this application.

[0123] S402: Determine a final score of the universal decoding text sequence based on the score of the universal decoding text sequence and the preset universal decoding weight;

[0124] S403: Determine a final score of the long-tail decoding text sequence based on the score of the long-tail decoding text sequence and the preset long-tail decoding weight;

[0125] S404: Determine the decoded text sequence with the highest final score as the final speech recognition result.

[0126] In summary, the present application proposes a speech recognition method, which degenerates the initial speech recognition results into smaller units during the decoding process of an end-to-end speech recognition model, and then combines them with a long-tail vocabulary language model for re-stimulation, thereby avoiding the problem that either directly using long-tail word excitation or long-tail vocabulary language model excitation cannot decode the correct result, and ultimately can effectively improve the long-tail vocabulary recognition effect of the end-to-end speech recognition model.

[0127] A speech recognition method provided in an embodiment of the present application is introduced above, and a device for executing the above speech recognition method will be introduced below.

[0128] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of a speech recognition device provided in an embodiment of the present application. Figure 5 As shown, the speech recognition device comprises:

[0129] A universal decoding text sequence determining unit 11 is used to determine a universal decoding text sequence of the speech data to be recognized; each element in the universal decoding text sequence is a text unit of a first granularity;

[0130] A text sequence conversion unit 12, configured to convert the universal decoded text sequence into a generalizable text sequence, wherein each element in the generalizable text sequence is a text unit of a second granularity, and the second granularity is smaller than the first granularity;

[0131] A text sequence expansion unit 13, used to expand the generalizable text sequence using similar elements corresponding to each element in the generalizable text sequence to obtain an expanded text sequence, wherein each element in the expanded text sequence is a text unit of the second granularity;

[0132] A long-tail decoded text sequence determining unit 14 is used to perform a re-segmentation operation on the generalizable text sequence and the extended text sequence based on a preset long-tail vocabulary language model to obtain a long-tail decoded text sequence;

[0133] The final speech recognition result determination unit 15 is used to determine the final speech recognition result based on the common decoded text sequence and the long-tail decoded text sequence.

[0134] In a possible implementation, the text sequence expansion unit includes:

[0135] A similarity matrix determination unit, used to determine a similarity matrix, wherein the similarity matrix includes similarities between text units of the second granularity;

[0136] A similar element determination unit is used to determine the similar elements corresponding to each element in the generalizable text sequence based on the similarity matrix.

[0137] In a possible implementation, the text sequence expansion unit is specifically used to:

[0138] The similar elements corresponding to the elements in the generalizable text sequence are used to replace the elements in the generalizable text sequence to obtain an expanded text sequence.

[0139] In a possible implementation, the similarity matrix determination unit is specifically used to:

[0140] Obtaining text sequence pairs, each text sequence pair comprising a decoded text sequence and a corresponding text sequence label; each element in the text sequence pair is a text unit of the first granularity;

[0141] Converting the text sequence pair into a generalizable text sequence pair, each element in the generalizable text sequence pair being a text unit of a second granularity, wherein the second granularity is smaller than the first granularity;

[0142] Based on the similarity between the pairs of error elements in the generalizable text sequence pairs, the similarity between the text units of the second granularity is statistically obtained;

[0143] The similarity matrix is ​​constructed based on the similarities between the text units of the second granularity.

[0144] In a possible implementation, the similar element determining unit is specifically configured to:

[0145] For each element in the generalizable text sequence, determining whether there is a text unit of a second granularity with a similarity greater than a preset threshold in the similarity matrix;

[0146] If it exists, the text unit of the second granularity with a similarity greater than a preset threshold is taken as a similar element corresponding to the element.

[0147] In a possible implementation, the final speech recognition result determination unit is specifically used to:

[0148] Obtaining the score of the universal decoding text sequence, the score of the long-tail decoding text sequence, a preset universal decoding weight, and a preset long-tail decoding weight;

[0149] Determining a final score of the universal decoding text sequence based on the score of the universal decoding text sequence and the preset universal decoding weight;

[0150] Determining a final score of the long-tail decoding text sequence based on the score of the long-tail decoding text sequence and the preset long-tail decoding weight;

[0151] The decoded text sequence with the highest final score is determined as the final speech recognition result.

[0152] The present application also provides an electronic device in an embodiment. Figure 6 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0153] like Figure 6As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0154] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0155] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any of the speech recognition methods provided in the embodiments of the present application.

[0156] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any speech recognition method provided in the embodiment of the present application.

[0157] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0158] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0159] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0160] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A speech recognition method, characterized in that: include: Determine a universal decoding text sequence of the speech data to be recognized, wherein each element in the universal decoding text sequence is a text unit of a first granularity; Converting the universal decoded text sequence into a generalizable text sequence, wherein each element in the generalizable text sequence is a text unit of a second granularity, and the second granularity is smaller than the first granularity; Expanding the generalizable text sequence using similar elements corresponding to each element in the generalizable text sequence to obtain an expanded text sequence, wherein each element in the expanded text sequence is a text unit of a second granularity; Based on a preset long-tail vocabulary language model, the generalizable text sequence and the expanded text sequence are re-partitioned to obtain a long-tail decoded text sequence; Based on the common decoded text sequence and the long-tail decoded text sequence, a final speech recognition result is determined.

2. The method according to claim 1, characterized in that The method for determining the similar elements corresponding to each element in the generalizable text sequence includes: Determine a similarity matrix, wherein the similarity matrix includes similarities between text units of the second granularity; Based on the similarity matrix, the similar elements corresponding to the elements in the generalizable text sequence are determined.

3. The method according to claim 1, characterized in that The step of expanding the generalizable text sequence by using similar elements corresponding to each element in the generalizable text sequence to obtain an expanded text sequence includes: The similar elements corresponding to the elements in the generalizable text sequence are used to replace the elements in the generalizable text sequence to obtain an expanded text sequence.

4. The method according to claim 2, characterized in that: The determining of the similarity matrix comprises: Obtaining text sequence pairs, each text sequence pair comprising a decoded text sequence and a corresponding text sequence label; each element in the text sequence pair is a text unit of the first granularity; Converting the text sequence pair into a generalizable text sequence pair, each element in the generalizable text sequence pair being a text unit of a second granularity, wherein the second granularity is smaller than the first granularity; Based on the similarity between the pairs of error elements in the generalizable text sequence pairs, the similarity between the text units of the second granularity is statistically obtained; The similarity matrix is ​​constructed based on the similarities between the text units of the second granularity.

5. The method according to claim 2, characterized in that: The determining, based on the similarity matrix, the similar elements corresponding to the elements in the generalizable text sequence comprises: For each element in the generalizable text sequence, determining whether there is a text unit of a second granularity with a similarity greater than a preset threshold in the similarity matrix; If it exists, the text unit of the second granularity with a similarity greater than a preset threshold is taken as a similar element corresponding to the element.

6. The method according to claim 1, characterized in that The determining of a final speech recognition result based on the universal decoded text sequence and the long-tail decoded text sequence includes: Obtaining the score of the universal decoding text sequence, the score of the long-tail decoding text sequence, a preset universal decoding weight, and a preset long-tail decoding weight; Determining a final score of the universal decoding text sequence based on the score of the universal decoding text sequence and the preset universal decoding weight; Determining a final score of the long-tail decoding text sequence based on the score of the long-tail decoding text sequence and the preset long-tail decoding weight; The decoded text sequence with the highest final score is determined as the final speech recognition result.

7. A speech recognition device, characterized in that: include: A universal decoding text sequence determination unit, used to determine a universal decoding text sequence of the speech data to be recognized; each element in the universal decoding text sequence is a text unit of a first granularity; A text sequence conversion unit, used for converting the universal decoding text sequence into a generalizable text sequence, each element in the generalizable text sequence is a text unit of a second granularity, and the second granularity is smaller than the first granularity; A text sequence expansion unit, used to expand the generalizable text sequence using similar elements corresponding to each element in the generalizable text sequence to obtain an expanded text sequence, wherein each element in the expanded text sequence is a text unit of a second granularity; A long-tail decoded text sequence determining unit, configured to perform a re-segmentation operation on the generalizable text sequence and the extended text sequence based on a preset long-tail vocabulary language model to obtain a long-tail decoded text sequence; The final speech recognition result determination unit is used to determine the final speech recognition result based on the common decoded text sequence and the long-tail decoded text sequence.

8. A computer program product, characterized in that It comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the speech recognition method as claimed in any one of claims 1 to 6.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the speech recognition method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the speech recognition method as described in any one of claims 1 to 6.

Images