Error correction method and apparatus for speech recognition

Abstract

Embodiments of the present application disclose a speech recognition error correction method and device, and relate to the technical field of speech processing and artificial intelligence. The method comprises: obtaining N first recognition results output by a speech recognition system for to-be-recognized speech, wherein N is a positive integer greater than 1; inputting the N first recognition results into an error correction model to obtain a second recognition result output by the error correction model; wherein the error correction model is obtained by pre-training based on a deep learning model and comprises an encoding network and a decoding network; the encoding network performs interactive encoding on the N first recognition results to obtain feature representations of M first recognition results in the N first recognition results, wherein M is a positive integer less than or equal to N; and the decoding network decodes the feature representations of the M first recognition results to obtain the second recognition result. The present application can accurately correct the result of speech recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech processing and artificial intelligence, and particularly relates to a speech recognition error correction method and device. BACKGROUND

[0002] The goal of ASR (Automatic Speech Recognition) is to recognize human speech as corresponding text. At present, there are speech recognition systems in various implementation manners, but the results output by the speech recognition system are often inaccurate, especially for fields with incomplete data coverage. For example, errors often occur in which the same pronunciation but different characters are recognized. Such errors have a great impact on user reading and understanding, and therefore an accurate speech recognition error correction method is urgently needed. SUMMARY

[0003] In view of this, the present application provides a speech recognition error correction method and device to accurately correct the results of speech recognition.

[0004] The present application provides the following solutions:

[0005] In a first aspect, a speech recognition error correction method is provided, and the method comprises the following steps:

[0006] obtaining N first recognition results output by a speech recognition system for to-be-recognized speech, wherein N is a positive integer greater than 1;

[0007] inputting the N first recognition results into an error correction model to obtain a second recognition result output by the error correction model; wherein the error correction model is obtained by pre-training based on a deep learning model and comprises an encoding network and a decoding network; the encoding network performs interactive encoding on the N first recognition results to obtain feature representations of M first recognition results in the N first recognition results, wherein M is a positive integer less than or equal to N; and the decoding network decodes the feature representations of the M first recognition results to obtain the second recognition result.

[0008] According to an implementable manner in the embodiments of the present application, before the N first recognition results are input into the error correction model, the method further comprises the following steps:

[0009] performing padding processing on the N first recognition results to obtain N first recognition results of equal length.

[0010] According to an implementable manner in the embodiments of the present application, the error correction model further comprises an embedding network.

[0011] The embedding network performs content-based embedding and position-based embedding on each element Token in the N first recognition results to obtain embedding features of each Token in the N first recognition results.

[0012] The encoding network performs the interaction encoding processing by using the embedding features of each Token in the N first recognition results.

[0013] According to an implementable manner in the embodiments of the present application, the encoding network performing the interaction encoding processing by using the embedding features of each Token in the N first recognition results comprises:

[0014] The encoding network performs, for each Token in the N first recognition results, the following processing: obtaining a query vector by using the embedding features of the current Token, obtaining a key matrix and a value matrix by using the embedding features of other Tokens associated with the current Token in the N first recognition results, performing attention processing, and then passing through a fully connected layer to obtain a feature vector of the current Token, wherein the other Tokens associated with the current Token include at least one Token belonging to the same first recognition result as the current Token and at least one Token belonging to a different first recognition result from the current Token.

[0015] The feature vectors of the Tokens included in each first recognition result are used to obtain a feature representation of each first recognition result.

[0016] The feature representations of M first recognition results are obtained from the feature representations of the N first recognition results.

[0017] According to an implementable manner in the embodiments of the present application, the error correction model further comprises a length prediction network and an editing processing module.

[0018] The length prediction network predicts an editing type corresponding to each Token in the M first recognition results by using the feature representations of the M first recognition results, and the editing type includes deletion, retention or insertion.

[0019] The editing processing module performs editing processing on the feature representations of the M first recognition results respectively by using the prediction results of the length prediction network, and provides the feature representations of the M first recognition results after the editing processing to the decoding network.

[0020] The decoding network performs decoding by using the feature representations of the M first recognition results after the editing processing.

[0021] According to an implementable manner in the embodiments of the present application, the error correction model further comprises a length prediction network and an editing processing module.

[0022] The length prediction network predicts an edit type corresponding to each Token in the M first recognition results using the feature representation of the M first recognition results, the edit type including deletion, retention or insertion.

[0023] The edit processing module performs edit processing on the embedding feature of each Token in the M first recognition results respectively using the prediction result of the length prediction network to obtain the embedding feature of the M first recognition results, wherein the embedding feature of each Token in the first recognition result after edit processing constitutes the embedding feature of the first recognition result.

[0024] The decoding network decodes using the feature representation and the embedding feature of the M first recognition results includes that the decoding network decodes using the feature representation and the embedding feature of the M first recognition results.

[0025] According to an implementable manner in the embodiments of the present application, the decoding network decodes using the feature representation and the embedding feature of the M first recognition results includes that:

[0026] The decoding network obtains a query matrix using the feature representation of the M first recognition results, obtains a key matrix and a value matrix using the embedding feature of the M recognition results, and obtains the second recognition result after self-attention processing and a fully connected layer.

[0027] According to an implementable manner in the embodiments of the present application, the M first recognition results are optimal recognition results output by the speech recognition system for the to-be-recognized speech.

[0028] According to an implementable manner in the embodiments of the present application, the error correction model is obtained by pre-training in the following manner:

[0029] Obtain training data, the training data including text samples and N texts obtained by performing edit processing on the text samples;

[0030] Train the error correction model using the training data; wherein the N texts are input into the error correction model as N first recognition results, and the training target includes minimizing the difference between the second recognition result output by the error correction model and the text sample corresponding to the N texts.

[0031] According to an implementable manner in the embodiments of the present application, the error correction model is obtained by pre-training in the following manner:

[0032] Obtain training data, the training data including text samples and N texts obtained by performing edit processing on the text samples, wherein the N texts are labeled with edit type labels of the edit processing;

[0033] training the error correction model with the training data; wherein the N texts are input into the error correction model as N first recognition results, and the training target comprises minimizing the difference between the second recognition results output by the error correction model and the text samples corresponding to the N texts, and minimizing the difference between the editing types corresponding to each Token of the M texts in the N texts predicted by the length prediction network and the corresponding editing type labels.

[0034] In a second aspect, an error correction device for speech recognition is provided, and the device comprises:

[0035] a result obtaining unit configured to obtain N first recognition results output by a speech recognition system for to-be-recognized speech, N being a positive integer greater than 1;

[0036] an error correction processing unit configured to input the N first recognition results into an error correction model and obtain a second recognition result output by the error correction model; wherein the error correction model is obtained by pre-training based on a deep learning model and comprises an encoding network and a decoding network; the encoding network performs interactive encoding on the N first recognition results to obtain feature representations of M first recognition results in the N first recognition results, M being a positive integer less than or equal to N; and the decoding network decodes the feature representations of the M first recognition results to obtain the second recognition result.

[0037] According to a third aspect, a computer readable storage medium is provided, and the medium stores a computer program, which, when executed by a processor, implements the steps of the method according to any one of the first aspect.

[0038] According to a fourth aspect, an electronic device is provided, and the device comprises:

[0039] one or more processors; and

[0040] a memory associated with the one or more processors, the memory being configured to store program instructions, which, when executed by the one or more processors, implement the steps of the method according to any one of the first aspect.

[0041] According to the embodiments provided in the present application, the following technical effects are disclosed:

[0042] 1) The application adopts an error correction model pre-trained based on a deep learning model to interactively encode and decode a plurality of first recognition results output by a speech recognition system, so as to obtain a recognition result after correction, i.e., a second recognition result. This way is not limited to the restriction of words in the error correction list, is more intelligent and flexible, and improves the accuracy of error correction. Moreover, based on the plurality of first recognition results output by the speech recognition system, the application can understand and learn from more effective information, improving the accuracy of the second recognition result obtained by error correction.

[0043] 2) In the application, the N first recognition results are not shallowly fused in an embedding network, but are interactively encoded in an encoding network. Not only the relationship between the Tokens in each first recognition result is learned, but also the relationship between the Tokens in different first recognition results is learned, so as to make full use of the useful information in the plurality of first recognition results, thereby improving the accuracy of error correction.

[0044] 3) In the application, when the encoding network of the error correction model encodes each Token, the embedding features of the context Tokens are considered. The i-th Token in the first recognition result can be associated with the i-t to i+t Tokens in other first recognition results, so as to reduce the influence of information confusion even without strict alignment processing of the N first recognition results, thereby improving the accuracy of error correction.

[0045] 4) In the application, a length prediction network and an editing processing module are introduced in the error correction model. The length prediction network predicts the editing type corresponding to each Token in the M first recognition results, and the editing processing module edits the feature representation of the corresponding first recognition result or the embedding features of the corresponding first recognition result, so that the decoding unit can perform decoding processing according to the more accurate feature information after editing, thereby further improving the accuracy of the obtained second recognition result.

[0046] Of course, any product implementing the application does not necessarily need to achieve all the advantages described above. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings in the following description only constitute some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0048] Figure 1 is a system structure diagram suitable for embodiments of the application;

[0049] Figure 2 is a flowchart of a voice recognition error correction method provided by an embodiment of the present application;

[0050] Figure 3 is a schematic diagram of a padding process provided by an embodiment of the present application;

[0051] Figure 4a is a schematic diagram of an error correction model provided by an embodiment of the present application;

[0052] Figure 4b is a schematic diagram of another error correction model provided by an embodiment of the present application;

[0053] Figure 4c is a schematic diagram of still another error correction model provided by an embodiment of the present application;

[0054] Figure 5 is a schematic diagram of a current Token related Token provided by an embodiment of the present application;

[0055] Figure 6 is a schematic diagram of an error correction device provided by an embodiment of the present application;

[0056] Figure 7 is a schematic diagram of a model training device provided by an embodiment of the present application;

[0057] Figure 8 is a schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present application.

[0059] The terms used in the embodiments of the present application are only for the purpose of describing the specific embodiments, and are not intended to limit the present application. The singular forms "a", "an" and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0060] It should be understood that the term "and / or" used herein is only to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are a "or" relationship.

[0061] Depending on context, the word "if" as used herein can be interpreted as meaning "when" or "while" or "in response to determining" or "in response to detecting." Similarly, the phrase "if it is determined" or "if [a stated condition or event] is detected" can be interpreted as "when it is determined" or "in response to determining" or "when [a stated condition or event] is detected" or "in response to detecting [a stated condition or event]."

[0062] Currently, there are mainly two kinds of error correction methods:

[0063] One is rule-based, that is, to provide an error correction list, and to determine whether there is a word in the speech recognition result that should be replaced by a word in the error correction list based on certain rules or models, and if so, to replace it with a word in the error correction list. The defect of this method is that it is limited to the words in the error correction list, and the effect is limited, and the accuracy is still poor.

[0064] The other is to use a neural network model to correct the optimal recognition result (One Best) obtained by the speech recognition system. The defect of this method is that the basis for error correction is only the optimal recognition result of the speech recognition system, and the reference information is limited, and the error correction accuracy is still insufficient.

[0065] Therefore, a new error correction method is adopted in the present application, which refers to multiple recognition results obtained by a speech recognition system for a to-be-recognized speech to perform error correction and obtain an error-corrected recognition result. In order to facilitate the understanding of the present application, the system architecture to which the present application is applied is first described briefly. Figure 1 An exemplary system architecture to which embodiments of the present application can be applied is shown, as shown in Figure 1 The system architecture includes a model training device that establishes an error correction model in an offline manner, and an error correction device that performs error correction on the recognition result of a speech recognition system online.

[0066] The model training device can perform model training using the method provided in the embodiments of the present application after obtaining training data to obtain an error correction model.

[0067] The speech recognition system is used to perform speech recognition on a to-be-recognized speech and output a recognition result.

[0068] The error correction device uses the established error correction model to perform error correction on the recognition result of the speech recognition system and obtains an error-corrected recognition result.

[0069] The model training apparatus, the speech recognition system, and the error correction apparatus can be respectively arranged as independent servers, can be arranged in a same server or a server group, and can also be arranged in independent or same cloud servers. The cloud server is also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, to solve the defects of great management difficulty and weak service scalability in traditional physical hosts and virtual private server (VPS, Virtual Private Server) services. The model training apparatus and the error correction apparatus can also be arranged in a computer terminal with strong computing capability.

[0070] It should be noted that, in addition to online error correction, the error correction apparatus can also perform error correction in an offline manner, for example, for a batch of speech recognition results.

[0071] It should be understood that, Figure 1 The number of the model training apparatus, the error correction model, the speech recognition system, and the error correction apparatus in the system is only illustrative. According to the implementation needs, there can be any number of model training apparatuses, error correction models, speech recognition systems, and error correction apparatuses.

[0072] Figure 2 A flowchart of a speech recognition error correction method provided by the embodiments of the present application is shown in the figure. The method can be executed by the error correction apparatus in the system shown in the figure. As shown in the figure, the method can include the following steps: Figure 1 Figure 2 The method can include the following steps:

[0073] Step 202: Obtain N first recognition results output by a speech recognition system for a to-be-recognized speech, N being a positive integer greater than 1.

[0074] Step 204: Input the N first recognition results into an error correction model to obtain a second recognition result output by the error correction model; wherein the error correction model is obtained by pre-training based on a deep learning model, and includes an encoding network and a decoding network; the encoding network performs interactive encoding on the N first recognition results to obtain feature representations of M first recognition results in the N first recognition results, M being a positive integer less than or equal to N; and the decoding network decodes the feature representations of the M first recognition results to obtain the second recognition result.

[0075] As can be seen from the above flow, the present application adopts an error correction model obtained by pre-training based on a deep learning model to perform interactive encoding and decoding on multiple first recognition results output by a speech recognition system, so as to obtain a recognition result after error correction, i.e., a second recognition result. This way is not limited by the words in the error correction list, is more intelligent and flexible, and improves the accuracy of error correction. Moreover, based on multiple first recognition results output by a speech recognition system, the present application can understand and learn from more effective information, and improve the accuracy of the second recognition result obtained by error correction.​

[0076] It should be noted that the "first", "second", and the like involved in the present disclosure do not have the limitations of size, order, and quantity, and are only used to distinguish in name. For example, "first recognition result" and "second recognition result" are used to distinguish two recognition results in name, where the first recognition result represents the recognition result output by the speech recognition system, and the second recognition result represents the recognition result output after error correction by the error correction model.

[0077] The present application is not limited to the implementation of the speech recognition system, and any implementation mechanism of the speech recognition system can be used. The present application is only used to correct the recognition result of the speech recognition system. Most correction methods only use the best recognition result in the recognition result for correction, but in fact, in addition to the best recognition result, other recognition results also contain a large amount of valid information that can be used to assist in error correction. Therefore, in the embodiment of the present application, multiple first recognition results output by the speech recognition system are obtained. As one of the implementable ways, the multiple first recognition results can be the recognition results with the top N confidence in the recognition results output by the speech recognition system, denoted as Nbest first recognition results, and the subsequent embodiments are described taking this implementation way as an example.

[0078] The above step 204, i.e., "inputting the N first recognition results into the error correction model to obtain the second recognition result output by the error correction model", will be described in detail below in combination with embodiments.

[0079] The error correction model can include an encoding network and a decoding network, and can further include an embedding network, a length prediction network, and an edit processing module.

[0080] As shown in Figure 4a , Figure 4b and Figure 4c , the encoding network is used to interactively encode the N first recognition results to obtain the feature representation of M first recognition results in the N first recognition results.

[0081] The decoding network decodes the feature representation of the M first recognition results to obtain the second recognition result.

[0082] The above error correction model is implemented based on a deep learning model, for example, the above encoding network and decoding network can be implemented based on Transformer, which will be described in detail later.

[0083] Since each first recognition result output by the speech recognition system can be of different lengths, in order to facilitate the subsequent interactive encoding of the error correction model, as one of the implementable ways, the length of each first recognition result is first normalized, and then the normalized length is used as the length of each first recognition result. Figure 3As shown in the middle, after obtaining the Nbest first recognition results output by the speech recognition system, padding processing can be performed on the Nbest first recognition results to obtain N equal-length first recognition results. Then the equal-length N first recognition results are provided to the error correction model for error correction processing.

[0084] The padding processing of the Nbest first recognition results can be performed by taking the longest first recognition result in the Nbest first recognition results as a reference, and padding zeros at the end of the other first recognition results to reach the length of the longest first recognition result. In this way, N equal-length first recognition results are obtained. This method is relatively simple to implement, but other padding methods can also be used, such as aligning the Nbest first recognition results and padding zeros in the empty positions.

[0085] The N first recognition results obtained after padding processing can first undergo embedding processing of the embedding network to obtain embedding features of each Token in the N first recognition results.

[0086] The embedding network can perform content-based embedding and position-based embedding on each element (Token) in the N first recognition results, i.e., word Embedding and position Embedding. Since the N first recognition results are actually a sequence of multiple Tokens, which include words, a start symbol [CLS] and a separator [SEP].

[0087] Word Embedding, i.e., word vector encoding of each Token to obtain word vector representation. Position Embedding, i.e., encoding the position of each Token in the first recognition result to obtain position representation. For example, a certain first recognition result is “[CLS] play [SEP] a [SEP] slow song [SEP]”, and each Token “[CLS]”, “play”, “[SEP]”, “a”, “[SEP]”, “slow song”, “[SEP]” is numbered as 0, 1, 2, 3, 4, 5 and 6 according to its position in the first recognition result.

[0088] The embedding feature of the Token can be obtained by adding the word vector representation of the Token and the representation of the position. After obtaining the embedding features of the Tokens in the N first recognition results, the encoding network can perform interactive encoding processing on the embedding features of the Tokens in the N first recognition results. The encoding network can be implemented by a plurality of Transformer layers, and each Transformer layer can include an attention layer and a fully connected layer. The fully connected layer can be, for example, a feed-forward network (FFN).

[0089] As one of the implementable manners, in order to enable the encoding network to learn the mutual relationship between the first recognition results, when performing attention processing on the Tokens, the encoding network not only considers the relationship between the Tokens in the same recognition result, but also considers the relationship between the Tokens in different recognition results. Specifically, the encoding network can perform the following operations on each Token in the N recognition results:

[0090] The embedding feature of the Token is obtained by using the embedding feature of the Token to obtain the query vector, the embedding features of the other Tokens associated with the Token in the N first recognition results to obtain the key matrix and the value matrix, and performing attention processing and then passing through the fully connected layer to obtain the feature vector of the Token. The other Tokens associated with the Token can be pre-set and can include at least one Token belonging to the same first recognition result as the Token and at least one Token belonging to a different first recognition result as the Token.

[0091] As shown in Figure 5 For the i-th Token (i.e., the current Token) in a certain first recognition result, the other Tokens associated with the current Token can be pre-set to include other Tokens belonging to the same first recognition result as the Token and Tokens corresponding to the positions of the Token and its context Tokens in other first recognition results. The context Tokens of the i-th Token can be the i-t-th Token to the i+t-th Token. t is a pre-set positive integer, which can be an empirical value or an experimental value. Figure 5 In the embodiment, t is taken as 2, and Figure 5The other tokens associated with the current token are shown in grayscale in the graph. The model can learn the association between the current token and the tokens shown in grayscale during the model training process, and use the association to obtain a feature representation of the current token during the error correction process. The feature representation of the current token considers not only the embedding features of the current token, but also the embedding features of the tokens shown in grayscale in the graph and the association with the current token. In other words, the embedding features of the tokens shown in grayscale in the graph affect the feature representation of the current token.

[0092] After obtaining the feature vectors of the tokens, the feature vectors of the tokens included in each first recognition result can be used to obtain a feature representation of each first recognition result. For example, the feature vectors of the tokens in the first recognition result 1 constitute the feature representation of the first recognition result 1, the feature vectors of the tokens in the first recognition result 2 constitute the feature representation of the first recognition result 2, and so on.

[0093] It can be seen that the cross-encoding manner of the encoding network uses self-attention learning to learn not only the relationship between the tokens in each first recognition result, but also the relationship between the N first recognition results. Moreover, the context of the tokens is expanded. This way of considering the context of the tokens is because the N first recognition results are processed to be equal in length by using the padding method in the embodiments of the present application, and no strict alignment processing is performed. This results in that the tokens in the N first recognition results are not strictly aligned, and therefore the embedding features of the context of the tokens are considered when obtaining the feature representation of each token. In the cross-encoding process, the i-th token in one first recognition result can be associated with the i-t-th to i+t-th tokens in other first recognition results. Even without strict alignment of the first recognition results, effective association relationships can be learned, thereby improving the error correction accuracy.

[0094] It should be noted that the above-mentioned form of expanding the context of the tokens is a preferred implementation manner, and other implementation manners can also be used. For example, only the tokens belonging to the same first recognition result as the current token and the tokens in the same position as the current token in other first recognition results are considered.

[0095] After obtaining the feature representations of the N first recognition results, the encoding network obtains the feature representations of the M first recognition results from the feature representations of the N first recognition results, where M is a positive integer less than or equal to N.

[0096] As one of the implementable manners, the feature representation of the optimal recognition result (which can be denoted as One best first recognition result) can be obtained from the feature representations of the N first recognition results, i.e., the feature representation of the first recognition result with the highest confidence output by the speech recognition system. In the description of subsequent embodiments, the feature representation of the One best first recognition result is taken as an example. In addition to this implementation manner, the feature representation of any one of the first recognition results can also be used for subsequent decoding processing, but the feature representation of the One best first recognition result is preferred.

[0097] As shown in Figures 4a-4c Each Transformer layer of the encoding network actually performs processing on the N first recognition results respectively, and the "N" beside the arrow in the figure shows that each layer outputs the feature representations of the N first recognition results respectively. The encoding network finally obtains the feature representation of the One best first recognition result from the N first recognition results, i.e., the encoding network finally outputs the feature representation of the One best first recognition result, and the "1" beside the arrow shows that the feature representation of one first recognition result.

[0098] Regarding the implementation of the decoding part, the following three manners can be adopted, but are not limited thereto:

[0099] Manner 1: The decoding network can directly utilize the feature representations of the M first recognition results output by the encoding network to perform decoding processing to obtain the second recognition result.

[0100] The decoding network can also be implemented by multiple Transformer layers, and each Transformer layer can include an attention layer and a fully connected layer. The fully connected layer can be, for example, FFN. The attention layer can utilize the feature representations of the M first recognition results to obtain query, key and value respectively, and after attention processing and passing through the fully connected layer, the second recognition result is obtained. The second recognition result is also a text composed of multiple Tokens, which is the recognition result after error correction.

[0101] As shown in Figure 4a Taking the feature representation of the One best first recognition result output by the encoding network as an example, the decoding network directly outputs the feature representation of the One best first recognition result to the decoding network, and the decoding network utilizes the feature representation of the One best first recognition result to perform decoding processing (including attention processing and fully connected layer processing) to obtain the second recognition result.

[0102] M first recognition results output by the encoding network can be first input into a length prediction network. The length prediction network predicts the edit type corresponding to each Token in the M first recognition results respectively using the feature representations of the M first recognition results inputted.

[0103] Then the feature representations of the M first recognition results are edited by the editing processing module respectively using the prediction results of the length prediction network, and the feature representations of the M first recognition results after editing are provided to the decoding network.

[0104] When editing, the following three types of edits can be faced:

[0105] If the edit type of a Token in a first recognition result is deletion, it means that the Token is redundant, so the editing processing module deletes the Token from the first recognition result. For the feature representation of the first recognition result, the vector corresponding to the Token can be deleted.

[0106] If the edit type of a Token in a first recognition result is retention, it means that the Token is correct, so the editing processing module retains the Token in the first recognition result, and keeps the vector corresponding to the Token in the feature representation of the first recognition result unchanged.

[0107] If the edit type of a Token in a first recognition result is insertion, it means that another Token needs to be inserted at the position of the Token. For the feature representation of the first recognition result, the vector of one of the adjacent Tokens of the Token can be copied and inserted into the position of the vector of the Token. Of course, other ways of insertion can also be used, such as averaging the vectors of several adjacent Tokens of the Token, inserting the averaged vector into the position of the vector of the Token, or inserting the vector at the position of the Token in a manner similar to interpolation, etc.

[0108] Similarly, the decoding network can also be implemented by multiple Transformer layers, and each Transformer layer can include an attention layer and a fully connected layer. The fully connected layer can be, for example, FFN. The attention layer can obtain query, key and value respectively using the feature representations of the M first recognition results after editing, and obtain the second recognition result after attention processing and passing through the fully connected layer. The second recognition result is also a text composed of multiple Tokens, which is the recognition result after error correction.

[0109] As Figure 4bAs shown in FIG. 1, taking the feature representation of the One best first recognition result output by the encoding network as an example, after the decoding network outputs the feature representation of the One best first recognition result to the length prediction network, the length prediction network predicts the editing type of each Token in the One best first recognition result, and outputs the editing type to the editing processing module. The editing processing module edits the feature representation of the One best first recognition result according to the predicted editing type, obtains the edited feature representation of the One best first recognition result, and outputs the edited feature representation to the decoding network. The decoding network decodes the feature representation of the One best first recognition result (including attention processing and full connection layer processing) to obtain a second recognition result.

[0110] Method 3: The feature representations of the M first recognition results output by the encoding network can be first input to the length prediction network. The length prediction network uses the input M first recognition results to predict the editing type of each Token in the M first recognition results, respectively. The editing type can include deletion, retention or insertion.

[0111] Then the editing processing module uses the prediction results of the length prediction network to edit the embedding features of each Token in the M first recognition results, respectively, to obtain the embedding features of the M first recognition results, wherein the embedding features of each Token in the edited first recognition result constitute the embedding features of the first recognition result.

[0112] When editing, the following three editing types can be faced:

[0113] If the editing type of a Token in a first recognition result is deletion, it means that the Token is redundant, so the editing processing module deletes the Token from the first recognition result. For the embedding features of the first recognition result, the vector corresponding to the Token can be deleted.

[0114] If the editing type of a Token in a first recognition result is retention, it means that the Token is correct, so the editing processing module retains the Token in the first recognition result, and keeps the vector corresponding to the Token in the embedding features of the first recognition result unchanged.

[0115] If the edit type of a token in a first recognition result is "insert," it means that another token needs to be inserted at that token's position. For the embedding features of that first recognition result, the vector of one of the adjacent tokens can be copied and inserted into the position of the token's vector. Alternatively, other methods can be used, such as averaging the vectors of several adjacent tokens and inserting the averaged vector into the position of the token's vector; or, for example, using an interpolation-like method to insert the vector at the token's position, and so on.

[0116] In this approach, the decoding network has two inputs: one is the feature representation of the M first recognition results output by the encoding network, and the other is the embedded features of the M first recognition results obtained through editing by the editing module. Therefore, the decoding network uses the feature representations and embedded features of the aforementioned M first recognition results for decoding.

[0117] Similarly, the decoding network can be implemented using multiple Transformer layers, each of which can include an attention layer and a fully connected layer. The fully connected layer can be, for example, an FFN (Full-Functional Network). Unlike the previous two methods, the attention layer can use one input to obtain the query and the other input to obtain the key and value. For example, the query can be obtained using the feature representations of M first recognition results, and the key and value can be obtained using the embedded features of the M recognition results. After attention processing, the result passes through a fully connected layer to obtain the second recognition result. This second recognition result is also a text composed of multiple tokens; this text is the error-corrected recognition result.

[0118] like Figure 4c As shown, taking the feature representation of the first recognition result of One Best output by the encoding network as an example, the decoding network outputs the feature representation of the first recognition result of One Best to the length prediction network. The length prediction network then predicts the edit type of each token in the first recognition result of One Best and outputs the edit type to the editing processing module. The editing processing module performs editing processing on the embedded features of the first recognition result of One Best according to the predicted edit type, obtaining the edited embedded features of the first recognition result of One Best, and outputs them to the decoding network. The decoding network uses the feature representation and embedded features of the first recognition result of One Best to perform decoding processing (including attention processing and fully connected layer processing) to obtain the second recognition result.

[0119] The training method of the above error correction model will be described in detail below with reference to the embodiments.

[0120] First, training data is acquired.

[0121] In the embodiments of the present application, the purpose of acquiring the training data is to obtain N first recognition results and texts after correcting the N first recognition results. As one of the implementable manners, N first recognition results output by a speech recognition system for a speech sample can be acquired as the input of the correction model, and the text obtained by correcting the N first recognition results manually is regarded as the target output of the speech recognition system for training. However, this manner needs to call the speech recognition system, and needs to correct each first recognition result according to the experience of the manual work, which is relatively complex to implement and depends on the experience of the manual work. Therefore, another preferred embodiment is provided in the present application to acquire the training data.

[0122] Specifically, a text sample can be acquired first. The text sample can be any correct sentence, that is, a sentence without grammatical errors, homophonic errors and the like. Such correct sentences are easy to acquire, and a large number of correct sentences can be acquired from the texts in various websites or databases as text samples. Then, N texts are obtained by performing different editing processes on the text sample. The editing process can be, for example, randomly selecting one or more words in the text sample for editing, and the editing process can include one or any combination of insertion, retention and deletion, so that N different texts can be generated, which can be used to simulate N recognition results output by the speech recognition system. When performing the editing process, some dictionaries can be additionally combined to perform the insertion process, for example, inserting some homophonic and synonymous words, to help the correction model learn the correction better.

[0123] For example, assuming that a text sample is acquired as “Today's weather is really good”, and after the editing process, texts such as “Today is really good”, “Today's weather is really good” and “Today's weather is really really good” can be obtained.

[0124] The N texts are used as N first recognition results input into the correction model, and the target is to enable the correction model to learn and correct the output correct sentence, that is, the text sample. That is, the training target includes minimizing the difference between the second recognition result output by the correction model and the text sample corresponding to the N texts. As one of the implementable manners, a loss function can be constructed according to the above training target, for example, a cross-entropy loss function can be used. In each iteration, the value of the loss function is used to update the parameters of the correction model by using methods such as gradient descent, until the preset training end condition is met. The training end condition can include, for example, the value of the loss function is less than or equal to a preset loss function threshold, the number of iterations reaches a preset number threshold, and the like. This implementation manner can be applied toFigures 4a-4c any one of the error correction models shown.

[0125] As another implementable manner, for the error correction model shown, Figure 4b and Figure 4c In the error correction model shown, when obtaining the training data, after obtaining N texts by performing different editing processes on the text sample, further label the N texts. The label here is the editing type label corresponding to the editing process, and the label is actually the label for each Token in the N texts.

[0126] Correspondingly, the training target can include two: the first training target is to minimize the difference between the second recognition result output by the error correction model and the text sample corresponding to the N texts, and in addition to the first training target, the second training target can be further included. The second training target is to minimize the difference between the editing type corresponding to each Token in the M texts in the N texts predicted by the length prediction network and the corresponding editing type label. For example, a loss function Loss1 can be constructed by using the first training target, a loss function Loss2 can be constructed by using the second training target, and a total loss function Loss can be obtained by using Loss1 and Loss2:

[0127] Loss = a1*Loss1 + a2*Loss2

[0128] Wherein, a1 and a2 are hyperparameters, which represent the weight values of the two loss functions, and can specifically use empirical values or experimental values.

[0129] In each iteration, the value of the total loss function is used to update the parameters of the error correction model using methods such as gradient descent, until the preset training end condition is met. The training end condition can include, for example, the value of the total loss function is less than or equal to the preset loss function threshold, the number of iterations reaches the preset number threshold, etc.

[0130] The above method provided by the embodiments of the present application can be applied to any speech recognition scene and connected to the output of the speech recognition system for error correction processing. For example, in the human-computer dialogue scene, the speech input scene in the input method application, the speech-to-text scene in the instant messaging application, the speech search scene in the search application, etc.

[0131] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be executed in an order different than the order in the embodiments and still achieve the desired result. In addition, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous or possible.

[0132] According to an embodiment of another aspect, an error correction apparatus is provided. Figure 6 A schematic block diagram of an error correction apparatus according to an embodiment is shown. As shown, the apparatus 600 comprises a result obtaining unit 601 and an error correction processing unit 602, and can further comprise a padding processing unit 603. The main functions of the constituent units are as follows: Figure 6

[0133] The result obtaining unit 601 is configured to obtain N first recognition results output by a speech recognition system for a to-be-recognized speech, N being a positive integer greater than 1.

[0134] The error correction processing unit 602 is configured to input the N first recognition results into an error correction model, and obtain a second recognition result output by the error correction model; wherein the error correction model is obtained by pre-training based on a deep learning model, and comprises an encoding network and a decoding network; the encoding network performs interactive encoding on the N first recognition results to obtain feature representations of M first recognition results among the N first recognition results, M being a positive integer less than or equal to N; the decoding network decodes the feature representations of the M first recognition results to obtain the second recognition result.

[0135] As one of the implementable manners, the padding processing unit 603 can perform padding processing on the N first recognition results obtained by the result obtaining unit 601 to obtain N first recognition results of equal length, and then provide the N first recognition results of equal length to the error correction processing unit 602.

[0136] As one of the implementable manners, when the encoding network performs interactive encoding on the N first recognition results to obtain the feature representations of the M first recognition results among the N first recognition results, the following steps are specifically performed:

[0137] A query vector is obtained by using the embedding feature of the current Token, a key matrix and a value matrix are obtained by using the embedding features of other Tokens associated with the current Token in the N first recognition results, and after attention processing and passing through a fully connected layer, a feature vector of the current Token is obtained, wherein the other Tokens associated with the current Token include at least one Token belonging to the same first recognition result as the current Token and at least one Token belonging to different first recognition results as the current Token;

[0138] The feature representations of the first recognition results are respectively obtained by using the feature vectors of the Tokens contained in each first recognition result;

[0139] The feature representations of the M first recognition results are obtained from the feature representations of the N first recognition results.

[0140] ​As one possible approach, the error correction model also includes a length prediction network and an editing processing module.

[0141] The length prediction network uses the feature representations of M first identification results to predict the edit type corresponding to each Token in the M first identification results. The edit type includes deletion, retention or insertion.

[0142] The editing module uses the prediction results of the length prediction network to edit the feature representations of the M first recognition results respectively, and provides the edited feature representations of the M first recognition results to the decoding network.

[0143] The decoding network uses the feature representations of the M first recognition results after editing to perform decoding.

[0144] As another possible approach, the error correction model also includes a length prediction network and an editing module;

[0145] The length prediction network uses the feature representations of M first identification results to predict the editing type corresponding to each Token in the M first identification results. The editing type includes deletion, retention or insertion.

[0146] The editing module uses the prediction results of the length prediction network to edit the embedding features of each token in the M first identification results to obtain the embedding features of the M first identification results. The embedding features of each token in the first identification results after editing constitute the embedding features of the first identification results.

[0147] The decoding network uses the feature representations of the M first recognition results to perform decoding, including: the decoding network uses the feature representations of the M first recognition results and embedded features to perform decoding.

[0148] In this approach, the decoding network can obtain a query matrix using the feature representations of the M first recognition results, obtain a key matrix and a value matrix using the embedded features of the M recognition results, and then obtain the second recognition result after self-attention processing and passing it through a fully connected layer.

[0149] As one possible approach, the aforementioned M first recognition results represent the optimal recognition results of the speech recognition system for the speech output to be recognized.

[0150] Figure 7 A schematic block diagram of a model training apparatus according to one embodiment is shown. Figure 7 As shown, the device 700 includes a sample acquisition unit 701 and a model training unit 702, and may further include a filling processing unit 703. The main functions of each component are as follows:

[0151] The sample acquisition unit 701 is configured to acquire training data, which includes text samples and N texts obtained by editing the text samples.

[0152] The model training unit 702 is configured to train an error correction model using training data; wherein, N texts are input into the error correction model as N first recognition results, and the training objective includes minimizing the difference between the second recognition result output by the error correction model and the text samples corresponding to the N texts.

[0153] As one possible approach, the padding processing unit 703 can pad the N texts in the training data to obtain N texts of equal length, which are then provided to the model training unit 702.

[0154] Furthermore, in the training data mentioned above, N texts are labeled with editing type tags. The training objective can also include minimizing the difference between the editing type and the corresponding editing type tag of each token of M texts out of the N texts predicted by the length prediction network.

[0155] For details on the specific structure and processing of the error correction model, please refer to the relevant descriptions in the method embodiments, which will not be elaborated here.

[0156] The above Figure 6 The error correction device shown and Figure 7 The model training device shown can be set to the same device or different devices.

[0157] It should be noted that the embodiments of this application may involve the use of user data. In practical applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, provided that it complies with the applicable laws and regulations of the country (e.g., with the user's explicit consent, with the user being properly notified, etc.).

[0158] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0159] And an electronic device, comprising:

[0160] One or more processors; and

[0161] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0162] The application also provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method according to any one of the preceding method embodiments.

[0163] wherein, Figure 8 An exemplary architecture of an electronic device is shown, which can specifically include a processor 810, a video display adapter 811, a disk drive 812, an input / output interface 813, a network interface 814, and a memory 820. The processor 810, the video display adapter 811, the disk drive 812, the input / output interface 813, the network interface 814, and the memory 820 can be communicatively connected through a communication bus 830.

[0164] The processor 810 can be implemented in the form of a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the application.

[0165] The memory 820 can be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, etc. The memory 820 can store an operating system 821 for controlling the operation of the electronic device 800, a basic input / output system (BIOS) 822 for controlling the low-level operation of the electronic device 800. In addition, a web browser 823, a data storage management system 824, and an error correction device 825, etc. can also be stored. The error correction device 825 can be an application program that specifically implements the above steps in the embodiments of the application. In summary, when the technical solutions provided by the application are implemented by software or firmware, the related program codes are stored in the memory 820 and executed by the processor 810.

[0166] The input / output interface 813 is configured to connect input / output modules to realize information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. The input devices can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output devices can include a display, a speaker, a vibrator, an indicator light, etc.

[0167] The network interface 814 is configured to connect a communication module (not shown in the figure) to realize the communication interaction between the device and other devices. The communication module can realize communication through wired mode (such as USB, network cable, etc.), or realize communication through wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0168] The bus 830 includes a path for transmitting information between various components (such as the processor 810, the video display adapter 811, the disk drive 812, the input / output interface 813, the network interface 814, and the memory 820) of the device.

[0169] It should be noted that although the above device only shows the processor 810, the video display adapter 811, the disk drive 812, the input / output interface 813, the network interface 814, the memory 820, the bus 830, etc., in the specific implementation process, the device can also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device can also only contain the components necessary to implement the scheme of the present application, and does not have to contain all the components shown in the figure.

[0170] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be realized by means of software and the necessary general hardware platform. Based on such understanding, the technical solutions of the present application can be embodied in the form of a computer program product, which can be stored in a storage medium such as ROM / RAM, magnetic disk, optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments.

[0171] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiments, since they are basically similar to the method embodiments, they are described more simply, and the relevant parts are referred to the part of the method embodiments. The above described system and system embodiments are only illustrative, and the units described as separate components can be or can not be physically separated, and the components displayed as units can be or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to the actual needs. Those skilled in the art can understand and implement without creative labor.

[0172] The technical solutions provided by the present application are described in detail above, and the principles and implementation manners of the present application are described by applying specific examples. The above description of the examples is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, the specific implementation manners and application ranges will be changed according to the idea of the present application. In conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A speech recognition error correction method, characterized in that, The method includes: Obtain N first recognition results output by the speech recognition system for the speech to be recognized, where N is a positive integer greater than 1; The N first recognition results are input into the error correction model to obtain the second recognition result output by the error correction model; wherein the error correction model is pre-trained based on a deep learning model and includes an encoding network and a decoding network; the encoding network performs interactive encoding on the N first recognition results to obtain feature representations of M first recognition results among the N first recognition results, where M is a positive integer less than or equal to N; the decoding network uses the feature representations of the M first recognition results to decode and obtain the second recognition result; The decoding network decodes the M first recognition results using feature representations to obtain the second recognition result, including: obtaining a query matrix, a key matrix, and a value matrix of the M first recognition results using feature representations; performing self-attention processing on the query matrix, the key matrix, and the value matrix of the M first recognition results; and passing the processed query matrix, the key matrix, and the value matrix through a fully connected layer in the decoding network to obtain the second recognition result.

2. The method according to claim 1, characterized in that, Before inputting the N first recognition results into the error correction model, the following steps are also included: The N first recognition results are padded to obtain N first recognition results of equal length.

3. The method according to claim 1, characterized in that, The error correction model also includes an embedded network; The embedding network performs content-based embedding and location-based embedding on each element Token in the N first identification results to obtain the embedding features of each Token in the N first identification results; The encoding network performs the interactive encoding process using the embedding features of each Token in the N first identification results.

4. The method according to claim 3, characterized in that, The encoding network performs the interactive encoding process using the embedding features of each token in the N first identification results, including: The encoding network performs the following operations for each Token in the N first identification results: obtaining a query vector using the embedding features of the current Token, obtaining the key matrix and value matrix of the current Token using the embedding features of other Tokens associated with the current Token in the N first identification results, performing attention processing, and then passing through a fully connected layer to obtain the feature vector of the current Token. The other Tokens associated with the current Token include: at least one Token that belongs to the same first identification result as the current Token, and at least one Token that belongs to a different first identification result as the current Token. The feature representation of each first identification result is obtained by using the feature vector of the Token contained in each first identification result; The feature representations of the M first identification results are obtained from the feature representations of the N first identification results.

5. The method according to claim 1, characterized in that, The error correction model also includes a length prediction network and an editing module; The length prediction network uses the feature representations of the M first identification results to predict the edit type corresponding to each Token in the M first identification results, where the edit type includes deletion, retention, or insertion. The editing module uses the prediction results of the length prediction network to edit the feature representations of the M first recognition results respectively, and provides the edited feature representations of the M first recognition results to the decoding network; The decoding network uses the feature representations of the M first recognition results after editing to perform decoding.

6. The method according to claim 3, characterized in that, The error correction model also includes a length prediction network and an editing module; The length prediction network uses the feature representations of the M first identification results to predict the edit type corresponding to each Token in the M first identification results, where the edit type includes deletion, retention, or insertion. The editing module uses the prediction results of the length prediction network to edit the embedding features of each Token in the M first identification results to obtain the embedding features of the M first identification results, wherein the embedding features of each Token in the first identification results after editing constitute the embedding features of the first identification results. The decoding network performs decoding using the feature representations of the M first recognition results, including: the decoding network performs decoding using the feature representations and embedded features of the M first recognition results.

7. The method according to claim 6, characterized in that, The decoding network performs decoding using the feature representations and embedded features of the M first recognition results, including: The decoding network obtains a query matrix using the feature representations of the M first recognition results, obtains the key matrix and value matrix of the M first recognition results using the embedding features of the M first recognition results, performs self-attention processing, and then passes through the fully connected layer to obtain the second recognition result.

8. The method according to any one of claims 1 to 7, characterized in that, The M first recognition results are the optimal recognition results of the speech recognition system for the speech output to be recognized.

9. The method according to any one of claims 1 to 7, characterized in that, The error correction model is pre-trained using the following method: Acquire training data, which includes text samples and N texts obtained by editing the text samples; The error correction model is trained using the training data; wherein the N texts are input into the error correction model as N first recognition results, and the training objective includes minimizing the difference between the second recognition result output by the error correction model and the text samples corresponding to the N texts.

10. The method according to claim 5, 6 or 7, characterized in that, The error correction model is pre-trained using the following method: Acquire training data, which includes text samples and N texts obtained by editing the text samples, wherein the N texts are labeled with the editing type label of the editing process; The error correction model is trained using the training data; wherein the N texts are input into the error correction model as N first recognition results, and the training objectives include: minimizing the difference between the second recognition result output by the error correction model and the text samples corresponding to the N texts, and minimizing the difference between the editing type and the corresponding editing type label of each Token of M texts among the N texts predicted by the length prediction network.

11. A speech recognition error correction device, characterized in that, The device includes: The result acquisition unit is configured to acquire N first recognition results output by the speech recognition system for the speech to be recognized, where N is a positive integer greater than 1; An error correction processing unit is configured to input the N first recognition results into an error correction model and obtain a second recognition result output by the error correction model; wherein the error correction model is pre-trained based on a deep learning model and includes an encoding network and a decoding network; the encoding network performs interactive encoding on the N first recognition results to obtain feature representations of M first recognition results among the N first recognition results, where M is a positive integer less than or equal to N; the decoding network uses the feature representations of the M first recognition results to decode and obtain the second recognition result; The error correction processing unit is further configured to perform the following steps to decode the decoding network using the feature representations of the M first identification results to obtain the second identification result: using the feature representations of the M first identification results to obtain the query matrix, key matrix, and value matrix of the M first identification results; performing self-attention processing on the query matrix, key matrix, and value matrix of the M first identification results; and passing the processed query matrix, key matrix, and value matrix through a fully connected layer in the decoding network to obtain the second identification result.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1 to 10.

13. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 10.

Images