Speech recognition error correction method and apparatus

Abstract

This invention discloses a speech recognition error correction method and apparatus, relating to the field of artificial intelligence technology. One specific implementation of the method includes: acquiring audio data and speech recognition text; inputting the audio data into an audio feature extraction model to obtain an audio feature vector; inputting the speech recognition text into a text feature extraction model to obtain a text feature vector; inputting the audio feature vector and the text feature vector into an associative attention module; in the associative attention module, performing associative attention calculation based on the audio feature vector and the text feature vector, outputting a bimodal representation vector that fuses the audio feature vector and the text feature vector; and inputting the bimodal representation vector into an error correction decoder to obtain the error correction result of the speech recognition text. This implementation improves error correction accuracy by introducing a bimodal feature that fuses speech and text into the error correction model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a speech recognition error correction method and apparatus. Background Technology

[0002] Speech recognition is a key technology in human-computer dialogue and is widely used in common services in daily life, such as speech-to-text conversion on mobile phones and automatic subtitle generation on video websites. However, current speech recognition models are not perfect and always contain some recognition errors. Therefore, error correction models are needed to correct errors in the speech-recognition text. Currently, most error correction models learn error correction information by comparing the speech-recognition text with manually annotated correct text. However, many recognition errors cannot be determined from the text level alone, resulting in low accuracy of existing error correction models. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a speech recognition error correction method and apparatus, which improves the error correction accuracy by introducing dual-modal features that fuse speech and text into the error correction model.

[0004] To achieve the above objectives, according to one aspect of the present invention, a speech recognition error correction method is provided.

[0005] The speech recognition error correction method of this invention includes: acquiring audio data and speech recognition text to be processed; inputting the audio data into a pre-trained audio feature extraction model to obtain an audio feature vector of the audio data; inputting the speech recognition text into a pre-trained text feature extraction model to obtain a text feature vector of the speech recognition text; inputting the audio feature vector and the text feature vector into a pre-trained associative attention module; performing associative attention calculation based on the audio feature vector and the text feature vector in the associative attention module to output a bimodal representation vector that fuses the audio feature vector and the text feature vector; and inputting the bimodal representation vector into a pre-trained error correction decoder to obtain the error correction result of the speech recognition text.

[0006] Optionally, the step of performing associative attention calculation based on the audio feature vector and the text feature vector to output a bimodal representation vector that fuses the audio feature vector and the text feature vector includes: forming a query vector of multiple attention heads based on the text feature vector; forming a key vector and a value vector of the multiple attention heads based on the audio feature vector; calculating a single-head attention output vector for each attention head based on the query vector, the key vector, and the value vector; concatenating the single-head attention output vectors of each attention head to form a multi-head attention output vector; and concatenating the multi-head attention output vector with the text feature vector to form the bimodal representation vector.

[0007] Optionally, the error correction decoder includes multiple decoding layers connected in sequence and having the same structure, each decoding layer including a self-attention sublayer, an associated attention sublayer, and a feedforward network sublayer connected in sequence; and the step of inputting the bimodal representation vector into the pre-trained error correction decoder includes: inputting the bimodal representation vector into the associated attention sublayer of each decoding layer.

[0008] Optionally, the method further includes: in any decoding layer of the error correction decoder, calculating a self-attention vector of the input vector of the decoding layer based on the self-attention sublayer of the decoding layer; normalizing the self-attention vector and combining it with the input vector to obtain a first intermediate vector, which is then passed to the associated attention sublayer of the decoding layer; in the associated attention sublayer, forming a query vector of multiple attention heads based on the first intermediate vector, forming a key vector and a value vector of the multiple attention heads based on the bimodal representation vector, and calculating a multi-head attention vector that fuses the first intermediate vector and the bimodal representation vector based on the query vector, key vector, and value vector; normalizing the multi-head attention vector and combining it with the first intermediate vector to obtain a second intermediate vector, which is then passed to the feedforward network sublayer of the decoding layer; calculating a mapping vector of the second intermediate vector based on the feedforward network sublayer, normalizing the mapping vector and combining it with the second intermediate vector to obtain the output vector of the decoding layer.

[0009] Optionally, the error correction decoder further includes a linear layer, a normalization layer, and an output layer connected to the last decoding layer; and the input vector of the frontmost decoding layer of the error correction decoder at any time step is the embedding vector of the currently generated morpheme, the output result of the output layer at any time step is the next morpheme of the currently generated morpheme, and the output results of the output layer at each time step constitute the error correction result.

[0010] Optionally, the associative attention module and the error correction decoder are jointly trained through the following steps: acquiring training audio data, corresponding training speech recognition text, and training error correction text as labels; inputting the training audio data into the audio feature extraction model to obtain the training audio feature vector of the training audio data; inputting the training speech recognition text into the text feature extraction model to obtain the training text feature vector of the training speech recognition text; inputting the training audio feature vector and the training text feature vector into the associative attention module; in the associative attention module, performing associative attention calculation based on the training audio feature vector and the training text feature vector, and outputting a training representation vector that fuses the training audio feature vector and the training text feature vector; inputting the training representation vector into the associative attention sublayer of each decoding layer in the error correction decoder, and inputting the embedding vector of the generated morphemes at the current time step into the front-end decoding layer of the error correction decoder to obtain the training output result of the error correction decoder; comparing the training output result with the training error correction text to form a loss function for training the associative attention module and the error correction decoder.

[0011] To achieve the above objectives, according to another aspect of the present invention, a speech recognition error correction device is provided.

[0012] The speech recognition error correction device of this invention may include: a data preparation unit for acquiring audio data and speech recognition text to be processed from the audio data; a feature extraction unit for inputting the audio data into a pre-trained audio feature extraction model to obtain an audio feature vector of the audio data; inputting the speech recognition text into a pre-trained text feature extraction model to obtain a text feature vector of the speech recognition text; a feature fusion unit for inputting the audio feature vector and the text feature vector into a pre-trained associative attention module; performing associative attention calculation based on the audio feature vector and the text feature vector in the associative attention module, and outputting a bimodal representation vector that fuses the audio feature vector and the text feature vector; and a decoding unit for inputting the bimodal representation vector into a pre-trained error correction decoder to obtain the error correction result of the speech recognition text.

[0013] Optionally, the feature fusion unit may be further configured to: form a query vector of multiple attention heads based on the text feature vector, and form a key vector and a value vector of the multiple attention heads based on the audio feature vector; calculate a single-head attention output vector for each attention head based on the query vector, key vector, and value vector, concatenate the single-head attention output vectors of each attention head to form a multi-head attention output vector; and concatenate the multi-head attention output vector with the text feature vector to form the bimodal representation vector.

[0014] To achieve the above objectives, according to another aspect of the present invention, an electronic device is provided.

[0015] An electronic device according to the present invention includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the speech recognition error correction method provided by the present invention.

[0016] To achieve the above objectives, according to another aspect of the present invention, a computer-readable storage medium is provided.

[0017] The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech recognition error correction method provided by the present invention.

[0018] According to the technical solution of the present invention, the embodiments described above have the following advantages or beneficial effects:

[0019] In the process of correcting errors in speech recognition text of audio data, a pre-trained feature extraction model is first used to extract features from both the audio data and its speech recognition text. Then, an associative attention mechanism is used to fuse the extracted audio and text feature vectors into a bimodal representation vector indicating the audio and text modalities. Finally, the bimodal representation vector is input into the error correction decoder to obtain the corrected speech recognition text. Thus, by using a multi-head attention mechanism to fuse the bimodal features (i.e., the bimodal representation vector) of audio and text during encoding, and by using a multi-head attention mechanism to jointly calculate the bimodal features and decoder input features during decoding, the error correction accuracy is improved.

[0020] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0021] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0022] Figure 1 This is a schematic diagram of the main steps of the speech recognition error correction method in this embodiment of the invention;

[0023] Figure 2 This is a schematic diagram of the error correction model in an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram of the error correction decoder in an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of the components of the speech recognition error correction device in an embodiment of the present invention;

[0026] Figure 5 This is an exemplary system architecture diagram that can be applied thereto according to embodiments of the present invention;

[0027] Figure 6 This is a schematic diagram of the electronic device structure used to implement the speech recognition error correction method in the embodiments of the present invention. Detailed Implementation

[0028] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0029] First, let me explain the technical concept of this invention. In the field of speech recognition within artificial intelligence, error correction models are needed to correct errors in the speech-recognition text. Most current error correction models learn error correction information by comparing the speech-recognition text with manually annotated correct text. However, many recognition errors cannot be determined correctly from the text alone. The inventors of this invention have discovered that it is necessary to combine the original audio information with the text to determine the correct result. Therefore, it is essential to introduce audio information into the error correction model. Furthermore, with the development of pre-trained models, by learning general representations from large amounts of unlabeled data and then fine-tuning them on labeled data, the model performance can be significantly improved. Therefore, a pre-trained model for extracting audio and text features can be introduced into the error correction model, and the extracted audio and text features can be fused before being input into the decoder, thereby improving the error correction accuracy of the model.

[0030] It should be noted that, unless otherwise specified, the embodiments of the present invention and the technical features thereof can be combined with each other.

[0031] Figure 1 This is a schematic diagram of the main steps of the speech recognition error correction method according to an embodiment of the present invention.

[0032] like Figure 1 As shown, the speech recognition error correction method of this invention can be executed according to the following steps:

[0033] Step S101: Acquire audio data and the speech recognition text to be processed.

[0034] In this step, the server used to execute the speech recognition error correction method acquires audio data (which may be the original audio signal) and the speech recognition text to be processed (i.e., to be corrected) obtained by the speech recognition model (i.e., the Automatic Speech Recognition (ASR) model, which is not included in the error correction model of this embodiment). Subsequently, the error correction model of this embodiment will combine the audio data and the speech recognition text to perform text error correction. The above speech recognition text may be composed of multiple morphemes, which are semantic-phonetic combinations in a language. They may be characters or words in Chinese, or words in alphabetic languages.

[0035] Step S102: Input the audio data into the pre-trained audio feature extraction model to obtain the audio feature vector of the audio data; input the speech recognition text into the pre-trained text feature extraction model to obtain the text feature vector of the speech recognition text.

[0036] To fuse features from both audio and text modalities, this step first involves feature extraction for each modality separately. In practical applications, a pre-trained audio feature extraction model can be used to extract audio feature vectors from the audio data, and a pre-trained text feature extraction model can be used to extract text feature vectors from the speech recognition text. These audio and text feature extraction models can retain their previous model parameters or be trained together with the associative attention module and error correction decoder in the error correction model (described later) to fine-tune the model parameters. For example, the audio feature extraction model could be Wave2vec2.0, and the text feature extraction model could be BERT. In the specific feature extraction process, the server can first segment the audio data into frames, then obtain the audio feature vector for each frame using the audio feature extraction model, and combine these audio feature vectors to form the audio feature vector of the audio data. Similarly, the server can obtain the text feature vector for each morpheme of the speech recognition text using the text feature extraction model, and combine these text feature vectors to form the text feature vector of the speech recognition text.

[0037] Step S103: Input the audio feature vector and the text feature vector into the pre-trained association attention module; in the association attention module, perform association attention calculation based on the audio feature vector and the text feature vector, and output a bimodal representation vector that fuses the audio feature vector and the text feature vector.

[0038] Figure 2 This is a schematic diagram of the error correction model in an embodiment of the present invention, as shown below. Figure 2As shown, the error correction model in this embodiment of the invention can be composed of the above-mentioned audio feature extraction model, text feature extraction model, cross-attention module, and error correction decoder. The cross-attention module is used to fuse audio features and text features through single-head or multi-head attention mechanisms and input the fused features into the error correction decoder, thereby combining the audio and text modalities to achieve error correction in speech recognition text and improve error correction accuracy. In specific scenarios, the cross-attention module and the error correction decoder can be jointly trained. As mentioned above, the cross-attention module and the error correction decoder can also be jointly trained with the audio feature extraction model and the text feature extraction model. The specific training process will be described below. The cross-attention module can be a pre-trained model with initial model parameters or an untrained model.

[0039] The attention calculation mechanism of the associative attention module can be a single-head attention mechanism, but a multi-head attention mechanism with better encoding performance is preferred. The attention mechanism used in the error correction decoder of this embodiment is also as described below, using a multi-head attention mechanism as an example. The attention calculation mechanism of the associative attention module can form a query vector based on text feature vectors or audio feature vectors. Since text features are generally considered to contribute more to the final error correction during the fusion of audio and text features, forming the query vector based on text feature vectors is a better approach. In a more optimized approach, within the associative attention module, the server generates query vectors for multiple attention heads based on text feature vectors (the text feature vectors can be used directly as query vectors, or they can be linearly transformed to form query vectors; the process of forming key and value vectors is similar). Based on audio feature vectors, it generates key and value vectors for these multiple attention heads. Then, based on the query vectors, key vectors, and value vectors, it calculates the single-head attention output vector for each attention head (using a known scaling dot product algorithm, a linear transformation can be performed after attention calculation). The single-head attention output vectors of each attention head are concatenated to form a multi-head attention output vector (this can be done by directly concatenating the single-head attention output vectors or by performing a linear transformation). The multi-head attention output vector can be understood as the result of aligning the audio feature vectors with the text feature vectors. Finally, the server concatenates the multi-head attention output vector with the text feature vectors to form a bimodal representation vector that fuses both audio and text modalities.

[0040] Optionally, if the audio feature vector is used as the query vector, then in the association attention module, the server forms query vectors for multiple attention heads based on the audio feature vector, and forms key vectors and value vectors for the above multiple attention heads based on the text feature vector. Then, the single-head attention output vector of each attention head is calculated based on the query vector, key vector, and value vector. The single-head attention output vectors of each attention head are concatenated to form a multi-head attention output vector. Finally, the multi-head attention output vector is concatenated with the audio feature vector to form another bimodal representation vector.

[0041] Step S104: Input the bimodal representation vector into the pre-trained error correction decoder to obtain the error correction result of the speech recognition text.

[0042] Figure 3 This is a schematic diagram of the error correction decoder in an embodiment of the present invention. See also... Figure 3 The error correction decoder comprises multiple sequentially connected decoding layers with identical structures. The layer closest to the input is the frontmost decoding layer, and the layer closest to the output is the rearmost decoding layer. Each decoding layer includes a self-attention sublayer, an associative attention sublayer, and a feedforward network sublayer, all connected sequentially. The self-attention sublayer performs single-head or multi-head self-attention calculations on the input vector of the decoding layer. The associative attention sublayer calculates the input data based on the output of the encoded portion (e.g., a bimodal representation vector). The feedforward network sublayer transforms the input data before outputting it. Generally, the error correction decoder may also include a linear layer, a normalization layer, and an output layer connected to the rearmost decoding layer. For the above error correction decoder structure, the bimodal representation vector obtained in step S103 can be input to the associative attention sublayer of each decoding layer. It is understood that the error correction decoder of this embodiment can also use any other applicable decoder structure.

[0043] In this embodiment of the invention, at any decoding layer of the error correction decoder, the server calculates the self-attention vector of the input vector of the decoding layer based on the self-attention sublayer of that decoding layer. After normalizing the self-attention vector (e.g., layer normalization, LN), it is combined with the input vector (e.g., added) to obtain a first intermediate vector, which is then passed to the associative attention sublayer of the decoding layer. In the associative attention calculation of this associative attention sublayer, a query vector can be formed based on the first intermediate vector or based on the bimodal representation vector.

[0044] In the first scenario, the server generates query vectors for multiple attention heads based on a first intermediate vector, and key and value vectors for these attention heads based on a bimodal representation vector. It then calculates a multi-head attention vector that fuses the first intermediate vector and the bimodal representation vector based on the query vector, key vector, and value vector. Subsequently, the server normalizes the multi-head attention vector (e.g., layer normalization, LN) and combines it with the first intermediate vector (e.g., by addition) to obtain a second intermediate vector, which is then passed to the feedforward sublayer of the decoding layer.

[0045] In the second scenario, the server generates a query vector for multiple attention heads based on the bimodal representation vector, and generates key and value vectors for these attention heads based on the first intermediate vector. It then calculates another multi-head attention vector by fusing the first intermediate vector and the bimodal representation vector, based on the query vector, key vector, and value vector. After normalizing the multi-head attention vector, the server combines it with the first intermediate vector to obtain a second intermediate vector, which is then passed to the feedforward sublayer of the decoding layer.

[0046] In the feedforward network sublayer, the server calculates the mapping vector of the second intermediate vector, that is, the second intermediate vector is transformed by the feedforward network to obtain the mapping vector. After that, the mapping vector is normalized (e.g., layer normalization LN) and then combined with the second intermediate vector (e.g., addition) to obtain the output vector of the decoding layer.

[0047] Furthermore, in practical applications, the input vector of the front-end decoding layer of the error correction decoder at any time step is the embedding vector of the currently generated morpheme, and the output result of the output layer at any time step is the next morpheme of the currently generated morpheme. The output results of the output layer at each time step constitute the error correction result of the speech recognition text.

[0048] Through the above structure of the error correction model, this embodiment of the invention uses an attention mechanism to fuse bimodal features of audio and text during the encoding process and uses an attention mechanism to jointly calculate the bimodal features and decoder input features during the decoding process, thereby improving the accuracy of speech recognition error correction.

[0049] Correspondingly, the associated attention module and the error correction decoder can be jointly trained through the following steps. If the audio feature extraction model and the text feature extraction model need to be fine-tuned, the associated attention module and the error correction decoder can also be jointly trained according to the following steps. First, the server obtains the training audio data, the corresponding training speech recognition text, and the training error correction text as labels. The training audio data is input into the audio feature extraction model to obtain the training audio feature vector of the training audio data; the training speech recognition text is input into the text feature extraction model to obtain the training text feature vector of the training speech recognition text. Then, the server inputs the training audio feature vector and the training text feature vector into the associated attention module. In the associated attention module, the server performs associated attention calculation based on the training audio feature vector and the training text feature vector, and outputs the training representation vector that fuses the training audio feature vector and the training text feature vector. After that, the server inputs the training representation vector into the associated attention sub-layer of each decoding layer in the error correction decoder, and inputs the embedding vector of the generated morpheme at the current time step into the frontmost decoding layer of the error correction decoder to obtain the training output result of the error correction decoder. The training output result is compared with the training error correction text to form a loss function for training the associated attention module and the error correction decoder. The specific calculation method during the training process is similar to the process of using the model, which will not be repeated here.

[0050] The following describes a specific embodiment of the present invention.

[0051] The first step is to construct training data. Use the trained fully supervised ASR model to recognize the training audio data in the target domain to obtain the training speech recognition text, such as "The delivery of buildings is not smooth" (there is an error). After manually recognizing the training audio data, the training error correction text as a label is formed, such as "The communication is not smooth" (the correct recognition result).

[0052] The second step is to train the error correction model. Send the training audio data into the pre-trained wav2vec2.0 model to obtain the training audio feature vector H a ; send the training speech recognition text into the pre-trained BERT model for encoding to obtain the training text feature vector H l .

[0053] After that, use the associated attention mechanism to fuse the features of the two modalities. Based on H l to form the query vector Q, based on H a to form the key vector K and the value vector V, and calculate the multi-head attention MHA. The calculation formula is:

[0054] MHA(Q, K, V) = Concat(head1, head2,..., head m )W o

[0055] Where concat means concatenation, W o Let be the transformation matrix, where each head represents the calculation result of a certain head, and m is the total number of attention heads.

[0056]

[0057] Where i is an integer between 1 and m, W i Q W i K W i V Let d be the transformation matrix, T denote the transpose, and d k This represents the dimension of the key vector.

[0058] The calculated MHA(Q,K,V) is an audio representation aligned with the training text feature vector. The concatenated representation H = (MHA(Q,K,V)); H... l ) is the training representation vector that integrates the audio and text modalities.

[0059] The decoder consists of 6 decoding layers, each with the structure described above. Assume the input to the i-th decoding layer is t. i The formula for calculating the self-attention sublayer is:

[0060] o i =MHA(t i ,t i ,t i )

[0061] r i =norm(o i )+t i

[0062] Where norm represents layer normalization, and the input to the associated attention sublayer is r. i It is based on r i The query vector is formed, and the key vector and value vector are formed based on H to calculate the association attention result.

[0063] During training, the decoder is first fed with sentence beginning identifiers. Then, the decoder's output is triggered at each time step. A single error correction training cycle ends when the decoder outputs sentence ending identifiers. The decoder's output is compared with the training text used as labels to form a loss function. Minimizing this loss function completes model training.

[0064] Step 3: Use of the error correction model. The audio data is sent into the wav2vec2.0 model to obtain audio feature vectors. The speech recognition text of the audio data, "That is, a fine of 200 yuan for three demerits", is sent into the BERT model for encoding to obtain text feature vectors. The associated attention module is used to fuse the features of the two modalities. After obtaining the bimodal representation vectors, they are sent into the error correction decoder. The error correction decoder adopts the autoregressive decoding method. First, the embedding vector of the sentence start token is input, and the decoder outputs the first character "记". Then, the embedding vectors of the sentence start token and "记" are used as inputs, and the second character "三" is output. After multiple iterations until the output is "记三分罚款两百元" <eos>Output sentence end marker. <eos>This means the error correction process is over. Finally, remove the sentence-end marker from the output to get the corrected result: "Three points deducted and a fine of two hundred yuan".

[0065] In the technical solution of this invention embodiment, the accuracy of the error correction model is improved by introducing a pre-trained model for audio and text feature extraction into the error correction model and by using a multi-head association attention mechanism to fuse bimodal information of text and speech.

[0066] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in the technical solution of this invention all comply with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0067] For the foregoing method embodiments, they are described as a series of actions for ease of description. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, and some steps may actually be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential for implementing the present invention.

[0068] To facilitate better implementation of the above-described solutions of the embodiments of the present invention, related apparatus for implementing the above-described solutions is also provided below.

[0069] Please see Figure 4 As shown, the speech recognition error correction device 400 provided in this embodiment of the invention may include: a data preparation unit 401, a feature extraction unit 402, a feature fusion unit 403, and a decoding unit 404.

[0070] The data preparation unit 401 can be used to acquire audio data and the speech recognition text to be processed from the audio data; the feature extraction unit 402 can be used to input the audio data into a pre-trained audio feature extraction model to obtain the audio feature vector of the audio data; and input the speech recognition text into a pre-trained text feature extraction model to obtain the text feature vector of the speech recognition text; the feature fusion unit 403 can be used to input the audio feature vector and the text feature vector into a pre-trained associative attention module; in the associative attention module, associative attention calculation is performed based on the audio feature vector and the text feature vector, and a bimodal representation vector fused from the audio feature vector and the text feature vector is output; the decoding unit 404 can be used to input the bimodal representation vector into a pre-trained error correction decoder to obtain the error correction result of the speech recognition text.

[0071] In this embodiment of the invention, the feature fusion unit 403 can be further used to: form a query vector of multiple attention heads based on the text feature vector, and form a key vector and a value vector of the multiple attention heads based on the audio feature vector; calculate a single-head attention output vector of each attention head according to the query vector, key vector and value vector, and concatenate the single-head attention output vector of each attention head to form a multi-head attention output vector; and concatenate the multi-head attention output vector with the text feature vector to form the bimodal representation vector.

[0072] In a specific application, the error correction decoder includes multiple decoding layers that are connected in sequence and have the same structure. Each decoding layer includes a self-attention sublayer, an associated attention sublayer, and a feedforward network sublayer that are connected in sequence. Furthermore, the decoding unit 404 can be used to input the bimodal representation vector into the associated attention sublayer of each decoding layer.

[0073] As a preferred embodiment, the decoding unit 404 may be further configured to: calculate a self-attention vector of the input vector of the decoding layer based on the self-attention sublayer of the decoding layer in any decoding layer; normalize the self-attention vector and combine it with the input vector to obtain a first intermediate vector, which is then passed to the associated attention sublayer of the decoding layer; in the associated attention sublayer, form a query vector of multiple attention heads based on the first intermediate vector, form a key vector and a value vector of the multiple attention heads based on the bimodal representation vector, and calculate a multi-head attention vector that fuses the first intermediate vector and the bimodal representation vector according to the query vector, key vector, and value vector; normalize the multi-head attention vector and combine it with the first intermediate vector to obtain a second intermediate vector, which is then passed to the feedforward network sublayer of the decoding layer; calculate a mapping vector of the second intermediate vector based on the feedforward network sublayer, normalize the mapping vector and combine it with the second intermediate vector to obtain the output vector of the decoding layer.

[0074] Preferably, the error correction decoder may further include a linear layer, a normalization layer, and an output layer connected to the last decoding layer; and the input vector of the frontmost decoding layer of the error correction decoder at any time step is the embedding vector of the currently generated morpheme, the output result of the output layer at any time step is the next morpheme of the currently generated morpheme, and the output results of the output layer at each time step constitute the error correction result.

[0075] Furthermore, in this embodiment of the invention, the device 400 may further include a model training unit, which is used to: acquire training audio data and corresponding training speech recognition text and training error correction text as labels; input the training audio data into the audio feature extraction model to obtain training audio feature vectors of the training audio data; input the training speech recognition text into the text feature extraction model to obtain training text feature vectors of the training speech recognition text; input the training audio feature vectors and the training text feature vectors into the associative attention module; in the associative attention module, perform associative attention calculation based on the training audio feature vectors and the training text feature vectors, and output a training representation vector that fuses the training audio feature vectors and the training text feature vectors; input the training representation vectors into the associative attention sublayers of each decoding layer in the error correction decoder, input the embedding vectors of the generated morphemes at the current time step into the frontmost decoding layer of the error correction decoder, and obtain the training output result of the error correction decoder; compare the training output result with the training error correction text to form a loss function for training the associative attention module and the error correction decoder.

[0076] According to the technical solution of the present invention, by using a multi-head attention mechanism to fuse the bimodal features of audio and text during the encoding process, and by using a multi-head attention mechanism to jointly calculate the bimodal features and decoder input features during the decoding process, the error correction accuracy is improved.

[0077] Figure 5 An exemplary system architecture 500 is shown that can be applied to the speech recognition error correction method or speech recognition error correction device of the present invention.

[0078] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, and 503, network 504, and server 505 (this architecture is merely an example; the components included in a specific architecture may be adjusted according to the specific application). Network 504 serves as the medium for providing a communication link between terminal devices 501, 502, and 503 and server 505. Network 504 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0079] Users can use terminal devices 501, 502, and 503 to interact with server 505 via network 504 to receive or send messages, etc. Various client applications can be installed on terminal devices 501, 502, and 503, such as speech recognition and error correction applications (for example only).

[0080] Terminal devices 501, 502, and 503 can be various electronic devices with displays that support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0081] Server 505 can be a server that provides various services, such as a backend server that supports speech recognition error correction applications operated by users using terminal devices 501, 502, and 503 (for example only). The backend server can process the received speech recognition error correction requests and feed back the processing results (such as error correction results - for example only) to terminal devices 501, 502, and 503.

[0082] It should be noted that the speech recognition error correction method provided in this embodiment of the invention is generally executed by server 505, and correspondingly, the speech recognition error correction device is generally set in server 505.

[0083] It should be understood that Figure 5 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0084] The present invention also provides an electronic device. The electronic device according to an embodiment of the present invention includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the speech recognition error correction method provided by the present invention.

[0085] The following is for reference. Figure 6 It shows a schematic diagram of the structure of a computer system 600 suitable for implementing an electronic device according to embodiments of the present invention. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0086] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage section 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the computer system 600. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0087] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0088] In particular, according to the embodiments disclosed in this invention, the processes described in the above main step diagrams can be implemented as computer software programs. For example, embodiments of this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the main step diagrams. In the above embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit 601, it performs the functions defined in the system of this invention.

[0089] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0090] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0091] The units described in the embodiments of the present invention can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor can be described as including a data preparation unit, a feature extraction unit, a feature fusion unit, and a decoding unit. The names of these units do not necessarily limit the specific unit; for example, the data preparation unit can also be described as "a unit that provides audio data and speech-recognized text to the feature extraction unit."

[0092] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to perform the following steps: acquiring audio data and speech-recognition text to be processed from the audio data; inputting the audio data into a pre-trained audio feature extraction model to obtain an audio feature vector of the audio data; inputting the speech-recognition text into a pre-trained text feature extraction model to obtain a text feature vector of the speech-recognition text; inputting the audio feature vector and the text feature vector into a pre-trained associative attention module; performing associative attention calculation based on the audio feature vector and the text feature vector in the associative attention module, and outputting a bimodal representation vector fusing the audio feature vector and the text feature vector; and inputting the bimodal representation vector into a pre-trained error correction decoder to obtain an error correction result for the speech-recognition text.

[0093] In the technical solution of this invention, when correcting speech recognition text errors for audio data, a pre-trained feature extraction model is first used to extract features from the audio data and its speech recognition text. Then, an associative attention mechanism is used to fuse the extracted audio feature vector and text feature vector into a bimodal representation vector indicating the audio and text modalities. Finally, the bimodal representation vector is input into the error correction decoder to obtain the corrected speech recognition text. Thus, by using a multi-head attention mechanism to fuse the bimodal features of audio and text during encoding and by using a multi-head attention mechanism to jointly calculate the bimodal features and decoder input features during decoding, the error correction accuracy is improved.

[0094] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.< / eos> < / eos>

Claims

1. A speech recognition error correction method, characterized by, The method comprises: obtaining audio data and speech recognition text to be processed of the audio data; inputting the audio data into a pre-trained audio feature extraction model to obtain an audio feature vector of the audio data; inputting the speech recognition text into a pre-trained text feature extraction model to obtain a text feature vector of the speech recognition text; inputting the audio feature vector and the text feature vector into a pre-trained correlation attention module; in the correlation attention module, performing correlation attention calculation based on the audio feature vector and the text feature vector, and outputting a dual-modal representation vector fusing the audio feature vector and the text feature vector; inputting the dual-modal representation vector into a pre-trained error correction decoder to obtain an error correction result of the speech recognition text; the error correction decoder comprises a plurality of decoding layers connected in sequence and having the same structure, and each decoding layer comprises a self-attention sublayer, a correlation attention sublayer and a feedforward network sublayer connected in sequence; and the inputting the dual-modal representation vector into the pre-trained error correction decoder comprises inputting the dual-modal representation vector into the correlation attention sublayer of each decoding layer; the correlation attention module and the error correction decoder are jointly trained by the following steps: obtaining training audio data, corresponding training speech recognition text and training error correction text as labels; inputting the training audio data into the audio feature extraction model to obtain a training audio feature vector of the training audio data; inputting the training speech recognition text into the text feature extraction model to obtain a training text feature vector of the training speech recognition text; inputting the training audio feature vector and the training text feature vector into the correlation attention module; in the correlation attention module, performing correlation attention calculation based on the training audio feature vector and the training text feature vector, and outputting a training representation vector fusing the training audio feature vector and the training text feature vector; inputting the training representation vector into the correlation attention sublayer of each decoding layer in the error correction decoder, inputting the embedding vector of the generated morpheme at the current time step into the frontmost decoding layer of the error correction decoder, and obtaining the training output result of the error correction decoder; comparing the training output result with the training error correction text to form a loss function for training the correlation attention module and the error correction decoder.

2. The method of claim 1, wherein, the performing correlation attention calculation based on the audio feature vector and the text feature vector, and outputting a dual-modal representation vector fusing the audio feature vector and the text feature vector, comprises: forming query vectors of a plurality of attention heads based on the text feature vector, and forming key vectors and value vectors of the plurality of attention heads based on the audio feature vector; calculating a single-head attention output vector of each attention head according to the query vectors, key vectors and value vectors, and concatenating the single-head attention output vectors of each attention head to form a multi-head attention output vector; concatenating the multi-head attention output vector and the text feature vector into the dual-modal representation vector.

3. The method of claim 1, wherein, The method further comprises: In any decoding layer of the error correction decoder, a self-attention vector of an input vector of the decoding layer is calculated based on a self-attention sublayer of the decoding layer, and after normalization is performed on the self-attention vector, the self-attention vector is combined with the input vector to obtain a first intermediate vector, which is transmitted to an associated attention sublayer of the decoding layer; In the associated attention sublayer, query vectors of a plurality of attention heads are formed based on the first intermediate vector, key vectors and value vectors of the plurality of attention heads are formed based on the dual-modal representation vector, and a multi-head attention vector fusing the first intermediate vector and the dual-modal representation vector is calculated according to the query vectors, the key vectors and the value vectors; after normalization is performed on the multi-head attention vector, the multi-head attention vector is combined with the first intermediate vector to obtain a second intermediate vector, which is transmitted to a feedforward network sublayer of the decoding layer; A mapping vector of the second intermediate vector is calculated based on the feedforward network sublayer, and after normalization is performed on the mapping vector, the mapping vector is combined with the second intermediate vector to obtain an output vector of the decoding layer.

4. The method of claim 3, wherein, The error correction decoder further comprises a linear layer, a normalization layer and an output layer connected at a last decoding layer; and An input vector of a frontmost decoding layer of the error correction decoder at any time step is an embedding vector of a current generated morpheme, and an output result of the output layer at any time step is a next morpheme of the current generated morpheme, and output results of the output layer at each time step constitute the error correction result.

5. A speech recognition error correction device, characterized by, Comprise: A data preparation unit configured to obtain audio data and a speech recognition text to be processed; A feature extraction unit configured to input the audio data into a pre-trained audio feature extraction model to obtain an audio feature vector of the audio data, and input the speech recognition text into a pre-trained text feature extraction model to obtain a text feature vector of the speech recognition text; A feature fusion unit configured to input the audio feature vector and the text feature vector into a pre-trained associated attention module; In the associated attention module, an associated attention calculation is performed based on the audio feature vector and the text feature vector, and a dual-modal representation vector fusing the audio feature vector and the text feature vector is output; A decoding unit configured to input the dual-modal representation vector into a pre-trained error correction decoder to obtain an error correction result of the speech recognition text; The error correction decoder comprises a plurality of decoding layers connected in sequence and having the same structure, each decoding layer comprising a self-attention sublayer, an associated attention sublayer and a feedforward network sublayer connected in sequence; And the inputting of the dual-modal representation vector into the pre-trained error correction decoder comprises inputting the dual-modal representation vector into the associated attention sublayer of each decoding layer. The association attention module and the error correction decoder are jointly trained by the following steps: obtaining training audio data, corresponding training speech recognition text and training error correction text as labels; inputting the training audio data into the audio feature extraction model to obtain a training audio feature vector of the training audio data; inputting the training speech recognition text into the text feature extraction model to obtain a training text feature vector of the training speech recognition text; inputting the training audio feature vector and the training text feature vector into the association attention module; performing association attention calculation based on the training audio feature vector and the training text feature vector in the association attention module to output a training representation vector fusing the training audio feature vector and the training text feature vector; inputting the training representation vector into the association attention sublayer of each decoding layer of the error correction decoder, inputting the embedding vector of the generated morpheme at the current time step into the frontmost decoding layer of the error correction decoder, and obtaining a training output result of the error correction decoder; comparing the training output result with the training error correction text to form a loss function for training the association attention module and the error correction decoder.

6. The apparatus of claim 5, wherein, The feature fusion unit is further used for: forming query vectors of a plurality of attention heads based on the text feature vector, forming key vectors and value vectors of the plurality of attention heads based on the audio feature vector; calculating a single-head attention output vector of each attention head according to the query vectors, key vectors and value vectors, and splicing the single-head attention output vectors of each attention head to form a multi-head attention output vector; and splicing the multi-head attention output vector and the text feature vector to form the dual-modal representation vector.

7. An electronic device, comprising: comprise: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-4.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1-4.

Images