A streaming speech recognition method incorporating text error correction

Abstract

This invention provides a streaming speech recognition method combined with text error correction. The method includes: segmenting the audio data to be recognized into blocks; inputting the audio data of each block into a streaming speech recognition system; the streaming speech recognition system outputting the recognized text of each block; dividing the recognized text of each block into equal-length text blocks, and then inputting them into an error correction model; the error correction model outputting the encoded representation of each block through an encoding module, and then decoding each block's encoded representation block by block through a decoding module, calculating the correlation between the current frame and historical frames using a memory bank, and outputting the error-corrected blocks; sending the error-corrected blocks to an edge compensation module, the edge compensation module using the common decoding results of historical blocks to correct and compensate the right edges of each block, obtaining the final recognized text of the audio data of each block. This invention achieves streaming output of the error correction model and solves the problem of edge error correction using an edge compensation mechanism.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to a streaming speech recognition method that incorporates text error correction. Background Technology

[0002] With the widespread adoption of smart devices, Automatic Speech Recognition (ASR) technology is gradually integrating into our work and daily lives. Speech recognition technology helps us convert speech into text, but typically, ASR systems need to receive the entire sentence before they can begin processing the audio. Therefore, recognizing any character within each speech segment requires waiting for the entire speech to be recorded. To reduce the latency caused by this waiting time to an acceptable level, researchers have proposed streaming speech recognition technology. In recent years, streaming speech recognition technology has been applied to various scenarios, including video conferencing, live sports or gaming events, voice input methods, and smart voice assistants on mobile devices.

[0003] Compared to non-streaming speech recognition, streaming speech recognition offers lower latency and higher memory efficiency, making it more suitable for tasks requiring real-time feedback or processing. Through streaming chunking, audio is segmented into shorter speech segments, each of which can be recognized independently, enabling rapid response and allowing streaming speech recognition systems to output results in real time. While current streaming speech recognition systems have achieved good recognition results, many errors still occur due to homophone and near-homophone confusion. To improve the language modeling capabilities of speech recognition systems and reduce homophone and near-homophone confusion through contextual information, common practices include adding language models or attention-based re-scoring decoding. After streaming decoding, the results are re-scored to improve the quality of the output text. However, this introduces high latency, and because re-scoring of the speech model or attention can only be performed after the entire sentence has been streamed, these methods compromise the system's real-time performance to some extent. Non-autoregressive text correction models have lower latency and can enhance the language modeling capabilities of speech recognition systems, but they lack real-time performance and require streaming improvements.

[0004] Because homophones and near-homophones have the same or similar pronunciation as the original words, and their phoneme composition is usually the same, they introduce significant interference to speech recognition tasks. The acoustic feature discrimination of homophones and near-homophones is generally low; even if the speech recognition system correctly identifies the phonemes, there is still a risk of misidentifying them as homophones or near-homophones in the correct text.

[0005] Non-autoregressive text correction models have low latency and can learn language knowledge from large-scale texts. They can detect and correct errors in speech recognition output text based on context and relevance. However, existing text correction models can only take whole sentences as input. When using streaming strategies to segment the input for error correction, insertion and deletion errors will occur because words at block boundaries are segmented.

[0006] End-to-end streaming models can be divided into three categories: CTC (Connectionist temporal classification), RNN-T (Recurrent Neural Network Transducer), and attention encoding / decoding streaming models. Among them, CTC and RNN-T are natural streaming speech recognition systems.

[0007] One existing streaming speech recognition scheme that incorporates a language model includes: the streaming speech recognition system that incorporates a language model mainly consists of three parts: an encoder, a decoder, and a language model.

[0008] The encoder is responsible for converting the input speech signal into a high-level representation of speech features. The encoder consists of multiple layers of Transformers or Conformers used to extract speech features and capture the temporal information of the speech signal. To achieve streaming output, this method divides the input frames into non-overlapping blocks and feeds them sequentially into the encoder.

[0009] The decoder receives the input feature sequence from the encoder and maps it to an output sequence using a neural network model. First, the decoder uses the CTC loss function to calculate the difference between the network output and the label sequence. It then uses a prefix beam search algorithm to generate the most probable output sequence, considering the probability distribution of the output sequence and searching for the optimal path through dynamic programming. Next, the language model predicts the next possible character based on the generated partial output sequence and contextual information. Mainstream language models are divided into statistical n-gram models and neural network-based models, such as recurrent neural network language models (RNNLM) or Transformer language models.

[0010] Language models are trained on large-scale text data and estimate the probability of the next character appearing in a given context based on statistical patterns and contextual information. Depending on the combination method, current language models can be divided into internal language models and external language models. Internal language models, as part of an end-to-end speech recognition model, are trained alongside the end-to-end speech recognition model on a domain-specific corpus. Although the model is smaller, the language knowledge it learns is limited. External language models can be trained independently on large-scale, multi-domain text corpora, thus providing the speech recognition model with language knowledge and contextual dependency information beyond the training corpus. However, this introduces higher latency, affecting the feedback experience of real-time speech recognition.

[0011] The disadvantages of the existing streaming speech recognition schemes that combine language models include:

[0012] Using language models can effectively improve the accuracy of streaming speech recognition, but it leads to a significant increase in computational complexity and latency. At the same time, the larger model size raises the deployment threshold for streaming speech recognition systems.

[0013] Because the language model decoding stage requires waiting for the complete CTC decoded text output, this compromises the real-time performance of streaming speech recognition to some extent.

[0014] One existing method for streaming speech recognition that incorporates attention rescoring includes: This method typically uses a Conformer or Transformer to directly label the input sequence, outputs a probability distribution of the label sequence based on the input sequence through the network, and then outputs the label sequence with the highest probability. Let the input sequence be x = (x1, x2, ..., x...). T The output label sequence is: L * = {L, blank}, where L is the character set and (blank) is the blank label. Given a training set... The goal of CTC model training is to make the network map x to z as realistically as possible. The length of the input feature sequence in a speech recognition system is usually greater than the length of the output text sequence. To address this, the CTC model inserts empty labels between output labels and allows label repetition, thus ensuring that the input and output sequence lengths are consistent. During decoding, path aggregation is used to achieve many-to-one alignment. The CTC model can calculate the conditional probability of each possible path in the output sequence, similar to the calculation in a Hidden Markov Model. When calculating conditional probabilities, the CTC loss can be efficiently calculated using a forward-backward algorithm, still predicting the target for each frame while assuming output independence, thus simplifying the computation process.

[0015] The low latency of CTC's non-autoregressive nature is a perfect match for the characteristics of streaming speech recognition. The streaming speech recognition framework that combines attention rescoring uses the CTC decoder for streaming output in the first round and attention rescoring technology in the second round to improve the quality of the complete sentence recognition output text.

[0016] Attention rescore is a technique that uses an attention mechanism to rescore the N-best candidate results of CTC prefix beam search decoding to obtain more accurate results. Attention rescore decoding is non-autoregressive, therefore its latency is relatively low, and it can significantly reduce the error rate of CTC results. Therefore, it can be applied to streaming scenarios to correct results after streaming decoding. In U2++, a bidirectional attention rescore decoder (Left2Right / Right2Left attention decoder) is introduced to model the left and right contexts.

[0017] Streaming speech recognition systems that incorporate attention rescoring require joint training. Their loss functions include three types: CTC loss, L2R AED loss, and R2L AED loss. The final loss is obtained by weighting these three losses.

[0018] Loss final =λL CTC +αL AED―L2R +(1―λ―α)L AED―R2L (1-1)

[0019] λ and α are used to balance the weights of CTC loss and attention loss. Joint training can accelerate the convergence speed of speech recognition model training and improve the stability of training.

[0020] The drawbacks of a current streaming speech recognition method that incorporates attention rescoring include:

[0021] The attention decoder is trained as a whole with the end-to-end streaming speech recognition system, so it cannot acquire language knowledge outside the training dataset of the streaming speech recognition system. When dealing with domains not included in the training dataset, the recognition accuracy of the streaming speech recognition system based on this method will drop significantly.

[0022] Attention rescoring also requires waiting for the complete CTC decoded text output, which to some extent disrupts the real-time performance of streaming speech recognition. Summary of the Invention

[0023] Embodiments of the present invention provide a streaming speech recognition method that combines text error correction to effectively improve the accuracy and readability of the output text of a streaming speech recognition system.

[0024] To achieve the above objectives, the present invention adopts the following technical solution.

[0025] A streaming speech recognition method incorporating text error correction includes:

[0026] After the audio data to be recognized is divided into blocks, the audio data of each block is input into the streaming speech recognition system, and the streaming speech recognition system outputs the recognized text of each block.

[0027] The recognized text of each block is divided into equal-length text blocks, and the equal-length text blocks are input into the error correction model.

[0028] The error correction model outputs the encoded representation of each block through the encoding module, and then decodes the encoded representation of each block one by one through the decoding module. It uses the memory bank to calculate the correlation between the current frame and the historical frames, and outputs the error-corrected blocks.

[0029] The error-corrected blocks are sent to the edge compensation module, which uses the common decoding results of historical blocks to correct and compensate the right edge of each block, thus obtaining the final recognized text of the audio data of each block.

[0030] Preferably, after segmenting the audio data to be recognized into blocks, the audio data of each block is input into the streaming speech recognition system, and the streaming speech recognition system outputs the recognized text of each block, including:

[0031] The streaming speech recognition system is configured to use the Conformer-CTC architecture, and the text error correction model is configured to use the Transformer codec architecture.

[0032] The acoustic feature sequence X = (x1, x2, ..., x...) is used to define the acoustic feature sequence X = (x1, x2, ..., x...) L The input is fed into the convolutional downsampling module of the streaming speech recognition system. The convolutional downsampling module acquires the feature sequence in the acoustic feature sequence X, divides the feature sequence into blocks, and sends them to the Conformer encoder. The Conformer encoder uses a self-attention mechanism to model the correlation between frames in the feature block. The encoded block output by the Conformer encoder is fed into the CTC decoder and converted into a text sequence. The above operation is represented by the following formula:

[0033] X1=Conv2d(X)(2-1)

[0034] H = Conformer(X1)(2-2)

[0035] Y = CTC(H) (2-3)

[0036] Where Conv2d represents a two-dimensional convolutional neural network, H is the encoded representation, and Y is the CTC decoded output sequence.

[0037] Preferably, the step of dividing the recognized text of each block into equal-length text blocks and inputting the equal-length text blocks into the error correction model includes:

[0038] The SACM streaming speech recognition framework divides the text sequence output by the streaming speech recognition system into equal-length text blocks, and then inputs these equal-length text blocks into the error correction model.

[0039] Let S be the text sequence output by Automatic Speech Recognition (ASR). Before feeding it to the error correction model, the text sequence is re-planned by dividing S into equal-length text blocks, each containing a fixed number of characters. Let there be n blocks, then the resulting block sequence is s1, s2, ..., s n .

[0040] Preferably, the error correction model outputs the encoded representation of each block through the encoding module, and then decodes the encoded representation of each block block by block through the decoding module. It uses a memory bank to calculate the correlation between the current frame and historical frames, and outputs the error-corrected blocks, including:

[0041] After the text block sequence is fed to the error correction module, the error correction module outputs the hidden sequence H through the Transformer encoder. The hidden sequence H is input to the error detector to obtain the error mark sequence L. Then H and L are fed together to the Transformer decoder to complete the correction process and obtain the output text O1.

[0042] S = re-chunk(Y) (2-4)

[0043] H=Transformer Encoder(S)(2-5)

[0044] L = errordetection(H) (2-6)

[0045] O1=TransformerDecoder(H,L)(2-7)

[0046] Where re-chunk represents re-chunking, H is the encoded representation, L is the error detector output sequence, and O1 is the decoded output;

[0047] The memory stores blocks that the error correction model has already computed. When processing the next block, the error correction model retrieves the intermediate variables needed for attention computation from the memory. Each Transformer layer performs a cache operation on historical blocks. The cache operation after the nth Transformer layer is represented as follows:

[0048] Bn =B n ∪{s i，n ′} (2-9)

[0049] Among them B n This represents the nth-level memory, which stores the processing results of all historical blocks at the nth level. It is cached after the current block's hidden representation is output by this Transformer layer. i,n ' represents the result obtained after the i-th encoded block passes through the n-th Transformer layer. The calculation formula for the attention part in the n-th Transformer layer is as follows:

[0050] Q n,i =W q X n―1,i (2-10)

[0051] K n,i =W k [X n―1,i ,X n―1,cache (2-11)

[0052] V n,i =W v [X n―1,i ,X n―1,cache (2-12)

[0053] A n, i = Attention(Q) n,i ,K n,i V n,i (2-13)

[0054] W represents the weight matrix, Q, K, and V represent the query, key, and value matrices, respectively, and A is the attention output;

[0055] Where Q is the query tensor, K and V are the key-value tensors, and the output of the i-th block of the n-th Transformer layer is obtained after passing through the feedforward layer:

[0056] s i,n ′=feed_forward(A n,i (2-14)

[0057] Where A n,i ' represents the normalized output of the attention layer, resulting in the error-corrected block s. i,n ' will be placed into cache library B' n middle.

[0058] Preferably, the step of sending the error-corrected blocks to the edge compensation module, wherein the edge compensation module uses the common decoding results of historical blocks to correct and compensate the right edge of each block, thereby obtaining the final recognized text of the audio data of each block, including:

[0059] Each time the edge compensation module corrects errors in the current block, it decodes the encoded representations of the historical blocks and the current block together, recovers characters that were erroneously deleted, replaced, or inserted on the right edge of adjacent historical blocks, deletes the characters on the right edge of the current block, and obtains the final output of the current block. At this point, the final recognized text of the audio data for each block is:

[0060] Y i = f(s1,s2,...,s i )―S i―1 ―π i,edge (2-15)

[0061] Where Y i Let f(·) represent the output sequence of the i-th block, f(·) represent the error correction model, and S i―1 Represents the output sequence of the history block, π i,edge This represents the right edge character of the decoding result of the i-th block. When a character is inserted on the right edge, two characters on the right edge need to be deleted. If a character is deleted, the right edge character does not need to be deleted.

[0062] As can be seen from the technical solutions provided by the embodiments of the present invention above, the method of the present invention realizes the streaming output of the error correction model, realizes the cascading combination of the error correction model and the streaming speech recognition system through re-blocking processing, and solves the problem of edge error correction by using an edge compensation mechanism, thus solving the problem that existing methods cannot fully guarantee the real-time performance of the system.

[0063] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of the invention. Attached Figure Description

[0064] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 A structural diagram of SACM (Streaming ASR with Correction Model), a novel streaming speech recognition framework that combines a text correction model, is provided for embodiments of the present invention.

[0066] Figure 2 A flowchart illustrating a streaming speech recognition method incorporating text correction, provided as an embodiment of the present invention;

[0067] Figure 3 This is a schematic diagram of a streaming error correction process provided in an embodiment of the present invention;

[0068] Figure 4 This is a schematic diagram illustrating edge compensation for audio data blocks according to an embodiment of the present invention. Detailed Implementation

[0069] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0070] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or couplings. The term “and / or” as used herein includes any and all combinations of one or more of the associated listed items.

[0071] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0072] To facilitate understanding of the embodiments of the present invention, the following will provide further explanation and description with reference to the accompanying drawings and several specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0073] This invention proposes a streaming speech recognition method incorporating text correction to improve the language modeling capability of Chinese streaming speech recognition systems. Firstly, this invention proposes a novel SACM (Streaming ASR with Correction Model) framework incorporating a text correction model, as shown below. Figure 1 As shown, by re-segmenting the output text of streaming speech recognition, a cascaded combination of the streaming speech recognition system and the text correction model is achieved. The streaming speech recognition system and the text correction module are trained independently, allowing the text correction model to learn richer linguistic knowledge from a large amount of text data. Secondly, this invention also proposes a method of segmented input and multi-block edge compensation error correction, reducing erroneous deletion and insertion at block edges, achieving streaming output of the error correction model, ensuring the real-time performance of the streaming speech recognition system, and reducing homophone and near-homophone recognition errors in Chinese streaming speech recognition.

[0074] The processing flowchart of a streaming speech recognition method combining text error correction provided in this embodiment of the invention is as follows: Figure 2 As shown, the processing steps include the following;

[0075] Step S10: After dividing the audio data to be recognized into blocks, the audio data of each block is input into the streaming speech recognition system, and the streaming speech recognition system outputs the recognized text of each block.

[0076] Step S20: Divide the recognized text of each block into equal-length text blocks and input the equal-length text blocks into the error correction model.

[0077] Step S30: The error correction model outputs the encoded representation of each block through TransformerEncoder, and then decodes the encoded representation of each block block by block through TransformerDecoder. It uses the memory bank to calculate the correlation between the current frame and the historical frames, and outputs the error-corrected blocks.

[0078] Step S40: Send each block after error correction to the edge compensation module. The edge compensation module uses the common decoding results of historical blocks to correct and compensate the right edge of each block, and obtain the final recognized text of the audio data of each block.

[0079] Specifically, step S10 includes: the streaming speech recognition system adopts the Conformer-CTC architecture, and the text correction model adopts the Transformer codec architecture. The two are trained independently so that the text correction model can learn richer language knowledge from a large amount of paired text data.

[0080] For a streaming speech recognition system, the acoustic feature sequence is first X = (x1, x2, ..., x...). L The input is fed into the convolutional downsampling module to capture local feature information and reduce the size of the feature map, thereby reducing the computational cost of subsequent networks. The feature sequence is then fed into the Conformer encoder in blocks, where a self-attention mechanism is used to model the correlation between frames within the feature block. Finally, the encoded block is fed into the CTC (Connectionist Temporal Classification) decoder to be converted into a text sequence representation. The above operations can be represented by the following formula:

[0081] X1=Conv2d(X)(2-1)

[0082] H = Conformer(X1)(2-2)

[0083] Y = CTC(H) (2-3)

[0084] Where Conv2d represents a two-dimensional convolutional neural network, H is the encoded representation, and Y is the CTC decoded output sequence.

[0085] Specifically, step S20 includes: the output text sequence of the streaming speech recognition system needs to be regrouped before being fed to the text correction module. The output text blocks of the streaming speech recognition system are of varying lengths, while the error correction model needs to receive text blocks of the same length as input. Therefore, SACM reorganizes the speech recognition output into text blocks of equal length before inputting them into the text correction module.

[0086] The actual input to streaming speech recognition is a speech segment of the same duration. Due to the differences in the pronunciation duration of different Chinese characters and the phenomenon of delayed tag emission, the lengths of different output text blocks of streaming speech recognition are also different. Therefore, SACM performs re-blocking processing on the output of streaming speech recognition, such as... Figure 3 As shown.

[0087] Let the output stream of ASR (Automatic Speech Recognition) be S. Before being fed into the error correction model, it is reorganized. Each block contains a fixed number of characters. Assuming it is divided into n blocks, we can obtain s1, s2, ..., s n Analysis of the audio file data revealed that the average duration of Chinese characters is approximately 180ms. This invention sets the block length to 4, corresponding to a block delay of approximately 720ms, which meets the real-time requirements.

[0088] S = re-chunk(Y) (2-4)

[0089] Where re-chunk means re-chunking.

[0090] Specifically, step S30 includes: after the text block is fed to the error correction module, the error correction module first outputs the hidden sequence H through the Transformer encoder. The hidden sequence H is input to the error detector to obtain the error marker sequence L. Then, H and L are fed together to the Transformer decoder to complete the correction process to obtain O1. Finally, the edge compensation module obtains the final output text.

[0091] H=Transformer Encoder(S)(2-5)

[0092] L = errordetection(H) (2-6)

[0093] O1=TransformerDecoder(H,L)(2-7)

[0094] H represents the encoded representation, L represents the error detector output sequence, O1 represents the decoded output, and O2 represents the output after edge compensation.

[0095] Existing error correction models require complete text input, which does not meet real-time requirements. Therefore, this invention proposes a streaming method for text error correction models. To achieve streaming encoding and decoding, block processing limits attention to within blocks. To improve global modeling capabilities without introducing dependencies on future contexts, the proposed streaming error correction model uses a cache library to store processed historical blocks for subsequent computation. Furthermore, to address the phenomenon of miscorrection at block edges, this invention proposes a multi-block edge compensation error correction method.

[0096] After input segmentation, the error correction model does not correct each block individually, but rather combines previous blocks to correct the current block, thus modeling the correlation between the current time step and historical time steps. To reduce computation, the memory stores already computed blocks, and retrieves the intermediate variables needed for attention operations from the memory when processing the next block.

[0097] Figure 3 This is a schematic diagram of a streaming error correction process provided in an embodiment of the present invention. Each Transformer layer needs to perform a caching operation on historical blocks. The caching operation after the nth Transformer layer can be represented as:

[0098] B n =B n ∪{s i,n ′} (2-9)

[0099] Among them B nThis represents the nth-level memory, which stores the processing results of all historical blocks at the nth level. It is cached after the current block's hidden representation is output by this Transformer layer. i,n ' represents the result obtained after the i-th encoded block passes through the n-th Transformer layer. The calculation formula for the attention part in the n-th Transformer layer is as follows:

[0100] Q n,i =W q X n―1,i (2-10)

[0101] K n,i =W k [X n―1,i ,X n―1,cache (2-11)

[0102] V n,i =W v [X n―1,i ,X n―1,cache (2-12)

[0103] A n,i =Attention(Q) n,i ,K n,i V n,i (2-13)

[0104] Where W represents the weight matrix, Q, K, and V represent the query, key, and value matrices, respectively, and A is the attention output.

[0105] Where Q is the query tensor, and K and V are the key-value tensors, respectively. The output of the i-th block of the n-th Transformer layer is obtained after passing through the feedforward layer:

[0106] s i,n ′=feed_forward(A n,i (2-14)

[0107] Where A n,i ' represents the normalized output of the attention layer, resulting in the error-corrected block s. i,n ' will be placed into cache library B' n middle.

[0108] Specifically, step S40 includes: Since the input text block accepted by the error correction model may have word segmentation, such as "multi-block recognition" may be segmented into "multi-block recognition" and "discrimination", causing "recognition" or "discrimination" to be deleted by the error correction model, the present invention adds an edge compensation module to the streaming text error correction model.

[0109] Figure 4This is a schematic diagram illustrating edge compensation for audio data blocks according to an embodiment of the present invention. Figure 4 As shown, each time error correction is performed on the current block, the encoded representations of the historical blocks and the current block are decoded together. Characters that were incorrectly deleted, replaced, or inserted on the right edge of adjacent historical blocks are recovered to avoid miscorrection caused by segmentation. Simultaneously, to prevent the output of characters on the right edge of the current block that have been incorrectly corrected, the characters on the right edge of the current block are deleted, thus obtaining the final output of the current block. At this point, the final recognized text of the audio data for each block is:

[0110] Y i = f(s1,s2,...,s i )―S i―1 ―π i,edge (2-15)

[0111] Where Y i Let f(·) represent the output sequence of the i-th block, f(·) represent the error correction model, and S i―1 Represents the output sequence of the history block, π i,edge This represents the right edge character of the decoding result of the i-th block. When a character is inserted on the right edge, two characters on the right edge need to be deleted. If a character is deleted, the right edge character does not need to be deleted.

[0112] This invention can be applied to meeting scenarios, game or sports event live streaming scenarios, and terminal smart assistants, etc., for real-time recording of meeting information, transcription of live streaming subtitles, and real-time output of transcribed text during human-computer interaction.

[0113] In summary, this invention proposes a streaming speech recognition method incorporating an error correction model, which effectively improves the language modeling capability of streaming speech recognition systems and reduces recognition errors caused by confusion between homophones and near-homophones. Compared with existing methods for improving language modeling capabilities, this invention uses a lightweight text error correction model to learn rich linguistic knowledge from large-scale text data, improving the model's domain adaptability while ensuring the system's low latency.

[0114] This invention realizes the streaming output of the error correction model, achieves the cascading combination of the error correction model and the streaming speech recognition system through re-blocking processing, and solves the problem of edge error correction by using an edge compensation mechanism, thus solving the problem that existing methods cannot fully guarantee the real-time performance of the system.

[0115] This invention achieves streaming output of the error correction model, combining text error correction with streaming speech recognition to improve the language modeling capability of streaming speech recognition. It uses a re-blocking method to cascade the error correction model and the streaming speech recognition system, adding an edge compensation module at the end of the error correction model to delete Chinese characters on the right edge of the current text block and perform compensation output in the next block, thus avoiding miscorrection.

[0116] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0117] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0118] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for apparatus or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The apparatus and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0119] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A streaming speech recognition method incorporating text error correction, characterized in that, include: After the audio data to be recognized is divided into blocks, the audio data of each block is input into the streaming speech recognition system, and the streaming speech recognition system outputs the recognized text of each block. The recognized text of each block is divided into equal-length text blocks, and the equal-length text blocks are input into the error correction model. The error correction model outputs the encoded representation of each block through the encoding module, and then decodes the encoded representation of each block one by one through the decoding module. It uses the memory bank to calculate the correlation between the current frame and the historical frames, performs error correction processing on each block based on the correlation between the current frame and the historical frames, and outputs the error-corrected blocks. The error-corrected blocks are sent to the edge compensation module, which uses the common decoding results of historical blocks to correct and compensate the right edge of each block, thus obtaining the final recognized text of the audio data of each block.

2. The method according to claim 1, characterized in that, The process of dividing the audio data to be recognized into blocks and inputting each block of audio data into a streaming speech recognition system, wherein the streaming speech recognition system outputs the recognized text of each block, includes: The streaming speech recognition system is configured to use the Conformer-CTC architecture, and the text error correction model is configured to use the Transformer codec architecture. The acoustic feature sequence X = (x1, x2, ..., x...) is used to define the acoustic feature sequence X = (x1, x2, ..., x...) L The input is fed into the convolutional downsampling module of the streaming speech recognition system. The convolutional downsampling module acquires the feature sequence in the acoustic feature sequence X, divides the feature sequence into blocks, and sends them to the Conformer encoder. The Conformer encoder uses a self-attention mechanism to model the correlation between frames in the feature block. The encoded block output by the Conformer encoder is fed into the CTC decoder and converted into a text sequence. The above operation is represented by the following formula: X1=Conv2d(X) (2-1) H = Conformer(X1) (2-2) Y = CTC(H) (2-3) Where Conv2d represents a two-dimensional convolutional neural network, H is the encoded representation, and Y is the CTC decoded output sequence.

3. The method according to claim 2, characterized in that, The process of dividing the recognized text of each block into equal-length text blocks and inputting the equal-length text blocks into the error correction model includes: The SACM streaming speech recognition framework divides the text sequence output by the streaming speech recognition system into equal-length text blocks, and then inputs these equal-length text blocks into the error correction model. Let S be the text sequence output by Automatic Speech Recognition (ASR). Before feeding it to the error correction model, the text sequence is re-planned by dividing S into equal-length text blocks, each containing a fixed number of characters. Let there be n blocks, then the resulting block sequence is s1, s2, ..., s n .

4. The method according to claim 3, characterized in that, The error correction model outputs the encoded representation of each block through the encoding module, and then decodes the encoded representation of each block block by block through the decoding module. It uses a memory bank to calculate the correlation between the current frame and historical frames, and outputs the error-corrected blocks, including: After the text block sequence is fed to the error correction module, the error correction module outputs the hidden sequence H through the Transformer encoder. The hidden sequence H is input to the error detector to obtain the error mark sequence L. Then H and L are fed together to the Transformer decoder to complete the correction process and obtain the output text O1. S = re-chunk(Y) (2-4) H=Transformer Encoder(S) (2-5) L = errordetection(H) (2-6) O1=TransformerDecoder(H,L) (2-7) Where re-chunk represents re-chunking, H is the encoded representation, L is the error detector output sequence, and O1 is the decoded output; The memory stores blocks that the error correction model has already computed. When processing the next block, the error correction model retrieves the intermediate variables needed for attention computation from the memory. Each Transformer layer performs a cache operation on historical blocks. The cache operation after the nth Transformer layer is represented as follows: B n =B n ∪{s i,n ′} (2-9) Among them B n This represents the nth-level memory, which stores the processing results of all historical blocks at the nth level. It is cached after the current block's hidden representation is output by this Transformer layer. i,n ' represents the result obtained after the i-th encoded block passes through the n-th Transformer layer. The calculation formula for the attention part in the n-th Transformer layer is as follows: Q n,i =W q X n―1,i (2-10) K n,i =W k [X n―1,i ,X n―1,cache ] (2-11) V n,i =W v [X n―1,i ,X n―1,cache ] (2-12) A n,i =Attention(Q n,i ,K n,i ,V n,i ) (2-13) W represents the weight matrix, X n―1,cache Let Q represent the input tensor from the history block, Q, K, and V represent the query, key, and value matrices, respectively, and A be the attention output. The output of the i-th block of the n-th Transformer layer is obtained after passing through the feedforward layer: s i,n ′=feed_forward(A n,i ′) (2-14) Where A n,i ' represents the normalized output of the attention layer, resulting in the error-corrected block s. i,n ' will be placed into cache library B' n middle.

5. The method according to claim 4, characterized in that, The process involves sending each block after error correction to an edge compensation module. The edge compensation module uses the shared decoding results of historical blocks to correct and compensate the right edges of each block, resulting in the final recognized text of the audio data for each block, including: Each time the edge compensation module corrects errors in the current block, it decodes the encoded representations of the historical blocks and the current block together, recovers characters that were erroneously deleted, replaced, or inserted on the right edge of adjacent historical blocks, deletes the characters on the right edge of the current block, and obtains the final output of the current block. At this point, the final recognized text of the audio data for each block is: Y i =f(s1,s2,...,s i )―S i―1 ―π i,edge (2-15) Where Y i Let f(·) represent the output sequence of the i-th block, f(·) represent the error correction model, and S i―1 Represents the output sequence of the history block, π i,edge This represents the right edge character of the decoding result of the i-th block. When a character is inserted on the right edge, two characters on the right edge need to be deleted. If a character is deleted, the right edge character does not need to be deleted.

Images