Non-streaming model streaming speech recognition method based on attention and boundary detection

Abstract

The present invention belongs to the field of speech recognition, and relates to a non-streaming model streaming speech recognition method based on attention and boundary detection, including: segmenting audio clips, extracting features from the audio clips, and generating a logarithmic Mel spectrum graph; using a non-streaming speech model Whisper to transcribe the audio clips, extracting the cross-attention of each frame of audio in the model, and obtaining the corresponding attention weight; setting a sliding window of length, and calculating the average attention weight of each frame in the window. When the maximum average attention weight of a window is less than the set threshold from the end frame of the audio, the model is controlled to stop decoding; using a word boundary detection module to detect the decoded content, and determine whether there are truncated transcribed words when decoding is stopped, so as to ensure the correct output of the transcription result. At the same time, the threshold is dynamically adjusted according to the boundary detection situation to achieve accurate and efficient real-time speech recognition.

Description

Technical Field

[0001] The invention belongs to the field of automatic speech recognition and relates to a non-streaming model streaming speech recognition method based on attention and boundary detection. Background Art

[0002] Automatic speech recognition technology plays an important role in scientific research and daily life by transcribing audio signals into text content. Streaming speech recognition can output transcribed text in real time during the audio stream input process. The Transformer-based speech recognition model converts the audio signal into a vector representation at one time, and then performs autoregressive decoding through the decoder module. This type of model performs well in non-streaming speech recognition tasks, but does not work well when used for streaming speech recognition and consumes more computing resources.

[0003] Some traditional methods use local transcription strategies to address the challenges of non-streaming models in streaming recognition. Usually, the audio is divided into small segments, and the model transcribes these segments separately, and finds the longest common prefix in the transcribed content of multiple segments. Although this type of method does not require modifying the parameters and structure of the model and can achieve relatively effective streaming speech recognition, it is weak in controlling transcription latency and uncertainty, and also brings a large computational overhead.

[0004] In the field of speech translation, streaming translation of non-streaming models is also a key challenge. Some research methods focus on the attention mechanism in the Transformer model to control the decoding timing and determine whether the maximum attention weight of the audio frame is close to the end of the audio segment, or the attention weight of the last frame reaches a certain threshold. This approach can also be applied to the field of speech recognition. Compared with the local transcription strategy, the decoding strategy based on cross attention can reduce the latency of transcription.

[0005] However, the method of controlling decoding based on the attention mechanism often focuses on the position of the maximum attention weight, but this approach has certain accuracy issues. Focusing only on the maximum attention weight may ignore background noise and cause the model to fall into a locally unstable area. Therefore, it should be extended to a wider range of weight values ​​for consideration. In addition, the reliability of the transcription results is not fully guaranteed. Multiple audio clips may cut a complete word, and a word boundary detection mechanism is needed to prevent this from happening. Boundary detection can also help correct the timing of decoding if premature stopping occurs during the decoding process. Some methods detect the boundaries of the transcribed content through linear layers or recurrent neural networks (RNNs), but these methods perform poorly and are inefficient when processing complex audio. At the same time, the unreliability of the transcribed content will also affect the effect of boundary detection. Summary of the invention

[0006] The purpose of the present invention is to solve the problems existing in the prior art and provide a non-streaming model streaming speech recognition method based on attention and boundary detection, including: segmenting audio clips, extracting features from the audio clips, and generating logarithmic Mel spectrograms; transcribing the audio clips using a non-streaming speech model Whisper, extracting the cross attention of each frame of audio in the model, and obtaining corresponding attention weights; setting the length to The sliding window is used to calculate the average attention weight of each frame in the window. When the maximum average attention weight of a window is less than the set threshold from the end frame of the audio When the word boundary detection module is used, the decoding content is detected to determine whether there are truncated transcription words when the decoding is stopped, so as to ensure the correct output of the transcription results. At the same time, the threshold is dynamically adjusted according to the boundary detection situation. , in order to achieve accurate and efficient real-time speech recognition.

[0007] The technical solution adopted by the present invention is as follows: a non-streaming model streaming speech recognition method based on attention and boundary detection, comprising the following steps:

[0008] Step 1, cutting a received real-time audio stream into multiple audio segments;

[0009] Step 2, converting the audio segment segments segmented in step 1 into logarithmic Mel spectrograms;

[0010] Step 3: Input the logarithmic Mel-spectrogram into the Whisper model to obtain the transcribed text and calculate the sum of the attention weights in the Whisper model;

[0011] The Whisper model includes a convolutional layer, a positional encoding, an Encoder module and a Decoder module, wherein the Encoder module includes a plurality of encoder blocks consisting of a self-attention mechanism and a multi-layer perceptron, and the Decoder module includes a plurality of decoder blocks consisting of a self-attention mechanism, a multi-layer perceptron and a cross-attention mechanism;

[0012] Step 4, starting from the start frame of the audio clip, set the length to The sliding window of length is In the audio clip, when the end of the window corresponding to the largest average attention weight in the sliding window is close to the end of the audio clip, and its distance to the end is less than the threshold When , the control model stops decoding;

[0013] Step 5: When decoding stops, the boundary detection model is used to detect whether the stop position is at the boundary position of the word;

[0014] Step 6, based on the detection of the word boundary in step 5, calculate the difference between the stop decoding position of the transcribed text in step 3 and the word boundary in step 5 , dynamically adjust the threshold in step 4 by adjusting the function value; when the audio frame position when the Whisper model stops decoding just reaches or exceeds the word boundary, it indicates that truncation has occurred. If truncation has occurred, the audio segment will not be transcribed directly to text, but will be transcribed together by caching and waiting for the next audio segment; if truncation has not occurred, go to step 7;

[0015] In step 7, the Decoder module of the Whisper model outputs the transcription of the audio clip.

[0016] Furthermore, in step 1, the received real-time audio stream is sampled at a specific sampling rate. Sampling is performed and then the size is The window is based on the step size Cut out multiple audio clips and capture the The calculation formula for the number of fragments is as follows:

[0017]

[0018] in, is the original audio signal sequence, The intercepted In the fragment audio sample values, where .

[0019] Furthermore, the specific implementation of step 2 is as follows;

[0020] For the time in the segment divided in step (1) and audio clip frequency , convert the audio signal into a spectrogram , and then the audio frequency Convert to Mel frequency , spectrum diagram Convert to Mel-spectrogram , and finally the Mel spectrum Convert to log mel spectrogram .

[0021] Furthermore, the processing of the Whisper model is as follows:

[0022] Step 3.1, the logarithmic Mel-spectrogram of the audio clip obtained in step 2 Downsampling is achieved through two convolutional layers to extract local time rate information, and GELU is used as the activation function to introduce nonlinearity;

[0023] Step 3.2, after the convolutional layer extracts features in step 3.1, add sinusoidal position encoding to enable the model to recognize inputs of different time steps;

[0024] Step 3.3, after being processed by the above steps, input into the Encoder module to obtain the hidden state representation of the audio feature;

[0025] Step 3.4, the Decoder module receives the hidden state representation of the audio features extracted in the Encoder module and obtains the transcribed text;

[0026] Step 3.5: At a certain time t in the audio clip, for the cross attention weight S in the decoder block i , calculate the sum of the cross attention weights of all decoder blocks, that is, calculate the multi-head cross attention set of the Decoder module The sum of the attention weights .

[0027] Furthermore, in a sliding window of length r Inside, for the moment To time The attention weights of the audio frames , calculate the average attention weight of the entire window ;

[0028] Average attention weight The calculation formula is as follows:

[0029] .

[0030] Furthermore, the stopping condition of the Whisper model in step 4 is as follows:

[0031] .

[0032] Furthermore, the specific implementation of step 5 is as follows:

[0033] Step 5.1: Input the logarithmic Mel spectrum in step 2 into the input layer of the boundary detection model. The tensor form of ;

[0034] Step 5.2, extract features from the tensor of the logarithmic Mel spectrum of the audio clip through the convolution layer Conv1, and then downsample it through the maximum pooling layer MP;

[0035] Step 5.3, the features after the maximum pooling layer enter the convolutional layers Conv2 and Conv3 for deeper feature extraction; then the features initially extracted by Conv1 are added to the output of Conv3 through the residual connection ADD1 to maintain the integrity of the feature information;

[0036] Step 5.4: pass the deep features obtained in step 5.3 to the convolutional layers Conv4 and Conv5 to continue extracting features, and perform residual connection ADD2 again. Then, enter the convolutional layers Conv6 and Conv7 in turn, and perform the third residual connection ADD3 to refine the feature extraction.

[0037] Step 5.5: The features extracted by the convolutional layer in the above steps are finally aggregated to the convolutional layer Conv8 to form a global feature representation. Then, the CBAM attention mechanism is introduced to highlight the significant features related to word boundaries in the speech signal.

[0038] In step 5.6, the average pooling layer AP is used to perform global dimensionality reduction, and the reduced features are passed to the fully connected layer to integrate the features. Finally, the output layer predicts the boundary detection position.

[0039] Further, according to the detection of the word boundary in step 5, the difference between the stop decoding position and the word boundary is calculated , by adjusting the function Calculate a new threshold ,Right now , adjust the function The calculation formula is as follows:

[0040]

[0041] Among them, the initial threshold and adjustable parameters Manual setting.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] The present invention controls the decoding process of the non-streaming model Whisper by introducing a decoding strategy based on cross-attention (by calculating the average attention weight in the sliding window), thereby realizing streaming recognition of the non-streaming speech model. At the same time, the word boundary detection mechanism based on the convolutional neural network and the attention mechanism ensures the integrity and accuracy of the transcribed content. Finally, according to the boundary detection results and the position where the decoding stops, the decoding threshold is dynamically adjusted by adjusting the function to further improve the accuracy of recognition. The invention can apply non-streaming speech models with higher recognition accuracy to the field of real-time speech recognition, significantly improving the accuracy and efficiency of streaming recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flow chart for implementing streaming speech recognition for the non-streaming speech model of the present invention;

[0045] Figure 2 Flow chart of the decoding strategy based on the cross-attention mechanism of the present invention;

[0046] Figure 3 This is a structural diagram of a word boundary detection model based on a convolutional neural network structure and an attention mechanism of the present invention;

[0047] Figure 4 This is the Chinese streaming speech recognition effect diagram of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0049] The embodiment of the present invention provides a non-streaming model streaming speech recognition method based on attention and boundary detection, such as Figure 1 As shown, the following steps are included:

[0050] (1) Split audio clips;

[0051] The received real-time audio stream is sampled at a specific rate Sampling is performed and then the size is The window is based on the step size , extract the audio sample value, and then cut out multiple audio segments. Above, intercept In the fragment Audio sample value The calculation formula is as follows:

[0052]

[0053] in .

[0054] (2) Feature extraction;

[0055] For the time in the segment divided in step (1) and audio clip frequency , convert the audio signal into a spectrogram . Then change the audio frequency Convert to Mel frequency , spectrum diagram Convert to Mel-spectrogram Finally, the Mel spectrum Convert to log mel spectrogram .

[0056] (3) Logarithmic Mel spectrum Input to the Whisper model, obtain the transcribed text, and calculate the multi-head cross attention weights in the Whisper model. The Whisper model includes convolutional layers, position encoding, Encoder modules and Decoder modules. The Decoder module includes a decoder block composed of multiple self-attention mechanisms, multi-layer perceptrons and cross attention mechanisms. Figure 2 , describe the network structure of the Whisper model, the process of processing spectrograms and implementing speech recognition;

[0057] (3.1) Logarithmic Mel-spectrogram of the audio clip obtained in step (2) , downsampling is achieved through two convolutional layers 2xConv1d to extract local time rate information. At the same time, GELU is used as the activation function to introduce nonlinearity.

[0058] (3.2) After the convolutional layer extracts features in step (3.1), sinusoidal position encoding is added to enable the model to recognize inputs of different time steps.

[0059] (3.3) After being processed by the above steps, the audio is input to the Encoder module. This module contains multiple encoder blocks, each of which is composed of a self-attention mechanism and a multi-layer perceptron to deeply represent the audio features.

[0060] (3.4) The Decoder module receives the hidden state representation of the audio features extracted by the Encoder module, and combines it with the Whisper model's own multi-task training format Tokens for prediction, and predicts the audio transcription text using an autoregressive method. The Decoder module consists of multiple decoder blocks, each of which consists of a self-attention mechanism, a multi-layer perceptron, and a cross-attention mechanism. Among them, the cross-attention mechanism combines the hidden state representation of the audio features extracted by the Encoder module.

[0061] (3.5) At a certain time t in the audio clip, for the cross attention weight S in the decoder block i , calculate the sum of the weights of the cross attention of all decoder blocks, that is, calculate the multi-head cross attention set in the Decoder module The sum of the attention weights .

[0062] (4) Starting from the start frame of the audio clip, set the length to The sliding window of length is In an audio clip of , when the end with the largest average attention weight in the sliding window is close to the end of the audio clip and its distance to the end is less than the threshold When , the control model stops decoding;

[0063] In a sliding window of length r Inside, for the moment To time The attention weights of the audio frames , calculate the average attention weight of the entire window The calculation formula is as follows:

[0064] ;

[0065] Length is In the audio clip, when the end of the window corresponding to the largest attention weight is close to the end of the audio clip and the distance from the end is less than the threshold When , the Whisper model stops decoding. The specific stopping conditions are as follows:

[0066] .

[0067] (5) When decoding stops, the boundary detection model detects whether the stop position is at the boundary of the word to ensure that the transcription segment is complete. Figure 3 , describes the process of the network structure of the boundary detection model processing the spectrum graph and realizing boundary detection;

[0068] (5.1) Input the logarithmic Mel spectrum in step (2) into the input layer of the boundary detection model The tensor form of .

[0069] (5.2) The tensor of the logarithmic Mel spectrum of the audio clip is extracted through the convolution layer Conv1, and then downsampled through the maximum pooling layer MP to reduce the dimension while retaining the significant part of the audio clip features.

[0070] (5.3) The features after the maximum pooling layer enter the convolutional layers Conv2 and Conv3 for deeper feature extraction, gradually capturing the complex features in the speech signal that may represent word boundaries. The features initially extracted by Conv1 are then added to the output of Conv3 through the residual connection ADD1 to maintain the integrity of the feature information.

[0071] (5.4) The deep features obtained in step (5.3) are passed to the convolutional layers Conv4 and Conv5 to continue extracting features, and the residual connection ADD2 is performed again. Next, the convolutional layers Conv6 and Conv7 are entered in turn, and the third residual connection ADD3 is performed to refine the feature extraction.

[0072] (5.5) The features extracted by the convolutional layer in the above steps are finally aggregated to Conv8 to form a global feature representation. The CBAM attention mechanism is then introduced to highlight the significant features related to word boundaries in the speech signal and reduce noise interference.

[0073] (5.6) Use the average pooling layer AP to perform global dimensionality reduction, and pass the reduced features to the fully connected layer to integrate the features. Finally, the output layer predicts the boundary detection position.

[0074] (6) Determine whether truncation has occurred during decoding based on the position of the word boundary detection obtained in step (5). Specifically, when the audio frame position when the Whisper model stops decoding just reaches or exceeds the word boundary position, it indicates that truncation has occurred. When the audio frame position has not reached the word boundary position during decoding, it indicates that truncation has occurred. If truncation occurs, the audio segment will not be directly transcribed into text, but will be transcribed into text by caching and waiting for the next audio segment.

[0075] Specifically, calculate the difference between the stop decoding position of the transcribed text in step (3) and the word boundary in step (5) , by adjusting the function Calculate a new threshold ,Right now For the manually set initial threshold and adjustable parameters , adjust the function The details are as follows:

[0076] .

[0077] (7) When decoding stops, if no truncation occurs, the decoder module of the Whisper model outputs the transcription of the audio segment. Figure 4 This is a streaming speech transcription effect diagram. The information printed in the effect diagram is divided into two parts. The first part is the current time of transcription, and the second part is the audio text transcribed by streaming recognition and the time information of the start and end of the content in the audio. The unit of the printed time information is milliseconds.

[0078] It should be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions executed on the computer or other programmable device for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0079] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A non-streaming model streaming speech recognition method based on attention and boundary detection, characterized in that: The steps include: Step 1, cutting a received real-time audio stream into multiple audio segments; Step 2, converting the audio segments segmented in step 1 into logarithmic Mel spectrograms; Step 3: Input the logarithmic Mel-spectrogram into the Whisper model to obtain the transcribed text and calculate the sum of the attention weights in the Whisper model; The Whisper model includes a convolutional layer, a positional encoding, an Encoder module and a Decoder module, wherein the Encoder module includes a plurality of encoder blocks consisting of a self-attention mechanism and a multi-layer perceptron, and the Decoder module includes a plurality of decoder blocks consisting of a self-attention mechanism, a multi-layer perceptron and a cross-attention mechanism; Step 4, starting from the start frame of the audio clip, set the length to The sliding window of length is In the audio clip, when the end of the window corresponding to the largest average attention weight in the sliding window is close to the end of the audio clip, and its distance to the end is less than the threshold When , the control model stops decoding; Step 5: When decoding stops, the boundary detection model is used to detect whether the stop position is at the boundary position of the word; Step 6, based on the detection of the word boundary in step 5, calculate the difference between the stop decoding position of the transcribed text in step 3 and the word boundary in step 5 , dynamically adjust the threshold in step 4 by adjusting the function value; when the audio frame position when the Whisper model stops decoding just reaches or exceeds the word boundary, it indicates that truncation has occurred. If truncation has occurred, the audio segment will not be transcribed directly to text, but will be transcribed together by caching and waiting for the next audio segment; if truncation has not occurred, go to step 7; In step 7, the Decoder module of the Whisper model outputs the transcription of the audio clip.

2. The non-streaming model streaming speech recognition method based on attention and boundary detection as claimed in claim 1, characterized in that: In step 1, the received real-time audio stream is sampled at a specific rate. Sampling is performed and then the size is The window is based on the step size Cut out multiple audio clips and capture the The calculation formula for the number of fragments is as follows: ； in, is the original audio signal sequence, The intercepted In the fragment audio sample values, where .

3. The non-streaming model streaming speech recognition method based on attention and boundary detection as claimed in claim 1, characterized in that: The specific implementation of step 2 is as follows; For the time in the segment divided in step (1) and audio clip frequency , convert the audio signal into a spectrogram , and then the audio frequency Convert to Mel frequency , spectrum diagram Convert to Mel-spectrogram , and finally the Mel spectrum Convert to log mel spectrogram .

4. The non-streaming model streaming speech recognition method based on attention and boundary detection as claimed in claim 1, characterized in that: The processing of the Whisper model is as follows: Step 3.1, the logarithmic Mel-spectrogram of the audio clip obtained in step 2 Downsampling is achieved through two convolutional layers to extract local time rate information, and GELU is used as the activation function to introduce nonlinearity; Step 3.2, after the convolutional layer extracts features in step 3.1, add sinusoidal position encoding to enable the model to recognize inputs of different time steps; Step 3.3, after being processed by the above steps, input into the Encoder module to obtain the hidden state representation of the audio feature; Step 3.4, the Decoder module receives the hidden state representation of the audio features extracted in the Encoder module and obtains the transcribed text; Step 3.5: At a certain time t in the audio clip, for the cross attention weight S in the decoder block i , calculate the sum of the cross attention weights of all decoder blocks, that is, calculate the multi-head cross attention set of the Decoder module The sum of the attention weights .

5. The non-streaming model streaming speech recognition method based on attention and boundary detection as claimed in claim 1, characterized in that: In a sliding window of length r Inside, for the moment To time The attention weights of the audio frames , calculate the average attention weight of the entire window ; Average attention weight The calculation formula is as follows: 。 6. The non-streaming model streaming speech recognition method based on attention and boundary detection as claimed in claim 1, characterized in that: The stopping conditions of the Whisper model in step 4 are as follows: 。 7. The non-streaming model streaming speech recognition method based on attention and boundary detection as claimed in claim 1, characterized in that: The specific implementation of step 5 is as follows: Step 5.1: Input the logarithmic Mel spectrum in step 2 into the input layer of the boundary detection model. The tensor form of ; Step 5.2, extract features from the tensor of the logarithmic Mel spectrum of the audio clip through the convolution layer Conv1, and then downsample it through the maximum pooling layer MP; Step 5.3, the features after the maximum pooling layer enter the convolutional layers Conv2 and Conv3 for deeper feature extraction; then the features initially extracted by Conv1 are added to the output of Conv3 through the residual connection ADD1 to maintain the integrity of the feature information; Step 5.4: pass the deep features obtained in step 5.3 to the convolutional layers Conv4 and Conv5 to continue extracting features, and perform residual connection ADD2 again. Then, enter the convolutional layers Conv6 and Conv7 in turn, and perform the third residual connection ADD3 to refine the feature extraction. Step 5.5: The features extracted by the convolutional layer in the above steps are finally aggregated to the convolutional layer Conv8 to form a global feature representation. Then, the CBAM attention mechanism is introduced to highlight the significant features related to word boundaries in the speech signal. In step 5.6, the average pooling layer AP is used to perform global dimensionality reduction, and the reduced features are passed to the fully connected layer to integrate the features. Finally, the output layer predicts the boundary detection position.

8. The non-streaming model streaming speech recognition method based on attention and boundary detection as claimed in claim 1, characterized in that: Based on the detection of the word boundary in step 5, calculate the difference between the stop decoding position and the word boundary , by adjusting the function Calculate a new threshold ,Right now , adjust the function The calculation formula is as follows: ； Among them, the initial threshold and adjustable parameters Manual setting.

Images