Streaming speech recognition method, device, electronic device and storage medium

Abstract

The present invention provides a streaming speech recognition method, apparatus, electronic device, and storage medium. The streaming speech recognition method includes: obtaining raw audio data and generating a raw feature sequence based on the raw audio data; inputting the raw feature sequence into an audio encoder to generate a target audio coding feature sequence, wherein the audio encoder includes a block self-attention mechanism encoding layer and a shifted block self-attention mechanism encoding layer; and inputting the target audio coding feature sequence into an audio decoder to generate speech recognition text. The raw audio data is encoded based on the block self-attention mechanism encoding layer of the audio encoder, and a shifted block self-attention mechanism encoding layer is introduced to implement cross-block calculation of self-attention scores, ensuring linear time complexity of the audio encoder. This method rapidly generates a target audio coding feature sequence with global context information, decodes the feature sequence, and generates speech recognition text, effectively improving the accuracy of streaming speech recognition.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a streaming speech recognition method, device, electronic device and storage medium. Background Art

[0002] With the development of artificial intelligence, end-to-end technology based on deep neural networks has propelled speech recognition into practical use. The current mainstream architecture is an end-to-end speech recognition system based on the Transformer codec framework. The Transformer model, due to its ability to better model global sequence information, has improved the performance of sequence-to-sequence prediction problems.

[0003] However, in related technologies, end-to-end speech recognition systems based on Transformers are difficult to directly apply to streaming speech recognition. The main reasons are: first, the Transformer requires all input information during decoding and cannot directly support streaming recognition; second, the computational complexity and memory usage of the Transformer's self-attention mechanism increase quadratically with the length of the input sequence.

[0004] In summary, the accuracy of streaming speech recognition using the Transformer model in related technologies is low. Therefore, improving the accuracy of streaming speech recognition is an important issue that needs to be addressed in the industry. Summary of the Invention

[0005] In order to solve the problems existing in the prior art, the present invention provides a streaming speech recognition method, device, electronic device and storage medium in an embodiment.

[0006] The present invention provides a streaming speech recognition method, comprising:

[0007] Acquire original audio data, and generate an original feature sequence based on the original audio data;

[0008] Inputting the original feature sequence into an audio encoder to generate a target audio coding feature sequence, wherein the audio encoder includes a block self-attention mechanism encoding layer and a shifted block self-attention mechanism encoding layer;

[0009] The target audio coding feature sequence is input into an audio decoder to generate speech recognition text.

[0010] Optionally, before generating an original feature sequence based on the original audio data, the method further includes:

[0011] Perform data expansion and data enhancement processing on the original audio data.

[0012] Optionally, generating an original feature sequence based on the original audio data includes:

[0013] Extracting a cepstral domain feature sequence of the original audio data;

[0014] Performing a downsampling operation on the cepstral domain feature sequence to generate a feature sequence after the downsampling operation;

[0015] The tail of the feature sequence after the downsampling operation is padded to generate an original feature sequence that meets the preset length.

[0016] Optionally, inputting the original feature sequence into an audio encoder to generate a target audio coding feature sequence includes:

[0017] Inputting the original feature sequence into the block self-attention mechanism encoding layer to generate a first audio encoding feature sequence; inputting the first audio encoding feature sequence into the shifted block self-attention mechanism encoding layer to generate a second audio encoding feature sequence;

[0018] The second audio coding feature sequence is input into the block self-attention mechanism coding layer for N loop iterations to obtain the target audio coding feature sequence output by the shifted block self-attention mechanism coding layer; N is a positive integer.

[0019] Optionally, inputting the original feature sequence into the block self-attention mechanism encoding layer to generate a first audio encoding feature sequence includes:

[0020] Evenly dividing the original feature sequence into blocks to obtain a plurality of first feature sequence blocks;

[0021] Calculate the first self-attention score of each first feature sequence block, and generate the first audio coding feature sequence based on the feature sequence in each first feature sequence block and the first self-attention score corresponding to each first feature sequence block.

[0022] Optionally, inputting the first audio coding feature sequence into the shift block self-attention mechanism encoding layer to generate a second audio coding feature sequence includes:

[0023] Re-blocking the first audio coding feature sequence by using a cyclic shift method to obtain a plurality of second feature sequence blocks;

[0024] Calculate the second self-attention score of each second feature sequence block, and generate the second audio coding feature sequence based on the feature sequence in each second feature sequence block and the second self-attention score corresponding to each second feature sequence block.

[0025] Optionally, generating the second audio coding feature sequence based on the feature sequence in each second feature sequence block and the second self-attention score corresponding to each second feature sequence block includes:

[0026] Identifying a timing constraint of a feature sequence between each of the second feature sequence blocks, and based on the timing constraint, using a preset masking mechanism to perform masking on the second self-attention score of each of the second feature sequence blocks;

[0027] Based on the masked second self-attention score, the second audio coding feature sequence is generated.

[0028] Optionally, the audio decoder includes a connected temporal classification decoding layer and a Transformer decoding layer;

[0029] The step of inputting the target audio coding feature sequence into an audio decoder to generate speech recognition text comprises:

[0030] Inputting the target audio coding feature sequence into the connection temporal classification decoding layer and the Transformer decoding layer respectively to obtain a first acoustic prediction score and a second acoustic prediction score;

[0031] The speech recognition text is generated based on a beam search algorithm according to the first acoustic prediction score and the second acoustic prediction score.

[0032] The present invention also provides a streaming speech recognition device, comprising:

[0033] An acquisition module, configured to acquire original audio data and generate an original feature sequence based on the original audio data;

[0034] A first generation module is configured to input the original feature sequence into an audio encoder to generate a target audio coding feature sequence, wherein the audio encoder includes a block self-attention mechanism encoding layer and a shifted block self-attention mechanism encoding layer;

[0035] The second generating module is used to input the target audio coding feature sequence into the audio decoder to generate speech recognition text.

[0036] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-described streaming speech recognition methods when executing the program.

[0037] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements any of the above-mentioned streaming speech recognition methods.

[0038] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned streaming speech recognition methods.

[0039] The streaming speech recognition method, device, electronic device, and storage medium provided by the present invention obtain raw audio data and generate an original feature sequence based on the raw audio data; input the original feature sequence into an audio encoder to generate a target audio coding feature sequence, wherein the audio encoder includes a block self-attention mechanism encoding layer and a shifted block self-attention mechanism encoding layer; and input the target audio coding feature sequence into an audio decoder to generate speech recognition text. The raw audio data is encoded based on the block self-attention mechanism encoding layer of the audio encoder. The shifted block self-attention encoding layer is introduced on this basis to achieve cross-block calculation of self-attention scores and ensure linear time complexity of the audio encoder. This allows for rapid generation of a target audio coding feature sequence with global context information, which is then decoded to generate speech recognition text, effectively improving the accuracy of streaming speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 This is one of the flow charts of the streaming speech recognition method provided by the present invention;

[0042] Figure 2 It is a structural diagram of the audio encoder provided by the present invention;

[0043] Figure 3 This is a schematic diagram of the calculation method of the shift block self-attention mechanism provided by the present invention;

[0044] Figure 4 Schematic diagram of the cyclic shift method provided by the present invention;

[0045] Figure 5 Schematic diagram of the masking method provided by the present invention;

[0046] Figure 6 This is the second flow chart of the streaming speech recognition method provided by the present invention;

[0047] Figure 7 It is a structural diagram of the streaming speech recognition device provided by the present invention;

[0048] Figure 8It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0050] With the development of artificial intelligence, end-to-end technologies based on deep neural networks have propelled speech recognition into practical use. Current mainstream architectures include: 1. End-to-end speech recognition systems based on Connectionist Temporal Classification (CTC); 2. End-to-end speech recognition systems based on Recurrent Neural Network Transducer (RNN-T); and 3. End-to-end speech recognition systems based on the Transformer codec framework. The Transformer model, due to its ability to better model global sequence information, has significantly improved the performance of sequence-to-sequence prediction problems, demonstrating enormous application potential.

[0051] However, end-to-end Transformer-based speech recognition systems are difficult to directly apply to streaming speech recognition. The main reasons are: first, the Transformer requires the entire input information during decoding, which cannot directly support streaming recognition; second, the computational complexity and memory usage of the Transformer's self-attention mechanism increase quadratically with the length of the input sequence.

[0052] Currently, there are three main solutions to the above problems: First, a method based on limited time length calculates attention between historical input information and future input information of limited length. Although this method can model global historical information, it will introduce a large delay because the receptive field grows linearly with the number of Transformer layers; the second is a method based on the chunking mechanism, which chunks the speech in the time series and calculates self-attention within the chunk. Although this method has linear time complexity under the condition of fixed chunk length, the recognition accuracy of this method often drops sharply because it ignores the relationship between different chunks; the third is a method based on the memory mechanism, which introduces a memory module to encode historical information and improves recognition performance by combining it with the chunking mechanism. However, since this method introduces a time-dependent memory mechanism, the parallel training feature of the Transformer is destroyed and usually requires longer training time.

[0053] It can be seen from this that there is currently no lack of Transformer-based end-to-end streaming speech recognition methods that have global historical sequence modeling, linear time complexity, and parallel training.

[0054] In view of the above problems, in order to improve the accuracy of streaming speech recognition, the present invention provides a streaming speech recognition method, thereby improving the accuracy of streaming speech recognition.

[0055] The following combination Figures 1-6 The streaming speech recognition method provided by the present invention is described in detail.

[0056] See also Figure 1 , Figure 1 This is one of the flow charts of the streaming speech recognition method provided by the present invention, which specifically includes steps 101 to 103.

[0057] Step 101: Acquire original audio data, and generate an original feature sequence based on the original audio data.

[0058] Specifically, in this embodiment, the original audio data refers to the audio data to be identified, and the format of the audio data can be various, such as WAV format, AIFF format, FLAC format, ALAC format, etc. The present invention does not limit the format of the original audio data.

[0059] After the original audio data is acquired, it is necessary to generate an original feature sequence based on the original audio data, wherein the original feature sequence is a feature sequence including original audio elements.

[0060] Optionally, in order to improve the accuracy of streaming speech recognition, after acquiring the original audio data and before generating the original feature sequence based on the original audio data, it is necessary to perform data expansion and data enhancement processing on the original audio data.

[0061] In practical applications, on the one hand, data expansion of original audio data can be achieved in a variety of ways. Specifically, the original audio data can be processed at a variable rate, for example, the playback rate of the original audio data can be accelerated or slowed down; the pitch of the original audio data can also be processed, for example, the pitch of the original audio data can be raised or lowered, etc.

[0062] Through the above method, the original audio data can be effectively expanded.

[0063] On the other hand, in addition to data expansion of the original audio data, the original audio data also needs to be enhanced. Data enhancement of the original audio data can be achieved in many ways.

[0064] Specifically, the SpecAugment algorithm can be used to perform data enhancement on the original audio data, or the original audio data can be subjected to audio noise processing for data enhancement.

[0065] Among them, the SpecAugment algorithm is a data enhancement algorithm applied to the log Mel spectrum level. That is, the original audio data is converted into a Mel spectrum, and then the Mel spectrum is used as the image data of a channel to perform translation data enhancement on the image. Several consecutive rows on the image are randomly masked to enhance the time dimension, and several consecutive columns on the image are randomly masked to enhance the frequency dimension, thereby achieving enhancement of the time dimension and frequency dimension.

[0066] In the above embodiment, data expansion and data enhancement processing are performed on the original audio data, thereby improving the robustness of the original audio data to noise data.

[0067] Optionally, in a possible implementation of the embodiment of the present invention, generating an original feature sequence based on the original audio data can be specifically implemented in the following manner:

[0068] Extracting a cepstral domain feature sequence of the original audio data;

[0069] Performing a downsampling operation on the cepstral domain feature sequence to generate a feature sequence after the downsampling operation;

[0070] The tail of the feature sequence after the downsampling operation is padded to generate an original feature sequence that meets the preset length.

[0071] In this embodiment, after obtaining the original audio data, it is necessary to extract the cepstral domain features of the original audio data, and then perform a downsampling operation on the cepstral domain features to generate a feature sequence after the downsampling operation. Finally, the tail of the feature sequence after the downsampling operation is padded to obtain the original feature sequence that meets the preset feature sequence length.

[0072] Specifically, the cepstral domain feature sequence is used to extract speech features of the original audio data, and the cepstral domain feature sequence is obtained by performing inverse Fourier transform on the logarithm of the power spectrum of the original audio data.

[0073] For example, the cepstral domain feature sequence is used to extract the vocal tract feature information (such as envelope features) of the original audio data, and use it as a feature parameter to describe the phonology and apply it to speech recognition; for another example, the cepstral domain feature sequence is used to extract the sound source information (such as fundamental pitch features) of the original audio data, and use it as an auxiliary parameter to describe the phonology and apply it to speech recognition.

[0074] Generally, in practical applications, when extracting the cepstral domain feature sequence of original audio data, an 80-dimensional Fbank feature sequence can be extracted.

[0075] After extracting the cepstral domain feature sequence of the original audio data, a downsampling operation is performed on the cepstral domain feature sequence.

[0076] Generally, in practical applications, when performing a downsampling operation on a cepstral domain feature sequence, a two-layer convolutional neural network (CNN) can be used to downsample the cepstral domain feature sequence by a factor of 4 to generate a feature sequence after the downsampling operation.

[0077] In this embodiment, the downsampling operation is performed on the cepstral domain feature sequence in order to reduce the sampling frequency of the cepstral domain feature sequence, thereby reducing the operation time and further improving the efficiency of speech recognition.

[0078] After generating the downsampled feature sequence, the tail of the downsampled feature sequence needs to be padded to generate an original feature sequence of a preset length. This ensures that the downsampled feature sequences are of equal length, reducing the complexity of speech recognition operations and improving speech recognition efficiency. The preset length is the preset feature sequence length, which is determined based on the length of the feature sequence after the downsampled batch.

[0079] Specifically, since the feature sequences after the downsampling operation in batch processing are of unequal length in streaming speech, it is necessary to pad the tails of the feature sequences after the downsampling operation in the same batch to the same length, so that the length of each feature sequence after the downsampling operation is slightly longer than the length of the longest feature sequence in the batch, and the feature sequence length is divisible by the chunk size in the audio encoder. It should be noted that the relevant embodiments of the audio encoder will be specifically described in step 102 and will not be repeated here.

[0080] In the above embodiment, by extracting the cepstral domain feature sequence of the original audio data, the vocal tract feature information of the original audio data can be extracted; the cepstral domain features are downsampled to generate a feature sequence after the downsampling operation, reducing the frequency of sampling the cepstral domain feature sequence, thereby reducing the operation time of speech recognition, and thus improving the efficiency of speech recognition; the tail of the feature sequence after the downsampling operation is padded to generate an original feature sequence that meets the preset length, so that the feature sequences after each downsampling operation are of equal length, reducing the complexity of the speech recognition operation, and thus improving the efficiency of speech recognition.

[0081] Step 102: Input the original feature sequence into an audio encoder to generate a target audio coding feature sequence, wherein the audio encoder includes a block self-attention mechanism coding layer and a shifted block self-attention mechanism coding layer.

[0082] Specifically, in this embodiment, after the original audio data is acquired and an original feature sequence is generated based on the original audio data, the original feature sequence needs to be input into an audio encoder to generate an audio coding feature sequence.

[0083] Among them, the audio encoder includes a block self-attention mechanism encoding layer and a shift block self-attention mechanism encoding layer, which is a network structure based on the shift block strategy.

[0084] See also Figure 2 , Figure 2 Schematic diagram of the structure of the audio encoder provided by the present invention. Figure 2 In

[15] , the network structure based on the shifted block strategy consists of two parts: the chunk-multihead self-attention mechanism encoding layer (C-MSA) and the shifted chunk-multihead self-attention mechanism encoding layer (SC-MSA).

[0085] Among them, Figure 2In the figure, C-MSA represents the block self-attention mechanism encoding unit; SC-MSA represents the shifted block self-attention mechanism encoding unit; FFN represents the forward propagation unit; LN represents the layer normalization unit; represents the output of the block self-attention mechanism encoding layer; z l Represents the output of the shift block self-attention mechanism encoding layer.

[0086] N single-layer network structures based on the shift block strategy are superimposed to form an audio encoder, where N is a positive integer.

[0087] It should be noted that the difference between the block self-attention mechanism encoding layer and the traditional self-attention mechanism layer is that the block self-attention mechanism encoding layer uses evenly divided blocks for self-attention calculation;

[0088] The difference between the shift block self-attention mechanism encoding layer and the traditional self-attention mechanism layer is that the shift block self-attention mechanism encoding layer performs self-attention calculations within the blocks divided by the shift blocks.

[0089] Optionally, in a possible implementation of the embodiment of the present invention, the original feature sequence is input into an audio encoder to generate a target audio coding feature sequence, which can be specifically implemented in the following manner:

[0090] Inputting the original feature sequence into the block self-attention mechanism encoding layer to generate a first audio encoding feature sequence; inputting the first audio encoding feature sequence into the shifted block self-attention mechanism encoding layer to generate a second audio encoding feature sequence;

[0091] The second audio coding feature sequence is input into the block self-attention mechanism coding layer for N loop iterations to obtain the target audio coding feature sequence output by the shifted block self-attention mechanism coding layer; N is a positive integer.

[0092] Specifically, in this embodiment, the audio encoder is generated by superimposing N block self-attention mechanism encoding layers and shifted block self-attention mechanism encoding layers.

[0093] Therefore, when the original feature sequence is input into the audio encoder for encoding, the original feature sequence needs to be first input into the block self-attention encoding layer for encoding, thereby generating a first audio encoding feature sequence, wherein the first audio encoding feature sequence is a block self-attention encoding feature sequence;

[0094] Then, the first audio coding feature sequence is input into the shift block self-attention mechanism encoding layer to generate a second audio coding feature sequence, wherein the second audio coding feature sequence is a shift block self-attention coding feature sequence;

[0095] Then the second audio coding feature sequence is iteratively input into the block self-attention mechanism encoding layer. After N loop iterations, the target audio coding feature sequence output by the shifted block self-attention mechanism encoding layer can be obtained.

[0096] That is to say, when the original feature sequence is input into the audio encoder for encoding, the original feature sequence is first input into the block self-attention mechanism encoding layer of the first layer of the audio encoder, and then the output of the first layer of block self-attention mechanism encoding layer is used as the input of the first layer of shifted block self-attention mechanism encoding layer;

[0097] Then, the output of the first-layer shifted block self-attention mechanism encoding layer is used as the input of the second-layer block self-attention mechanism encoding layer, and the output of the second-layer block self-attention mechanism encoding layer is used as the input of the second-layer shifted block self-attention mechanism encoding layer. This is iterated N times in sequence until the output of the N-1th layer shifted block self-attention mechanism encoding layer is used as the input of the N-layer block self-attention mechanism encoding layer, and the output of the N-layer block self-attention mechanism encoding layer is used as the input of the N-layer shifted block self-attention mechanism encoding layer, so that the target audio coding feature sequence output by the N-layer shifted block self-attention mechanism encoding layer can be obtained.

[0098] Through the above implementation, the original audio data is encoded based on the audio encoder's block self-attention mechanism encoding layer. On this basis, a shifted block self-attention encoding layer is introduced, ensuring the linear time complexity of the audio encoder and enabling cross-block calculation of self-attention scores. This allows for the rapid generation of a target audio encoding feature sequence with global context information. Furthermore, this target audio encoding feature sequence is decoded and speech recognition text is generated, effectively improving the accuracy of streaming speech recognition.

[0099] Optionally, in a possible implementation of the embodiment of the present application, the original feature sequence is input into the block self-attention mechanism encoding layer to generate a first audio encoding feature sequence, which can be specifically implemented in the following manner:

[0100] Evenly dividing the original feature sequence into blocks to obtain a plurality of first feature sequence blocks;

[0101] Calculate the first self-attention score of each first feature sequence block, and generate the first audio coding feature sequence based on the feature sequence in each first feature sequence block and the first self-attention score corresponding to each first feature sequence block.

[0102] Specifically, in this embodiment, the original feature sequence is first input into the block self-attention mechanism encoding layer. Then, the original feature sequence is evenly divided into blocks using fixed-size blocks in the block self-attention mechanism encoding layer, thereby obtaining multiple feature sequence blocks as the first feature sequence block. In practical applications, the block size can be set to 16.

[0103] Then, the first self-attention score in each first feature sequence block is calculated, and based on the first self-attention score of each first feature sequence block, a first audio coding feature sequence is generated, wherein the first self-attention score refers to the self-attention score of the feature sequence within each first feature sequence block calculated based on the self-attention mechanism.

[0104] Specifically, the first self-attention score in each first feature sequence block can be calculated by the following formulas (1)-(2):

[0105]

[0106]

[0107] Among them, C-MSA represents the block self-attention mechanism encoding unit in the block self-attention mechanism encoding layer; FFN represents the forward propagation unit; LN represents the layer normalization unit; represents the output of the block self-attention mechanism encoding layer; z l Represents the output of the shift block self-attention mechanism encoding layer.

[0108] After calculating the first self-attention score in each first feature sequence block using formulas (1)-(2), the first self-attention score in each first feature sequence block is multiplied by the feature sequence in each first feature sequence block to generate a first audio coding feature sequence.

[0109] In the above embodiment, by inputting the original feature sequence into the block self-attention mechanism encoding layer, a first audio coding feature sequence with local context information can be generated.

[0110] Optionally, in a possible implementation of the embodiment of the present invention, the first audio coding feature sequence is input into the shift block self-attention mechanism encoding layer to generate a second audio coding feature sequence, which can be specifically implemented in the following manner:

[0111] Re-blocking the first audio coding feature sequence by using a cyclic shift method to obtain a plurality of second feature sequence blocks;

[0112] Calculate the second self-attention score of each second feature sequence block, and generate the second audio coding feature sequence based on the feature sequence in each second feature sequence block and the second self-attention score corresponding to each second feature sequence block.

[0113] Specifically, in this embodiment, after the first audio coding feature sequence is generated, the fixed-size blocks in the block self-attention mechanism encoding layer will still divide the first audio coding feature sequence into multiple feature sequence blocks.

[0114] At this time, it is necessary to use a cyclic shift method to re-uniformly block the first audio coding feature sequence divided into multiple feature sequence blocks to obtain a second feature sequence block, wherein the second self-attention score refers to the self-attention score of the feature sequence within each second feature sequence block calculated based on the self-attention mechanism.

[0115] Then, a second self-attention score in each second feature sequence block is calculated, and a second audio coding feature sequence is generated based on the second self-attention score of each second feature sequence block.

[0116] Specifically, the second self-attention score in each second feature sequence block can be calculated by the following formulas (3)-(4):

[0117]

[0118]

[0119] Among them, SC-MSA represents the shift block self-attention mechanism encoding unit in the shift block self-attention mechanism encoding layer; FFN represents the forward propagation unit; LN represents the layer normalization unit; represents the output of the block self-attention mechanism encoding layer; z l Represents the output of the shift block self-attention mechanism encoding layer.

[0120] After calculating the second self-attention score in each second feature sequence block using formulas (3)-(4), the second self-attention score in each second feature sequence block is multiplied by the feature sequence in each second feature sequence block to generate a second audio coding feature sequence.

[0121] Figure 3 This is a schematic diagram of the calculation method of the shift block self-attention mechanism provided by the present invention. Figure 3 As shown, in the Lth layer, the block self-attention mechanism encoding layer is used to evenly block the original feature sequence to obtain a first audio coding feature sequence and multiple first feature sequence blocks corresponding to the first audio coding feature sequence, and the first self-attention score is calculated in each first feature sequence block;

[0122] In the L+1 layer, the shift block self-attention mechanism encoding layer is used to re-block the multiple first feature sequence blocks corresponding to the first audio coding feature sequence to obtain the second audio coding feature sequence and the multiple second feature sequence blocks corresponding to the second audio coding feature sequence, and the second self-attention score is calculated in each second feature sequence block.

[0123] Since the second feature sequence block spans the boundary of the first feature sequence block in the L layer, it can provide cross-block context information, and through multi-layer iterative operations, a target audio coding feature sequence with global context information can be obtained.

[0124] Specifically, re-blocking the first audio coding feature sequence by using a cyclic shift method can be achieved by the following methods:

[0125] First, the feature sequence of each feature sequence block in the first audio coding feature sequence is reversely moved from the tail to the head, and the number of moving bits is 1 / 2 of the fixed size block length, so as to realize re-block processing of the first audio coding feature sequence.

[0126] The specific cyclic shift process is as follows Figure 4 As shown, Figure 4 This is a schematic diagram of the cyclic shift method provided by the present invention. Figure 4 In the original block partition (i.e., the multiple first feature sequence blocks mentioned above), A, B, C, and D are the feature sequences of the first feature sequence block in the multiple first feature sequence blocks; E, F, G, and H are the feature sequences of the second feature sequence block in the multiple first feature sequence blocks; I, J, K, and L are the feature sequences of the third feature sequence block in the multiple first feature sequence blocks; M, PAD1, PAD2, and PAD3 are the feature sequences of the fourth feature sequence block in the multiple first feature sequence blocks, wherein PAD1, PAD2, and PAD3 are the feature sequences for filling the tail of the first audio coding feature sequence.

[0127] Then, the first 1 / 2 feature sequence of the first feature sequence block in the multiple first feature sequence blocks is moved to the tail of the fourth feature sequence block, and the last 1 / 2 feature sequence of the first feature sequence block is shifted left to the original first 1 / 2 feature sequence. The feature sequences in all subsequent feature sequence blocks are deduced in this way, and finally the shifted block partitions (i.e., the multiple second feature sequence blocks mentioned above) are generated.

[0128] Specifically, C, D, E, and F are the feature sequences of the first feature sequence block in the multiple second feature sequence blocks; G, H, I, and J are the feature sequences of the second feature sequence block in the multiple second feature sequence blocks; K, L, M, and PAD1 are the feature sequences of the third feature sequence block in the multiple second feature sequence blocks; and PAD2, PAD3, A, and B are the feature sequences of the fourth feature sequence block in the multiple second feature sequence blocks.

[0129] The advantage of performing cyclic shifting by the above method is that the number of feature sequence blocks after the shift is consistent with the number of feature sequence blocks before the shift, which facilitates batch parallel calculation of the original feature sequence, thereby improving the efficiency of generating the target audio coding feature sequence.

[0130] Optionally, in a possible implementation of the embodiment of the present invention, generating the second audio coding feature sequence based on the second self-attention score can be specifically implemented in the following manner:

[0131] Identifying a timing constraint of a feature sequence between each of the second feature sequence blocks, and based on the timing constraint, using a preset masking mechanism to perform masking on the second self-attention score of each of the second feature sequence blocks;

[0132] Based on the masked second self-attention score, the second audio coding feature sequence is generated.

[0133] Specifically, in this embodiment, since the original feature sequence input to the audio encoder is generated from streaming audio data, the streaming speech has a strong logical relationship in terms of time sequence. If the second self-attention score is directly calculated within multiple blocks of the second feature sequence after cyclic shift, the logical relationship in terms of time sequence of the streaming speech will be destroyed.

[0134] For example, according to the logical relationship of streaming voice in time sequence, Figure 4 Among the multiple second feature sequence blocks after the shift, the first feature sequence block contains feature sequences C, D, E, and F, among which C and D cannot participate in the second self-attention calculation of E and F.

[0135] The reason is that E and F are future feature sequences relative to C and D, whereas E and F can participate in the second self-attention calculation of C and D. This is because C and D belong to historical feature sequences relative to E and F.

[0136] Therefore, in this embodiment, when generating the second audio coding feature sequence based on the second self-attention score, it is necessary to first identify the timing constraints of the feature sequences between each second feature sequence block, use a preset mask mechanism to mask the self-attention scores of future feature sequences in each second feature sequence block, and then generate the second audio coding feature sequence based on the masked second self-attention scores.

[0137] The specific process of using the mask mechanism for mask processing is as follows Figure 5 As shown, Figure 5 This is a schematic diagram of the mask method provided by the present invention. Figure 5 In the figure, the horizontal coordinate is the input of the shift block self-attention mechanism encoding layer, and the vertical coordinate is the output of the shift block self-attention mechanism encoding layer. In several second feature sequence blocks, the area with a black background is the second self-attention score of the historical feature sequence of a second feature sequence block, and the area with a white background is the second self-attention score of the future feature sequence of a second feature sequence block.

[0138] When masking, it is necessary to mask the future feature sequence of the area with a white background, and then use the masked second self-attention score to generate the second audio coding feature sequence.

[0139] Through the above method, a preset mask mechanism is used to mask the second self-attention score of each second feature sequence block; based on the masked second self-attention score, a second audio coding feature sequence is generated, which can ensure the logical relationship of the streaming speech in time sequence, thereby improving the accuracy of streaming speech recognition.

[0140] Step 103: Input the target audio coding feature sequence into an audio decoder to generate a speech recognition result.

[0141] In this embodiment, after the original feature sequence is input into the audio encoder to generate the target audio coding feature sequence, the target audio coding feature sequence needs to be input into the audio decoder to generate a speech recognition result, where the speech recognition result is the recognition text corresponding to the original audio data.

[0142] Specifically, the audio decoder includes a connected temporal classification decoding layer and a Transformer decoding layer;

[0143] The target audio coding feature sequence is input into the audio decoder to generate a speech recognition result, which can be specifically achieved by the following methods:

[0144] Inputting the target audio coding feature sequence into the connection temporal classification decoding layer and the Transformer decoding layer respectively to obtain a first acoustic prediction score and a second acoustic prediction score;

[0145] The speech recognition text is generated based on a beam search algorithm according to the first acoustic prediction score and the second acoustic prediction score.

[0146] In this embodiment, the audio decoder includes a connectionist temporal classification decoding layer (i.e., a connectionist temporal classification model (CTC)) and a transformer decoding layer; the target audio coding features are input into the connectionist temporal classification decoding layer and the transformer decoding layer respectively, so as to obtain a first acoustic prediction score and a second acoustic prediction score.

[0147] Specifically, the CTC decoding layer consists of a linear layer and a log softmax layer. The target audio encoding features are input into the CTC decoding layer. During the test phase of training the CTC decoding layer, a CTC acoustic prediction score, also known as the first acoustic prediction score, is obtained. The first acoustic prediction score refers to the acoustic prediction score of each word in the multiple recognized texts corresponding to the original audio data output by the CTC decoding layer.

[0148] At the same time, the target audio coding features are input into the Transformer decoding layer, and the network parameters of the Transformer decoding layer are trained. During the test phase of training the Transformer decoding layer, the acoustic prediction score of the Transformer decoding is obtained, that is, the second acoustic prediction score, where the second acoustic prediction score refers to the acoustic prediction score of each word in the multiple recognized texts corresponding to the original audio data output by the Transformer decoding layer.

[0149] After obtaining the first acoustic prediction score and the second acoustic prediction score, based on the beam search algorithm, a preset number of texts with the highest prediction scores are selected from the first acoustic prediction score and the second acoustic prediction score as the speech recognition texts corresponding to the final original speech data.

[0150] In the above embodiment, the first acoustic prediction score and the second acoustic prediction score are obtained by connecting the temporal classification decoding layer and the Transformer decoding layer, and a beam search algorithm is used to select a preset number of texts with the highest prediction scores from the first acoustic prediction score and the second acoustic prediction score as the speech recognition texts corresponding to the final original speech data, which can effectively improve the accuracy of speech recognition.

[0151] The streaming speech recognition method provided by the present invention obtains raw audio data and generates an original feature sequence based on the raw audio data; inputs the original feature sequence into an audio encoder to generate a target audio coding feature sequence, wherein the audio encoder includes a block self-attention mechanism encoding layer and a shifted block self-attention mechanism encoding layer; and inputs the target audio coding feature sequence into an audio decoder to generate speech recognition text. The raw audio data is encoded based on the block self-attention mechanism encoding layer of the audio encoder. On this basis, the shifted block self-attention encoding layer is introduced to implement cross-block calculation of self-attention scores while ensuring linear time complexity of the audio encoder. This method rapidly generates a target audio coding feature sequence with global context information, decodes the feature sequence, and generates speech recognition text, thereby effectively improving the accuracy of streaming speech recognition.

[0152] See also Figure 6 , Figure 6 This is the second flow chart of the streaming speech recognition method provided by the present invention, which specifically includes steps 601 to 609:

[0153] Step 601: Obtain original audio data.

[0154] Step 602: Perform data expansion and data enhancement processing on the original audio data to obtain the original audio data after data expansion and data enhancement.

[0155] Step 603: extract the cepstral domain feature sequence of the original audio data after data expansion and data enhancement.

[0156] Step 604: downsample the cepstral domain feature sequence to generate a feature sequence after the downsampling operation.

[0157] Step 605: pad the tail of the feature sequence after the downsampling operation to generate an original feature sequence that meets the preset length.

[0158] Step 606: Evenly divide the original feature sequence into blocks to obtain multiple first feature sequence blocks; calculate the first self-attention score of each first feature sequence block, and generate a first audio coding feature sequence based on the feature sequence in each first feature sequence block and the first self-attention score corresponding to each first feature sequence block.

[0159] Step 607: Re-block the first audio coding feature sequence using a cyclic shift method to obtain multiple second feature sequence blocks; identify the timing constraints of the feature sequences between each second feature sequence block, and based on the timing constraints, use a preset masking mechanism to mask the second self-attention score of each second feature sequence block; generate a second audio coding feature sequence based on the masked second self-attention score.

[0160] Step 608: Input the second audio coding feature sequence into the block self-attention mechanism coding layer for N loop iterations to obtain the target audio coding feature sequence output by the shifted block self-attention mechanism coding layer; N is a positive integer.

[0161] Step 609: Input the target audio coding feature sequence into the audio decoder to generate speech recognition text.

[0162] The streaming speech recognition method provided by the present invention encodes the original audio data based on the block self-attention mechanism encoding layer of the audio encoder, and introduces a shift block self-attention encoding layer on this basis, and re-blocks the first audio coding feature sequence using a cyclic shift method to obtain multiple second feature sequence blocks; realizes cross-block calculation of self-attention scores, and can ensure the linear time complexity of the audio encoder, thereby significantly enhancing the modeling capability of global context information; at the same time, identifies the timing constraints of the feature sequences between each second feature sequence block, and based on the timing constraints, adopts a preset mask mechanism to mask the second self-attention score of each second feature sequence block, which can ensure the logical relationship of the streaming speech in time, thereby quickly generating a target audio coding feature sequence that has the linear time complexity of streaming speech, global context information and a temporal logical relationship.

[0163] Furthermore, the feature sequence is decoded and speech recognition text is generated, thereby improving the accuracy of streaming speech recognition.

[0164] The following combination Figure 7 The streaming speech recognition device provided by the present invention is described below. The streaming speech recognition device described below and the streaming speech recognition method described above can be referred to in correspondence with each other. Figure 7 7 is a structural diagram of a streaming speech recognition device 700 provided by the present invention.

[0165] An acquisition module 701 is configured to acquire original audio data and generate an original feature sequence based on the original audio data;

[0166] A first generating module 702 is configured to input the original feature sequence into an audio encoder to generate a target audio coding feature sequence, wherein the audio encoder includes a block self-attention mechanism encoding layer and a shifted block self-attention mechanism encoding layer;

[0167] The second generating module 703 is configured to input the target audio coding feature sequence into the audio decoder to generate speech recognition text.

[0168] The streaming speech recognition device provided by the present invention obtains raw audio data and generates an original feature sequence based on the raw audio data; inputs the original feature sequence into an audio encoder to generate a target audio coding feature sequence, wherein the audio encoder includes a block self-attention mechanism encoding layer and a shifted block self-attention mechanism encoding layer; and inputs the target audio coding feature sequence into an audio decoder to generate speech recognition text. The raw audio data is encoded based on the block self-attention mechanism encoding layer of the audio encoder. The shifted block self-attention encoding layer is introduced on this basis to achieve cross-block calculation of self-attention scores while ensuring linear time complexity of the audio encoder. This allows for rapid generation of a target audio coding feature sequence with global context information, decoding the feature sequence to generate speech recognition text, and thereby improving the accuracy of streaming speech recognition.

[0169] Optionally, the acquisition module 701 is further configured to:

[0170] Perform data expansion and data enhancement on the original audio data.

[0171] Optionally, the acquisition module 701 is further configured to:

[0172] Extracting a cepstral domain feature sequence of the original audio data;

[0173] Performing a downsampling operation on the cepstral domain feature sequence to generate a feature sequence after the downsampling operation;

[0174] The tail of the feature sequence after the downsampling operation is padded to generate an original feature sequence that meets the preset length.

[0175] Optionally, the first generating module 702 is further configured to:

[0176] Input the original feature sequence into the block self-attention mechanism encoding layer to generate a first audio encoding feature sequence; input the first audio encoding feature sequence into the shifted block self-attention mechanism encoding layer to generate a second audio encoding feature sequence;

[0177] The second audio coding feature sequence is input into the block self-attention mechanism encoding layer for N loop iterations to obtain the target audio coding feature sequence output by the shifted block self-attention mechanism encoding layer.

[0178] Optionally, the first generating module 702 is further configured to:

[0179] Evenly dividing the original feature sequence into blocks to obtain multiple first feature sequence blocks;

[0180] A first self-attention score of each first feature sequence block is calculated, and a first audio coding feature sequence is generated based on the feature sequence in each first feature sequence block and the first self-attention score corresponding to each first feature sequence block.

[0181] Optionally, the first generating module 702 is further configured to:

[0182] Re-blocking the first audio coding feature sequence by using a cyclic shift method to obtain a plurality of second feature sequence blocks;

[0183] A second self-attention score of each second feature sequence block is calculated, and a second audio coding feature sequence is generated based on the feature sequence in each second feature sequence block and the second self-attention score corresponding to each second feature sequence block.

[0184] Optionally, the first generating module 702 is further configured to:

[0185] Identify the temporal constraints of the feature sequences between each second feature sequence block, and based on the temporal constraints, use a preset masking mechanism to mask the second self-attention score of each second feature sequence block;

[0186] Based on the masked second self-attention score, a second audio coding feature sequence is generated.

[0187] Optionally, the audio decoder includes a connected temporal classification decoding layer and a Transformer decoding layer;

[0188] Optionally, the second generating module 703 is further configured to:

[0189] Input the target audio coding feature sequence into the connected temporal classification decoding layer and the Transformer decoding layer respectively to obtain the first acoustic prediction score and the second acoustic prediction score;

[0190] The speech recognition text is generated according to the first acoustic prediction score and the second acoustic prediction score based on a beam search algorithm.

[0191] Figure 8 800 is a schematic diagram of the structure of the electronic device provided by the present invention. Figure 8As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute a streaming speech recognition method, which includes: obtaining original audio data, generating an original feature sequence based on the original audio data; inputting the original feature sequence into an audio encoder to generate a target audio coding feature sequence, wherein the audio encoder includes a block self-attention mechanism encoding layer and a shifted block self-attention mechanism encoding layer; inputting the target audio coding feature sequence into an audio decoder to generate speech recognition text.

[0192] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0193] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the streaming speech recognition method provided by the above methods, which includes: obtaining original audio data, generating an original feature sequence based on the original audio data; inputting the original feature sequence into an audio encoder to generate a target audio coding feature sequence, wherein the audio encoder includes a block self-attention mechanism coding layer and a shift block self-attention mechanism coding layer; inputting the target audio coding feature sequence into an audio decoder to generate speech recognition text.

[0194] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the streaming speech recognition method provided by the above-mentioned methods, the method comprising: obtaining original audio data, generating an original feature sequence based on the original audio data; inputting the original feature sequence into an audio encoder to generate a target audio coding feature sequence, wherein the audio encoder comprises a block self-attention mechanism coding layer and a shifted block self-attention mechanism coding layer; inputting the target audio coding feature sequence into an audio decoder to generate speech recognition text.

[0195] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0196] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0197] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A streaming speech recognition method, characterized in that: include: Obtaining original audio data, extracting a cepstral domain feature sequence of the original audio data for extracting speech features of the original audio data, wherein the cepstral domain feature sequence is obtained by performing an inverse Fourier transform on the logarithm of the power spectrum of the original audio data; downsampling the cepstral domain feature sequence by a factor of 4 using a two-layer convolutional neural network to generate a feature sequence after the downsampling operation; padding the tail of the feature sequence after the downsampling operation to generate an original feature sequence of a preset length; wherein the length of each feature sequence after the downsampling operation is slightly larger than the length of the longest feature sequence in the batch, and the feature sequence length is divisible by the block size in the audio encoder; Inputting the original feature sequence into an audio encoder to generate a target audio coding feature sequence, wherein the audio encoder includes a block self-attention mechanism encoding layer and a shifted block self-attention mechanism encoding layer; Inputting the target audio coding feature sequence into an audio decoder to generate speech recognition text; The audio decoder includes a concatenated temporal classification decoding layer and a Transformer decoding layer; the concatenated temporal classification decoding layer is composed of a linear layer and a log softmax layer, and the target audio coding features are input to the concatenated temporal classification decoding layer. During the test phase of training the concatenated temporal classification decoding layer, a first acoustic prediction score is obtained, where the first acoustic prediction score refers to the acoustic prediction score of each word in the multiple recognized texts corresponding to the original audio data output by the concatenated temporal classification decoding layer; At the same time, the target audio coding features are input into the Transformer decoding layer, the network parameters of the Transformer decoding layer are trained, and a second acoustic prediction score is obtained during the test phase of training the Transformer decoding layer. The second acoustic prediction score refers to the acoustic prediction score of each word in the multiple recognized texts corresponding to the original audio data output by the Transformer decoding layer; After obtaining the first acoustic prediction score and the second acoustic prediction score, based on the beam search algorithm, a preset number of texts with the highest prediction scores are selected from the first acoustic prediction score and the second acoustic prediction score as the speech recognition texts corresponding to the final original speech data; After obtaining the original audio data, before generating the original feature sequence based on the original audio data, the original audio data is subjected to variable rate processing and the pitch of the original audio data is processed; the original audio data is converted into a Mel spectrum, and the Mel spectrum is used as the image data of one channel, and the image is subjected to translation data enhancement, and several consecutive rows on the image are randomly masked to enhance the time dimension, and several consecutive columns on the image are randomly masked to enhance the frequency dimension, thereby achieving enhancement of the time dimension and the frequency dimension, or the original audio data is subjected to audio noise processing for data enhancement.

2. The streaming speech recognition method according to claim 1, wherein: The step of inputting the original feature sequence into an audio encoder to generate a target audio coding feature sequence comprises: Inputting the original feature sequence into the block self-attention mechanism encoding layer to generate a first audio encoding feature sequence; inputting the first audio encoding feature sequence into the shifted block self-attention mechanism encoding layer to generate a second audio encoding feature sequence; The second audio coding feature sequence is input into the block self-attention mechanism coding layer for N loop iterations to obtain the target audio coding feature sequence output by the shifted block self-attention mechanism coding layer; N is a positive integer.

3. The streaming speech recognition method according to claim 2, wherein: The step of inputting the original feature sequence into the block self-attention mechanism encoding layer to generate a first audio encoding feature sequence includes: Evenly dividing the original feature sequence into blocks to obtain a plurality of first feature sequence blocks; Calculate the first self-attention score of each first feature sequence block, and generate the first audio coding feature sequence based on the feature sequence in each first feature sequence block and the first self-attention score corresponding to each first feature sequence block.

4. The streaming speech recognition method according to claim 2, wherein: The step of inputting the first audio coding feature sequence into the shift block self-attention mechanism encoding layer to generate a second audio coding feature sequence includes: Re-blocking the first audio coding feature sequence by using a cyclic shift method to obtain a plurality of second feature sequence blocks; Calculate the second self-attention score of each second feature sequence block, and generate the second audio coding feature sequence based on the feature sequence in each second feature sequence block and the second self-attention score corresponding to each second feature sequence block.

5. The streaming speech recognition method according to claim 4, characterized in that: The generating the second audio coding feature sequence based on the feature sequence in each second feature sequence block and the second self-attention score corresponding to each second feature sequence block includes: Identifying a timing constraint of a feature sequence between each of the second feature sequence blocks, and based on the timing constraint, using a preset masking mechanism to perform masking on the second self-attention score of each of the second feature sequence blocks; Based on the masked second self-attention score, the second audio coding feature sequence is generated.

6. A streaming speech recognition device, characterized in that: include: An acquisition module is configured to acquire raw audio data and extract a cepstral domain feature sequence of the raw audio data to extract speech features of the raw audio data, wherein the cepstral domain feature sequence is obtained by performing an inverse Fourier transform on the logarithm of the power spectrum of the raw audio data; downsample the cepstral domain feature sequence by a factor of 4 using a two-layer convolutional neural network to generate a feature sequence after the downsampling operation; and pad the tail of the feature sequence after the downsampling operation to generate an original feature sequence of a preset length; wherein the length of each feature sequence after the downsampling operation is slightly larger than the length of the longest feature sequence in the batch, and the feature sequence length is divisible by the block size in the audio encoder; A first generation module is configured to input the original feature sequence into an audio encoder to generate a target audio coding feature sequence, wherein the audio encoder includes a block self-attention mechanism encoding layer and a shifted block self-attention mechanism encoding layer; A second generation module is configured to input the target audio coding feature sequence into an audio decoder to generate speech recognition text; the audio decoder includes a connection time series classification decoding layer and a Transformer decoding layer; the connection time series classification decoding layer is composed of a linear layer and a log softmax layer, the target audio coding feature is input into the connection time series classification decoding layer, and a first acoustic prediction score is obtained during a test phase of training the connection time series classification decoding layer, wherein the first acoustic prediction score refers to the acoustic prediction score of each word in the multiple recognition texts corresponding to the original audio data output by the connection time series classification decoding layer; At the same time, the target audio coding features are input into the Transformer decoding layer, the network parameters of the Transformer decoding layer are trained, and a second acoustic prediction score is obtained during the test phase of training the Transformer decoding layer. The second acoustic prediction score refers to the acoustic prediction score of each word in the multiple recognized texts corresponding to the original audio data output by the Transformer decoding layer; After obtaining the first acoustic prediction score and the second acoustic prediction score, based on the beam search algorithm, a preset number of texts with the highest prediction scores are selected from the first acoustic prediction score and the second acoustic prediction score as the speech recognition texts corresponding to the final original speech data; After obtaining the original audio data, before generating the original feature sequence based on the original audio data, the original audio data is subjected to variable rate processing and the pitch of the original audio data is processed; the original audio data is converted into a Mel spectrum, and the Mel spectrum is used as the image data of one channel, and the image is subjected to translation data enhancement, and several consecutive rows on the image are randomly masked to enhance the time dimension, and several consecutive columns on the image are randomly masked to enhance the frequency dimension, thereby achieving enhancement of the time dimension and the frequency dimension, or the original audio data is subjected to audio noise processing for data enhancement.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the streaming speech recognition method according to any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the streaming speech recognition method according to any one of claims 1 to 5 is implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the streaming speech recognition method according to any one of claims 1 to 5 is implemented.

Images