Streaming speech recognition system and method based on non-autoregressive model

By introducing a non-autoregressive model in streaming speech recognition, using acoustic feature sequence extraction, streaming acoustic encoder and CTC linear mapping layer, combined with non-autoregressive decoder and cross entropy loss, the problem of low speech recognition decoding efficiency is solved. , achieving faster inference speed and better real-time.

CN114203170BActive Publication Date: 2024-11-05董立波
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111541837.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-16
Publication Date
2024-11-05
Estimated Expiration
2041-12-16

AI Technical Summary

Technical Problem

The existing speech recognition decoding efficiency is low and the real-time performance is poor, especially the problem of slow autoregressive decoding in streaming speech recognition.

Method used

Using a streaming speech recognition system based on non-autoregressive model, including acoustic feature sequence extraction module, streaming acoustic encoder, CTC linear mapping layer and non-autoregressive decoder, the block acoustic coding state subs are encoded through non-autoregressive decoder. The sequence is decoded, and the Viterbi algorithm and cross-entropy loss are combined for training and decoding.

Benefits of technology

It achieves the improvement of streaming inference speed, avoids the loss of language modeling capabilities, and improves the efficiency and real-timeness of speech recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114203170B_ABST
    Figure CN114203170B_ABST
Patent Text Reader

Abstract

The present invention discloses a streaming speech recognition system and method based on a non-autoregressive model, the method comprising: S11, extracting an acoustic feature sequence; S12, generating an acoustic coding state sequence; S13, generating an acoustic coding state sequence; S14, calculating a CTC output probability distribution and a connection timing loss; S15, aligning using a Viterbi algorithm; S16, inputting segment by segment and calculating a joint cross entropy loss; S17, calculating a gradient according to a joint loss of a connection timing loss and a joint cross entropy loss, and performing back propagation; S18, looping through S12 to S17 until training is completed; the system comprises: an acoustic feature sequence extraction module, a streaming acoustic encoder, a CTC linear transformation layer, and a non-autoregressive decoder that are sequentially interconnected. The present invention achieves an improvement in streaming reasoning speed by performing segment-by-segment non-autoregressive decoding on the input audio segment. The loss of language modeling capability is avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of electronic signal processing, and in particular relates to a streaming speech recognition system and method based on a non-autoregressive model. Background Art

[0002] As the entrance to human-computer interaction, speech recognition has important application value in assisting machines to obtain external information and improving the human-computer interaction experience. Streaming speech recognition methods are usually implemented using models based on autoregressive models. Common models include RNN-Transducer models and encoding and decoding models based on attention mechanisms. The decoder starts from the start symbol and predicts the corresponding text sequence step by step or frame by frame based on the editor output until the end tag is predicted. The decoding method of this autoregressive decoding depends on the tags generated in the past. This time-dependent characteristic seriously affects the efficiency of decoding, and it is also difficult to further optimize the reasoning speed through parallel computing. Non-autoregressive speech recognition methods have achieved breakthrough progress in non-streaming speech recognition tasks, and their extremely fast reasoning efficiency has attracted a lot of attention. The present invention introduces non-autoregressive technology to address the problem of slow autoregressive decoding in streaming recognition, and pioneered a streaming speech recognition method based on a non-autoregressive model. Summary of the invention

[0003] The present invention aims to provide a streaming speech recognition system and method based on a non-autoregressive model to solve the technical problems of low efficiency of existing speech recognition decoding and poor real-time performance of speech recognition.

[0004] In order to solve the above technical problems, the specific technical solutions of a streaming speech recognition system and method based on a non-autoregressive model of the present invention are as follows:

[0005] A streaming speech recognition system based on a non-autoregressive model, comprising: an acoustic feature sequence extraction module, a streaming acoustic encoder, a CTC linear mapping layer and a non-autoregressive decoder which are sequentially connected to each other;

[0006] The acoustic feature sequence extraction module is used to extract features from the input speech sequence;

[0007] The streaming acoustic encoder is used to convert the input speech sequence into streaming encoding and output an acoustic coding state sequence;

[0008] The CTC linear mapping layer is used to calculate the CTC output probability distribution and assist in determining the text mark position;

[0009] The non-autoregressive decoder is used to perform non-autoregressive decoding on the block acoustic coding state subsequence.

[0010] The present invention also discloses a streaming speech recognition method based on a non-autoregressive model, comprising the following steps:

[0011] S11, acquiring speech training data and corresponding text data to form a speech-text pair as training data, and extracting an acoustic feature sequence;

[0012] S12, inputting the acoustic feature sequence into the streaming acoustic encoder, and outputting the acoustic coding state sequence through forward calculation;

[0013] S13, input the acoustic coding state sequence into the CTC linear transformation layer to calculate the CTC output probability distribution, and combine it with the text annotation to calculate the connection time series loss, that is, the CTC loss;

[0014] S14, aligning the acoustic coding state sequence with the corresponding text tag sequence using the CTC output probability distribution according to the Viterbi algorithm, determining the position of the text tag sequence in the acoustic coding state sequence, and recording it;

[0015] S15, dividing the acoustic coding state into acoustic coding feature blocks of equal length and non-overlapping, if there is a text tag inside this block, then using the acoustic feature subsequence of this block, the text tag sequence corresponding to the previous acoustic coding feature block, and the fixed-length blank-filled tag sequence as inputs to the non-autoregressive decoder, and calculating the non-autoregressive probability distribution;

[0016] S16, calculate the cross entropy loss of the non-autoregressive probability distribution corresponding to each block and the text tag corresponding to each block respectively, and sum them up to obtain the joint non-autoregressive cross entropy loss L CE ;

[0017] S17, weight the connection time series loss and the joint non-autoregressive cross entropy loss, calculate the gradient according to the joint loss of the two, and perform back propagation;

[0018] S18, loop through S12 to S17 until the preset training end condition is reached, and the training is completed.

[0019] Furthermore, in the step S11, the speech waveform signal in the time domain is windowed and framed and then discrete Fourier transformed to extract coefficients of specific frequency components to form feature vectors. A series of feature vectors constitute a speech feature sequence, and the coefficients are Mel-frequency cepstral coefficients or Mel filter group coefficients, or low-level or high-level feature representations generated by various pre-trained models.

[0020] Furthermore, the streaming acoustic encoder in step S12 is an acoustic encoder based on a convolutional network and a unidirectional self-attention network, the convolution part is responsible for modeling local acoustic features and temporal downsampling; the unidirectional self-attention network is responsible for modeling the contextual dependencies of the input sequence; the streaming acoustic encoder is constructed by a unidirectional recurrent network model or a convolutional neural network.

[0021] Furthermore, the CTC linear transformation layer in step S13 includes only one layer of linear mapping, which is responsible for mapping the output of the streaming acoustic encoder to the output vocabulary size, and each output unit thereof corresponds one-to-one to a label in the vocabulary, and the vocabulary includes a space label, which corresponds to the space mark in the CTC model.

[0022] Furthermore, the input of the non-autoregressive decoder in step S15 includes three parts, namely, the acoustic feature subsequence of this block, the text tag sequence corresponding to the previous acoustic coding feature block, and the blank padding tag sequence of fixed length; the acoustic feature subsequence of this block is used to calculate the attention mechanism to obtain acoustic information; the text tag sequence corresponding to the previous acoustic coding feature block and the blank padding tag sequence of fixed length are concatenated in sequence as input; the length of the padding sequence is equal to the maximum number of labels contained in a segmentation; the model only calculates the non-autoregressive probability distribution at the padding tag position.

[0023] Furthermore, the calculation of the cross entropy loss in step S16 follows a commonly used cross entropy loss calculation method.

[0024] Furthermore, in step S17, the joint loss function L=αL CTC +(1-α)L CE , α represents the weight of the joint temporal loss in the joint loss function, L CTC represents the connection timing loss, L CE represents the joint non-autoregressive cross entropy loss, and α is 0.5.

[0025] Furthermore, after the training is completed, the voice data is decoded, including the following steps:

[0026] S21, obtaining a fixed-length voice packet from an external audio acquisition device and inputting it into the recognition system packet by packet, wherein the voice packet length is equal to the audio length corresponding to a block in the training process;

[0027] S22, extracting features from the input voice packet voice data to obtain a voice feature sequence;

[0028] S23, inputting the speech feature sequence into a streaming acoustic encoder to calculate an acoustic coding state sequence;

[0029] S24, using the condition that whether there is a non-space mark probability greater than 0.5 in the CTC output probability distribution to determine whether there is a mark to be decoded in the acoustic coding subsequence of this block; passing the acoustic coding state sequence through the CTC linear transformation layer, calculating the CTC output probability distribution of the mark, and calculating the probability of the non-space mark, if the probability of the non-space mark is greater than 0.5, it is considered that there is a mark in this segmentation, and then continue to step S25, otherwise jump to S21, and repeat steps S21 to S24;

[0030] S25, concatenating the previously predicted tag sequence and the fixed-length padding sequence, and using them together with the acoustic coding state subsequence of the current block as the input of the non-autoregressive decoder, and calculating the non-autoregressive probability distribution; if the previously predicted tag sequence is empty, only using the fixed-length padding sequence together with the acoustic coding state subsequence of the current block as the input of the non-autoregressive decoder;

[0031] S26, parsing the predicted non-autoregressive probability distribution, starting from the first position of the filling token, obtaining the token in the vocabulary corresponding to the maximum probability output predicted at each position as the output token, and splicing them from left to right until the output of the last filled position, or the space token or the end token;

[0032] S27, repeat S21 to S26 until the last voice packet is input and the last decoding position is calculated, and the decoding is completed.

[0033] The streaming speech recognition system and method based on the non-autoregressive model of the present invention has the following advantages: the present invention realizes the improvement of streaming reasoning speed by performing non-autoregressive decoding on the input audio segments segment by segment. Each block has a certain dependency relationship, thereby avoiding the loss of language modeling ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the overall structure of the present invention.

[0035] Figure 2 It is a schematic diagram of the training process in the present invention.

[0036] Figure 3 It is a schematic diagram of the decoding process in the present invention. DETAILED DESCRIPTION

[0037] In order to better understand the purpose, structure and function of the present invention, the following further describes in detail a streaming speech recognition system and method based on a non-autoregressive model of the present invention in conjunction with the accompanying drawings.

[0038] like Figure 1As shown, a streaming speech recognition system based on a non-autoregressive model of the present invention includes: an acoustic feature sequence extraction module, a streaming acoustic encoder, a CTC linear mapping layer, and a non-autoregressive decoder which are sequentially connected to each other.

[0039] An acoustic feature sequence extraction module, used for extracting features from an input speech sequence;

[0040] A streaming acoustic encoder converts the input speech sequence into streaming encoding and outputs an acoustic coding state sequence;

[0041] The CTC linear mapping layer is used to calculate the CTC output probability distribution and assist in determining the text mark position;

[0042] A non-autoregressive decoder is used to perform non-autoregressive decoding on the block-like acoustically coded state subsequence. Figure 2 As shown, the streaming speech recognition method based on the non-autoregressive model includes the following steps:

[0043] S11, acquiring speech training data and corresponding text data to form a speech-text pair as training data, and extracting an acoustic feature sequence; performing discrete Fourier transform on the speech waveform signal in the time domain after windowing and framing, extracting coefficients of specific frequency components to form a feature vector, a series of feature vectors constitute a speech feature sequence, and the coefficients are Mel-frequency cepstrum coefficients or Mel filter group coefficients, or low-level or high-level feature representations generated by various pre-training models.

[0044] S12, inputs the acoustic feature sequence into the streaming acoustic encoder, and outputs the acoustic coding state sequence through forward calculation; the streaming acoustic encoder is an acoustic encoder based on a convolutional network and a unidirectional self-attention network. The convolution part is responsible for modeling local acoustic features and temporal downsampling. The unidirectional self-attention network is responsible for modeling the contextual dependency of the input sequence. The streaming acoustic encoder can also be constructed by a unidirectional recurrent network model or a convolutional neural network.

[0045] S13, input the acoustic coding state sequence into the CTC linear transformation layer to calculate the CTC output probability distribution, and calculate the connection temporal loss (CTC loss) in combination with the text annotation; the CTC linear transformation layer contains only one layer of linear mapping, which is responsible for mapping the output of the streaming acoustic encoder to the output vocabulary size, and each of its output units corresponds to a label in the vocabulary. The vocabulary contains space labels, which correspond to the space mark in the CTC model.

[0046] S14, aligning the acoustic coding state sequence with the corresponding text tag sequence using the CTC output probability distribution according to the Viterbi algorithm, determining the position of each tag in the acoustic coding state sequence, and recording it;

[0047] S15, divide the acoustic coding state into acoustic coding feature blocks of equal length and non-overlapping. If there is a text tag inside this block, the acoustic feature subsequence of this block, the text tag sequence corresponding to the previous acoustic coding feature block, and the fixed-length blank padding tag sequence are used as the input of the non-autoregressive decoder to calculate the non-autoregressive probability distribution; the input of the non-autoregressive decoder includes three parts, namely the acoustic feature subsequence of this block, the text tag sequence corresponding to the previous acoustic coding feature block, and the fixed-length blank padding tag sequence. The acoustic feature subsequence of this block is used to calculate the attention mechanism to obtain acoustic information. The text tag sequence corresponding to the previous acoustic coding feature block and the fixed-length blank padding tag sequence are concatenated in sequence as input; the length of the padding sequence is equal to the maximum number of labels contained in a segmentation; the model only calculates the non-autoregressive probability distribution at the padding tag position;

[0048] S16, calculate the cross entropy loss of the non-autoregressive probability distribution corresponding to each block and the text tag corresponding to each block respectively, and sum them up to obtain the joint non-autoregressive cross entropy loss L CE ; The calculation of cross entropy loss follows the commonly used cross entropy loss calculation method.

[0049] S17, weight the joint temporal loss and the joint non-autoregressive cross entropy loss, calculate the gradient according to the joint loss of the two, and perform back propagation; the joint loss function L = αL CTC +(1-α)L CE , α represents the weight of the joint temporal loss in the joint loss function, L CTC represents the connection timing loss, L CE represents the joint non-autoregressive cross entropy loss, and α is 0.5.

[0050] S18, loop through S12 to S17 until the preset training end condition is reached, and the training is completed.

[0051] like Figure 3 As shown, the streaming speech recognition method based on the non-autoregressive model is to decode the speech data after completing the training, including the following steps:

[0052] S21, obtaining a fixed-length voice packet from an external audio acquisition device and inputting it into the recognition system packet by packet, wherein the voice packet length is equal to the audio length corresponding to a block in the training process;

[0053] S22, extracting features from the input voice packet voice data to obtain a voice feature sequence;

[0054] S23, inputting the speech feature sequence into a streaming acoustic encoder to calculate an acoustic coding state sequence;

[0055] S24, using the condition that whether there is a non-space mark probability greater than 0.5 in the CTC output probability distribution to determine whether there is a mark to be decoded in the acoustic coding subsequence of this block; passing the acoustic coding state sequence through the CTC linear transformation layer, calculating the CTC output probability distribution of the mark, and calculating the probability of the non-space mark (1-the probability of the space mark), if the probability of the non-space mark is greater than 0.5, it is considered that there is a mark in this segmentation, and then continue to step S25, otherwise jump to S21, and repeat steps S21 to S24;

[0056] S25, concatenating the previously predicted tag sequence and the fixed-length padding sequence, and using them together with the acoustic coding state subsequence of the current block as the input of the non-autoregressive decoder, and calculating the non-autoregressive probability distribution; if the previously predicted tag sequence is empty, only using the fixed-length padding sequence together with the acoustic coding state subsequence of the current block as the input of the non-autoregressive decoder;

[0057] S26, parsing the predicted non-autoregressive probability distribution, starting from the first position of the filling token, obtaining the token in the vocabulary corresponding to the maximum probability output predicted at each position as the output token, and splicing them from left to right until the output of the last filled position, or the space token or the end token;

[0058] S27, repeat the above steps until the last voice packet is input and the last decoding position is calculated, and the decoding is completed.

[0059] Embodiment 1:

[0060] The streaming speech recognition system training method based on the non-autoregressive model includes an acoustic feature sequence extraction module, a streaming acoustic encoder, a CTC linear mapping layer, and a non-autoregressive decoder, such as Figure 1 As shown, the training process includes the following steps:

[0061] Step 1, obtaining speech training data and corresponding text annotation training data, and extracting features of a series of speech training data to form a speech feature sequence;

[0062] The goal of speech recognition is to convert continuous speech signals into text sequences. During the recognition process, the waveform signal in the time domain is windowed and framed, and then a discrete Fourier transform is performed to extract the coefficients of specific frequency components to form a feature vector. A series of feature vectors constitute a speech feature sequence. The speech features are Mel-frequency cepstral coefficients (MFCC) or Mel filter bank coefficients (FBANK). This embodiment uses 80-dimensional FBANK features as input, and other features may be used in other embodiments.

[0063] Step 2, input the acoustic features into a streaming acoustic encoder and output an acoustic coding state sequence;

[0064] In this embodiment, the streaming acoustic encoder is composed of an acoustic encoder based on a convolutional network and a unidirectional self-attention mechanism. The convolutional network is used to perform local modeling and temporal downsampling of input features. In other implementations, the encoder can be replaced by other types of encoders such as a unidirectional long short-term memory network (LSTM). The acoustic encoder based on the unidirectional self-attention mechanism is composed of a stack of 6 or 12 layers of encoding modules of the unidirectional self-attention mechanism. Each module includes a multi-head unidirectional self-attention layer (Multi-Head Attention) and a feed-forward network layer (Feed-Forward Network). Through empirical parameters, 4 or 8 attention heads are used. The feed-forward network layer includes two layers of linear transformations. Nonlinear activation functions are used between linear layers. According to experience, a gated linear unit (GLU) is used as the activation function.

[0065] The implementation of the multi-head attention mechanism is:

[0066] MultiHeadAttention(Q,K,V)=Concat(head1,...,head n )W O

[0067]

[0068]

[0069] Among them, MultiHeadAttention represents a multi-head attention mechanism, which converts the output vector head of multiple self-attention mechanisms i After being spliced ​​together, they are mapped to a fixed dimension through a linear layer, and then the output result is obtained through a feed-forward network FFN. WO are all learnable weight matrices. For the self-attention encoding module, Q = K = V = X, which represent the convolution downsampling sequence of the first layer of the self-attention encoding module input, or the output of the previous layer of the self-attention encoding module. k is the dimension of vector K (equal to the dimension of X);

[0070] The implementation of the feedforward calculation is:

[0071] FFN(x)=GLU(xW1+b1)W2+b2

[0072] x represents the input of the feedforward network layer, W1 and W2 represent the weight matrices, and b1 and b2 represent the weight bias vectors.

[0073] Step 3: Input the acoustic coding state sequence into a CTC linear transformation layer, calculate the CTC output probability distribution and calculate the connection temporal loss (CTC loss) in combination with the text annotation training data;

[0074] The calculation of CTC loss follows the general CTC loss calculation method. The probabilities of all possible alignment paths are summed through the forward and backward algorithms, and the negative logarithmic form of the probability sum is optimized.

[0075] Step 4: Align the acoustic coding state sequence with the corresponding text tag sequence using the CTC output probability distribution according to the Viterbi algorithm, determine the position of each tag in the acoustic coding state sequence, and record it;

[0076] The method of using the Viterbi algorithm to generate an alignment path is commonly used in speech recognition. The alignment path is generated by backtracking the path with the highest probability in the output probability distribution.

[0077] Step 5: Divide the acoustic coding state into acoustic coding feature blocks of equal length and without overlap. If there is a text tag inside this block, the acoustic feature subsequence of this block, the text tag sequence corresponding to the previous acoustic coding feature block, and the fixed-length blank-filled tag sequence are used as inputs of the non-autoregressive decoder to calculate the non-autoregressive probability distribution.

[0078] The length of the acoustic coding feature block usually corresponds to a speech of about 1 second in length, and its length can be set according to the real-time requirements of the system.

[0079] Step 6, calculate the cross entropy loss of the non-autoregressive probability distribution corresponding to each block and the text tag corresponding to each block respectively, and sum them up to obtain the joint non-autoregressive cross entropy loss;

[0080] The cross entropy loss needs to be calculated for each encoding block. The output mark of each block is determined by the position of the alignment mark generated by the CTC output probability distribution. Each block only uses the marks and blank marks within the block to calculate the cross entropy loss. The cross entropy losses of different block outputs are summed as the joint non-autoregressive cross entropy loss L CE .

[0081] Step 7: Weight the joint temporal loss and the joint non-autoregressive cross entropy loss, calculate the gradient based on the joint loss of the two, and perform back propagation;

[0082] Joint loss function L = αL CTC +(1-α)L CE , α represents the weight of the joint temporal loss in the joint loss function, L CTC represents the connection timing loss, L CE represents the joint non-autoregressive cross entropy loss, and α is 0.5.

[0083] Step 8, loop through S12 to S17 until the preset training end condition is reached and the training is completed.

[0084] The streaming speech recognition method based on the non-autoregressive model includes the following steps in decoding:

[0085] Decoding step 1, obtain a fixed-length voice packet from an external audio acquisition device, where the length of the voice packet is equal to the audio length corresponding to a block during the training process;

[0086] The external audio acquisition device can be a smartphone or other embedded device.

[0087] Decoding step 2, extracting features from the input voice packet voice data to obtain a voice feature sequence;

[0088] The speech features are Mel-frequency cepstral coefficients (MFCC) or Mel filter bank coefficients (FBANK), and the feature processing method is consistent with the training process.

[0089] Decoding step 3, inputting the speech feature sequence into the streaming acoustic encoder to calculate the acoustic coding state sequence;

[0090] Decoding step 4, passing the acoustic coding state sequence through the CTC linear transformation layer, calculating the CTC output probability distribution of the mark, and calculating the probability of the non-space mark (1-the probability of the space mark). If the probability of the non-space mark is greater than 0.5, it is considered that there is a mark in this segmentation, and then continue to step S25, otherwise jump to S21 and repeat steps S21 to S24;

[0091] Decoding step 5, concatenating the previously predicted tag sequence and the fixed-length padding sequence together with the acoustic coding state subsequence of this block as the input of the non-autoregressive decoder, and calculating the non-autoregressive probability distribution; if the previously predicted tag sequence is empty, only the fixed-length padding sequence and the acoustic coding state subsequence of this block are used as the input of the non-autoregressive decoder;

[0092] Decoding step 6: parse the predicted non-autoregressive probability distribution, starting from the first position of the filled token, obtain the token in the vocabulary corresponding to the maximum probability output predicted at each position as the output token, and concatenate them from left to right until the output of the last filled position, or the space token or the end token;

[0093] Decoding step 7, repeat the above steps until the last voice packet is input and the last decoding position is calculated, and the decoding is completed.

[0094] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A method for streaming speech recognition of a streaming speech recognition system based on a non-autoregressive model, the system comprising: The acoustic feature sequence extraction module, streaming acoustic encoder, CTC linear mapping layer and non-autoregressive decoder are sequentially connected to each other; The acoustic feature sequence extraction module is used to extract features from the input speech sequence; The streaming acoustic encoder is used to convert the input speech sequence into streaming encoding and output an acoustic coding state sequence; The CTC linear mapping layer is used to calculate the CTC output probability distribution and assist in determining the text mark position; The non-autoregressive decoder is used to perform non-autoregressive decoding on the block acoustic coding state subsequence; Characterized in that the method comprises the following steps: S11, acquiring speech training data and corresponding text data to form a speech-text pair as training data, and extracting an acoustic feature sequence; S12, inputting the acoustic feature sequence into the streaming acoustic encoder, and outputting the acoustic coding state sequence through forward calculation; S13, input the acoustic coding state sequence into the CTC linear transformation layer to calculate the CTC output probability distribution, and combine it with the text annotation to calculate the connection time series loss, that is, the CTC loss; S14, aligning the acoustic coding state sequence with the corresponding text tag sequence using the CTC output probability distribution according to the Viterbi algorithm, determining the position of the text tag sequence in the acoustic coding state sequence, and recording it; S15, dividing the acoustic coding state into acoustic coding feature blocks of equal length and non-overlapping, if there is a text tag inside this block, then using the acoustic feature subsequence of this block, the text tag sequence corresponding to the previous acoustic coding feature block, and the fixed-length blank-filled tag sequence as inputs to the non-autoregressive decoder, and calculating the non-autoregressive probability distribution; The input of the non-autoregressive decoder in step S15 includes three parts, namely, the acoustic feature subsequence of the current block, the text tag sequence corresponding to the previous acoustic coding feature block, and the blank padding tag sequence of fixed length; the acoustic feature subsequence of the current block is used to calculate the attention mechanism to obtain acoustic information; the text tag sequence corresponding to the previous acoustic coding feature block and the blank padding tag sequence of fixed length are spliced ​​together in order as input; the length of the padding sequence is equal to the maximum number of labels contained in a segment; the model only calculates the non-autoregressive probability distribution at the padding tag position; S16, calculate the cross entropy loss of the non-autoregressive probability distribution corresponding to each block and the text tag corresponding to each block respectively, and sum them up to obtain the joint non-autoregressive cross entropy loss ; S17, weight the connection time series loss and the joint non-autoregressive cross entropy loss, calculate the gradient according to the joint loss of the two, and perform back propagation; S18, loop through S12 to S17 until the preset training end condition is reached, and the training is completed.

2. The streaming speech recognition method based on a non-autoregressive model according to claim 1, characterized in that: In the step S11, the speech waveform signal in the time domain is windowed and framed and then discrete Fourier transformed to extract the coefficients of specific frequency components to form feature vectors. A series of feature vectors constitute a speech feature sequence, and the coefficients are Mel-frequency cepstral coefficients or Mel filter group coefficients, or low-level or high-level feature representations generated by various pre-trained models.

3. The streaming speech recognition method based on a non-autoregressive model according to claim 1, characterized in that: The streaming acoustic encoder in step S12 is an acoustic encoder based on a convolutional network and a unidirectional self-attention network. The convolution part is responsible for modeling local acoustic features and temporal downsampling; the unidirectional self-attention network is responsible for modeling the contextual dependencies of the input sequence; the streaming acoustic encoder is constructed by a unidirectional recurrent network model or a convolutional neural network.

4. The streaming speech recognition method based on a non-autoregressive model according to claim 1, characterized in that: The CTC linear transformation layer in step S13 includes only one layer of linear mapping, which is responsible for mapping the output of the streaming acoustic encoder to the output vocabulary size, and each output unit thereof corresponds one-to-one to a label in the vocabulary, and the vocabulary includes a space label, which corresponds to the space mark in the CTC model.

5. The streaming speech recognition method based on a non-autoregressive model according to claim 1, characterized in that: The joint loss function in step S17 , α represents the weight of the joint temporal loss in the joint loss function, L CTC represents the connection timing loss, L CE represents the joint non-autoregressive cross entropy loss, α is 0.

5.

6. The streaming speech recognition method based on a non-autoregressive model according to claim 1, characterized in that: After training, the voice data is decoded, including the following steps: S21, obtaining a fixed-length voice packet from an external audio acquisition device and inputting it into the recognition system packet by packet, wherein the voice packet length is equal to the audio length corresponding to a block in the training process; S22, extracting features from the input voice packet voice data to obtain a voice feature sequence; S23, inputting the speech feature sequence into a streaming acoustic encoder to calculate an acoustic coding state sequence; S24, using the condition that whether there is a non-space mark probability greater than 0.5 in the CTC output probability distribution to determine whether there is a mark to be decoded in the acoustic coding subsequence of this block; passing the acoustic coding state sequence through the CTC linear transformation layer, calculating the CTC output probability distribution of the mark, and calculating the probability of the non-space mark, if the probability of the non-space mark is greater than 0.5, it is considered that there is a mark in this segmentation, and then continue to step S25, otherwise jump to S21, and repeat steps S21~S24; S25, concatenating the previously predicted tag sequence and the fixed-length padding sequence, and using them together with the acoustic coding state subsequence of the current block as the input of the non-autoregressive decoder, and calculating the non-autoregressive probability distribution; if the previously predicted tag sequence is empty, only using the fixed-length padding sequence together with the acoustic coding state subsequence of the current block as the input of the non-autoregressive decoder; S26, parsing the predicted non-autoregressive probability distribution, starting from the first position of the filling token, obtaining the token in the vocabulary corresponding to the maximum probability output predicted at each position as the output token, and splicing them from left to right until the output of the last filled position, or the space token or the end token; S27, repeat S21 to S26 until the last voice packet is input and the last decoding position is calculated, and the decoding is completed.