A method for draft sequence multiplexing for automatic speech recognition hypothesis decoding

Abstract

The application provides a draft sequence multiplexing method for automatic speech recognition speculation decoding, and belongs to the field of deep learning. First, a lightweight draft model is used for speculation decoding, and a target model is used for verification. Then, two sub-sequences that pass the verification are predicted in parallel through attention masking. Finally, the decoding of the part that does not pass the verification is multiplexed through the connection of the two sub-sequences. The application can improve the decoding efficiency of automatic speech recognition.

Description

Technical Field

[0001] This invention relates to the field of deep learning, and more specifically to a draft sequence recycling method for speculative decoding in automatic speech recognition. Background Technology

[0002] Automatic speech recognition based on large language models has attracted widespread attention due to its high recognition accuracy and enhanced multi-dialect support. When large language models are combined with audio encoders as decoders, the resulting automatic speech recognition models have achieved significant breakthroughs in recognition accuracy, effectively handling complex situations involving multiple languages, dialects, and accents. However, the introduction of large language models inevitably leads to a substantial increase in the scale and computational load of automatic speech recognition models, resulting in improved performance but also increasingly prominent decoding latency issues. Currently, the number of parameters in large language model decoders used for automatic speech recognition is generally high, typically 7 billion or 13 billion, while the number of parameters in the accompanying audio encoder models is relatively low, mostly below 1 billion, and some even only around 100 million. The autoregressive decoding characteristics of large language models further exacerbate computational latency. In this situation, large language model decoders produce more significant inference latency compared to audio encoders, becoming a key bottleneck restricting the efficiency of automatic speech recognition.

[0003] To address the challenge of low decoding efficiency in large language models, speculative decoding techniques, after extensive research and practical exploration in natural language processing tasks, have recently been introduced into the architecture of automatic speech recognition models. The core innovation of this technique lies in introducing a lightweight draft model and constructing a highly efficient "draft-re-verification" decoding process, thereby significantly improving the decoding speed of the target model. The specific operation is as follows: First, a large language model with a small parameter size and fast decoding speed is used to generate a draft word sequence via autoregression. Then, the target large language model, with a larger parameter size but relatively slower decoding speed, is used to perform a single decoding verification of the draft sequence. During the verification process, if the draft word sequence is consistent with the output of the target model, it is accepted. Through this design, a significant improvement in the decoding efficiency of the target model is achieved without sacrificing decoding quality.

[0004] Although speculative decoding can effectively reduce the number of decoding iterations in the target model and improve decoding efficiency, the number of tokens received in each round of prediction and validation is still relatively small, limiting the speedup and failing to meet the requirements of real-time speech transcription. The root cause of this problem lies in directly applying speculative decoding to automatic speech recognition, ignoring the decoding characteristics of large language models in speech recognition scenarios. Automatic speech recognition is an audio conditional generation task, and the audio encoder plays a strong guiding role in decoding. This leads to a key feature: even if the outputs of the large and small models mismatch at a certain token, subsequent decoding of the two models can still align due to the guidance of the audio encoding. This encourages the reuse of unvalidated draft tokens rather than simple discarding.

[0005] Therefore, it is necessary to perform correctness assessment and selective reuse of unreceived lexical units on the draft sequences in speculative decoding for automatic speech recognition. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention proposes a speculative decoding framework for a large language model decoder for automatic speech recognition, which has a core mechanism for a reusable draft lexical prediction algorithm.

[0007] The technical solution provided by this invention is as follows:

[0008] A method for reusing draft sequences for automatic speech recognition and speculative decoding, such as Figure 1 As shown, the feature is that it includes the following steps:

[0009] Step 1: For a target autoregressive large language model decoder M with a large parameter size and slow decoding speed... t A lightweight draft model M was selected, which has the same vocabulary as the model but fewer parameters and faster decoding speed. d Assuming a batch size of 1, the audio encoder is used to extract and compress features from the speech signal. The encoded speech features are denoted as H_speech, and when fed into the large language model, they are projected as a tensor of shape (speech feature sequence length, hidden layer dimension). The text prompts and transcribed text are denoted as text, which is a tensor of shape (1, initial text length). H_speech and text are fed into model M. d The calculation model outputs logits1 = M dFor (H_speech,text), select logits1[:,-1,:], a tensor of shape (1, 1, vocabulary size), where the index of the maximum value in the vocabulary is the newly transcribed text term. Set the prediction length n for speculative decoding, update the key-value cache (KV cache) autoregressively and decode, and denote the output sequence of newly transcribed terms as the prediction sequence seq_1, which is a tensor of shape (1,n).

[0010] Step 2: Feed H_speech, text, and seq_1 into the target large language model decoder M. t The calculation model outputs logits2 = M t (H_speech, concat([text, seq_1], dim = -1)). Choosing a tensor of shape logits2[:, -n-1:-1,:] with shape (1, n, vocabulary size), the index of the validation word sequence can be represented as:

[0011] seq_2=argmax(logits2[:,-n-1:-1,∶],dim=-1)

[0012] It is a tensor of shape (1, n). The predicted sequence seq_1 and the validation sequence seq_2 are sequentially compared, with the term seq_1[:,i] and seq_2[:,i] at the same position (denoted as i). If they are identical, the term prediction is correct, and it is added to the transcribed text, updating text = concat([text,seq_1[:,i]], dim = -1). If the terms in the two sequences are completely identical, the validation passes, and we return to step 1 for the next round of prediction. If seq_1[:,i] ≠ seq_2[:,i] at the i-th term, the term seq_2[:,i] is added to the transcribed text, updating text = concat([text,seq_2[:,i]], dim = -1). The comparison and validation are stopped, and we proceed to step 3 to re-predict.

[0013] Step 3: Draft Model M d The currently predicted complete text sequence is text1 = concat([text[:,:-1], seq_1[:,i:]], dim = -1), which is a tensor of shape (1, initial text length + n). The target model M is then added. tThe sequence of corrected tokens is text2 = concat([text1, seq_2[:,i]], dim = -1), which is a tensor of shape (1, initial text length + n + 1). The dimensions of text1 are padded to match those of text2, and the two are concatenated along the batch size dimension, denoted as text_new = concat([text1, text2], dim = 0), which is a tensor of shape (2, initial text length + n + 1). To make M... d It can simultaneously perform forward prediction of seq_1 and correct the original prediction error positions, hide the re-prediction decoding overhead, and construct an attention mask:

[0014] Attn = torch.ones(2, initial text length + n + 1)

[0015] Attn = (Attn[1, initial text length + i: -1] = 0)

[0016] The attention mask for unverified words in the draft sequence is set to 0, and the attention mask for the remaining words is set to 1. Then proceed to step 4 for re-decoding and reuse.

[0017] Step 4: Feed the attention-masked text_new and H_speech into the draft model M d Autoregression is used to re-decode the model. In each decoding round (denoted as round j), the model output logits = M is calculated. d (H_speech, Attn(h(text_new))) selects a tensor of shape (2, 1, vocabulary size) with logits[:,-1,:], and takes the term [term1, term2] corresponding to the index of the maximum value on the third dimension, updating text_new. Term1 is the next term to be decoded using the original predicted seq_1 with partial errors as the preceding text, and term2 is the next term to be decoded using the actually verified and updated text as the preceding text. To determine whether term2 has corrected the errors in the original predicted seq_1 and can be connected to the subsequent predicted parts of seq_1, compare whether the re-predicted term2 and the original predicted sequence text_new[0, initial text length + i + j - 1: initial text length + i + j + 2] have the same term:

[0018] text_new[1,-1]==text_new[0,k]

[0019] k∈[initial text length + i + j - 1, initial text length + i + j + 2)

[0020] If identical terms exist, the correction is complete, and the original predicted sequence can be reused in subsequent decoding, proceeding to step 5. Otherwise, after the number of autoregressive rounds j reaches the set speculative decoding prediction length n, seq_1 = text_new[1,-n:], return to step 2 to start the next round of verification.

[0021] Step 5: After executing j rounds in step 4, the part new_seq = text_new[1,-j:] that is re-predicted with the verified text as the preceding text is obtained. The part re_seq = text_new[0,k:] that skipped the error in the sequence that continues to be predicted with the original predicted seq_1 as the preceding text can be reused. The complete predicted seq_1 = concat([new_seq,re_seq],dim =-1) is updated, which is a tensor of shape (1, 2j+ni). Then, step 2 is entered to submit for verification again.

[0022] The technical effects of this invention are as follows:

[0023] This invention first uses a lightweight draft model for speculative decoding, which is then validated by the target model. Subsequently, attention masks are used to predict in parallel the two sub-sequences submitted for validation and those that passed validation. Finally, the decoding of the failed-validation portion is reused by concatenating the two sub-sequences. This invention improves the speedup of speculative decoding in automatic speech recognition tasks by reusing draft sequences during the parallel decoding and comparison process in the draft model regeneration process, hiding the regeneration overhead. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the overall workflow of the present invention.

[0025] Figure 2 The present invention describes the speculative decoding workflow. (a) describes the autoregressive serial decoding workflow, and (b) describes the speculative decoding workflow.

[0026] Figure 3 This is an example of parallel decoding during the draft model regeneration process. Among them, (a) is an example image of the draft model regeneration, which adds the continued prediction of the original draft sequence compared to the original re-decoding; (b) shows the construction method of the attention mask.

[0027] Figure 4 This is a schematic diagram illustrating the estimated revenue from reusing draft sequences. Detailed Implementation

[0028] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0029] like Figure 2As shown, speculative decoding can reduce decoding latency by decreasing the number of decoding iterations of the target model. (a) is an autoregressive decoding method, where only one word can be generated per decoding iteration. (b) is a speculative decoding method, where multiple words can be received per validation iteration. Speculative decoding for automatic speech recognition consists of steps 1 to 2:

[0030] Step 1: Select Whisper Encoder as the audio encoder to extract and compress features from the speech signal. Denote the encoded speech features as H_speech and project them as a tensor of shape (speech feature sequence length, hidden layer dimension). Use Llama 7B or Vicuna 13B, which have larger model parameters and slower decoding speeds, as the target autoregressive large language model decoder M. t TinyLlama, with its smaller parameter size and faster decoding speed, was selected as the draft autoregressive large language model decoder M. d M t and M d Both were fine-tuned for the automatic speech recognition task. The text prompt and the transcribed text are denoted as `text`, which is a tensor of shape (1, initial text length); `H_speech` and `text` are fed into model M. d The calculation model outputs logits1 = M d For (H_speech,text), select logits1[:,-1,:], a tensor of shape (1, 1, vocabulary size), where the index of the maximum value in the vocabulary is the newly transcribed text term. Set the prediction length n for speculative decoding, update the key-value cache (KV cache) autoregressively on the draft model and decode, and denote the output sequence of newly transcribed terms as the prediction sequence seq_1, which is a tensor of shape (1,n).

[0031] Step 2: Feed H_speech, text, and seq_1 into the target large language model decoder M. t The calculation model outputs logits2 = M t (H_speech, concat([text, seq_1], dim = -1)). Choosing a tensor of shape logits2[:, -n-1:-1,:] with shape (1, n, vocabulary size), the index of the validation word sequence can be represented as:

[0032] seq_2=argmax(logits2[:,-n-1:-1,∶],dim=-1)

[0033] It is a tensor of shape (1, n). The tokens at the same position (denoted as i) in the predicted sequence seq_1 and the verification sequence seq_2 are compared sequentially: seq_1[:,i] and seq_2[:,i]. If they are the same, the token prediction is correct; it is added to the transcribed text, and text is updated to text = concat([text, seq_1[:,i]], dim = -1). If the tokens in the two sequences are completely identical, the verification passes, and the process returns to step 1 to continue the next round of prediction. Figure 3 As shown in (a), at the i=2 terminus, t2≠T2. The terminus T2 is added to the transcribed text, and text=concat([text,seq_2[:,i]],dim=-1). The alignment and verification are stopped, and step 3 is initiated to re-predict.

[0034] The method for reusing draft sequences consists of steps 3 through 5:

[0035] Step 3: As Figure 3 As shown in (a), the draft model M d The currently predicted complete text sequence is text1 = concat([text[:,:-1], seq_1[:,i:]], dim = -1), which is a tensor of shape (1, initial text length + n). The target model M is then added. t The sequence of corrected tokens is text2 = concat([text1, seq_2[:,i]], dim = -1), which is a tensor of shape (1, initial text length + n + 1). The dimensions of text1 are padded to match those of text2, and the two are concatenated along the batch size dimension, denoted as text_new = concat([text1, text2], dim = 0), which is a tensor of shape (2, initial text length + n + 1). To make M... d It can simultaneously perform forward prediction of seq_1 and correct the original prediction error positions, hide the re-prediction decoding overhead, and construct an attention mask:

[0036] Attn = torch.ones(2, initial text length + n + 1)

[0037] Attn = (Attn[1, initial text length + i: -1] = 0)

[0038] The attention mask for unverified words in the draft sequence is set to 0, and the attention mask for the remaining words is set to 1. Then proceed to step 4 for re-decoding and reuse.

[0039] Step 4: As Figure 3As shown in (b), the attention-masked text_new and H_speech are fed into the draft model M. d Autoregression is used to re-decode the model. In each decoding round (denoted as round j), the model output logits = M is calculated. d (H_speech, Attn(h(text_new))) selects a tensor of shape (2, 1, vocabulary size) with the value of logits1[:,-1,:]. It takes the token [token 1, token 2] corresponding to the index of the maximum value on the third dimension and updates text_new. Token 1 is the next token to be decoded using the original predicted seq_1 with partial errors as the preceding text. Token 2 is the next token to be decoded using the actually verified and updated text as the preceding text. To determine whether token 2 has corrected the errors in the original predicted seq_1 and can be connected to the subsequent predicted parts of seq_1, the re-predicted token 2 and the original predicted sequence text_new[0, initial text length + i + j - 1: initial text length + i + j + 2] are compared to see if they contain the same token.

[0040] text_new[1,-1]==text_new[0,k]

[0041] k∈[initial text length + i + j - 1, initial text length + i + j + 2)

[0042] If the same term exists (t2 = t′4), it means the correction is complete, and the original predicted sequence can be reused in subsequent decoding, proceeding to step 5. Otherwise, after the number of autoregressive rounds j reaches the set speculative decoding prediction length n, seq_1 = text_new[1,-n:], return to step 2 to start the next round of verification.

[0043] Step 5: After executing j rounds in Step 4, the part new_seq = text_new[1,-j:], which is re-predicted using the verified text as the preceding context, can be reused. The part re_seq = text_new[0,k:], which skips errors, can be reused in the sequence predicted using the original predicted seq_1 as the preceding context. The complete predicted seq_1 = concat([new_seq,re_seq],dim =-1), which is a tensor of shape (1, 2j + ni). Then, proceed to Step 2 and submit for verification again. Figure 4 As shown, after two rounds of re-decoding, all four terms in the second round passed the verification, and the expected receive-prediction ratio reached 2.0.

[0044] Finally, it should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the claims.

Claims

1. A method for reusing draft sequences for automatic speech recognition speculative decoding, comprising the following steps: Step 1: For the target large model decoder, use a lightweight draft model. Autoregressive prediction of draft sequence of speech transcription seq_1; Step 2: Validate the draft sequence seq_1 using the target model. If all validations pass, update the transcribed text; otherwise, add the validated parts to the text sequence. Step 3: Create the draft model The current predicted complete text sequence and the sequence with corrected words added to the target model are filled into the same dimension and then connected in the batch size dimension. The attention mask for words that have not been validated in the draft sequence is assigned a value of 0, and the attention mask for the remaining words is assigned a value of 1. Step 4: Feed the attention-masked sequence and audio features into the draft model. The autoregressive process performs re-decoding, parallel execution of draft sequence expansion and validation failure word regeneration, and compares the next word in the re-decoded text with the corresponding and adjacent positions in the original draft sequence seq_1. If they are the same, it means that the correction is complete and the original predicted sequence can be reused for subsequent decoding; otherwise, it is re-validated after the re-decoding reaches the set length. Step 5: After executing j rounds in Step 4, the part that is re-predicted with the verified text as the preceding text is obtained. It can reuse the skipped part in the sequence that is predicted with the original prediction draft sequence seq_1 as the preceding text, update the complete prediction draft sequence seq_1, and enter Step 2 to submit for verification again.

2. The draft sequence multiplexing method for automatic speech recognition speculative decoding as described in claim 1, characterized in that, Step 1 specifically involves, assuming a batch size of 1, using an audio encoder to extract and compress features from the speech signal, denoting the encoded speech features as H_speech, and projecting them into a tensor of shape (speech feature sequence length, hidden layer dimension) when fed into the large language model; denoting the text prompts and transcribed text as text, and feeding H_speech and text into the draft model. The calculation model outputs logits1= (H_speech, text) selects the index of the maximum value in logits1[:, -1, :] in the vocabulary as the newly transcribed text term.

3. The draft sequence multiplexing method for automatic speech recognition speculative decoding as described in claim 2, characterized in that, Set the prediction length of speculative decoding to n, update the key value cache and decode using autoregression, and record the output new transcribed word sequence as the prediction sequence.

Images