Large model-based speculative decoding method and device, equipment and storage medium

Abstract

The present disclosure provides a large model-based speculation decoding method and device, equipment and a storage medium, relating to the technical field of artificial intelligence, in particular to the technical field of computer vision, deep learning, large model and the like. The specific implementation scheme is as follows: inputting a first token sequence into a first large model to obtain a second token sequence; wherein the first large model is obtained by quantizing a second large model, and the first token sequence is obtained based on multi-modal data; inputting the first token sequence and the second token sequence into the second large model to obtain a third token sequence; and verifying the second token sequence and the third token sequence to obtain a speculation decoding result.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to the fields of computer vision, deep learning, and large models. Background Technology

[0002] Speculative decoding is a parallel decoding technique, also known as inference decoding. It accelerates inference by increasing the parallelism of language model computation in each decoding step, thereby reducing the total number of decoding steps. Speculative decoding typically uses two models: a large main model and a small auxiliary model. The auxiliary model first generates a sequence. Then, the main model verifies this sequence in a single forward pass. Summary of the Invention

[0003] This disclosure provides a method, apparatus, device, and storage medium for speculative decoding based on large models.

[0004] According to one aspect of this disclosure, a speculative decoding method based on large models is provided, including:

[0005] The first word sequence is input into the first large model to obtain the second word sequence; the first large model is obtained by quantization based on the second large model, and the first word sequence is obtained based on multimodal data;

[0006] Input the first word sequence and the second word sequence into the second large model to obtain the third word sequence;

[0007] The inferred decoding result is obtained by verifying the second and third word sequences.

[0008] According to another aspect of this disclosure, a speculative decoding apparatus based on a large model is provided, comprising:

[0009] The first input module is used to input the first word sequence into the first large model to obtain the second word sequence; wherein the first large model is obtained by quantization based on the second large model, and the first word sequence is obtained based on multimodal data;

[0010] The second input module is used to input the first word sequence and the second word sequence into the second large model to obtain the third word sequence.

[0011] The verification module is used to verify the second and third word sequences to obtain the inferred decoding result.

[0012] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0013] At least one processor; and

[0014] The memory is communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0016] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0017] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0019] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0020] Figure 1 This is a flowchart illustrating a large-model-based speculative decoding method according to an embodiment of the present disclosure;

[0021] Figure 2 This is a flowchart illustrating a large-model-based speculative decoding method according to another embodiment of the present disclosure;

[0022] Figure 3 This is a flowchart illustrating a large-model-based speculative decoding method according to another embodiment of the present disclosure;

[0023] Figure 4 This is a flowchart illustrating a large-model-based speculative decoding method according to another embodiment of the present disclosure;

[0024] Figure 5 This is a flowchart illustrating a large-model-based speculative decoding method according to another embodiment of the present disclosure;

[0025] Figure 6 This is a flowchart illustrating a large-model-based speculative decoding method according to another embodiment of the present disclosure;

[0026] Figure 7 This is a flowchart illustrating a large-model-based speculative decoding method according to another embodiment of the present disclosure;

[0027] Figure 8This is a schematic diagram of the structure of a self-cooperative inference decoding multimodal model acceleration framework according to an embodiment of the present disclosure;

[0028] Figure 9 This is a flowchart illustrating a draft model generation method according to an embodiment of the present disclosure;

[0029] Figure 10 This is a schematic diagram of the structure of a large-model-based speculative decoding apparatus according to an embodiment of the present disclosure;

[0030] Figure 11 This is a schematic diagram of the structure of a large-model-based speculative decoding apparatus according to another embodiment of the present disclosure;

[0031] Figure 12 This is a block diagram of an electronic device used to implement embodiments of the present disclosure. Detailed Implementation

[0032] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0033] Figure 1 This is a flowchart illustrating a large-model-based speculative decoding method 100 according to an embodiment of the present disclosure. In one embodiment, the method includes:

[0034] S101. Input the first word sequence into the first large model to obtain the second word sequence; wherein, the first large model is obtained by quantization based on the second large model, and the first word sequence is obtained based on multimodal data;

[0035] S102. Input the first word sequence and the second word sequence into the second large model to obtain the third word sequence;

[0036] S103. Verify based on the second and third word sequences to obtain the inferred decoding result.

[0037] In this embodiment, a first large model, such as a draft large model, and a second large model, such as a base large model, can be used for speculative decoding. The draft large model can also be called a draft large language model, auxiliary large model, auxiliary model, draft model, etc. The base large model can also be called a base large language model, benchmark large model, etc. The draft large model can be obtained by quantization based on the base large model. Both the base large model and the draft large model can be large language models (LLMs). Inputting a first word sequence into the draft large model can predict a second word sequence. The first word sequence includes multiple input words, and the second word sequence includes multiple output words. The draft large model obtained by quantization based on the base large model is a lightweight large language model. The quantization precision can be selected according to requirements, such as 4 bits, 8 bits, etc. Since the draft large model is obtained by quantization based on the base large model, their word segmentation methods, vocabulary lists, etc., are consistent, and the output distribution is similar. Compared to the base large model, the quantized draft large model can reduce computational load, obtain calculation results faster, and improve computational efficiency.

[0038] In this embodiment, the first and second word sequences are concatenated and input into a base model to predict a third word sequence. The third word sequence may include multiple output words. The third word sequence output by the base model can be used to validate the second word sequence, resulting in a more accurate descriptive text.

[0039] In this embodiment of the disclosure, a basic large model can be used to compare each word in the second word sequence and the third word sequence, and the inferred decoding result can be output based on the comparison result. For example, if each word in the second word sequence and the third word sequence is consistent, either the second word sequence or the third word sequence can be output. If a word in the second word sequence and the third word sequence is inconsistent, the consistent word can be output.

[0040] According to the embodiments of this disclosure, the word segmentation method and vocabulary of the draft big model quantized based on the basic big model are basically the same as those of the basic big model. The output results of the two are similar, which can process the input words faster. In subsequent word verification, it is easier to pass the verification and obtain accurate inference decoding results quickly.

[0041] Figure 2 This is a flowchart illustrating a large-model-based speculative decoding method 200 according to another embodiment of the present disclosure. Method 200 can be used to implement large-model-based speculative decoding method 100. In one embodiment, method 200 further includes:

[0042] S201. Normalize and scale the weight parameters of the second largest model to obtain the quantized first largest model; wherein the first largest model includes the draft large model and the second largest model includes the basic large model.

[0043] In this embodiment of the disclosure, the quantization method for the basic large model may include: normalizing the weight parameters of each layer of the basic large model to obtain the normalized result of each weight parameter. There are various normalization methods, such as min-max normalization, Z-score normalization, and maximum value normalization. An example of maximum value normalization is as follows: a certain weight parameter is W... i The maximum value of the weight parameter is Max(|W i |), then the normalization formula can be:

[0044]

[0045] Then, the normalization result can be scaled according to a certain quantization precision, such as 4 bits or 8 bits, to complete the quantization of the weight parameters. The scaling factor is related to the quantization precision. An example formula for 4-bit quantization is as follows:

[0046]

[0047] According to embodiments of this disclosure, the computational complexity of the quantized draft large model is lower than that of the base large model, which can improve computational speed with almost no loss of generation accuracy and quickly output word sequences. Since there is no need to retrain or filter the draft large model, time can be saved in obtaining the draft large model.

[0048] Figure 3 This is a flowchart illustrating a large-model-based speculative decoding method 300 according to another embodiment of the present disclosure. Method 300 can be used to implement step S201 in the large-model-based speculative decoding method 200. In one embodiment, method 300 includes: normalizing and scaling the weight parameters of the second large model to obtain the quantized first large model, and further includes:

[0049] S301. Iterate through the weight parameters of each linear layer of the second largest model; wherein the tensor formed by the weight parameters includes the first channel and the second channel;

[0050] S302. Calculate the maximum absolute value of the weight parameter in the first channel to obtain the maximum value of the first channel;

[0051] S303. Normalize the weight parameter on the first channel according to the maximum value to obtain the normalized result;

[0052] S304. Scale the normalization result according to the scaling ratio to obtain the quantized first large model.

[0053] In this embodiment, the basic large model may include multiple linear layers. By traversing each linear layer of the basic large model, multiple weight parameters included in each linear layer can be obtained. The weight parameters of the basic large model can be formed into a tensor, which may include two channels. The weight parameters can be quantized based on the two channels of the tensor. For example, the basic large language model is a structure of 32 stacked transformer blocks. Traversing the weight parameters of each linear layer of the model can obtain a tensor [W, H]. Here, W and H each represent a channel. The maximum absolute value of the weight parameters can be obtained by statistically analyzing the H channel, thus obtaining the maximum value of the H channel (or the H group maximum value). The weight parameters are normalized according to the maximum value on the H channel. An example of the formula for normalization calculation can be found in Equation 1 of the previous embodiment.

[0054] In this embodiment, the scaling ratio is related to the quantization precision. For example, the scaling ratio corresponding to 4-bit quantization is 15, and the scaling ratio corresponding to 8-bit quantization is 255. After 4-bit quantization, the length of each weight parameter of the model occupies 4 bits; after 8-bit quantization, the length of the weight parameter of the model occupies 8 bits (1 byte). The scaling ratio can be set before quantization or determined based on the normalization result of the weight parameters. By scaling the scaling ratio, the weight parameters on each linear layer of the normalized basic large model can be scaled to the encoding length corresponding to the quantization precision, thereby simplifying the basic large model and obtaining a draft large model. For example, the normalized weight parameters can be scaled to the range of 4-bit values ​​[-15, 15] to complete the quantization of the weight parameters, as shown in Equation 2 above.

[0055] According to embodiments of this disclosure, the weight parameters of each linear layer in the basic large model can be normalized and scaled to obtain a quantized draft large model, thereby improving the speed of inference decoding.

[0056] Figure 4 This is a flowchart illustrating a large-model-based speculative decoding method 400 according to another embodiment of the present disclosure. Method 400 can be used to implement large-model-based speculative decoding method 100. The multimodal data includes images and text. In one embodiment, method 400 further includes:

[0057] S401. Input the image into the visual encoder to obtain the visual word sequence;

[0058] S402. Input the text into the text encoder to obtain the text word sequence;

[0059] S403. Concatenate the visual word sequence and the text word sequence to obtain the first word sequence.

[0060] In this embodiment of the disclosure, speculative decoding can be performed on multimodal data. Images from the multimodal data can be input into a visual encoder, and after encoding, a visual word sequence can be obtained. Text from the multimodal data can be input into a text encoder, and after encoding, a text word sequence can be obtained. The visual encoder can exist independently of the underlying large model, or it can be part of the underlying large model. The text encoder can exist independently of the underlying large model, or it can be part of the underlying large model.

[0061] In this embodiment, the visual lexical sequence and the text lexical sequence can be simply concatenated to obtain the first multimodal lexical sequence; alternatively, the visual lexical sequence and the text lexical sequence can be concatenated by inputting different attention heads into the concatenation layer of the draft model to obtain the first multimodal lexical sequence. Then, the first multimodal lexical sequence can be input into the draft model for processing to obtain the second lexical sequence.

[0062] According to embodiments of this disclosure, image word sequences and text word sequences can be obtained by encoding multimodal data, and then concatenated to obtain a first word sequence of multimodal data. The process of speculative decoding of multimodal data can then be accelerated by co-processing a draft large model and a base large model.

[0063] Figure 5 This is a flowchart illustrating a large-model-based speculative decoding method 500 according to another embodiment of the present disclosure. Method 500 can be used to implement step S101 in the large-model-based speculative decoding method 100. In one embodiment, method 500 includes: inputting a first lexical sequence into a first large model to obtain a second lexical sequence, and further includes:

[0064] S501. Input the first word sequence into the first large model;

[0065] S502. Using the first major model, a self-attention mask matrix is ​​used to perform self-attention calculation on the first word sequence to obtain the second word sequence.

[0066] In this embodiment of the disclosure, the draft large model may include a self-attention layer. In the self-attention layer, the feature information of the last word of the input sequence and all words before it can be obtained through the self-attention mask matrix, thereby predicting the next word of the second word sequence.

[0067] A casual attention mask matrix can include a lower triangular mask matrix. An example of a lower triangular mask matrix is ​​shown below:

[0068]

[0069] The size of the matrix is ​​n×n, where n can be determined based on the number of words in the first word sequence. For example, for a first word sequence A with 3 words, a 3×3 lower triangular mask matrix can be used for masking; for a first word sequence B with 20 words, a 20×20 lower triangular mask matrix can be used for masking.

[0070] According to embodiments of this disclosure, the second word sequence can be obtained quickly and accurately by performing masking and self-attention calculations on the first word sequence using a draft large model.

[0071] Figure 6 This is a flowchart illustrating a large-model-based speculative decoding method 600 according to another embodiment of this disclosure. In one embodiment, such as Figure 6 As shown, S502 uses the first large model to perform self-attention calculation on the first word sequence using a self-attention mask matrix to obtain the second word sequence, which further includes:

[0072] S601. The first large model uses a self-attention mask matrix to obtain information about the last word and the words preceding it in the first word sequence, so as to predict an output word in the second word sequence.

[0073] S602. The output word is concatenated to the last word of the first word sequence to obtain a new first word sequence.

[0074] S603. Input the new first word sequence into the first large model. Use the self-attention mask matrix of the first large model to obtain the information of the last word and the words before it in the new first word sequence, so as to predict the next output word in the second word sequence.

[0075] S604. Repeat the word concatenation step S602 and the prediction step S603 until the second word sequence is output. The number of repetitions can be flexibly set according to requirements, or according to relevant parameters such as the length of the first word sequence.

[0076] In this embodiment, the length of the first word sequence is t. The entire first word sequence can be input into the draft model. The semantic feature information of the last word t and the preceding words t-1, t-2, ..., up to the first word, is obtained through the self-attention mask matrix of the self-attention layer. After self-attention calculation, the features of the t-th word are used to predict the (t+1)-th word, which can be understood as the first word of the second word sequence. After obtaining the (t+1)-th word, it can be concatenated to the end of the original first word sequence to obtain a new first word sequence of length t+1. The new first word sequence with the concatenated first word is then input into the self-attention layer of the draft model for calculation. After obtaining the next word, for example, the (t+2)-th word, the above concatenation and prediction steps are repeated. This process can be repeated a certain number of times, for example, n times, to predict a second word sequence containing n words.

[0077] According to embodiments of this disclosure, by concatenating the previously predicted word sequence with the original input word sequence and processing them together in the draft large model, the coherence between each predicted word sequence is improved.

[0078] In one implementation, such as Figure 5 As shown, method 500 can be used to implement step S102 in the large model-based inference decoding method 100. In one embodiment, method 500 further includes: inputting the first lexical sequence and the second lexical sequence into the second large model to obtain a third lexical sequence, further including:

[0079] S503. Input the concatenated sequence of the first word sequence and the second word sequence into the second large model;

[0080] S504. The third word sequence is obtained by using the self-attention mask matrix to perform self-attention calculation on the concatenated sequence through the second major model.

[0081] In this embodiment, the first and second word sequences can be concatenated directly, or they can be input into the concatenation layer of a base model for concatenation. For example, the first word sequence [image word A, image word B, text word C] can be concatenated with the second word sequence [predicted word D, predicted word E, predicted word F] to obtain the concatenated sequence [image word A, image word B, text word C, predicted word D, predicted word E, predicted word F]. After obtaining the concatenated sequence, it can be input into the base model for self-attention mask calculation to obtain the third word sequence. The words in the third word sequence can be called verification words.

[0082] In this embodiment, a basic large model can be used to mask the concatenated words using a self-attention mask matrix, such as a lower triangular mask matrix. The masked result is then subjected to self-attention calculation using the basic large model to obtain the third word sequence. The specific method for predicting the third word sequence in S504 is similar in principle to the specific method for predicting the second word sequence in S502, and can be found in the relevant description of S502. The size of the self-attention mask matrix used for the concatenated words can be determined based on the number of words in the concatenated word sequence.

[0083] According to embodiments of this disclosure, a more accurate third word sequence can be quickly obtained by performing masking and self-attention calculations on the concatenated sequence of the first and second word sequences using a basic large model. The second word sequence can then be verified using the third word sequence, thereby improving the accuracy of the inferred decoding results.

[0084] Figure 7 This is a flowchart illustrating a large-model-based speculative decoding method according to another embodiment of the present disclosure. In one implementation, such as Figure 7 As shown, S504 uses the second major model to perform self-attention calculation on the concatenated sequence using a self-attention mask matrix, resulting in the third word sequence, which further includes:

[0085] S701. The second major model uses a self-attention mask matrix to obtain information about the last word and the words preceding it in the concatenated sequence, in order to predict an output word in the third word sequence.

[0086] S702. The output word is appended to the last word of the concatenation sequence to obtain a new concatenation sequence;

[0087] S703. Input the new concatenated sequence into the second large model. The second large model uses a self-attention mask matrix to obtain information about the last word and the words before it in the new concatenated sequence, so as to predict the next output word in the third word sequence.

[0088] S704. Repeat the word concatenation step S702 and the prediction step S703 until the third word sequence is output.

[0089] In this embodiment, the length of the concatenated sequence is t+n. The entire concatenated sequence can be input into the basic large model. The semantic feature information of the last word t+n and the preceding words t+n-1, t+n-2, ..., up to the first word, is obtained through the self-attention mask matrix of the self-attention layer. After self-attention calculation, the features of the t+n word are used to predict the t+n+1 word, which can be understood as the first word of the third word sequence. After obtaining the t+n+1 word, it can be concatenated to the end of the original concatenated sequence to obtain a new concatenated sequence of length t+n+1. The new concatenated sequence with the first word concatenated is then input into the self-attention layer of the basic large model for calculation. After obtaining the next word, for example, the t+2 word, the above concatenation and prediction steps are repeated. This process can be repeated a certain number of times, for example, n times, to predict a third word sequence containing n words. The number of words in the third word sequence can be the same as or different from the number of words in the second word sequence.

[0090] According to the embodiments of this disclosure, in each calculation of the basic large model, the coherence between the output words is improved by concatenating the previously output word sequence with the previously input word sequence and then processing them together in the basic large model; when using the basic large model for calculation, the word predicted by the draft large model is referenced, which improves the calculation speed of the basic large model.

[0091] In one implementation, such as Figure 5 As shown, method 500 can be used to implement step S103 in the large model-based speculative decoding method 100. In one embodiment, method 500 includes: verifying based on the second lexical sequence and the third lexical sequence to obtain a speculative decoding result, further including:

[0092] S505. Compare the word values ​​at the current positions of the second word sequence and the third word sequence in turn;

[0093] S506. If the word values ​​at the current positions are the same, continue to compare the word values ​​at the next position after the current position of the second word sequence and the third word sequence.

[0094] In this embodiment, since the draft large model and the basic large model have essentially the same structure, the word segmentation method and vocabulary can also be consistent, resulting in a similar distribution of output results. Therefore, the word values ​​of the second and third word sequences can be compared at the same positions. When the word values ​​at the same positions in the second and third word sequences are the same, the word values ​​of the same portion can be output as the fourth word sequence. For example, the second word sequence includes {n1,n2,n3,n4,n5}, and the third word sequence includes {n1,n2,n6,n7,n8}. At the first position, word n1 in the second word sequence is the same as n1 in the third word sequence. At the second position, word n2 in the second word sequence is the same as n2 in the third word sequence. At the third position, word n3 in the second word sequence is different from n6 in the third word sequence. In this case, partially inferred decoding results word n1 and word n2 can be obtained.

[0095] According to embodiments of this disclosure, more accurate speculative decoding results can be obtained based on the verification results of the second and third word sequences.

[0096] In one implementation, such as Figure 5 As shown, method 500 can be used to implement step S103 in the large model-based speculative decoding method 100. In one embodiment, method 500 includes: verifying based on the second lexical sequence and the third lexical sequence to obtain a speculative decoding result, and further includes:

[0097] S507. If the word values ​​at the current position are inconsistent, obtain the word values ​​that are consistent with those at the current position.

[0098] S508. Repeat the steps of generating the visual lexical sequence (S401), the text lexical sequence (S402), the first lexical sequence (S403), the second lexical sequence (S501 and S502), and the third lexical sequence (S503 and S504).

[0099] S509. Continue verification from the next position after the current position of the regenerated second and third word sequences (e.g., treat the next position as the current position and return to execute S507) until all inference decoding results are obtained.

[0100] In this embodiment, the lexical values ​​at the same positions in the second and third lexical sequences can be compared sequentially, starting from the initial position. If the lexical values ​​at a certain position are different, the verification fails. All lexical values ​​before the position where verification failed can be obtained first. The next verification can continue from the position where verification failed. For example, if the regenerated second lexical sequence has 20 lexical elements, and the first 10 lexical elements have passed verification after the previous one, the regenerated second lexical sequence will start verification from the 11th lexical element. If 6 lexical elements pass verification this time, the next generated second lexical sequence will start verification from the 17th lexical element, and so on, until all 20 lexical elements pass verification. The speculative decoding result in this example can include the 20 verified lexical elements.

[0101] According to embodiments of this disclosure, by repeatedly performing speculative decoding and performing multiple inferences and verifications on the speculative decoding results, the accuracy of the speculative decoding results can be further improved.

[0102] The input (denoted as the query) to a multimodal large language model typically includes both text and image modalities. Text queries are usually questions about image content, and the model's output (denoted as the response) is the text of the answer. The model structure of a multimodal large language model (or multi-module large language model) can include an image encoder, a text encoder, a mapping module, and a large language model. The image encoder and text encoder encode the image query and text query into image lexical features and text lexical features, respectively. Due to the differences between image and text modalities, the mapping module can be used to transform the image lexical features into the text lexical feature space. The image lexical features and text lexical features are concatenated and fed into the large language model, which outputs the text answer. Some multimodal large language models can achieve high-resolution image input, multi-image and video content understanding, multimodal data accumulation, and multimodal evaluation datasets. Other multimodal large language models provide image modality-related capabilities, such as image content understanding, image creation, document image content parsing, and image reasoning. However, the inference performance of multimodal models needs improvement. Multimodal large language models output text results serially (called serial decoding), resulting in high output latency.

[0103] Speculative decoding is a parallel decoding technique used to accelerate inference in Large Language Models (LLMs). In speculative decoding, a high-performance draft model (or auxiliary model) pre-outputs the results, and then a more accurate base model (or main model) verifies these outputs in parallel to ensure quality. The combination of the draft and base models balances the accuracy of the base model with the performance of the draft model, accelerating the decoding process. However, the requirement for a matching draft model for the base model increases development costs. Furthermore, if the word segmentation methods, vocabulary sizes, or output distributions of the draft and base models are inconsistent, the verification calculations may fail, potentially even slowing down the decoding process.

[0104] The self-cooperative inference decoding technology for multimodal models disclosed in this embodiment can achieve self-cooperative speculative decoding without the need for additional training of draft models, thereby reducing the cost of using inference decoding, improving the verification effect in the speculative decoding process, and accelerating the decoding speed of multimodal models.

[0105] Figure 8 This is a schematic diagram of the structure of a self-cooperative inference decoding multimodal model acceleration framework according to an embodiment of the present disclosure, including: a draft model production module 801, a visual encoder 802, a text encoder 803, a verification module 804, a draft large language model 805, and a basic large language model 806. Figure 8 As shown, image A and the text "Please describe the animals in the picture" are processed by visual encoder 802 and text encoder 803 respectively to obtain visual lexical units and text lexical units. These visual and text lexical units are then input into the draft model production module 801, which generates a draft large language model based on the quantized basic large language model. The draft large language model outputs a draft lexical unit sequence. The basic large language model in the verification module 804 verifies the draft lexical unit sequence, obtaining and outputting the verified lexical unit sequence, for example, "There are two giraffes standing on the grass, and several zebras in the distance."

[0106] Draft model production module 801: Figure 9This is a flowchart illustrating a draft model generation method according to an embodiment of the present disclosure. The process may include: performing 4-bit model weight quantization on a basic large language model 901, and using the quantized model as a draft large language model 902. For example, the basic large language model is a structure of 32 stacked transformer blocks. The weights of each linear layer of the basic large language model are traversed. If the weights are tensors of shape [W, H], the maximum absolute value of the weight parameters is calculated on the H channel to obtain H sets of maximum values. The weight parameters are normalized on the H channel according to the maximum value, and then scaled to the 4-bit value range [-15, 15] to complete the quantization of the weight parameters.

[0107] Visual encoder 802 and text encoder 803: The visual encoder can convert images into image words, and the text encoder can convert text into text words. The concatenated image and text words can then be fed into a large language model.

[0108] Draft Big Language Model 805: The Draft Big Language Model is a quantized basic big language model. Its input is a sequence concatenated from visual and textual word units. The Draft Big Language Model outputs a draft word unit sequence through serial decoding. Specifically, assuming the input word unit sequence is of length t, a lower triangular Casual Attention Mask is used to ensure that the t-th word unit can acquire the semantic feature information of the (t-1), (t-2), (t-3), up to the first word unit—that is, the information of all word units before it. After performing the self-attention calculation, the features of the t-th word unit are used to predict the (t+1)-th word unit. After obtaining the (t+1)-th word unit, it is concatenated to the original input sequence to obtain a word unit sequence of length t+1. This sequence is then input into the Draft Big Language Model again to predict the (t+2)-th word unit, and this process is repeated n times to predict n word units.

[0109] Validation Module 804: The validation module can be composed of the basic large language model 806. For example, the basic large language model is a structure of 32 stacked transformer blocks. The input of the basic large language model is a sequence concatenated from visual lexical units, text lexical units, and draft lexical units. The output is the validated lexical sequence. After the sequence concatenated from visual lexical units, text lexical units, and draft lexical units enters the basic large language model, self-attention calculation is performed using a lower triangular style casual attention mask, which can output the lexical sequence to be validated. Assume the draft lexical sequence is t1, t2, t3, t4, and t5, and the lexical sequence to be validated is s1, s2, s3, s4, and s5. Starting from t1 and s1, they are compared sequentially. If the lexical values ​​are the same, the validation passes; otherwise, the validation terminates, and the validated lexical sequence is retained as the final output.

[0110] Figure 10 This is a schematic diagram of a large-model-based speculative decoding apparatus 1000 according to an embodiment of the present disclosure. In one embodiment, the apparatus includes:

[0111] The first input module 1001 is used to input the first word sequence into the first large model to obtain the second word sequence; wherein the first large model is obtained by quantization based on the second large model, and the first word sequence is obtained based on multimodal data;

[0112] The second input module 1002 is used to input the first word sequence and the second word sequence into the second large model to obtain the third word sequence.

[0113] The verification module 1003 is used to verify the second word sequence and the third word sequence to obtain the inferred decoding result.

[0114] Figure 11 This is a schematic diagram of a large-model-based speculative decoding device 1100 according to another embodiment of the present disclosure. The device 1100 includes a first input module 1101, a second input module 1102, and a verification module 1103. The functions of these modules are the same as those of the modules in the large-model-based speculative decoding device 1000 described in the previous embodiment. In one embodiment, the device further includes:

[0115] The quantization module 1104 is used to normalize and scale the weight parameters of the second largest model to obtain the quantized first largest model; wherein the first largest model includes the draft large model and the second largest model includes the base large model.

[0116] In one implementation, such as Figure 11 As shown, the quantization module 1104 includes:

[0117] The traversal submodule 11041 is used to traverse the weight parameters of each linear layer of the second large model; wherein the tensor formed by the weight parameters includes a first channel and a second channel.

[0118] The statistics submodule 11042 is used to calculate the maximum value of the absolute value of the weight parameter on the first channel, so as to obtain the maximum value of the first channel;

[0119] The normalization submodule 11043 is used to perform normalization calculation on the weight parameter according to the maximum value on the first channel to obtain the normalization result;

[0120] The scaling submodule 11044 is used to scale the normalization result according to the scaling ratio to obtain the quantized first large model.

[0121] In one implementation, such as Figure 11 As shown, the multimodal data includes images and text, and the device also includes:

[0122] The visual encoding module 1105 is used to input an image into a visual encoder to obtain a visual word sequence;

[0123] The text encoding module 1106 is used to input text into the text encoder to obtain a text word sequence;

[0124] The splicing module 1107 is used to splice the visual word sequence and the text word sequence to obtain the first word sequence.

[0125] In one implementation, such as Figure 11 As shown, the first input module 1101 includes:

[0126] The first input submodule 11011 is used to input the first word sequence into the first large model;

[0127] The first calculation submodule 11012 is used to perform self-attention calculation on the first word sequence using the self-attention mask matrix of the first large model to obtain the second word sequence.

[0128] In one embodiment, the first computing submodule 11012 is further configured to:

[0129] The first major model uses a self-attention mask matrix to obtain information about the last word and the words preceding it in the first word sequence, in order to predict an output word in the second word sequence.

[0130] The output word is concatenated to the last word of the first word sequence to obtain a new first word sequence;

[0131] The new first word sequence is input into the first large model. The first large model uses a self-attention mask matrix to obtain information about the last word and the words before it in the new first word sequence, so as to predict the next output word in the second word sequence.

[0132] Repeat the word concatenation and prediction steps until the second word sequence is output.

[0133] In one implementation, such as Figure 11 As shown, the second input module 1102 includes:

[0134] The second input submodule 11021 is used to input the concatenated sequence of the first word sequence and the second word sequence into the second large model.

[0135] The second computational submodule 11022 is used to perform self-attention computation on the concatenated sequence using the self-attention mask matrix through the second major model to obtain the third word sequence.

[0136] In one embodiment, the second computing submodule 11022 is further configured to:

[0137] The second major model uses a self-attention mask matrix to obtain information about the last word and the words preceding it in the concatenated sequence, in order to predict an output word in the third word sequence;

[0138] The output word is appended to the last word of the concatenation sequence to obtain a new concatenation sequence;

[0139] The new concatenated sequence is input into the second largest model, which uses a self-attention mask matrix to obtain information about the last word and the words preceding it in the new concatenated sequence in order to predict the next output word in the third word sequence.

[0140] Repeat the word concatenation and prediction steps until the third word sequence is output.

[0141] In one implementation, such as Figure 11 As shown, the verification module 1103 includes:

[0142] The comparison submodule 11031 is used to compare the word values ​​at the current positions of the second word sequence and the third word sequence in turn;

[0143] The comparison submodule 11031 is further configured to continue comparing the word values ​​at the next position of the current position of the second word sequence and the third word sequence if the word values ​​at the current position are consistent.

[0144] In one implementation, such as Figure 11 As shown, the verification module 1103 also includes:

[0145] The submodule 11032 is used to retrieve the lexical units with consistent values ​​before the current position when the lexical unit values ​​at the current position are inconsistent.

[0146] The regeneration submodule 11033 is used to re-execute the steps of generating the visual lexical sequence, the text lexical sequence, the first lexical sequence, the second lexical sequence, and the third lexical sequence;

[0147] The verification module 1103 is also used to continue verification from the next position of the current position in the regenerated second and third word sequences until all inference decoding results are obtained.

[0148] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0149] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0150] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0151] Figure 12 A schematic block diagram of an example electronic device 1200 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0152] like Figure 12 As shown, device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1202 or a computer program loaded from storage unit 1208 into random access memory (RAM) 1203. The RAM 1203 may also store various programs and data required for the operation of device 1200. The computing unit 1201, ROM 1202, and RAM 1203 are interconnected via bus 1204. Input / output (I / O) interface 1205 is also connected to bus 1204.

[0153] Multiple components in device 1200 are connected to I / O interface 1205, including: input unit 1206, such as keyboard, mouse, etc.; output unit 1207, such as various types of monitors, speakers, etc.; storage unit 1208, such as disk, optical disk, etc.; and communication unit 1209, such as network card, modem, wireless transceiver, etc. Communication unit 1209 allows device 1200 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0154] The computing unit 1201 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1201 performs the various methods and processes described above, such as the large model-based speculative decoding method. For example, in some embodiments, the large model-based speculative decoding method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1208. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1200 via ROM 1202 and / or communication unit 1209. When the computer program is loaded into RAM 1203 and executed by the computing unit 1201, one or more steps of the large model-based speculative decoding method described above may be performed. Alternatively, in other embodiments, computing unit 1201 may be configured by any other suitable means (e.g., by means of firmware) to perform a large-model-based speculative decoding method.

[0155] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0156] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0157] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0158] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0159] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0160] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0161] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0162] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A speculative decoding method based on a large model, comprising: The image is input into the visual encoder to obtain a visual word sequence; Input the text into the text encoder to obtain the text word sequence; The visual word sequence and the text word sequence are concatenated to obtain the first word sequence; The first word sequence is input into the first large model to obtain the second word sequence; wherein the first large model is obtained by quantization based on the second large model, and the first word sequence is obtained based on multimodal data; the length of the first word sequence is t, the length of the second word sequence is n, and the second word sequence includes the (t+1)th word to the (t+n)th word predicted by the first large model based on the first word sequence; the word sequence input into the first large model in this instance is concatenated with the word sequence predicted by the first large model in this instance to obtain the word sequence input into the first large model in the next instance; wherein the multimodal data includes images and text; The first word sequence and the second word sequence are input into the second large model to obtain the third word sequence; wherein, the length of the third word sequence is n, and the third word sequence includes the (t+n+1)th word to the (t+n+n)th word predicted by the second large model based on the concatenated sequence of the first word sequence and the second word sequence. The word sequence input into the second large model this time is concatenated with the word sequence predicted by the second large model this time as the word sequence input into the second large model next time. The inferred decoding result is obtained by verifying the second and third word sequences.

2. The method according to claim 1, further comprising: The weight parameters of the second large model are normalized and scaled to obtain the quantized first large model; wherein the first large model includes the draft large model and the second large model includes the base large model.

3. The method according to claim 2, wherein, The weight parameters of the second largest model are normalized and scaled to obtain the quantized first largest model, which includes: Iterate through the weight parameters of each linear layer of the second large model; wherein the tensor formed by the weight parameters includes a first channel and a second channel; The maximum value of the absolute value of the weight parameter is calculated on the first channel to obtain the maximum value of the first channel; The weight parameters are normalized according to the maximum value on the first channel to obtain the normalized result; The normalization result is scaled according to a scaling ratio to obtain the first largest quantized model.

4. The method according to any one of claims 1 to 3, wherein, Inputting the first word sequence into the first large model yields the second word sequence, which includes: Input the first word sequence into the first large model; The second word sequence is obtained by performing self-attention calculation on the first word sequence using the first large model with a self-attention mask matrix.

5. The method according to claim 4, wherein, The second word sequence is obtained by performing self-attention calculation on the first word sequence using a self-attention mask matrix in the first large model, including: The first large model uses a self-attention mask matrix to obtain information about the last word and the words preceding it in the first word sequence, in order to predict an output word in the second word sequence; The output word is concatenated to the last word of the first word sequence to obtain a new first word sequence; The new first word sequence is input into the first large model. The first large model uses a self-attention mask matrix to obtain information about the last word and the words before it in the new first word sequence, so as to predict the next output word in the second word sequence. Repeat the word concatenation and prediction steps until the second word sequence is output.

6. The method according to any one of claims 1 to 3, wherein, Inputting the first and second word sequences into the second large model yields the third word sequence, which includes: Input the concatenated sequence of the first word sequence and the second word sequence into the second large model; The third word sequence is obtained by performing self-attention calculation on the concatenated sequence using the second major model with a self-attention mask matrix.

7. The method according to claim 6, wherein, The third word sequence is obtained by performing self-attention calculation on the concatenated sequence using the second major model with a self-attention mask matrix, including: The second major model uses a self-attention mask matrix to obtain information about the last word and the words preceding it in the concatenated sequence, in order to predict an output word in the third word sequence; The output word is appended to the last word of the concatenation sequence to obtain a new concatenation sequence; The new concatenated sequence is input into the second large model, and the second large model uses a self-attention mask matrix to obtain information about the last word and the words before it in the new concatenated sequence, so as to predict the next output word in the third word sequence; Repeat the word concatenation and prediction steps until the third word sequence is output.

8. The method according to any one of claims 1 to 3, wherein, Verification is performed based on the second and third word sequences to obtain the inferred decoding result, including: The lexical values ​​at the current positions of the second lexical sequence and the third lexical sequence are compared sequentially. If the lexical values ​​at the current position are the same, continue comparing the lexical values ​​at the next position of the second lexical sequence and the third lexical sequence.

9. The method according to claim 8, wherein, Verification based on the second and third word sequences yields a speculative decoding result, which also includes: If the lexical values ​​at the current position are inconsistent, obtain the lexical values ​​that are consistent before the current position; Repeat the steps of generating the visual lexical sequence, the text lexical sequence, the first lexical sequence, the second lexical sequence, and the third lexical sequence; The verification continues from the next position after the current position in the regenerated second and third word sequences until all inference decoding results are obtained.

10. A speculative decoding device based on a large model, comprising: The visual encoding module is used to input images into the visual encoder to obtain a visual word sequence; The text encoding module is used to input text into the text encoder to obtain a text word sequence; The concatenation module is used to concatenate the visual word sequence and the text word sequence to obtain a first word sequence. The first input module is used to input the first word sequence into the first large model to obtain the second word sequence; wherein the first large model is obtained by quantization based on the second large model, and the first word sequence is obtained based on multimodal data; the length of the first word sequence is t, the length of the second word sequence is n, and the second word sequence includes the (t+1)th word to the (t+n)th word predicted by the first large model based on the first word sequence; the word sequence input to the first large model in this instance is concatenated with the word sequence predicted by the first large model in this instance to obtain the word sequence input to the first large model in the next instance; wherein the multimodal data includes images and text; The second input module is used to input the first word sequence and the second word sequence into the second large model to obtain a third word sequence; wherein, the length of the third word sequence is n, and the third word sequence includes the (t+n+1)th word to the (t+n+n)th word predicted by the second large model based on the concatenated sequence of the first word sequence and the second word sequence; the word sequence input into the second large model this time is concatenated with the word sequence predicted by the second large model this time as the word sequence input into the second large model next time; The verification module is used to verify the second lexical sequence and the third lexical sequence to obtain the inferred decoding result.

11. The apparatus of claim 10, further comprising: The quantization module is used to normalize and scale the weight parameters of the second large model to obtain the quantized first large model; wherein the first large model includes the draft large model and the second large model includes the base large model.

12. The apparatus according to claim 11, wherein, The quantization module includes: The traversal submodule is used to traverse the weight parameters of each linear layer of the second large model; wherein the tensor formed by the weight parameters includes a first channel and a second channel; The statistics submodule is used to calculate the maximum value of the absolute value of the weight parameter on the first channel, so as to obtain the maximum value of the first channel; The normalization submodule is used to perform normalization calculation on the weight parameters according to the maximum value on the first channel to obtain the normalization result; The scaling submodule is used to scale the normalization result according to a scaling ratio to obtain the quantized first large model.

13. The apparatus according to any one of claims 10 to 12, wherein, The first input module includes: The first input submodule is used to input the first word sequence into the first large model; The first calculation submodule is used to perform self-attention calculation on the first word sequence using the first large model with a self-attention mask matrix to obtain the second word sequence.

14. The apparatus according to claim 13, wherein, The first calculation submodule is further configured to: The first large model uses a self-attention mask matrix to obtain information about the last word and the words preceding it in the first word sequence, in order to predict an output word in the second word sequence; The output word is concatenated to the last word of the first word sequence to obtain a new first word sequence; The new first word sequence is input into the first large model. The first large model uses a self-attention mask matrix to obtain information about the last word and the words before it in the new first word sequence, so as to predict the next output word in the second word sequence. Repeat the word concatenation and prediction steps until the second word sequence is output.

15. The apparatus according to any one of claims 10 to 12, wherein, The second input module includes: The second input submodule is used to input the concatenated sequence of the first word sequence and the second word sequence into the second large model; The second calculation submodule is used to perform self-attention calculation on the concatenated sequence using the second large model with a self-attention mask matrix to obtain the third word sequence.

16. The apparatus according to claim 15, wherein, The second calculation submodule is also used for: The second major model uses a self-attention mask matrix to obtain information about the last word and the words preceding it in the concatenated sequence, in order to predict an output word in the third word sequence; The output word is appended to the last word of the concatenation sequence to obtain a new concatenation sequence; The new concatenated sequence is input into the second large model, and the second large model uses a self-attention mask matrix to obtain information about the last word and the words before it in the new concatenated sequence, so as to predict the next output word in the third word sequence; Repeat the word concatenation and prediction steps until the third word sequence is output.

17. The apparatus according to any one of claims 10 to 12, wherein, The verification module includes: The comparison submodule is used to sequentially compare the lexical values ​​at the current positions of the second lexical sequence and the third lexical sequence; The comparison submodule is further configured to, if the lexical values ​​at the current position are consistent, continue to compare the lexical values ​​at the next position of the second lexical sequence and the third lexical sequence at the current position.

18. The apparatus according to claim 17, wherein, The verification module further includes: The acquisition submodule is used to acquire the lexical values ​​that are consistent before the current position when the lexical values ​​at the current position are inconsistent. The regeneration submodule is used to re-execute the steps of generating the visual lexical sequence, the text lexical sequence, the first lexical sequence, the second lexical sequence, and the third lexical sequence; The verification module is further configured to continue verification from the next position of the current position in the regenerated second and third word sequences until all inference decoding results are obtained.

19. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-9.

21. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-9.

Images