Speculation decoding method and device based on large model, equipment and storage medium

Abstract

The invention provides a large-model-based speculation decoding method, device and equipment and a storage medium, and relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like. According to the specific implementation scheme, the method comprises the steps of inputting a first lexical element sequence into a first large model to obtain a second lexical element sequence; wherein the first large model is obtained through quantification according to a second large model, and the first lexical element sequence is obtained based on multi-modal data; inputting the first lexical element sequence and the second lexical element sequence into the second large model to obtain a third lexical element sequence; and performing verification according to the second lexical element sequence and the third lexical element sequence to obtain a speculative decoding result.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to technical fields such as computer vision, deep learning, and large models. Background Art

[0002] Speculative decoding is a parallel decoding technique, also known as inference decoding. Speculative decoding can achieve inference acceleration by increasing the parallelism of the language model calculation at each decoding step and reducing the total number of decoding steps. Speculative decoding usually uses two models, a large main model and a small auxiliary model. The auxiliary model first generates a sequence. Then, the main model verifies this sequence in a forward pass. Summary of the invention

[0003] The present disclosure provides a large model-based speculative decoding method, apparatus, device, and storage medium.

[0004] According to one aspect of the present disclosure, a large model-based speculative decoding method is provided, comprising:

[0005] Inputting the first word-gram sequence into the first large model to obtain a second word-gram sequence; wherein the first large model is quantized according to the second large model, and the first word-gram sequence is obtained based on multimodal data;

[0006] Inputting the first word sequence and the second word sequence into the second large model to obtain a third word sequence;

[0007] Verification is performed according to the second word-gram sequence and the third word-gram sequence to obtain a deduced decoding result.

[0008] According to another aspect of the present disclosure, a large model-based speculative decoding device is provided, comprising:

[0009] A first input module is used to input a first word sequence into a first large model to obtain a second word sequence; wherein the first large model is quantized according to the second large model, and the first word sequence is obtained based on multimodal data;

[0010] A second input module, used for inputting the first word sequence and the second word sequence into the second large model to obtain a third word sequence;

[0011] The verification module is used to perform verification according to the second word-unit sequence and the third word-unit sequence to obtain a deduced decoding result.

[0012] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0016] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0017] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0018] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0020] Figure 1 is a flowchart of a large model-based inference decoding method according to an embodiment of the present disclosure;

[0021] Figure 2 is a flowchart of a large model-based inference decoding method according to another embodiment of the present disclosure;

[0022] Figure 3 is a flowchart of a large model-based inference decoding method according to another embodiment of the present disclosure;

[0023] Figure 4 is a flowchart of a large model-based inference decoding method according to another embodiment of the present disclosure;

[0024] Figure 5 is a flowchart of a large model-based inference decoding method according to another embodiment of the present disclosure;

[0025] Figure 6 is a flowchart of a large model-based inference decoding method according to another embodiment of the present disclosure;

[0026] Figure 7 is a flowchart of a large model-based inference decoding method according to another embodiment of the present disclosure;

[0027] Figure 8It is a structural diagram of a self-collaborative inference decoding multimodal model acceleration framework according to an embodiment of the present disclosure;

[0028] Fig. 9 is a flowchart of a method for generating a draft model according to an embodiment of the present disclosure;

[0029] Fig.10 is a structural diagram of a large model-based speculative decoding device according to an embodiment of the present disclosure;

[0030] Fig.11 is a structural diagram of a large model-based speculative decoding device according to another embodiment of the present disclosure;

[0031] Fig.12 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0033] Figure 1 FIG. 1 is a flow chart of a method 100 for inferential decoding based on a large model according to an embodiment of the present disclosure. In one implementation, the method includes:

[0034] S101, inputting a first word-gram sequence into a first large model to obtain a second word-gram sequence; wherein the first large model is obtained by quantization according to the second large model, and the first word-gram sequence is obtained based on multimodal data;

[0035] S102, inputting the first word-gram sequence and the second word-gram sequence into the second large model to obtain a third word-gram sequence;

[0036] S103: Verify according to the second word-gram sequence and the third word-gram sequence to obtain an inferred decoding result.

[0037] In the disclosed embodiment, a first large model such as a draft large model and a second large model such as a basic large model can be used for speculative decoding. The draft large model can also be called a draft large language model, an auxiliary large model, an auxiliary model, a draft model, etc. The basic large model can also be a basic large language model, a benchmark large model, etc. The draft large model can be obtained by quantization based on the basic large model. Both the basic large model and the draft large model can be large language models (LLM). The first word unit sequence is input into the draft large model, and the second word unit sequence can be predicted. Among them, the first word unit sequence includes multiple input words, and the second word unit sequence includes multiple output words. The draft large model obtained by quantization according to the basic large model is a lightweight large language model. The quantization accuracy can be selected according to the needs, such as 4 bits, 8 bits, etc. Since the draft large model is quantized based on the basic large model, the word segmentation method, vocabulary, etc. of the two are consistent, and the output result distribution is close. Compared with the basic large model, the quantized draft large model can reduce the amount of calculation, obtain calculation results faster, and improve calculation efficiency.

[0038] In the disclosed embodiment, the first word-gram sequence and the second word-gram sequence are concatenated and input into the basic large model, and a third word-gram sequence can be predicted. The third word-gram sequence may include multiple output words. The third word-gram sequence output by the basic large model can be used to verify the second word-gram sequence to obtain a more accurate description text.

[0039] In the disclosed embodiment, the basic large model can be used to compare each word in the second word sequence and the third word sequence, and the inferred decoding result can be output according to the comparison result. For example, if each word in the second word sequence and the third word sequence is consistent, the second word sequence or the third word sequence can be output. If a word in the second word sequence and the third word sequence is inconsistent, the consistent partial word can be output.

[0040] According to the embodiments of the present disclosure, the word segmentation method and vocabulary of the draft large model quantized according to the basic large model are basically consistent with the basic large model, and the output results of the two are distributed similarly, so the input word units can be processed more quickly, and the subsequent word unit verification is easier to pass, so that accurate inferred decoding results can be quickly obtained.

[0041] Figure 2 2 is a flow chart of a method 200 for speculative decoding based on a large model according to another embodiment of the present disclosure. The method 200 may be used to implement the method 100 for speculative decoding based on a large model. In one implementation, the method 200 further includes:

[0042] S201. Normalize and scale the weight parameters of the second largest model to obtain the quantized first largest model; wherein the first largest model includes a draft large model, and the second largest model includes a basic large model.

[0043] In the embodiment of the present disclosure, the method of quantizing the basic large model may include: performing normalization calculation on the weight parameters of each layer of the basic large model to obtain the normalized result of each weight parameter. There are many ways of normalization calculation, such as minimum and maximum normalization, Z score normalization, maximum value normalization, etc. An example of maximum value normalization is as follows: a certain weight parameter is W i The maximum value of the weight parameter is Max(|W i |), then the normalization formula can be:

[0044]

[0045] Then, the normalized result can be scaled according to a certain quantization precision, such as 4 bits or 8 bits, to complete the quantization of the weight parameter. The scaling multiple is related to the quantization precision. An example of a 4-bit quantization formula can be:

[0046]

[0047] According to the embodiments of the present disclosure, the computational complexity of the quantized draft large model is lower than that of the basic large model, and the computational speed can be increased without losing almost any generation accuracy, and the word unit sequence can be quickly output. Since there is no need to retrain or screen the draft large model, the time for obtaining the draft large model can be saved.

[0048] Figure 3 3 is a flow chart of a method 300 for inferential decoding based on a large model according to another embodiment of the present disclosure. The method 300 can be used to implement step S201 in the method 200 for inferential decoding based on a large model. In one embodiment, the method 300 includes: normalizing and scaling the weight parameters of the second large model to obtain the quantized first large model, and further includes:

[0049] S301, traversing the weight parameters of each linear layer of the second largest model; wherein the tensor formed by the weight parameters includes a first channel and a second channel;

[0050] S302, counting the maximum absolute value of the weight parameter on the first channel to obtain the maximum value of the first channel;

[0051] S303, performing normalization calculation on the weight parameter on the first channel according to the maximum value to obtain a normalized result;

[0052] S304: Scale the normalized result according to the scaling ratio to obtain the quantized first large model.

[0053] In the disclosed embodiment, the basic large model may include multiple linear layers, and by traversing each linear layer of the basic large model, multiple weight parameters included in each linear layer can be obtained. The weight parameters of the basic large model can form a tensor, and the tensor can include two channels. The weight parameters can be quantized based on the two channels of the tensor. For example, the basic large language model is a structure of 32 layers of transformer blocks (transformer blocks) stacked, and the weight parameters of each linear layer of the model can be traversed to obtain a tensor [W, H]. Among them, W and H represent a channel respectively. The maximum value of the absolute value of the weight parameter can be obtained by statistics on the H channel, thereby obtaining the maximum value of the H channel (or called the H group maximum value). The weight parameter is normalized and calculated according to the maximum value on the H channel. An example of the formula for normalization calculation can be found in Formula 1 of the previous embodiment.

[0054] In the disclosed embodiment, the scaling ratio is related to the quantization accuracy, for example, the scaling ratio corresponding to 4-bit quantization is 15, the scaling ratio corresponding to 8-bit quantization is 255, and so on. After 4-bit quantization, the length of each weight parameter of the model is 4 bits; after 8-bit quantization, the length of the weight parameter of the model is 8 bits (1 byte). The scaling ratio can be set before quantization, or it can be determined based on the normalization result of the weight parameter. Through the scaling ratio, the weight parameters on each linear layer of the normalized basic large model can be scaled to the coding length corresponding to the quantization accuracy, thereby simplifying the basic large model and obtaining a draft large model. For example, the normalized weight parameter is scaled to the value range [-15,15] of the 4-bit (4-bit) value to complete the quantization of the weight parameter, see the above formula 2.

[0055] According to the embodiments of the present disclosure, the weight parameters of each linear layer in the basic large model can be normalized and scaled to obtain a quantized draft large model, thereby improving the inferred decoding speed.

[0056] Figure 4 4 is a flow chart of a method 400 for speculative decoding based on a large model according to another embodiment of the present disclosure. The method 400 can be used to implement the method 100 for speculative decoding based on a large model. The multimodal data includes images and text. In one embodiment, the method 400 further includes:

[0057] S401, inputting the image into a visual encoder to obtain a visual word sequence;

[0058] S402, inputting the text into a text encoder to obtain a text word sequence;

[0059] S403: Concatenate the visual word-gram sequence and the text word-gram sequence to obtain the first word-gram sequence.

[0060] In the disclosed embodiments, multimodal data may be speculatively decoded. Images in the multimodal data may be input into a visual encoder, and a visual word-meta sequence may be obtained after encoding. Text in the multimodal data may be input into a text encoder, and a text word-meta sequence may be obtained after encoding. The visual encoder may exist independently of the basic large model, or may be a part of the basic large model. The text encoder may exist independently of the basic large model, or may be a part of the basic large model.

[0061] In the disclosed embodiment, the visual word-gram sequence and the text word-gram sequence can be simply spliced ​​to obtain a multimodal first word-gram sequence; the visual word-gram sequence and the text word-gram sequence can also be spliced ​​by inputting the splicing layer of the draft large model through different attention heads to obtain a multimodal first word-gram sequence. Then, the multimodal first word-gram sequence can be input into the draft large model for processing to obtain a second word-gram sequence.

[0062] According to an embodiment of the present disclosure, an image word sequence and a text word sequence can be obtained by encoding multimodal data, and then concatenating them to obtain a multimodal first word sequence, which is then processed collaboratively by the draft large model and the basic large model to speed up the speculative decoding process of the multimodal data.

[0063] Figure 5 is a flow chart of a method 500 for speculative decoding based on a large model according to another embodiment of the present disclosure. The method 500 can be used to implement step S101 in the method 100 for speculative decoding based on a large model. In one embodiment, the method 500 includes: inputting a first word-unit sequence into a first large model to obtain a second word-unit sequence, and further includes:

[0064] S501, inputting the first word-gram sequence into the first large model;

[0065] S502. Perform self-attention calculation on the first word-unit sequence using a self-attention mask matrix through the first large model to obtain the second word-unit sequence.

[0066] In the disclosed embodiment, the draft large model may include a self-attention layer, in which the feature information of the last word of the input sequence and all previous words can be obtained through the self-attention mask matrix, and then the next word of the second word sequence can be predicted.

[0067] The Casual Attention Mask matrix may include a lower triangular mask matrix. An example of a lower triangular mask matrix is ​​as follows:

[0068]

[0069] The size of the matrix is ​​n×n, where n can be determined according to the number of words in the first word sequence. For example, for a first word sequence A with 3 words, a 3×3 lower triangular mask matrix can be used for masking; for a first word sequence B with 20 words, a 20×20 lower triangular mask matrix can be used for masking.

[0070] According to the embodiment of the present disclosure, the first word-unit sequence can be masked and self-attention calculated using the draft large model, so that the second word-unit sequence can be obtained quickly and accurately.

[0071] Figure 6 FIG. 6 is a flow chart of a large model-based inference decoding method 600 according to another embodiment of the present disclosure. Figure 6 As shown, S502 uses the first large model to perform self-attention calculation on the first word-unit sequence using the self-attention mask matrix to obtain the second word-unit sequence, further comprising:

[0072] S601, using the first large model to use a self-attention mask matrix to obtain information about the last word in the first word sequence and the words before it, so as to predict an output word in the second word sequence;

[0073] S602, concatenating the output word unit to the last word unit of the first word unit sequence to obtain a new first word unit sequence;

[0074] S603, inputting the new first word-gram sequence into the first large model, and obtaining information of the last word-gram and the word-gram before it in the new first word-gram sequence by using the self-attention mask matrix through the first large model to predict the next output word-gram in the second word-gram sequence;

[0075] S604, repeatedly executing the word unit concatenation step S602 and the prediction step S603 until the second word unit sequence is output. The number of repetitions can be flexibly set according to the requirements, and can also be set according to the length of the first word unit sequence and other related parameters.

[0076] In the disclosed embodiment, the length of the first word-unit sequence is t, and the first word-unit sequence can be input into the draft large model in its entirety, and the semantic feature information of the last word-unit t in the first word-unit sequence and the words t-1, t-2, etc. before it in the first word-unit sequence can be obtained through the self-attention mask matrix of the self-attention layer, until the first word-unit. After the self-attention calculation is performed, the t-th word-unit feature is used to predict the t+1-th word-unit, and the t+1-th word-unit can be understood as the first word-unit of the second word-unit sequence. After obtaining the t+1-th word-unit, it can be spliced ​​to the end of the original first word-unit sequence to obtain a new first word-unit sequence of t+1 length. Then, the new first word-unit sequence spliced ​​with the first word-unit is input into the self-attention layer of the draft large model for calculation, and after obtaining the next word-unit, such as the t+2-th word-unit, the above-mentioned splicing step and prediction step are executed again in a loop. It can be executed in a loop for a certain number of times, such as n times, to predict a second word-unit sequence including n words.

[0077] According to the embodiment of the present disclosure, the coherence between the predicted words each time is improved by concatenating the last predicted word with the original input word sequence and processing them together in the draft large model.

[0078] In one embodiment, Figure 5 As shown, the method 500 can be used to implement step S102 in the large model-based inference decoding method 100. In one embodiment, the method 500 further includes: inputting the first word-gram sequence and the second word-gram sequence into the second large model to obtain a third word-gram sequence, further including:

[0079] S503, inputting a concatenated sequence obtained by concatenating the first word sequence and the second word sequence into the second large model;

[0080] S504. Perform self-attention calculation on the concatenated sequence using a self-attention mask matrix through the second largest model to obtain the third word-unit sequence.

[0081] In the disclosed embodiment, the first word-gram sequence and the second word-gram sequence can be spliced ​​by directly splicing, or the first word-gram sequence and the second word-gram sequence can be input into the splicing layer of the basic large model for splicing. For example, the first word-gram sequence [image word-gram A, image word-gram B, text word-gram C] is spliced ​​with the second word-gram sequence [prediction word-gram D, prediction word-gram E, prediction word-gram F] to obtain a spliced ​​sequence [image word-gram A, image word-gram B, text word-gram C, prediction word-gram D, prediction word-gram E, prediction word-gram F]. After obtaining the spliced ​​sequence, the spliced ​​sequence can be input into the basic large model for self-attention mask calculation to obtain a third word-gram sequence. The word-grams in the third word-gram sequence can be called verification word-grams.

[0082] In the disclosed embodiment, the concatenated word-units can be masked using a self-attention mask matrix such as a lower triangular mask matrix using a basic large model, and the masked result can be self-attention calculated using the basic large model to obtain a third word-unit sequence. The specific method of S504 predicting the third word-unit sequence is similar to the principle of the specific method of S502 predicting the second word-unit sequence, and reference can be made to the relevant description of S502. Among them, the size of the self-attention mask matrix used for the concatenated word-units can be determined according to the number of word-units in the concatenated word-unit sequence.

[0083] According to the embodiments of the present disclosure, the concatenation sequence of the first word-gram sequence and the second word-gram sequence can be masked and self-attention calculated through the basic large model to quickly obtain a more accurate third word-gram sequence, and the second word-gram sequence can be verified through the third word-gram sequence, thereby improving the accuracy of the inferred decoding result.

[0084] Figure 7 FIG. 1 is a flow chart of a method for speculative decoding based on a large model according to another embodiment of the present disclosure. In one implementation, Figure 7 As shown, S504 uses the second largest model to perform self-attention calculation on the concatenated sequence using the self-attention mask matrix to obtain the third word-unit sequence, further comprising:

[0085] S701, using the second largest model to obtain information about the last word and the word before it in the concatenated sequence using a self-attention mask matrix to predict an output word in the third word sequence;

[0086] S702, concatenating the output word unit to the last word unit of the concatenated sequence to obtain a new concatenated sequence;

[0087] S703, inputting the new concatenated sequence into the second largest model, and obtaining information of the last word and the word before it in the new concatenated sequence by using the self-attention mask matrix through the second largest model to predict the next output word in the third word sequence;

[0088] S704, repeatedly executing the word unit concatenation step S702 and the prediction step S703 until the third word unit sequence is output.

[0089] In the disclosed embodiment, the length of the spliced ​​sequence is t+n, and the spliced ​​sequence can be input into the basic large model in its entirety, and the semantic feature information of the last word t+n in the spliced ​​sequence and the words t+n-1, t+n-2, etc. before it, can be obtained through the self-attention mask matrix of the self-attention layer, until the first word. After the self-attention calculation is performed, the t+n+1th word is predicted using the t+nth word feature, and the t+n+1th word can be understood as the first word of the third word sequence. After obtaining the t+n+1th word, it can be spliced ​​to the end of the original spliced ​​sequence to obtain a new spliced ​​sequence of t+n+1 length. Then, the new spliced ​​sequence spliced ​​with the first word is input into the self-attention layer of the basic large model for calculation, and after obtaining the next word, such as the t+2th word, the above splicing step and prediction step are executed again in a loop. It can be executed a certain number of times, such as n times, to predict a third word sequence including n words. The number of words in the third word sequence and the second word sequence can be the same or different.

[0090] According to the disclosed embodiment, each time the basic large model is calculated, the word element output last time is concatenated with the word element sequence input last time, and then processed together in the basic large model, thereby improving the coherence between the word elements output each time; when using the basic large model for calculation, the word elements predicted by the draft large model are referred to, thereby improving the calculation speed of the basic large model.

[0091] In one embodiment, Figure 5 As shown, the method 500 can be used to implement step S103 in the large model-based inference decoding method 100. In one embodiment, the method 500 includes: performing verification according to the second word-unit sequence and the third word-unit sequence to obtain the inference decoding result, and further includes:

[0092] S505, comparing the word value of the current position of the second word-gram sequence and the word-gram value of the third word-gram sequence in sequence;

[0093] S506: When the word-gram values ​​at the current position are consistent, continue to compare the word-gram values ​​at the next position of the current position of the second word-gram sequence and the third word-gram sequence.

[0094] In the disclosed embodiment, since the structures of the draft large model and the basic large model are basically the same, the word segmentation method and the vocabulary can also be consistent, and the output results are distributed similarly. Therefore, the respective word values ​​can be compared at the same position of the second word-unit sequence and the third word-unit sequence. When the word-unit values ​​at the same position of the second word-unit sequence and the third word-unit sequence are the same, the word-unit values ​​of the same part can be output as the fourth word-unit sequence. For example, the second word-unit sequence includes {n1, n2, n3, n4, n5}, and the third word-unit sequence includes {n1, n2, n6, n7, n8}. At the first position, the word n1 of the second word-unit sequence is the same as n1 of the third word-unit sequence. At the second position, the word n2 of the second word-unit sequence is the same as n2 of the third word-unit sequence. At the third position, the word n3 of the second word-unit sequence is different from n6 of the third word-unit sequence. In this case, partial inference decoding results word n1 and word n2 can be obtained.

[0095] According to the embodiment of the present disclosure, a more accurate inferred decoding result can be obtained based on the verification results of the second word-gram sequence and the third word-gram sequence.

[0096] In one embodiment, Figure 5 As shown, the method 500 can be used to implement step S103 in the large model-based inference decoding method 100. In one embodiment, the method 500 includes: performing verification according to the second word-unit sequence and the third word-unit sequence to obtain the inference decoding result, and further includes:

[0097] S507: If the word-unit value at the current position is inconsistent, obtain the word-unit with the same word-unit value before the current position;

[0098] S508, re-execute the steps of generating the visual word-gram sequence (S401), the text word-gram sequence (S402), the first word-gram sequence (S403), the second word-gram sequence (S501 and S502), and the third word-gram sequence (S503 and S504);

[0099] S509 , continue verification from the next position of the current position of the regenerated second word-gram sequence and the third word-gram sequence (for example, regard the next position as the current position and return to execute S507 ) until all inference decoding results are obtained.

[0100] In the disclosed embodiment, the word values ​​at the same position of the second word-element sequence and the third word-element sequence can be compared in sequence starting from the initial position. If the word-element values ​​of the two are different at a certain position, the verification fails. All word-element before the position where the verification fails can be obtained first. The verification can be continued from the position where the verification fails next time. For example, the regenerated second word-element sequence has 20 word-element. After the last verification, the first 10 word-element have been verified. The second word-element generated again will be verified from the 11th word-element. If 6 word-element are verified again this time, the second word-element generated next time will be verified from the 17th word-element, and so on, until all 20 word-element are verified. The inferred decoding result in this example may include 20 word-element that have passed the verification.

[0101] According to the embodiment of the present disclosure, by cyclically executing speculative decoding and performing multiple inferences and verifications on the speculative decoding results, the accuracy of the speculative decoding results can be further improved.

[0102] The input of a multimodal large model (referred to as a query) usually includes two modalities: text and image. A text query is usually a question about the content of an image, and the output of the model (referred to as a response) is the text of the answer. The model structure of a multimodal large model (or a multi-module large language model) may include an image encoder, a text encoder, a mapping module, and a large language model. The image encoder and the text encoder encode the image query and the text query into image word features and text word features, respectively. Due to the difference between image and text modalities, a mapping module can be used to transform the image word features into the text word feature space. The image word features and the text word features are concatenated and sent to the large language model, which outputs the text answer result. Some multimodal large language models can realize functions such as high-resolution image input, multi-image and video content understanding, multimodal data accumulation, and multimodal evaluation data sets. Other multimodal large language models provide image modality-related capabilities, such as image content understanding, image creation, document image internal analysis, image reasoning, etc. However, the inference performance of multimodal models needs to be improved. Multimodal large language models output text results in a serial manner (called serial decoding), and the output delay is high.

[0103] Speculative Decoding is a parallel decoding technology that can be used to accelerate the reasoning of large language models (LLM). In speculative decoding, a high-performance draft model (or auxiliary model) can be used to output the results in advance, and then a basic model (or main model) with better accuracy can be used to verify the output results in parallel to ensure the quality of the output results. The combination of the draft model and the basic model can take into account the accuracy of the basic model and the performance of the draft model, and can accelerate the decoding process. However, since the basic model needs to have a matching draft model, the R&D cost is increased; if the word segmentation method and vocabulary size of the draft model and the basic model are inconsistent, or the output result distribution of the draft model and the basic model are not close, the above verification calculation may not pass, or even reduce the decoding speed.

[0104] The self-collaborative inference decoding technology of the multimodal model of the disclosed embodiment can realize self-collaborative inference decoding without the need for additional training of the draft model, thereby reducing the cost of using inference decoding, improving the verification effect during the inference decoding process, and accelerating the decoding speed of the multimodal model.

[0105] Figure 8 8 is a schematic diagram of a self-cooperative inference decoding multimodal model acceleration framework according to an embodiment of the present disclosure, including: a draft model generation module 801, a visual encoder 802, a text encoder 803, a verification module 804, a draft large language model 805, and a basic large language model 806. Figure 8 As shown, after image A and text "Please introduce the animals in the picture" are processed by visual encoder 802 and text encoder 803 respectively, visual word units and text word units are obtained. The visual word units and text word units are input into the draft model production module 801 based on the draft large language model quantized by the basic large language model. The draft large language model outputs a draft word unit sequence. The basic large language model in the verification module 804 verifies the draft word unit sequence, obtains the verified word unit sequence and outputs it, for example, "There are two giraffes standing on the grass, and there are several zebras in the distance."

[0106] Draft Model Production Module 801: Fig. 9It is a flow chart of a draft model generation method according to an embodiment of the present disclosure, and the flow may include: implementing 4-bit model weight quantization on the basic large language model 901, and using the quantized model as the draft large language model 902. For example, the basic large language model is a structure of 32 layers of transformer blocks stacked, and the weights of each linear layer of the basic large language model are traversed. If the weight is a tensor of shape [W, H], the maximum value of the absolute value of the weight parameter is counted on the H channel to obtain the H group maximum value. The weight parameter is normalized and calculated according to the maximum value on the H channel, and then scaled to the value range of the 4-bit value [-15,15] to complete the quantization of the weight parameter.

[0107] Visual encoder 802 and text encoder 803: The visual encoder can convert images into image words, and the text encoder can convert text into text words. The image and text words can be concatenated and sent to the large language model.

[0108] Draft large language model 805: The draft large language model is a quantized basic large language model, and its input is a sequence of visual words and text words. The draft large language model can output a draft word sequence by serial decoding. Specifically, assuming that the length of the input word sequence is t, by setting a lower triangular style self-attention mask (CasualAttention Mask), it is ensured that the tth word can obtain the semantic feature information of the t-1th, t-2th, t-3th, and up to the 1st word, that is, the information of all words before it. After the self-attention calculation, the tth word feature is used to predict the t+1th word. After obtaining the t+1th word, it is spliced ​​to the original input sequence to obtain a word sequence of length t+1. The draft large language model is input again to predict the t+2th word, and n words are predicted in this way by looping n times.

[0109] Verification module 804: The verification module can be composed of a basic large language model 806. For example, the basic large language model is a structure of 32 layers of transformer blocks stacked together. The input of the basic large language model is a sequence of visual words, text words and draft words. The output is a verified word sequence. After the sequence of visual words, text words and draft words enters the basic large language model, a self-attention mask (Casual Attention Mask) in a lower triangle style is used to perform self-attention calculation, and the word sequence to be verified can be output. Assume that the draft word sequence is t1, t2, t3, t4 and t5, and the word sequence to be verified is s1, s2, s3, s4 and s5. Starting from t1 and s1, compare them one by one. If the word value is the same, the verification is passed, otherwise the verification is terminated and the verified word sequence is retained as the final output.

[0110] Fig.10 1 is a schematic diagram of a large model-based inference decoding device 1000 according to an embodiment of the present disclosure. In one implementation, the device includes:

[0111] A first input module 1001 is used to input a first word-gram sequence into a first large model to obtain a second word-gram sequence; wherein the first large model is obtained by quantization according to the second large model, and the first word-gram sequence is obtained based on multimodal data;

[0112] A second input module 1002, configured to input the first word sequence and the second word sequence into the second large model to obtain a third word sequence;

[0113] The verification module 1003 is used to perform verification according to the second word-gram sequence and the third word-gram sequence to obtain a deduced decoding result.

[0114] Fig.11 1 is a schematic diagram of a large model-based inference decoding device 1100 according to another embodiment of the present disclosure. The device 1100 includes: a first input module 1101, a second input module 1102 and a verification module 1103. The functions of the above modules can refer to the functions of the modules of the large model-based inference decoding device 1000 in the above embodiment. In one embodiment, the device further includes:

[0115] The quantization module 1104 is used to normalize and scale the weight parameters of the second large model to obtain the quantized first large model; wherein the first large model includes a draft large model, and the second large model includes a basic large model.

[0116] In one embodiment, Fig.11 As shown, the quantization module 1104 includes:

[0117] A traversal submodule 11041 is used to traverse the weight parameters of each linear layer of the second large model; wherein the tensor formed by the weight parameters includes a first channel and a second channel;

[0118] A statistical submodule 11042 is used to count the maximum absolute value of the weight parameter on the first channel to obtain the maximum value of the first channel;

[0119] A normalization submodule 11043 is used to perform normalization calculation on the weight parameter on the first channel according to the maximum value to obtain a normalized result;

[0120] The scaling submodule 11044 is used to scale the normalized result according to a scaling ratio to obtain the quantized first large model.

[0121] In one embodiment, Fig.11 As shown, the multimodal data includes images and texts, and the device also includes:

[0122] A visual encoding module 1105 is used to input the image into a visual encoder to obtain a visual word sequence;

[0123] The text encoding module 1106 is used to input the text into a text encoder to obtain a text word sequence;

[0124] The concatenation module 1107 is configured to concatenate the visual word-gram sequence and the text word-gram sequence to obtain the first word-gram sequence.

[0125] In one embodiment, Fig.11 As shown, the first input module 1101 includes:

[0126] A first input submodule 11011, used for inputting the first word sequence into the first large model;

[0127] The first calculation submodule 11012 is used to perform self-attention calculation on the first word sequence using the self-attention mask matrix through the first large model to obtain the second word sequence.

[0128] In one implementation, the first calculation submodule 11012 is further configured to:

[0129] Using the first large model and a self-attention mask matrix to obtain information about the last word and the word before it in the first word sequence, so as to predict an output word in the second word sequence;

[0130] splicing the output word unit to the last word unit of the first word unit sequence to obtain a new first word unit sequence;

[0131] Input the new first word-unit sequence into the first large model, and obtain information of the last word-unit and the word-unit before it in the new first word-unit sequence by using the self-attention mask matrix through the first large model to predict the next output word-unit in the second word-unit sequence;

[0132] The word unit concatenation step and the prediction step are repeatedly performed until the second word unit sequence is output.

[0133] In one embodiment, Fig.11 As shown, the second input module 1102 includes:

[0134] The second input submodule 11021 is used to input a concatenated sequence of the first word sequence and the second word sequence into the second large model;

[0135] The second calculation submodule 11022 is used to perform self-attention calculation on the concatenated sequence using the self-attention mask matrix through the second large model to obtain the third word sequence.

[0136] In one implementation, the second calculation submodule 11022 is further used to:

[0137] Using the second largest model to obtain information about the last word and the word before it in the concatenated sequence, so as to predict an output word in the third word sequence;

[0138] The output word is concatenated to the last word of the concatenated sequence to obtain a new concatenated sequence;

[0139] The new concatenated sequence is input into the second largest model, and the information of the last word and the word before it in the new concatenated sequence is obtained by the second largest model using the self-attention mask matrix to predict the next output word in the third word sequence;

[0140] The word unit concatenation step and the prediction step are repeatedly performed until the third word unit sequence is output.

[0141] In one embodiment, Fig.11 As shown, the verification module 1103 includes:

[0142] A comparison submodule 11031 is used to compare the word value of the current position of the second word sequence and the third word sequence in sequence;

[0143] The comparison submodule 11031 is further configured to, when the word value at the current position is consistent, continue to compare the word value at the next position of the current position of the second word sequence and the third word sequence.

[0144] In one embodiment, Fig.11 As shown, the verification module 1103 further includes:

[0145] The acquisition submodule 11032 is used to acquire the word element with the same word element value before the current position when the word element value at the current position is inconsistent;

[0146] A regeneration submodule 11033, for re-executing the steps of generating a visual word-gram sequence, a text word-gram sequence, a first word-gram sequence, a second word-gram sequence, and a third word-gram sequence;

[0147] The verification module 1103 is further configured to continue verification from the next position of the regenerated second word-gram sequence and the third word-gram sequence after the current position until all inference decoding results are obtained.

[0148] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0149] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0150] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0151] Fig.12 A schematic block diagram of an example electronic device 1200 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0152] like Fig.12 As shown, the device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a random access memory (RAM) 1203. In the RAM 1203, various programs and data required for the operation of the device 1200 can also be stored. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0153] A number of components in the device 1200 are connected to the I / O interface 1205, including: an input unit 1206, such as a keyboard, a mouse, etc.; an output unit 1207, such as various types of displays, speakers, etc.; a storage unit 1208, such as a disk, an optical disk, etc.; and a communication unit 1209, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1209 allows the device 1200 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0154] The computing unit 1201 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1201 performs the various methods and processes described above, such as a speculative decoding method based on a large model. For example, in some embodiments, the speculative decoding method based on a large model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1208. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded into the RAM 1203 and executed by the computing unit 1201, one or more steps of the speculative decoding method based on the large model described above may be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured to perform the large model-based speculative decoding method in any other appropriate manner (eg, by means of firmware).

[0155] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0156] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0157] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0158] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0159] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0160] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0161] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0162] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A large model-based inference decoding method, comprising: Inputting the first word-gram sequence into the first large model to obtain a second word-gram sequence; wherein the first large model is quantized according to the second large model, and the first word-gram sequence is obtained based on multimodal data; Inputting the first word-gram sequence and the second word-gram sequence into the second large model to obtain a third word-gram sequence; Verification is performed based on the second word-gram sequence and the third word-gram sequence to obtain a deduced decoding result.

2. The method according to claim 1, further comprising: The weight parameters of the second large model are normalized and scaled to obtain the quantized first large model; wherein the first large model includes a draft large model, and the second large model includes a basic large model.

3. The method according to claim 2, wherein: Normalizing and scaling the weight parameters of the second largest model to obtain a quantized first largest model, including: Traversing the weight parameters of each linear layer of the second large model; wherein the tensor formed by the weight parameters includes a first channel and a second channel; Counting the maximum absolute value of the weight parameter on the first channel to obtain the maximum value of the first channel; Performing normalization calculation on the weight parameter on the first channel according to the maximum value to obtain a normalized result; The normalized result is scaled according to a scaling ratio to obtain the quantized first large model.

4. The method according to any one of claims 1 to 3, wherein: The multimodal data includes images and texts, and the method further includes: Input the image into the visual encoder to obtain a visual word sequence; Input the text into the text encoder to obtain a text word sequence; The visual word-gram sequence and the text word-gram sequence are concatenated to obtain the first word-gram sequence.

5. The method according to any one of claims 1 to 4, wherein: Inputting the first word sequence into the first large model to obtain a second word sequence includes: Inputting the first word sequence into the first large model; The first large model uses a self-attention mask matrix to perform self-attention calculation on the first word-unit sequence to obtain the second word-unit sequence.

6. The method according to claim 5, wherein: The self-attention calculation is performed on the first word-unit sequence by using the self-attention mask matrix through the first large model to obtain the second word-unit sequence, including: Using the first large model and the self-attention mask matrix to obtain information about the last word and the word before it in the first word sequence, so as to predict an output word in the second word sequence; splicing the output word unit to the last word unit of the first word unit sequence to obtain a new first word unit sequence; Inputting the new first word-unit sequence into the first large model, and obtaining information of the last word-unit and the word-unit before it in the new first word-unit sequence by using the self-attention mask matrix of the first large model to predict the next output word-unit in the second word-unit sequence; The word unit concatenation step and the prediction step are repeatedly performed until the second word unit sequence is output.

7. The method according to any one of claims 1 to 6, wherein: Inputting the first word-gram sequence and the second word-gram sequence into the second large model to obtain a third word-gram sequence includes: Inputting a concatenated sequence obtained by concatenating the first word sequence and the second word sequence into the second large model; The second largest model uses a self-attention mask matrix to perform self-attention calculation on the concatenated sequence to obtain the third word sequence.

8. The method according to claim 7, wherein: The self-attention calculation is performed on the concatenated sequence by using the self-attention mask matrix through the second largest model to obtain the third word-unit sequence, including: Using the second largest model and the self-attention mask matrix to obtain information about the last word and the word before it in the concatenated sequence, so as to predict an output word in the third word sequence; splicing the output word unit to the last word unit of the splicing sequence to obtain a new splicing sequence; Inputting the new concatenated sequence into the second large model, and obtaining information of the last word unit and the word units before it in the new concatenated sequence by using the self-attention mask matrix of the second large model to predict the next output word unit in the third word unit sequence; The word unit concatenation step and the prediction step are repeatedly performed until the third word unit sequence is output.

9. The method according to any one of claims 1 to 8, wherein: Verifying according to the second word-gram sequence and the third word-gram sequence to obtain an inferred decoding result includes: Comparing the word value of the current position of the second word-unit sequence and the third word-unit sequence in sequence; When the word-gram values ​​at the current position are consistent, the word-gram values ​​at the position next to the current position of the second word-gram sequence and the third word-gram sequence are continuously compared.

10. The method according to claim 9, wherein: Verifying according to the second word-gram sequence and the third word-gram sequence to obtain an inferred decoding result also includes: When the word-unit value of the current position is inconsistent, obtain the word-unit with the same word-unit value before the current position; Re-execute the steps of generating a visual word-gram sequence, a text word-gram sequence, a first word-gram sequence, a second word-gram sequence, and a third word-gram sequence; Verification continues from the next position of the current position of the regenerated second word-gram sequence and the third word-gram sequence until all inference decoding results are obtained.

11. A large model-based speculative decoding device, comprising: A first input module, configured to input a first word-gram sequence into a first large model to obtain a second word-gram sequence; wherein the first large model is quantized according to the second large model, and the first word-gram sequence is obtained based on multimodal data; A second input module, configured to input the first word-gram sequence and the second word-gram sequence into the second large model to obtain a third word-gram sequence; A verification module is used to perform verification according to the second word-gram sequence and the third word-gram sequence to obtain an inferred decoding result.

12. The apparatus according to claim 11, further comprising: A quantization module is used to normalize and scale the weight parameters of the second large model to obtain the quantized first large model; wherein the first large model includes a draft large model, and the second large model includes a basic large model.

13. The device according to claim 12, wherein: The quantization module comprises: A traversal submodule, used for traversing the weight parameters of each linear layer of the second large model; wherein the tensor formed by the weight parameters includes a first channel and a second channel; A statistical submodule, used for counting the maximum absolute value of the weight parameter on the first channel to obtain the maximum value of the first channel; A normalization submodule, configured to perform normalization calculation on the weight parameter according to the maximum value on the first channel to obtain a normalized result; The scaling submodule is used to scale the normalized result according to a scaling ratio to obtain the quantized first large model.

14. The device according to any one of claims 11 to 13, wherein: The multimodal data includes images and texts, and the method further includes: A visual encoding module, which is used to input the image into a visual encoder to obtain a visual word sequence; A text encoding module, used for inputting text into a text encoder to obtain a text word sequence; A concatenation module is used to concatenate the visual word-gram sequence and the text word-gram sequence to obtain the first word-gram sequence.

15. The device according to any one of claims 11 to 14, wherein: The first input module comprises: A first input submodule, used for inputting the first word-unit sequence into the first large model; The first calculation submodule is used to perform self-attention calculation on the first word-unit sequence using the self-attention mask matrix through the first large model to obtain the second word-unit sequence.

16. The device according to claim 15, wherein: The first computing submodule is further configured to: Using the first large model and the self-attention mask matrix to obtain information about the last word and the word before it in the first word sequence, so as to predict an output word in the second word sequence; splicing the output word unit to the last word unit of the first word unit sequence to obtain a new first word unit sequence; Inputting the new first word-unit sequence into the first large model, and obtaining information of the last word-unit and the word-unit before it in the new first word-unit sequence by using the self-attention mask matrix of the first large model to predict the next output word-unit in the second word-unit sequence; The word unit concatenation step and the prediction step are repeatedly performed until the second word unit sequence is output.

17. The device according to any one of claims 11 to 16, wherein: The second input module comprises: A second input submodule, configured to input a concatenated sequence of the first word sequence and the second word sequence into the second large model; The second calculation submodule is used to perform self-attention calculation on the spliced ​​sequence using the second large model and a self-attention mask matrix to obtain the third word sequence.

18. The device according to claim 17, wherein: The second computing submodule is further configured to: Using the second largest model and the self-attention mask matrix to obtain information about the last word and the word before it in the concatenated sequence, so as to predict an output word in the third word sequence; splicing the output word unit to the last word unit of the splicing sequence to obtain a new splicing sequence; Inputting the new concatenated sequence into the second large model, and obtaining information of the last word unit and the word units before it in the new concatenated sequence by using the self-attention mask matrix of the second large model to predict the next output word unit in the third word unit sequence; The word unit concatenation step and the prediction step are repeatedly performed until the third word unit sequence is output.

19. The device according to any one of claims 11 to 18, wherein: The verification module comprises: a comparison submodule, used for comparing the word value of the current position of the second word-unit sequence and the third word-unit sequence in sequence; The comparison submodule is further configured to, when the word value at the current position is consistent, continue to compare the word value at the next position of the current position of the second word sequence and the third word sequence.

20. The device according to claim 19, wherein The verification module further includes: An acquisition submodule, for acquiring a word element with a consistent word element value before the current position when the word element value at the current position is inconsistent; a regeneration submodule, for re-executing the steps of generating a visual word-gram sequence, a text word-gram sequence, a first word-gram sequence, a second word-gram sequence, and a third word-gram sequence; The verification module is further used to continue verification from the next position of the current position of the regenerated second word-unit sequence and the third word-unit sequence until all inference decoding results are obtained.

21. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

23. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images