Method and device for accelerating inference without training of multimodal large models based on visual token pruning

Abstract

The present application provides a method and apparatus for accelerating inference without training of a multimodal large model based on visual token pruning, which relates to the field of artificial intelligence technology and aims to accelerate the inference speed of the multimodal large model. The method includes: inputting a target image into a visual encoder for multi-layer attention interaction to obtain a visual token sequence composed of multiple visual tokens and the attention score distribution of multiple class tokens to the visual tokens in multiple layers; determining the final attention score of each class token to each visual token according to the attention score distribution of the multiple class tokens to the visual tokens in multiple layers, where the final attention score represents the importance of the visual token; selecting multiple target visual tokens with high importance from the visual token sequence according to the final attention score of each class token to each visual token; and inputting the multiple target visual tokens and a text prompt into a language model for inference to obtain an inference result.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method and device for accelerating non-training reasoning of a multimodal large model based on visual token pruning. Background Art

[0002] Large multimodal models, with their superior understanding and generation capabilities, have greatly changed the way we obtain information on the Internet. However, when reasoning, large multimodal models need to process a large number of visual tokens derived from images, which greatly increases the computational complexity and greatly increases the time cost of model reasoning.

[0003] Existing methods for accelerating inference on large multimodal models include solutions that improve the model structure and solutions that prune during the inference process. The model structure improvement solution uses a lightweight multimodal model to accelerate inference. This method requires a lot of time and computing resources to retrain the model and cannot be immediately applied to new models. Although the pruning solution during the inference process can be used without additional training, the model's task performance is poor.

[0004] Therefore, how to accelerate the inference speed of large multimodal models is a technical problem that needs to be solved urgently. Summary of the Invention

[0005] In view of the above problems, an embodiment of the present application provides a multimodal large model untrained reasoning acceleration method and device based on visual token pruning to overcome the above problems or at least partially solve the above problems.

[0006] In a first aspect of an embodiment of the present application, a method for accelerating inference of a multimodal large model without training based on visual token pruning is disclosed, the method comprising:

[0007] The target image is input into the visual encoder for multi-layer attention interaction, resulting in a visual token sequence consisting of multiple visual tokens and a multi-layer class token-to-visual token attention score distribution. Each layer of attention interaction integrates the global and local information of the visual token based on the class token. The class token-to-visual token attention score distribution is similar to the visual token attention score distribution of the multimodal large model.

[0008] Determining a final attention score of the class token to each visual token based on the attention score distribution of the multi-layer class token to the visual token, wherein the final attention score represents the importance of the visual token;

[0009] selecting a plurality of target visual tokens with high importance from the visual token sequence according to the final attention scores of the class tokens on each visual token;

[0010] Input the multiple target visual tokens and the text prompt into a language model for inference to obtain an inference result.

[0011] Optionally, the visual encoder includes a token embedding layer and multiple layers of attention interaction layers; input the target image into the visual encoder for multiple layers of attention interaction to obtain a visual token sequence composed of multiple visual tokens, and the attention score distribution of the multiple layers of class tokens to the visual tokens, including:

[0012] Process the target image into an initial visual token sequence composed of multiple visual tokens based on the token embedding layer;

[0013] Input the initial visual token sequence and the class token into the multiple layers of attention interaction layers for attention interaction to obtain a visual token sequence composed of multiple visual tokens, and the attention score distribution of the multiple layers of class tokens to the visual tokens.

[0014] Optionally, each layer of attention interaction layer includes multiple attention heads, and the attention score distribution of each layer of class tokens to the visual tokens is obtained by the following method:

[0015] For each visual token, take the average of the attention scores of the class tokens to this visual token output by the multiple attention heads as the attention score of the class token to this visual token;

[0016] Obtain the attention score distribution of the class tokens to the visual tokens according to the attention scores of the class tokens to each visual token.

[0017] Optionally, determine the final attention score of the class token to each visual token according to the attention score distribution of the multiple layers of class tokens to the visual tokens, including:

[0018] For each visual token, determine multiple attention scores of the class token to this visual token from the attention score distributions of the multiple layers of class tokens to the visual tokens corresponding to the penultimate layer to the previous K layers;

[0019] Calculate the final attention score of the class token to this visual token according to the multiple attention scores of the class token to this visual token.

[0020] Optionally, calculate the final attention score of the class token to this visual token according to the multiple attention scores of the class token to this visual token, including:

[0021] Take the average of the multiple attention scores of the class token to this visual token as the final attention score of the class token to this visual token; or,

[0022] Among the multiple attention scores of the class token for the visual token, the maximum attention score is used as the final attention score of the class token for the visual token.

[0023] Optionally, input the multiple target visual tokens and the text prompt into a multi-modal large model for inference to obtain an inference result, including:

[0024] Project the multiple target visual tokens into the semantic space of the text prompt through a projection connector to obtain visually tokens aligned in the semantic space;

[0025] Input the visually tokens aligned in the semantic space and the text prompt processed by a tokenizer into a multi-modal large model for inference to obtain an inference result.

[0026] In a second aspect of the embodiments of the present application, an apparatus for accelerating the inference without training of a multi-modal large model based on visual token pruning is disclosed. The apparatus includes:

[0027] A first input module, configured to input a target image into a visual encoder for multi-layer attention interaction to obtain a visual token sequence composed of multiple visual tokens, and a distribution of attention scores of the class token for the visual token in each layer; each layer of attention interaction integrates the global information and local information of the visual token based on the class token, and the distribution of attention scores of the class token for the visual token is similar to the distribution of attention scores of the visual token of the multi-modal large model;

[0028] A first determination module, configured to determine the final attention score of the class token for each visual token according to the distribution of attention scores of the class token for the visual token in each layer, where the final attention score represents the importance of the visual token;

[0029] A first selection module, configured to select multiple target visual tokens with high importance from the visual token sequence according to the final attention score of the class token for each visual token;

[0030] A second input module, configured to input the multiple target visual tokens and the text prompt into a language model for inference to obtain an inference result.

[0031] In a third aspect of the embodiments of the present application, an electronic device is disclosed, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the steps of the method for accelerating the inference without training of a multi-modal large model based on visual token pruning described in the first aspect of the embodiments of the present application are implemented.

[0032] In the fourth aspect of the embodiments of the present application, a computer-readable storage medium is disclosed, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for accelerating inference without training of a multimodal large model based on visual token pruning described in the first aspect of the embodiments of the present application are implemented.

[0033] In the fifth aspect of the embodiments of the present application, a computer program product is disclosed, including a computer program. When the computer program is executed by a processor, the steps of the method for accelerating inference without training of a multimodal large model based on visual token pruning described in the first aspect of the embodiments of the present application are implemented.

[0034] The embodiments of the present application have the following advantages:

[0035] In the embodiments of the present application, a target image is input into a visual encoder for multi-layer attention interaction to obtain a visual token sequence composed of multiple visual tokens, and a distribution of attention scores of multi-layer class tokens on visual tokens; according to the distribution of attention scores of the multi-layer class tokens on visual tokens, the final attention score of each visual token by the class token is determined, and the final attention score represents the importance of the visual token; and according to the final attention score of each visual token by the class token, multiple target visual tokens with high importance are selected from the visual token sequence; the multiple target visual tokens and a text prompt are input into a language model for inference to obtain an inference result.

[0036] Since the attention interaction of each layer of the visual encoder integrates the global information and local information of visual tokens based on class tokens, and the distribution of attention scores of class tokens on visual tokens is similar to the distribution of attention scores of visual tokens of the multimodal large model, the class token pair has a good fitting ability for the part it pays attention to without seeing text tokens. Based on the attention information of the class token pair on visual tokens, the global content and local features of visual information can be effectively grasped at the same time to correctly capture important visual tokens containing necessary information (i.e., multiple target visual tokens), and inference is performed based on the multiple target visual tokens. In this way, while effectively accelerating the inference process of the multimodal large model, the performance of the model is maximally improved. Description of the Drawings

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0038] Figure 1It is a flowchart of the steps of a method for accelerating inference without training of a multimodal large model based on visual token pruning provided by an embodiment of the present application;

[0039] Figure 2 It is a flowchart of the steps of another method for accelerating inference without training of a multimodal large model based on visual token pruning provided by an embodiment of the present application;

[0040] Figure 3 It is a flowchart of another inference without training of a multimodal large model based on visual token pruning provided by an embodiment of the present application;

[0041] Figure 4 It is a schematic structural diagram of a device for accelerating inference without training of a multimodal large model based on visual token pruning provided by an embodiment of the present application;

[0042] Figure 5 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Specific embodiments

[0043] To make the above objects, features, and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0044] Existing methods for accelerating inference of multimodal large models include a scheme for improving the model structure and a scheme for pruning during the inference process. 1) The scheme for improving the model structure uses a lightweight multimodal model to accelerate inference. The lightweight multimodal model uses a lightweight projection connector and methods such as downsampling to reduce the number of visual tokens input to the language model to achieve an acceleration effect, thereby achieving the goal of being deployed to the mobile terminal.

[0045] 2) The existing pruning schemes in the inference process include the following: The implementation of the first pruning method in the inference process is as follows: Based on the fact that visual tokens receive less attention compared to other tokens such as language tokens and special symbols, removing most of the relatively unimportant visual tokens in the shallow layer of the language model part of the multi-modal large model will not have a decisive impact on the overall performance of the model. Therefore, this method measures the importance of each visual token by the attention score between the visual token and the prompt token in the language model, and then retains several tokens with the highest importance and discards the rest. The second pruning method in the inference process adopts the method of pruning and fusion of visual tokens after the visual encoder. First, several most important tokens are selected through the attention mechanism and mixed with tokens uniformly sampled in the space; then, the unselected tokens are fused into the selected important tokens through the nearest neighbor judgment method, so as to effectively extract visual information in a two-stage method.

[0046] The above two pruning methods in the inference process can be adopted without additional training. In addition, there are some schemes that must be used after retraining the model. One pruning method that requires training is to combine the understanding mechanism of the language model to compress visual tokens into specific VoCo tokens; another pruning method that requires training is to compare the compression effects at different layers of the language model with different compression rates and design a multi-level hierarchical compression method, using a simple pooling strategy to compress visual tokens; there are also some pruning methods that stack visual tokens and inject them into the layers of the model from the bottom layer to the top layer, allowing the model to process more visual tokens under the same conditions without changing the model structure.

[0047] The current schemes for improving the model structure and the pruning schemes in the inference process both have their own drawbacks. The schemes for improving the model structure use lightweight multi-modal models to accelerate inference. Such methods require a large amount of time and computing resources to retrain the model and cannot be immediately applied to new models. Their plug-and-play nature is not as good as the way without training. And the currently proposed schemes without training have a relatively serious impact on the model performance in the scenario without training. For example, the first pruning method in the above inference process mainly uses the attention between the prompt and visual tokens as the inference information, which may ignore the relationship between the content to be generated and the input picture, and thus may discard visual tokens containing important visual information, resulting in a large cognitive deviation in the model's understanding and a decline in the performance of multi-modal tasks; while the second pruning method in the inference process is not an optimal scheme in the scenario without training, which may lead to a significant decline in the model's performance and requires new technical solutions for improvement.

[0048] To overcome the limitations of related technologies, the embodiments of the present application provide a method for accelerating inference without training of a multimodal large model based on visual token pruning. This method can make full use of the intermediate information generated during the inference process of the multimodal large model, effectively select the relatively more important visual tokens, improve the performance of the model on various multimodal tasks after pruning operations, and make it as close as possible to the performance of the model without pruning. Specifically, this method uses the attention information of the class token (CLS token), and after fully demonstrating its accuracy in selecting tokens, prunes the visual tokens after the visual encoder, and effectively accelerates the inference of the model on the premise of minimizing the impact on the model performance.

[0049] The following will combine with the accompanying drawings to provide a detailed description of the method for accelerating inference without training of a multimodal large model based on visual token pruning provided by the embodiments of the present application.

[0050] Refer to Figure 1 as shown Figure 1 is the flowchart of the steps of a method for accelerating inference without training of a multimodal large model based on visual token pruning provided by the embodiments of the present application. As Figure 1 shown, the method for accelerating inference without training of a multimodal large model based on visual token pruning may include steps S110 to step S140:

[0051] Step S110: Input the target image into the visual encoder for multi-layer attention interaction to obtain a visual token sequence composed of multiple visual tokens, and the distribution of attention scores of multi-layer class tokens to visual tokens; each layer of attention interaction integrates the global information and local information of visual tokens based on the class token, and the distribution of attention scores of the class token to visual tokens is similar to the distribution of attention scores of visual tokens of the multimodal large model.

[0052] Among them, the target image refers to the visual information input into the multimodal large model, and the visual encoder is the encoder in the multimodal large model for processing visual information.

[0053] In the multimodal large model, both the visual encoder and the language model (used for content understanding and content generation) can measure the importance of different tokens through attention information. In the process of the visual encoder processing the target image, the visual encoder has not seen the tokens of the text (including prompt words and generated words); at the same time, although the language model part is visible to the tokens of the prompt words, it cannot see the future generated content during the generation process, which makes it difficult to select the optimal token set during the reasoning process. However, the embodiment of the present application takes into account that the visual encoder uses class tokens to collect global information and local information during forward propagation, which is a structure that does not exist in the language model, and by comparing the attention score distribution of the visual tokens of the language model and the attention score distribution of the class tokens, it is found that the two are similar. Therefore, the embodiment of the present application believes that the class token can have a better fitting ability for the part to which its attention is focused without seeing the text token, and thus measures the importance of the visual token based on the attention score of the class token to the visual token.

[0054] The visual encoder consists of multiple layers of attention interaction. First, the target image is input into the visual encoder for multi-layer attention interaction, resulting in a visual token sequence consisting of multiple visual tokens and a multi-layer distribution of attention scores between class tokens and visual tokens. The visual token sequence refers to the feature representation output by the last layer of attention interaction of the visual encoder. Each layer of attention interaction will generate a corresponding distribution of attention scores between class tokens and visual tokens, that is, the attention scores of class tokens on each visual token in the attention interaction of that layer.

[0055] Step S120: Determine the final attention score of the class token to each visual token based on the attention score distribution of the multi-layer class token to the visual token, and the final attention score represents the importance of the visual token.

[0056] Since the attention score distribution of class tokens to visual tokens may have different distributions in the attention interactions of different layers of the visual encoder, that is, different layers of attention interactions focus on different features of the image, if only the attention information of one layer of class tokens is used (that is, the final attention score distribution of one layer of class tokens to each visual token), it may lead to inaccurate capture of visual information.

[0057] Therefore, an embodiment of the present application combines the attention score distribution of multi-layer class tokens to visual tokens in the visual encoder to select visual tokens that carry important visual information, that is, based on the attention score distribution of multi-layer class tokens to visual tokens, the final attention score of the class token to each visual token is determined, and the importance of the visual token is measured by the final attention score.

[0058] Step S130: Select multiple target visual tokens with high importance from the visual token sequence according to the final attention scores of each visual token with respect to the class token.

[0059] In the embodiments of the present application, the final attention score represents the importance of a visual token. Thus, according to the final attention scores of each visual token with respect to the class token, multiple target visual tokens with high importance are selected for subsequent reasoning, and the remaining visual tokens are discarded. In some embodiments, each visual token in the visual token sequence can be sorted according to the final attention score of the class token for each visual token, and the top N visual tokens in the sorting are used as the target visual tokens.

[0060] Step S140: Input the multiple target visual tokens and the text prompt into a language model for reasoning to obtain a reasoning result.

[0061] In the embodiments of the present application, the text prompt is the text information input into the multimodal large model, and the language model is the model in the multimodal large model used for content understanding and content generation. By inputting the multiple target visual tokens and the text prompt into the language model for reasoning, a reasoning result is obtained; among them, the reasoning result mainly depends on the text prompt. For example, if the text prompt is a question, the reasoning result is the answer to that question.

[0062] Among them, the language model is stacked by multiple layers of transformer architectures (attention interaction layers). The internal structure of each layer is similar to that of the visual encoder, but the number of parameters is relatively larger and the number of stacked layers is also more. The number of parameters of the language model occupies the vast majority of the number of parameters of the multimodal large model. Therefore, most of the computational overhead is also generated in this link. And the number of tokens corresponding to visual information is usually one order of magnitude more than the number of tokens corresponding to the text prompt. Therefore, pruning before the information is transmitted to the language model can effectively reduce the computational overhead while retaining as much visual information as possible.

[0063] Adopting the technical solution of the embodiments of the present application, since the attention interaction of each layer of the visual encoder integrates the global information and local information of the visual tokens based on the class token, and the attention score distribution of the class token for the visual tokens is similar to the attention score distribution of the visual tokens of the multimodal large model, the class token pair has a good fitting ability for the part that its attention focuses on in the case of not seeing the text token. Based on the attention information of the class token for the visual tokens, the global content and local features of the visual information can be effectively grasped at the same time to correctly capture important visual tokens containing necessary information (i.e., multiple target visual tokens), and reasoning is performed based on the multiple target visual tokens. In this way, while effectively accelerating the reasoning process of the multimodal large model, the performance of the model is maximally improved.

[0064] Combined with the above embodiments, an embodiment of the present application provides a method for accelerating inference without training of a multimodal large model based on visual token pruning. In this method, the visual encoder includes a token embedding layer and multiple layers of attention interaction layers. In step S110 above, "inputting the target image into the visual encoder for multi-layer attention interaction to obtain a visual token sequence composed of multiple visual tokens, and the attention score distribution of multiple layers of class tokens to visual tokens" specifically includes steps S110-1 to S110-2:

[0065] Step S110-1: Based on the token embedding layer, process the target image into an initial visual token sequence composed of multiple visual tokens.

[0066] Step S110-2: Input the initial visual token sequence and class tokens into the multiple layers of attention interaction layer for attention interaction to obtain a visual token sequence composed of multiple visual tokens, and the attention score distribution of multiple layers of class tokens to visual tokens.

[0067] Among them, the visual encoder in the multimodal large model can be a pre-trained visual encoder, and the parameters of the visual encoder do not need to be changed again during the entire process of training, fine-tuning, and inference of the multimodal large model. When the visual encoder processes the target image, first input the target image into the token embedding layer for data augmentation and size scaling, and then split the processed image into multiple small blocks, each small block representing a visual token, thus obtaining an initial visual token sequence. Then input the initial visual token sequence and class tokens into the multiple layers of attention interaction layer for attention interaction (forward propagation). The multiple layers of attention interaction layer can capture short-distance and long-distance dependencies in different regions of the image, and finally obtain a visual token sequence composed of multiple visual tokens, and the attention score distribution of multiple layers of class tokens to visual tokens.

[0068] Exemplarily, taking the multimodal large model LLaVA-v1.5 as an example, the visual encoder it uses is CLIP-ViT / L-14-336, and the processing process of the target image is described. Specifically, when processing a target image, it will first go through the processes of data augmentation (normalization, standardization) and size scaling to obtain a tensor with dimensions and number of channels of and then split it into 576 drag_maildrag_mail Small patches (visual tokens), when flattened, are fed into a transformer structure (attention interaction layer), where the attention interaction layer can capture short-range and long-range dependencies in different regions of the image. In the attention interaction layer, in addition to the visual tokens of the 576 images mentioned above, there is also a class token (CLS token) artificially added and participating in the entire training and inference process. A tensor composed of a total of 577 tokens performs forward propagation. Among them, each attention interaction layer consists of a self-attention mechanism, a multi-layer perceptron, and two layer normalization modules. In forward propagation, in order to retain the original features, the residual connection method is adopted.

[0069] Furthermore, each layer of the attention interaction layer includes multiple attention heads. The attention score distribution of the class token for the visual tokens in each layer is obtained through the following method:

[0070] For each visual token, the average value of the attention scores of the class token for this visual token output by multiple attention heads is used as the attention score of the class token for this visual token; according to the attention scores of the class token for each visual token, the attention score distribution of the class token for the visual tokens is obtained.

[0071] In the embodiment of this application, each layer of the attention interaction layer includes multiple attention heads. The initial visual token sequence and the class token are input into each attention head for processing. For each visual token, each attention head can output the attention score of the class token for this visual token. The average value of the attention scores of the class token for this visual token input by multiple attention heads is used as the attention score of the class token for this visual token. For example, the attention score of the class token for the visual token output by the th attention head is denoted as , then the average value of the attention scores of the class token for this visual token input by multiple attention heads is used as the attention score of the class token for this visual token, where is the number of heads of the attention in the attention interaction layer of the visual encoder.

[0072] In this way, in each layer of attention interaction, the average value of the attention scores of the class token for this visual token output by multiple attention heads is used as the attention score of the class token for this visual token, so as to obtain the attention score distribution of the class token for the visual tokens in each layer, and to measure the importance of each visual token based on the attention score distribution of the class token for the visual tokens in each layer.

[0073] Combined with the above implementation manners, the embodiment of this application provides a method for accelerating the non-training inference of a multi-modal large model based on visual token pruning. In this method, in the above step S120, "determine the final attention score of the class token for each visual token according to the attention score distribution of the multi-layer class token for the visual tokens", specifically includes:

[0074] For each visual token, based on the attention score distribution of the multi-layer class tokens corresponding to the last second layer to the previous K layers for the visual token, multiple attention scores of the class token for the visual token are determined; based on the multiple attention scores of the class token for the visual token, the final attention score of the class token for the visual token is calculated.

[0075] In the embodiments of the present application, in order to accurately capture visual information, starting from the last second layer, the last second layer and the previous K layers are selected, and the importance of each visual token is measured by combining the attention score distribution of the corresponding class tokens for the visual token in these layers.

[0076] Specifically, for each visual token, the class token has an attention score for the visual token in each layer, so multiple attention scores of the class token for the visual token are determined, and then the final attention score is calculated based on the multiple attention scores of the class token for the visual token.

[0077] In some embodiments, calculating the final attention score of the class token for the visual token based on the multiple attention scores of the class token for the visual token includes: taking the average value of the multiple attention scores of the class token for the visual token as the final attention score of the class token for the visual token; or, taking the maximum attention score among the multiple attention scores of the class token for the visual token as the final attention score of the class token for the visual token.

[0078] That is to say, the final attention score of the class token for each visual token can be calculated by the average value method, or the final attention score of the class token for each visual token can be calculated by the maximum value method, and finally the final attention score distribution of the class token for the visual token is obtained.

[0079] Exemplarily, the final attention score distribution of the class token for the visual token is expressed as:

[0080] ,

[0081] where, represents the final attention score distribution of the class token for the visual token, including the final attention scores of the class token for each visual token; is a function for data integration between different layers, is the number of layers of the visual encoder, represents the attention score distribution of the class token in the last second layer (i.e., the th layer) for the visual token, represents the attention score distribution of the class token in the th layer for the visual token, Indicating the attention score distribution of the class tokens in the

[0082] layer with respect to the visual tokens.

[0083] Combining the above embodiments, the embodiments of the present application provide a method for accelerating inference without training for a multimodal large model based on visual token pruning. In this method, in the above step S140, "inputting the multiple target visual tokens and the text prompt into the multimodal large model for inference to obtain an inference result" specifically includes steps S140-1 to S140-2:

[0084] Step S140-2: Project the multiple target visual tokens into the semantic space of the text prompt through a projection connector to obtain visually tokens aligned in the semantic space.

[0085] Step S140-2: Input the visually tokens aligned in the semantic space and the text prompt processed by the tokenizer into the multimodal large model for inference to obtain an inference result.

[0086] In the embodiments of the present application, after the target image is processed by the visual encoder, the dependencies between different information are captured, and a visual token sequence composed of multiple visual tokens is obtained. Among them, each visual token is equivalent to a feature vector, that is, the visual token sequence is equivalent to a tensor composed of multiple feature vectors. In order for the semantic model to understand and process the visual token sequence (visual information), it is necessary to align the semantic spaces of the visual token sequence and the prompt text.

[0087] Specifically, a trained projection connector is used to project multiple target visual tokens into the semantic space of the text prompt. The projection connector includes two linear layers and a GeLU activation function. The multiple target visual tokens are processed through one linear layer, then processed through the GeLU activation function, and finally the processing result of the GeLU activation function is input into another linear layer for processing to achieve the purpose of aligning with the semantic space of the text prompt.

[0088] Exemplarily, projecting the multiple target visual tokens into the semantic space of the text prompt through the projection connector can be expressed as:

[0089] ,

[0090] where is the weight of the first linear layer, is the bias parameter of the first linear layer, is the weight of the second linear layer, is the bias parameter of the second linear layer, are multiple target visual tokens, are visual tokens aligned with the semantic space.

[0091] Finally, use a tokenizer (i.e., a text editor) to process the text prompt into semantic tokens that the language model can directly assist with, and then input the visually token-aligned semantic tokens and semantic tokens (i.e., the text prompt processed by the tokenizer) into the multimodal large model for inference to obtain the inference result.

[0092] In this way, by projecting multiple target visual tokens into the semantic space of the text prompt to align the visual information with the semantic space of the text prompt, the language model can understand and process visual information.

[0093] Refer to Figure 2 as shown, Figure 2 is a flowchart of the steps of another method for accelerating the untrained inference of a multimodal large model based on visual token pruning provided by an embodiment of the present application. Specifically, the method includes steps S210 to S250:

[0094] Step S210: Based on the token embedding layer, process the target image into an initial visual token sequence composed of multiple visual tokens; input the initial visual token sequence and the class token into the multi-layer attention interaction layer for attention interaction to obtain a visual token sequence composed of multiple visual tokens, and a multi-layer class token attention score distribution for the visual tokens; each layer of attention interaction integrates the global information and local information of the visual tokens based on the class token, and the class token attention score distribution for the visual tokens is similar to the visual token attention score distribution of the multimodal large model.

[0095] Step S220: For each visual token, determine multiple attention scores of the class token for the visual token from the attention score distributions of the multi-layer class tokens for the visual token corresponding to the last second layer to the previous K layers; calculate the final attention score of the class token for the visual token according to the multiple attention scores of the class token for the visual token.

[0096] Step S230: Determine the final attention scores of the class token for each visual token according to the attention score distribution of the multi-layer class token for the visual token, and the final attention score represents the importance of the visual token.

[0097] Step S240: Project the multiple target visual tokens into the semantic space of the text prompt through a projection connector to obtain visually token-aligned semantic tokens.

[0098] Step S250: Input the visually token aligned with the semantic space and the text prompt processed by the tokenizer into the multi-modal large model for inference to obtain an inference result.

[0099] In the embodiment of the present application, since the attention interaction of each layer of the visual encoder integrates the global information and local information of the visual tokens based on the class tokens, and the attention score distribution of the class tokens to the visual tokens is similar to the attention score distribution of the visual tokens of the multi-modal large model, the class token pair has a good fitting ability for the part it pays attention to without seeing the text tokens. Based on the attention information of the class token pair to the visual tokens, the global content and local features of the visual information can be effectively grasped at the same time to correctly capture important visual tokens containing necessary information (i.e., multiple target visual tokens), and inference is performed based on the multiple target visual tokens. In this way, while effectively accelerating the inference process of the multi-modal large model, the performance of the model is maximally improved.

[0100] Refer to Figure 3 as shown in Figure 3 FIG. is a flowchart of another multi-modal large model untrained inference based on visual token pruning provided by the embodiment of the present application. The multi-modal large model includes a visual encoder, a projection connector, a tokenizer, and a language model. Specifically, the visual encoder includes a token embedding layer and multiple layers of attention interaction layers. The target image is input into the visual encoder for processing. The visual encoder processes the target image into an initial visual token sequence composed of multiple visual tokens based on the token embedding layer. Then, the initial visual token sequence and the class tokens are input into the multiple layers of attention interaction layers for attention interaction to obtain a visual token sequence composed of multiple visual tokens, and the attention score distribution of the multiple layers of class tokens to the visual tokens.

[0101] After that, for each visual token, multiple attention scores of the class token pair to the visual token are determined from the attention score distribution of the multiple layers of class tokens to the visual token corresponding to the last second layer to the previous K layers, and the final attention score of the class token pair to the visual token is calculated according to the multiple attention scores of the class token pair to the visual token. Furthermore, multiple target visual tokens with high importance are selected from the visual token sequence according to the final attention scores of the class token pair to each visual token.

[0102] Finally, the multiple target visual tokens are projected into the semantic space of the text prompt through the projection connector to obtain visually tokens aligned with the semantic space; the visually tokens aligned with the semantic space and the text prompt processed by the tokenizer are input into the multi-modal large model for inference to obtain an inference result.

[0103] In this way, through the attention information of class tokens, the global content and local features of visual information can be effectively grasped simultaneously, important visual tokens containing necessary information can be correctly captured, and while effectively accelerating the inference process of the multi-modal large model, the performance of the model is maximally improved.

[0104] The embodiment of the present application also provides a device for accelerating the inference without training of a multi-modal large model based on visual token pruning. Refer to Figure 4 as shown in Figure 4 FIG. 7 is a schematic structural diagram of a device for accelerating the inference without training of a multi-modal large model based on visual token pruning provided by an embodiment of the present application. The device includes:

[0105] A first input module 410, configured to input a target image into a visual encoder for multi-layer attention interaction, obtain a visual token sequence composed of multiple visual tokens, and a distribution of attention scores of multi-layer class tokens for visual tokens; each layer of attention interaction integrates the global information and local information of visual tokens based on class tokens, and the distribution of attention scores of class tokens for visual tokens is similar to the distribution of attention scores of visual tokens of the multi-modal large model;

[0106] A first determination module 420, configured to determine the final attention score of a class token for each visual token according to the distribution of attention scores of multi-layer class tokens for visual tokens, where the final attention score represents the importance of the visual token;

[0107] A first selection module 430, configured to select multiple target visual tokens with high importance from the visual token sequence according to the final attention score of the class token for each visual token;

[0108] A second input module 440, configured to input the multiple target visual tokens and a text prompt into a language model for inference to obtain an inference result.

[0109] In an optional embodiment, the visual encoder includes a token embedding layer and a multi-layer attention interaction layer; the first input module includes:

[0110] A first processing module, configured to process the target image into an initial visual token sequence composed of multiple visual tokens based on the token embedding layer;

[0111] A first input sub-module, configured to input the initial visual token sequence and class tokens into the multi-layer attention interaction layer for attention interaction, obtain a visual token sequence composed of multiple visual tokens, and a distribution of attention scores of multi-layer class tokens for visual tokens.

[0112] In an alternative embodiment, each layer of the attention interaction layer includes a plurality of attention heads, and the first input sub-module is further configured to: for each visual token, take the average of the attention scores of the class tokens output by the plurality of attention heads for the visual token as the attention score of the class token for the visual token; and obtain the attention score distribution of the class token for the visual token according to the attention scores of the class token for each visual token.

[0113] In an alternative embodiment, the first determination module includes:

[0114] The first determination sub-module is configured to, for each visual token, determine a plurality of attention scores of the class token for the visual token from the attention score distributions of the class tokens for the visual token corresponding to the last second layer to the previous K layers.

[0115] The first calculation module is configured to calculate the final attention score of the class token for the visual token according to the plurality of attention scores of the class token for the visual token.

[0116] In an alternative embodiment, the first calculation module is further configured to: take the average of the plurality of attention scores of the class token for the visual token as the final attention score of the class token for the visual token; or take the maximum attention score among the plurality of attention scores of the class token for the visual token as the final attention score of the class token for the visual token.

[0117] In an alternative embodiment, the second input module includes:

[0118] The projection module is configured to project the plurality of target visual tokens into the semantic space of the text prompt through a projection connector to obtain visually tokens aligned in the semantic space.

[0119] The second input sub-module is configured to input the visually tokens aligned in the semantic space and the text prompt processed by a tokenizer into a multi-modal large model for inference to obtain an inference result.

[0120] The embodiment of the present application further provides an electronic device. Refer to Figure 5 , Figure 5 is a schematic structural diagram of an electronic device provided by the embodiment of the present application. As Figure 5 shown, the electronic device 500 includes: a memory 510 and a processor 520. The memory 510 is communicatively connected to the processor 520 through a bus. A computer program is stored in the memory 510, and the computer program can run on the processor 520 to implement the steps of the method for accelerating inference without training of the multi-modal large model based on visual token pruning described in the embodiment of the present application.

[0121] The embodiments of the present application also provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for accelerating inference without training of a multimodal large model based on vision token pruning described in the embodiments of the present application are implemented.

[0122] The embodiments of the present application also provide a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the method for accelerating inference without training of a multimodal large model based on vision token pruning described in the embodiments of the present application are implemented.

[0123] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other.

[0124] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods and devices according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general computer, a special computer, an embedded processor or other programmable data processing terminal devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal devices generate a device for implementing the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0125] These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing terminal device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured product including an instruction device, and the instruction device implements the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0126] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0127] Although the preferred embodiments of the embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to these embodiments once they learn the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications that fall within the scope of the embodiments of the present application.

[0128] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or terminal device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or terminal device comprising the element.

[0129] The above has introduced in detail a method and device for accelerating inference without training of a multimodal large model based on visual token pruning provided by the present application. Specific examples are used in this text to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A multimodal large model training-free inference acceleration method based on visual token pruning, characterized by: The method comprises: The target image is input into the visual encoder for multi-layer attention interaction, resulting in a visual token sequence consisting of multiple visual tokens and a multi-layer class token-to-visual token attention score distribution. Each layer of attention interaction integrates the global and local information of the visual token based on the class token. The class token-to-visual token attention score distribution is similar to the visual token attention score distribution of the multimodal large model. Determining a final attention score of the class token to each visual token based on the attention score distribution of the multi-layer class token to the visual token, wherein the final attention score represents the importance of the visual token; selecting a plurality of target visual tokens with high importance from the visual token sequence according to the final attention scores of the class tokens on each visual token; The multiple target visual tokens and text prompts are input into a language model for reasoning to obtain a reasoning result.

2. The method according to claim 1, characterized in that The visual encoder includes a token embedding layer and a multi-layer attention interaction layer; a target image is input into the visual encoder for multi-layer attention interaction, thereby obtaining a visual token sequence consisting of multiple visual tokens and a multi-layer class token attention score distribution for the visual tokens, including: processing the target image into an initial visual token sequence consisting of a plurality of visual tokens based on the token embedding layer; The initial visual token sequence and class tokens are input into the multi-layer attention interaction layer for attention interaction to obtain a visual token sequence consisting of multiple visual tokens and an attention score distribution of the multi-layer class tokens to the visual tokens.

3. The method according to claim 2, characterized in that Each attention interaction layer includes multiple attention heads, and the attention score distribution of each layer’s class tokens to visual tokens is obtained as follows: For each visual token, the average of the attention scores of the class tokens output by multiple attention heads on the visual token is used as the attention score of the class token on the visual token; According to the attention scores of the class tokens to each visual token, the attention score distribution of the class tokens to the visual tokens is obtained.

4. The method according to claim 1, wherein Determining a final attention score of the class token to each visual token based on the attention score distribution of the multi-layer class token to the visual token includes: For each visual token, the attention score distribution of multiple class tokens from the penultimate layer to the previous K layers on the visual token is used to determine the multiple attention scores of the class tokens on the visual token; According to the multiple attention scores of the class token to the visual token, a final attention score of the class token to the visual token is calculated.

5. The method according to claim 4, characterized in that Calculating a final attention score of the class token to the visual token based on the multiple attention scores of the class token to the visual token includes: The average of multiple attention scores of the class token to the visual token is used as the final attention score of the class token to the visual token; or The maximum attention score among the multiple attention scores of the class token to the visual token is used as the final attention score of the class token to the visual token.

6. The method according to any one of claims 1 to 5, characterized in that: Inputting the plurality of target visual tokens and text prompts into the multimodal large model for reasoning to obtain reasoning results, including: Projecting the plurality of target visual tokens into the semantic space of the text prompt by a projection connector to obtain semantic space-aligned visual tokens; The visual tokens aligned in the semantic space and the text prompts processed by the word segmenter are input into the multimodal large model for reasoning to obtain the reasoning results.

7. A multimodal large model non-training inference acceleration device based on visual token pruning, characterized in that: The device comprises: The first input module is used to input the target image into the visual encoder for multi-layer attention interaction, thereby obtaining a visual token sequence consisting of multiple visual tokens and a multi-layer class token-to-visual token attention score distribution. Each layer of attention interaction integrates the global and local information of the visual token based on the class token. The class token-to-visual token attention score distribution is similar to the visual token attention score distribution of the multimodal large model. A first determining module is configured to determine a final attention score of a class token to each visual token based on the attention score distribution of the multi-layer class tokens to the visual tokens, wherein the final attention score represents the importance of the visual token; a first selection module, configured to select a plurality of target visual tokens with high importance from the visual token sequence according to the final attention scores of the class tokens on the respective visual tokens; The second input module is used to input the multiple target visual tokens and text prompts into the language model for reasoning to obtain reasoning results.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the multimodal large model untrained reasoning acceleration method based on visual token pruning according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multimodal large model untrained reasoning acceleration method based on visual token pruning according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the multimodal large model untrained reasoning acceleration method based on visual token pruning are implemented as described in any one of claims 1 to 6.

Images