Multi-modal large model non-training reasoning acceleration method and device based on visual token pruning

Abstract

The invention provides a multi-modal large model non-training reasoning acceleration method and device based on visual token pruning, relates to the technical field of artificial intelligence, and aims to accelerate the reasoning speed of a multi-modal large model. The method comprises the following steps: inputting a target image into a visual encoder for multilayer attention interaction to obtain a visual token sequence consisting of a plurality of visual tokens and attention score distribution of multilayer class tokens to the visual tokens; according to the attention score distribution of the multi-layer class tokens to the visual tokens, final attention scores of the class tokens to the visual tokens are determined, and the final attention scores represent the importance degrees of the visual tokens; according to the final attention score of the class token to each visual token, selecting a plurality of target visual tokens with high importance from the visual token sequence; and inputting the plurality of target visual tokens and the text prompts into a language model for reasoning to obtain a reasoning result.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method and device for accelerating non-training reasoning of a multimodal large model based on visual token pruning. Background Art

[0002] Large multimodal models, with their superior understanding and generation capabilities, have greatly changed the way we obtain information on the Internet. However, when reasoning, large multimodal models need to process a large number of visual tokens derived from images, which greatly increases the computational complexity and greatly increases the time cost of model reasoning.

[0003] Existing methods for accelerating inference on large multimodal models include solutions that improve the model structure and solutions that prune during the inference process. The model structure improvement solution uses a lightweight multimodal model to accelerate inference. This method requires a lot of time and computing resources to retrain the model and cannot be immediately applied to new models. Although the pruning solution during the inference process can be used without additional training, the model's task performance is poor.

[0004] Therefore, how to accelerate the inference speed of large multimodal models is a technical problem that needs to be solved urgently. Summary of the Invention

[0005] In view of the above problems, an embodiment of the present application provides a multimodal large model untrained reasoning acceleration method and device based on visual token pruning to overcome the above problems or at least partially solve the above problems.

[0006] In a first aspect of an embodiment of the present application, a method for accelerating inference of a multimodal large model without training based on visual token pruning is disclosed, the method comprising: The target image is input into the visual encoder for multi-layer attention interaction, resulting in a visual token sequence consisting of multiple visual tokens and a multi-layer class token-to-visual token attention score distribution. Each layer of attention interaction integrates the global and local information of the visual token based on the class token. The class token-to-visual token attention score distribution is similar to the visual token attention score distribution of the multimodal large model. Determining a final attention score of the class token to each visual token based on the attention score distribution of the multi-layer class token to the visual token, wherein the final attention score represents the importance of the visual token; selecting a plurality of target visual tokens with high importance from the visual token sequence according to the final attention scores of the class tokens on each visual token; The multiple target visual tokens and text prompts are input into a language model for reasoning to obtain a reasoning result.

[0007] Optionally, the visual encoder includes a token embedding layer and a multi-layer attention interaction layer; the target image is input into the visual encoder for multi-layer attention interaction to obtain a visual token sequence consisting of multiple visual tokens and a multi-layer class token attention score distribution to the visual tokens, including: processing the target image into an initial visual token sequence consisting of a plurality of visual tokens based on the token embedding layer; The initial visual token sequence and class tokens are input into the multi-layer attention interaction layer for attention interaction to obtain a visual token sequence composed of multiple visual tokens and an attention score distribution of the multi-layer class tokens to the visual tokens.

[0008] Optionally, each attention interaction layer includes multiple attention heads, and the attention score distribution of each layer's class tokens to visual tokens is obtained as follows: For each visual token, the average of the attention scores of the class tokens output by multiple attention heads on the visual token is used as the attention score of the class token on the visual token; According to the attention scores of the class tokens to each visual token, the attention score distribution of the class tokens to the visual tokens is obtained.

[0009] Optionally, determining a final attention score of a class token to each visual token based on the attention score distribution of the multi-layer class tokens to the visual tokens includes: For each visual token, the attention score distribution of multiple class tokens from the penultimate layer to the previous K layers on the visual token is used to determine the multiple attention scores of the class tokens on the visual token; According to the multiple attention scores of the class token to the visual token, a final attention score of the class token to the visual token is calculated.

[0010] Optionally, calculating a final attention score of the class token to the visual token based on multiple attention scores of the class token to the visual token includes: The average of multiple attention scores of the class token to the visual token is used as the final attention score of the class token to the visual token; or The maximum attention score among the multiple attention scores of the class token to the visual token is used as the final attention score of the class token to the visual token.

[0011] Optionally, the plurality of target visual tokens and text prompts are input into a multimodal large model for reasoning to obtain reasoning results, including: Projecting the plurality of target visual tokens into the semantic space of the text prompt by a projection connector to obtain semantic space-aligned visual tokens; The visual tokens aligned in the semantic space and the text prompts processed by the word segmenter are input into the multimodal large model for reasoning to obtain the reasoning results.

[0012] A second aspect of the embodiments of the present application discloses a multimodal large model training-free inference acceleration device based on visual token pruning, the device comprising: The first input module is used to input the target image into the visual encoder for multi-layer attention interaction, thereby obtaining a visual token sequence consisting of multiple visual tokens and a multi-layer class token-to-visual token attention score distribution; each layer of attention interaction integrates the global and local information of the visual token based on the class token, and the class token-to-visual token attention score distribution is similar to the visual token attention score distribution of the multimodal large model; A first determining module is configured to determine a final attention score of a class token to each visual token based on the attention score distribution of the multi-layer class tokens to the visual tokens, wherein the final attention score represents the importance of the visual token; a first selection module, configured to select a plurality of target visual tokens with high importance from the visual token sequence according to the final attention scores of the class tokens on the respective visual tokens; The second input module is used to input the multiple target visual tokens and text prompts into the language model for reasoning to obtain reasoning results.

[0013] According to a third aspect of an embodiment of the present application, an electronic device is disclosed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the multimodal large model untrained inference acceleration method based on visual token pruning described in the first aspect of the embodiment of the present application are implemented.

[0014] The fourth aspect of the embodiments of the present application discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the multimodal large model untrained inference acceleration method based on visual token pruning described in the first aspect of the embodiments of the present application are implemented.

[0015] The fifth aspect of the embodiments of the present application discloses a computer program product, including a computer program, which, when executed by a processor, implements the steps of the multimodal large model untrained inference acceleration method based on visual token pruning described in the first aspect of the embodiments of the present application.

[0016] The embodiments of the present application include the following advantages: In an embodiment of the present application, a target image is input into a visual encoder for multi-layer attention interaction to obtain a visual token sequence consisting of multiple visual tokens, and a multi-layer class token attention score distribution to the visual tokens; based on the multi-layer class token attention score distribution to the visual tokens, a final attention score of the class token to each visual token is determined, and the final attention score represents the importance of the visual token; and based on the final attention score of the class token to each visual token, a plurality of target visual tokens with high importance are selected from the visual token sequence; the plurality of target visual tokens and text prompts are input into a language model for reasoning to obtain an inference result.

[0017] Since each layer of attention interaction of the visual encoder integrates the global and local information of visual tokens based on class tokens, and the attention score distribution of class tokens to visual tokens is similar to the attention score distribution of visual tokens in the multimodal large model, the class token pairs can have a good fitting ability for the parts of their attention without seeing the text tokens. The attention information based on the class token pairs to visual tokens can effectively grasp the global content and local features of the visual information at the same time, so as to correctly capture important visual tokens (i.e., multiple target visual tokens) containing necessary information, and perform reasoning based on multiple target visual tokens. In this way, it can effectively accelerate the reasoning process of the multimodal large model while maximizing the performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 This is a flowchart of a method for accelerating multimodal large-model training-free reasoning based on visual token pruning provided by an embodiment of the present application; Figure 2 This is a flowchart of another method for accelerating multimodal large model training-free reasoning based on visual token pruning provided by an embodiment of the present application; Figure 3 This is another flowchart of untrained reasoning of a multimodal large model based on visual token pruning provided by an embodiment of the present application; Figure 4 1 is a schematic structural diagram of a multimodal large model untrained inference device based on visual token pruning provided by an embodiment of the present application; Figure 5 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] To make the above-mentioned purposes, features, and advantages of this application more clearly understood, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of this application.

[0021] Existing methods for accelerating inference on large multimodal models include improving the model structure and pruning during inference. 1) The model structure improvement approach utilizes lightweight multimodal models to accelerate inference. These models employ lightweight projection connectors and methods such as downsampling to reduce the number of visual tokens fed into the language model, achieving acceleration and enabling mobile deployment.

[0022] 2) Existing pruning schemes during inference include the following: The first pruning method during inference is implemented based on the fact that visual tokens receive less attention than other tokens corresponding to language tokens and special symbols. Removing most of the relatively unimportant visual tokens in the shallow layer of the language model part of the multimodal large model will not have a decisive impact on the overall performance of the model. Therefore, this method measures the importance of each visual token by the attention score between the visual token and the prompt word token in the language model, and then retains the most important tokens and discards the rest. The second pruning method during inference uses a pruning-fusion method for visual tokens after the visual encoder. First, the most important tokens are selected through the attention mechanism and mixed with the tokens uniformly sampled in the space. Then, the unselected tokens are merged into the selected important tokens through the nearest neighbor judgment method. In this way, a two-stage method is used to effectively extract visual information.

[0023] Both of the above pruning methods can be used without additional training during the inference process. In addition, some solutions must be retrained before they can be used. One pruning method that requires training is to compress visual tokens into specific VoCo tokens in combination with the understanding mechanism of the language model; another pruning method that requires training compares the effects of compression at different layers of the language model using different compression rates, and designs a multi-level hierarchical compression method that uses a simple pooling strategy to compress visual tokens; some other pruning methods that require training are to stack visual tokens and inject them into the layers of the model from the bottom to the top, allowing the model to process more visual tokens under the same circumstances without changing the model structure.

[0024] Current approaches to improving model structure and pruning during inference both have their own shortcomings. Model structure improvement approaches utilize lightweight multimodal models to accelerate inference. However, these approaches require significant time and computational resources to retrain the models and cannot be immediately applied to new models. Their plug-and-play nature is inferior to approaches that do not require training. Currently proposed approaches that do not require training can significantly impact model performance in untrained scenarios. For example, the first pruning approach in the inference process described above primarily utilizes the inference information between the cue word and the visual token. This may overlook the relationship between the content to be generated and the input image, potentially discarding visual tokens containing important visual information. This can lead to significant cognitive biases in the model's understanding and degraded multimodal task performance. The second pruning approach in the inference process is not a particularly effective approach in untrained scenarios and may significantly degrade model performance, necessitating new technical solutions.

[0025] In order to overcome the limitations of related technologies, an embodiment of the present application provides a method for accelerating untrained inference of a multimodal large model based on visual token pruning. This method can fully utilize the intermediate information generated by the inference during the inference process of the multimodal large model, effectively select the relatively important part of the visual tokens, and improve the performance of the model on various multimodal tasks after the pruning operation, making it as close as possible to the performance of the model without pruning. Specifically, the method utilizes the attention information of the class token (CLS token), and prunes the visual token after the visual encoder, while fully demonstrating its accuracy in selecting tokens, and effectively accelerates the inference of the model while minimizing the impact on the model performance.

[0026] The following is a detailed description of the multimodal large model untrained reasoning acceleration method based on visual token pruning provided in the embodiment of the present application in conjunction with the accompanying drawings.

[0027] Reference Figure 1 As shown, Figure 1 This is a flowchart of a multimodal large model non-training reasoning acceleration method based on visual token pruning provided by an embodiment of the present application. Figure 1 As shown, the multimodal large model non-training reasoning acceleration method based on visual token pruning may include steps S110 to S140: Step S110: Input the target image into the visual encoder for multi-layer attention interaction to obtain a visual token sequence composed of multiple visual tokens, and a multi-layer class token attention score distribution to the visual token; each layer of attention interaction integrates the global information and local information of the visual token based on the class token, and the attention score distribution of the class token to the visual token is similar to the visual token attention score distribution of the multimodal large model.

[0028] Among them, the target image refers to the visual information input into the multimodal large model, and the visual encoder is the encoder used to process visual information in the multimodal large model.

[0029] In the multimodal large model, both the visual encoder and the language model (used for content understanding and content generation) can measure the importance of different tokens through attention information. In the process of the visual encoder processing the target image, the visual encoder has not seen the tokens of the text (including prompt words and generated words); at the same time, although the language model part is visible to the tokens of the prompt words, it cannot see the future generated content during the generation process, which makes it difficult to select the optimal token set during the reasoning process. However, the embodiment of the present application takes into account that the visual encoder uses class tokens to collect global information and local information during forward propagation, which is a structure that does not exist in the language model, and by comparing the attention score distribution of the visual tokens of the language model and the attention score distribution of the class tokens, it is found that the two are similar. Therefore, the embodiment of the present application believes that the class token can have a better fitting ability for the part to which its attention is focused without seeing the text token, and thus measures the importance of the visual token based on the attention score of the class token to the visual token.

[0030] The visual encoder consists of multiple layers of attention interaction. First, the target image is input into the visual encoder for multi-layer attention interaction, resulting in a visual token sequence consisting of multiple visual tokens and a multi-layer distribution of attention scores between class tokens and visual tokens. The visual token sequence refers to the feature representation output by the last layer of attention interaction of the visual encoder. Each layer of attention interaction will generate a corresponding distribution of attention scores between class tokens and visual tokens, that is, the attention scores of class tokens on each visual token in the attention interaction of that layer.

[0031] Step S120: Determine the final attention score of the class token to each visual token based on the attention score distribution of the multi-layer class token to the visual token, and the final attention score represents the importance of the visual token.

[0032] Since the attention score distribution of class tokens to visual tokens may have different distributions in the attention interactions of different layers of the visual encoder, that is, different layers of attention interactions focus on different features of the image, if only the attention information of one layer of class tokens is used (that is, the final attention score distribution of one layer of class tokens to each visual token), it may lead to inaccurate capture of visual information.

[0033] Therefore, an embodiment of the present application combines the attention score distribution of multi-layer class tokens to visual tokens in the visual encoder to select visual tokens that carry important visual information, that is, based on the attention score distribution of multi-layer class tokens to visual tokens, the final attention score of the class token to each visual token is determined, and the importance of the visual token is measured by the final attention score.

[0034] Step S130: Select multiple target visual tokens with high importance from the visual token sequence based on the final attention score of the class token to each visual token.

[0035] In the embodiment of the present application, the final attention score represents the importance of the visual token, and then based on the final attention score of each visual token by the class token, multiple target visual tokens with high importance are selected for subsequent reasoning, and the remaining visual tokens are discarded. In some embodiments, the visual tokens in the visual token sequence can be sorted according to the final attention score of each visual token by the class token, and the top N visual tokens are used as target visual tokens.

[0036] Step S140: Input the multiple target visual tokens and text prompts into a language model for reasoning to obtain reasoning results.

[0037] In an embodiment of the present application, a text prompt is text information input into a multimodal large model, and a language model is a model in the multimodal large model used for content understanding and content generation. An inference result is obtained by inputting multiple target visual tokens and text prompts into the language model for inference; wherein the inference result mainly depends on the text prompt. For example, if the text prompt is a question, the inference result is the answer to the question.

[0038] The language model is constructed from a stack of multi-layer transformer architectures (with attention interaction layers). The internal structure of each layer is similar to that of the visual encoder, but with a higher number of parameters and stacked layers. The language model's parameters account for the vast majority of the parameters in a large multimodal model, and therefore incur the majority of the computational overhead. Since the number of tokens corresponding to visual information is typically an order of magnitude greater than that corresponding to textual cues, pruning before passing information to the language model effectively reduces computational overhead while preserving as much visual information as possible.

[0039] By adopting the technical solution of the embodiment of the present application, since each layer of attention interaction of the visual encoder integrates the global information and local information of the visual token based on the class token, and the attention score distribution of the class token to the visual token is similar to the attention score distribution of the visual token of the multimodal large model, the class token pair can have a better fitting ability for the part of its attention without seeing the text token. The attention information of the class token to the visual token can effectively grasp the global content and local features of the visual information at the same time, so as to correctly capture important visual tokens (i.e., multiple target visual tokens) containing necessary information, so as to perform reasoning based on the multiple target visual tokens. In this way, the performance of the model can be maximized while effectively accelerating the reasoning process of the multimodal large model.

[0040] In combination with the above embodiments, an embodiment of the present application provides a multimodal large model untrained inference acceleration method based on visual token pruning. In this method, the visual encoder includes a token embedding layer and a multi-layer attention interaction layer. The above step S110 "inputs the target image into the visual encoder for multi-layer attention interaction to obtain a visual token sequence consisting of multiple visual tokens, and a multi-layer class token attention score distribution for the visual tokens" specifically includes steps S110-1 to S110-2: Step S110 - 1 : processing the target image into an initial visual token sequence consisting of a plurality of visual tokens based on the token embedding layer.

[0041] Step S110-2: Input the initial visual token sequence and class tokens into the multi-layer attention interaction layer for attention interaction to obtain a visual token sequence composed of multiple visual tokens and an attention score distribution of the multi-layer class tokens to the visual tokens.

[0042] The visual encoder in the multimodal large model can be a pre-trained visual encoder, and the parameters of the visual encoder do not need to be changed again during the entire process of multimodal large model training, fine-tuning, and inference. When the visual encoder processes the target image, it first inputs the target image into the token embedding layer for data augmentation and size scaling, and then divides the processed image into multiple small blocks, each of which represents a visual token, thereby obtaining an initial visual token sequence. The initial visual token sequence and class tokens are then input into the multi-layer attention interaction layer for attention interaction (forward propagation). The multi-layer attention interaction layer can capture the short-range and long-range dependencies between different areas of the image, and ultimately obtain a visual token sequence consisting of multiple visual tokens, as well as the attention score distribution of the multi-layer class tokens on the visual tokens.

[0043] For example, the multimodal large model LLaVA-v1.5 is used as an example. The visual encoder used is CLIP-ViT / L-14-336. The processing process of the target image is explained. When processing the target image of , it will first go through the process of data enhancement (normalization, standardization) and size scaling to obtain a size and number of channels of The tensor is then divided into 576 drag_maildrag_mail The small blocks (visual tokens) are flattened and fed into the transformer structure (attention interaction layer). The attention interaction layer captures short-range and long-range dependencies between different image regions. In addition to the 576 visual tokens mentioned above, the attention interaction layer also includes a manually added class token (CLS token) that participates in the entire training and inference process. A total of 577 tokens are propagated forward. Each attention interaction layer consists of a self-attention mechanism, a multi-layer perceptron, and two layer normalization modules. Residual connections are used during forward propagation to preserve the original features.

[0044] Furthermore, each attention interaction layer includes multiple attention heads, and the attention score distribution of each layer of class tokens to visual tokens is obtained as follows: For each visual token, the average of the attention scores of the class token output by multiple attention heads to the visual token is used as the attention score of the class token to the visual token; based on the attention scores of the class token to each visual token, the attention score distribution of the class token to the visual token is obtained.

[0045] In the embodiment of the present application, each attention interaction layer includes multiple attention heads, and the initial visual token sequence and the class token are input to each attention head for processing. For each visual token, each attention head can output the attention score of the class token to the visual token, and the average of the attention scores of the class token input by multiple attention heads to the visual token is used as the attention score of the class token to the visual token. The attention score of the class token output by the attention head to the visual token is recorded as , then the average of the attention scores of the class tokens input by multiple attention heads on the visual token is , as the attention score of the class token to the visual token, where is the number of attention heads in the attention interaction layer of the visual encoder.

[0046] In this way, in each layer of attention interaction, the average of the attention scores of the class token output by multiple attention heads to the visual token is used as the attention score of the class token to the visual token, so as to obtain the attention score distribution of the class tokens of each layer to the visual token, so as to measure the importance of each visual token based on the attention score distribution of the class tokens of each layer to the visual token.

[0047] In combination with the above embodiments, the embodiment of the present application provides a multimodal large model untrained inference acceleration method based on visual token pruning. In this method, the above step S120 of "determining the final attention score of the class token to each visual token based on the attention score distribution of the multi-layer class token to the visual token" specifically includes: For each visual token, the attention score distribution of the multi-layer class tokens corresponding to the penultimate layer to the previous K layers on the visual token is used to determine the multiple attention scores of the class token on the visual token; based on the multiple attention scores of the class token on the visual token, the final attention score of the class token on the visual token is calculated.

[0048] In an embodiment of the present application, in order to accurately capture visual information, starting from the penultimate layer, the penultimate layer and the K layers before it are selected, and the importance of each visual token is measured by combining the attention score distribution of the class tokens corresponding to these layers to the visual tokens.

[0049] Specifically, for each visual token, the class token has an attention score for the visual token at each layer, thereby determining multiple attention scores of the class token for the visual token, and then calculating the final attention score based on the multiple attention scores of the class token for the visual token.

[0050] In some embodiments, the final attention score of the class token to the visual token is calculated based on the multiple attention scores of the class token to the visual token, including: taking the average of the multiple attention scores of the class token to the visual token as the final attention score of the class token to the visual token; or taking the maximum attention score among the multiple attention scores of the class token to the visual token as the final attention score of the class token to the visual token.

[0051] That is to say, the final attention score of the class token to each visual token can be calculated by the average value, or the final attention score of the class token to each visual token can be calculated by the maximum value, and finally the final attention score distribution of the class token to the visual token is obtained.

[0052] For example, the final attention score distribution of class tokens over visual tokens is expressed as: , in, represents the final attention score distribution of class tokens to visual tokens, including the final attention scores of class tokens to each visual token; is the function for integrating data between different layers, is the number of layers of the visual encoder, Indicates the second to last layer (i.e. The attention score distribution of class tokens to visual tokens at the layer, Indicates the The attention score distribution of the class tokens to the visual tokens of the layer, Indicates the Attention score distribution of class tokens to visual tokens of a layer.

[0053] In this way, according to the final attention score distribution of class tokens to visual tokens, multiple target visual tokens with the highest attention scores are retained to participate in the subsequent reasoning process, and the rest are all discarded.

[0054] In combination with the above embodiments, the present application provides a method for accelerating untrained inference of a multimodal large model based on visual token pruning. In this method, the above step S140 of "inputting the multiple target visual tokens and text prompts into the multimodal large model for inference to obtain inference results" specifically includes steps S140-1 to S140-2: Step S140 - 2 : Projecting the multiple target visual tokens into the semantic space of the text prompt through a projection connector to obtain semantic space-aligned visual tokens.

[0055] Step S140 - 2 : Inputting the semantically space aligned visual tokens and the text prompts processed by the word segmenter into the multimodal large model for reasoning to obtain reasoning results.

[0056] In the embodiment of the present application, after the target image is processed by the visual encoder, the dependencies between different information are captured, resulting in a visual token sequence consisting of multiple visual tokens, where each visual token is equivalent to a feature vector. In other words, the visual token sequence is equivalent to a tensor composed of multiple feature vectors. In order for the semantic model to understand and process the visual token sequence (visual information), the visual token sequence needs to be aligned with the semantic space of the prompt text.

[0057] Specifically, a trained projection connector is used to project multiple target visual tokens into the semantic space of the textual prompt. The projection connector consists of two linear layers and a GeLU activation function. The multiple target visual tokens are processed through a linear layer, then through a GeLU activation function, and finally the result of the GeLU activation function is input into another linear layer for processing to achieve the purpose of alignment with the semantic space of the textual prompt.

[0058] For example, projecting the multiple target visual tokens into the semantic space of the text prompt through the projection connector can be expressed as: , in, is the weight of the first linear layer, is the bias parameter of the first linear layer, is the weight of the second linear layer, is the bias parameter of the second linear layer, For multiple target visual tokens, Aligning visual tokens for semantic space.

[0059] Finally, the text prompts are processed using a word segmenter (i.e., a text editor) to transform them into semantic tokens that can directly assist the language model. The visual tokens and semantic tokens (i.e., the text prompts processed by the word segmenter) aligned in the semantic space are then input into the multimodal large model for reasoning to obtain the inference results.

[0060] In this way, by projecting multiple target visual tokens into the semantic space of textual prompts, the semantic space of visual information and textual prompts is aligned, so that the language model can understand and process visual information.

[0061] Reference Figure 2 As shown, Figure 2 1 is a flowchart of another method for accelerating multimodal large model training-free reasoning based on visual token pruning provided by an embodiment of the present application. Specifically, the method includes steps S210 to S250: Step S210: Based on the token embedding layer, the target image is processed into an initial visual token sequence consisting of multiple visual tokens; the initial visual token sequence and the class token are input into the multi-layer attention interaction layer for attention interaction to obtain a visual token sequence consisting of multiple visual tokens, and the attention score distribution of the multi-layer class tokens to the visual tokens; each layer of attention interaction integrates the global information and local information of the visual token based on the class token, and the attention score distribution of the class token to the visual token is similar to the visual token attention score distribution of the multimodal large model.

[0062] Step S220: For each visual token, determine the multiple attention scores of the class token to the visual token from the second-to-last layer to the previous K layers corresponding to the multi-layer class tokens, and calculate the final attention score of the class token to the visual token based on the multiple attention scores of the class token to the visual token.

[0063] Step S230: Determine the final attention score of the class token to each visual token based on the attention score distribution of the multi-layer class token to the visual token, and the final attention score represents the importance of the visual token.

[0064] Step S240: Projecting the multiple target visual tokens into the semantic space of the text prompt through a projection connector to obtain semantic space-aligned visual tokens.

[0065] Step S250: inputting the semantic space aligned visual tokens and the text prompts processed by the word segmenter into the multimodal large model for reasoning to obtain reasoning results.

[0066] In an embodiment of the present application, since each layer of attention interaction of the visual encoder integrates the global information and local information of the visual token based on the class token, and the attention score distribution of the class token to the visual token is similar to the attention score distribution of the visual token of the multimodal large model, the class token pair can have a better fitting ability for the part of its attention without seeing the text token. The attention information of the class token to the visual token can effectively grasp the global content and local features of the visual information at the same time, so as to correctly capture important visual tokens (i.e., multiple target visual tokens) containing necessary information, so as to perform reasoning based on multiple target visual tokens. In this way, the performance of the model can be maximized while effectively accelerating the reasoning process of the multimodal large model.

[0067] Reference Figure 3 As shown, Figure 3 This is a flowchart of another multimodal large model untrained reasoning based on visual token pruning provided by an embodiment of the present application. Among them, the modal large model includes a visual encoder, a projection connector, a word segmenter and a language model. Specifically, the visual encoder includes a token embedding layer and a multi-layer attention interaction layer. The target image is input into the visual encoder for processing. The visual encoder processes the target image into an initial visual token sequence composed of multiple visual tokens based on the token embedding layer. Then, the initial visual token sequence and the class token are input into the multi-layer attention interaction layer for attention interaction, and a visual token sequence composed of multiple visual tokens and an attention score distribution of the multi-layer class tokens on the visual tokens are obtained.

[0068] Next, for each visual token, the attention scores of the class tokens corresponding to the penultimate layer up to the previous K layers are distributed. Multiple attention scores of the class token on the visual token are determined, and based on these multiple attention scores, the final attention score of the class token on the visual token is calculated. Furthermore, based on the final attention scores of the class token on each visual token, multiple target visual tokens with high importance are selected from the visual token sequence.

[0069] Finally, multiple target visual tokens are projected into the semantic space of the text prompt through a projection connector to obtain semantic space-aligned visual tokens; the semantic space-aligned visual tokens and the text prompts processed by the word segmenter are input into the multimodal large model for reasoning to obtain the reasoning results.

[0070] In this way, through token-like attention information, we can effectively grasp the global content and local features of visual information at the same time, correctly capture important visual tokens containing necessary information, and maximize the performance of the model while effectively accelerating the inference process of large multimodal models.

[0071] The embodiment of the present application also provides a multimodal large model non-training reasoning acceleration device based on visual token pruning, referring to Figure 4 As shown, Figure 4 : This is a structural diagram of a multimodal large model non-training inference acceleration device based on visual token pruning provided by an embodiment of the present application, the device comprising: A first input module 410 is configured to input a target image into a visual encoder for multi-layer attention interaction, thereby obtaining a visual token sequence consisting of multiple visual tokens and a multi-layer attention score distribution of class tokens to the visual tokens; each layer of attention interaction integrates global and local information of the visual tokens based on the class tokens, and the attention score distribution of the class tokens to the visual tokens is similar to the attention score distribution of the visual tokens in the multimodal large model; A first determining module 420 is configured to determine a final attention score of a class token to each visual token based on the attention score distribution of the multi-layer class tokens to the visual tokens, wherein the final attention score represents the importance of the visual token; A first selection module 430 is configured to select a plurality of target visual tokens with high importance from the visual token sequence according to the final attention scores of the class tokens on each visual token; The second input module 440 is configured to input the plurality of target visual tokens and text prompts into a language model for inference to obtain an inference result.

[0072] In an optional embodiment, the visual encoder includes a token embedding layer and a multi-layer attention interaction layer; the first input module includes: a first processing module, configured to process the target image into an initial visual token sequence consisting of a plurality of visual tokens based on the token embedding layer; The first input submodule is used to input the initial visual token sequence and class tokens into the multi-layer attention interaction layer for attention interaction, so as to obtain a visual token sequence composed of multiple visual tokens and an attention score distribution of the multi-layer class tokens to the visual tokens.

[0073] In an optional embodiment, each attention interaction layer includes multiple attention heads, and the first input submodule is also used to: for each visual token, take the average of the attention scores of the class token output by multiple attention heads to the visual token as the attention score of the class token to the visual token; and obtain the attention score distribution of the class token to the visual token based on the attention scores of the class token to each visual token.

[0074] In an optional embodiment, the first determining module includes: The first determination submodule is used to determine, for each visual token, the attention score distribution of the multi-layer class tokens corresponding to the penultimate layer to the previous K layers on the visual token, and determine multiple attention scores of the class tokens on the visual token; The first calculation module is used to calculate a final attention score of the class token to the visual token based on multiple attention scores of the class token to the visual token.

[0075] In an optional embodiment, the first calculation module is further used to: take the average of multiple attention scores of the class token to the visual token as the final attention score of the class token to the visual token; or take the maximum attention score among the multiple attention scores of the class token to the visual token as the final attention score of the class token to the visual token.

[0076] In an optional embodiment, the second input module includes: a projection module, configured to project the plurality of target visual tokens into the semantic space of the text prompt via a projection connector to obtain semantic space-aligned visual tokens; The second input submodule is used to input the visual tokens aligned in the semantic space and the text prompts processed by the word segmenter into the multimodal large model for reasoning to obtain the reasoning results.

[0077] The present application also provides an electronic device, Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the electronic device 500 includes: a memory 510 and a processor 520. The memory 510 and the processor 520 are connected via a bus communication. A computer program is stored in the memory 510, and the computer program can be run on the processor 520, thereby implementing the steps of the multimodal large model untrained inference acceleration method based on visual token pruning described in the embodiment of the present application.

[0078] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the multimodal large model untrained inference acceleration method based on visual token pruning described in the embodiment of the present application are implemented.

[0079] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the multimodal large model untrained inference acceleration method based on visual token pruning described in the embodiment of the present application.

[0080] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0081] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods and devices according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0082] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0083] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0084] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0085] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0086] The above is a detailed introduction to the multimodal large model untrained inference acceleration method and device based on visual token pruning provided by this application. This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method of this application and its core idea; at the same time, for general technical personnel in this field, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this application.

Claims

1. A multimodal large model non-training reasoning acceleration method based on visual token pruning, characterized in that: The method comprises: Input the target image into the visual encoder for multi-layer attention interaction, and obtain a visual token sequence consisting of multiple visual tokens, and a multi-layer class token attention score distribution to the visual token; each layer of attention interaction integrates the global information and local information of the visual token based on the class token, and the class token attention score distribution to the visual token is similar to the visual token attention score distribution of the multimodal large model; Determine, according to the distribution of the attention scores of the multi-layer class tokens to the visual tokens, a final attention score of the class tokens to each visual token, wherein the final attention score represents the importance of the visual token; Selecting a plurality of target visual tokens with high importance from the visual token sequence according to the final attention scores of the class tokens to each visual token; The multiple target visual tokens and text prompts are input into a language model for reasoning to obtain a reasoning result.

2. The method according to claim 1, characterized in that: The visual encoder includes a token embedding layer and a multi-layer attention interaction layer; the target image is input into the visual encoder for multi-layer attention interaction, and a visual token sequence consisting of a plurality of visual tokens and an attention score distribution of the multi-layer class tokens to the visual tokens are obtained, including: processing the target image into an initial visual token sequence consisting of a plurality of visual tokens based on the token embedding layer; The initial visual token sequence and class tokens are input into the multi-layer attention interaction layer for attention interaction to obtain a visual token sequence composed of multiple visual tokens and an attention score distribution of the multi-layer class tokens to the visual tokens.

3. The method according to claim 2, characterized in that Each attention interaction layer includes multiple attention heads, and the attention score distribution of each layer of class tokens to visual tokens is obtained as follows: For each visual token, the average of the attention scores of the class tokens output by multiple attention heads on the visual token is used as the attention score of the class token on the visual token; According to the attention scores of the class tokens to each visual token, the attention score distribution of the class tokens to the visual tokens is obtained.

4. The method according to claim 1, characterized in that: According to the distribution of the attention scores of the multi-layer class tokens to the visual tokens, determining the final attention scores of the class tokens to each visual token includes: For each visual token, the attention score distribution of the multi-layer class tokens corresponding to the second-to-last layer to the previous K layers on the visual token is used to determine multiple attention scores of the class tokens on the visual token; According to the multiple attention scores of the class token to the visual token, a final attention score of the class token to the visual token is calculated.

5. The method according to claim 4, characterized in that According to the multiple attention scores of the class token to the visual token, a final attention score of the class token to the visual token is calculated, including: Taking the average of multiple attention scores of the class token to the visual token as the final attention score of the class token to the visual token; or, The maximum attention score among the multiple attention scores of the class token to the visual token is taken as the final attention score of the class token to the visual token.

6. The method according to any one of claims 1 to 5, characterized in that: Inputting the plurality of target visual tokens and text prompts into the multimodal large model for reasoning to obtain reasoning results, including: Projecting the plurality of target visual tokens into the semantic space of the text prompt through a projection connector to obtain semantic space-aligned visual tokens; The visual tokens aligned in the semantic space and the text prompts processed by the word segmenter are input into the multimodal large model for reasoning to obtain reasoning results.

7. A multimodal large model non-training reasoning acceleration device based on visual token pruning, characterized in that: The device comprises: The first input module is used to input the target image into the visual encoder for multi-layer attention interaction to obtain a visual token sequence composed of multiple visual tokens and a multi-layer class token attention score distribution to the visual token; each layer of attention interaction integrates the global information and local information of the visual token based on the class token, and the class token attention score distribution to the visual token is similar to the visual token attention score distribution of the multimodal large model; A first determination module is used to determine a final attention score of the class token to each visual token according to the attention score distribution of the multi-layer class token to the visual token, wherein the final attention score represents the importance of the visual token; A first selection module, configured to select a plurality of target visual tokens with high importance from the visual token sequence according to the final attention scores of the class tokens to each visual token; The second input module is used to input the multiple target visual tokens and text prompts into the language model for reasoning to obtain reasoning results.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the multimodal large model non-training reasoning acceleration method based on visual token pruning according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multimodal large model untrained reasoning acceleration method based on visual token pruning described in any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the multimodal large model untrained reasoning acceleration method based on visual token pruning are implemented as described in any one of claims 1-6.

Images