A visual token pruning method based on graph information propagation

Abstract

The application provides a visual token pruning method based on graph information propagation, comprising the following steps: visual extraction is performed on an input image to obtain visual tokens; importance scores of the visual tokens are initialized; a graph structure about the visual tokens is constructed, each visual token is taken as a node, an adjacency matrix is calculated to construct connections between the visual tokens, and the graph structure is initialized; the adjacency matrix is updated through a preset similarity threshold to obtain visual token subgraph structures of different regions; each row of the adjacency matrix is normalized, node information is iteratively propagated, and final scores of each visual token are calculated; k visual tokens with the highest scores are selected according to the final scores of the visual tokens and are projected; the visual tokens obtained through the projection are spliced with text tokens, input into a large language model, and output results are obtained. The method can improve the calculation efficiency of the model, significantly reduces the calculation cost while maintaining the performance of the visual task.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a visual token pruning method based on graph information propagation. Background Technology

[0002] With the rapid development of digital media and artificial intelligence technologies, large multimodal language models (MLLM) have shown unprecedented application potential in fields such as human-computer interaction, automatic content generation, and data analysis. Especially in the intelligent processing of image and video content, MLLM, leveraging its powerful language understanding and multimodal data processing capabilities, is gradually driving the technology towards broader application prospects.

[0003] In the development of multimodal models, the visual modality is particularly important. Traditional methods require models to process a large number of visual tokens to understand and generate content, which not only consumes significant computational resources but also increases the time required for model training and inference. Especially in tasks requiring high-resolution and large-scale visual data processing, the consumption of computational resources becomes a major bottleneck limiting model application.

[0004] Furthermore, many visual tokens contain redundant information, which not only increases computational burden but also reduces model processing speed and efficiency. The introduction of pruning techniques, particularly in the optimization of visual tokens, aims to select tokens that have the greatest impact on the final task, while discarding tokens that are redundant or have little impact on model performance. This method can significantly reduce model complexity and improve model efficiency and speed.

[0005] For example, an existing Chinese invention patent (publication number CN116933859A) – a dynamic pruning method for visual Transformers – discloses the following steps: Step 1: Constructing an image dataset, which will be used for subsequent model training and testing; Step 2: Building a visual Transformer network model by stacking multiple encoder blocks after a location embedding module, inserting a token pruning module after a specific encoder block, and embedding a channel sparsity module in each layer to sparsify the feature channels; Step 3: Training the visual Transformer network model, while optimizing the model's weights and structural parameters through a dynamic pruning method during training. The pruning targets include the token sequence and the channel parameters of the fully connected layers; Step 4: Selecting the sub-token sequence with the highest global information score and the channel with the highest importance weight from the training process to reconstruct a lightweight network model, thereby reducing the model's computational resource requirements and improving execution efficiency. And the existing Chinese invention patent (publication number CN115310607A) – a model optimization method based on dynamic pruning technology and visual Transformer, which discloses: In a machine vision inference system, firstly, the ViT model is initially trained for several rounds using a data-driven method to generate complete attention maps; then, the information entropy of these attention maps is calculated, and the attention heads are pruned based on the obtained information entropy values; next, the weight parameters corresponding to the pruned attention heads are removed to construct a new ViT model; then, the parameters of this new ViT model are fine-tuned; by pruning the modules in the multi-head attention mechanism, removing feature maps and their corresponding attention heads with high uncertainty, the parameters and complexity of the ViT model can be reduced, the computational complexity and the total number of parameters can be reduced, thereby effectively reducing the size of the ViT model and achieving lightweighting of the model with minimal performance loss.

[0006] These patents, by introducing pruning techniques in image processing tasks, reduce the amount of data processed to some extent, but still have the following drawbacks and shortcomings: One dynamic pruning method for visual Transformers (publication number CN116933859A) implements dynamic pruning on the visual Transformer model, learning the importance of features at different levels through training, thereby achieving efficient network operation. However, this method relies on constructing high-quality image datasets for training and updating parameters, resulting in high data and training costs. Another model optimization method based on dynamic pruning and visual Transformers (publication number CN115310607A) prunes based on the information entropy of attention maps, mainly focusing on salient features of the image, which may lead to the loss of background or contextual information. This insignificant information may be equally important in the multimodal model's understanding of the overall scene, and it also requires fine-tuning to update parameters.

[0007] Therefore, how to effectively prune visual tokens and reduce unnecessary computational overhead while ensuring model performance has become a pressing technical problem. Summary of the Invention

[0008] The technical problem to be solved by this invention is to provide a visual token pruning method based on graph information propagation, which improves the computational efficiency of multimodal large language models and significantly reduces computational costs while maintaining the performance of visual tasks.

[0009] This invention is implemented as follows: a visual token pruning method based on graph information propagation, the method comprising:

[0010] Step S1: Visual tokens are obtained by visually extracting the input image using a visual encoder;

[0011] Step S2: Initialize the importance score of each visual token by calculating the L2 norm of each visual token, and obtain the importance scores of all tokens;

[0012] Step S3: Construct a graph structure for visual tokens, treating each visual token as a node, and use importance scores to calculate an adjacency matrix to build connections between visual tokens, thus initializing the graph structure.

[0013] Step S4: Update the adjacency matrix using a preset similarity threshold to obtain the visual token subgraph structure for different regions;

[0014] Step S5: Normalize each row of the adjacency matrix, iteratively propagate the node information according to the preset number of iterations n, and calculate the final score of each visual token by combining the degree of the visual token.

[0015] Step S6: Select the k visual tokens with the highest scores based on the final scores of the visual tokens and project them.

[0016] Step S7: Concatenate the visual token and text token obtained from the projection, input them into the large language model, and obtain the output result of the question answering task.

[0017] Furthermore, step S2 specifically involves: processing the N d-dimensional visual token matrix X extracted by the visual encoder. v =[v1,v2,…,v N ]∈ N×d Using the L2 norm to assign initial information to each visual token, we obtain the importance score vector W∈ for all visual tokens. N×1 ;

[0018] The l2 norm of the visual token is calculated as follows:

[0019]

[0020] W = [||v1||2,||v2||2,…,||v N ||2] T

[0021] Among them, v ij This represents the j-th dimension component of the i-th token.

[0022] Furthermore, step S3 specifically involves: defining an adjacency matrix A∈ based on the importance score of the visual token using cosine similarity. N×N :

[0023]

[0024] Among them, v i and v j Let i and j be the vector representations of the i-th and j-th visual tokens, respectively.

[0025] Treating each visual token as a node, we model the interactions between visual tokens in the visual context using an adjacency matrix A, thus obtaining a graph structure about the visual tokens.

[0026] Furthermore, step S4 specifically includes:

[0027] Set a similarity threshold s, and update the adjacency matrix through the similarity threshold s, so that only visual tokens with high mutual similarity can be connected, and obtain the visual token subgraph structure of different regions. The update formula of the adjacency matrix is ​​as follows;

[0028]

[0029] Among them, A′ij This is the updated adjacency matrix.

[0030] Furthermore, step S5 specifically includes:

[0031] Use the Softmax() function to normalize each row of the adjacency matrix A':

[0032]

[0033] Where T is a temperature parameter used to control the smoothness of the Softmax(·) function;

[0034] Perform an nth-degree exponentiation operation:

[0035] A (n) =(A t ) n

[0036] Where n represents the number of iterations, this process is used to model the multi-round iterative propagation of node information;

[0037] Calculate the number of edges connected to each visual token, i.e., the degree D∈ 1×N Define it as:

[0038]

[0039]

[0040] The final score of the computational visual token after graph information iteration is as follows:

[0041]

[0042] Where S is the final score vector of the visual token, and W is the importance score vector of the visual token.

[0043] Furthermore, step S6 specifically includes:

[0044] Select the k visual tokens with the highest scores:

[0045] X g =[v1,v2,…,v k ]

[0046] Among them, each v i From the visual token matrix X v Choose from the options that satisfy:

[0047]

[0048] Among them, v k This indicates that according to ||v i||2 The visual token located at position k in a list sorted in descending order.

[0049] This invention has the following advantages: By proposing a visual token pruning method based on graph information propagation, a relationship graph between visual tokens is constructed, and the importance scores of the tokens are propagated and updated through an iterative algorithm, thereby accurately selecting the tokens that have the greatest impact on the task. This achieves the goal of reducing the consumption of computing resources while ensuring the processing efficiency of multimodal models, enabling the model to run efficiently in resource-constrained environments. The pruning method based on graph information propagation in this invention requires no training and is plug-and-play, making it particularly suitable for application scenarios that require real-time processing of large amounts of data. Attached Figure Description

[0050] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0051] Figure 1 This is a flowchart illustrating the execution of a visual token pruning method based on graph information propagation according to the present invention.

[0052] Figure 2 This is a schematic diagram of the framework structure of a visual token pruning method based on graph information propagation according to the present invention.

[0053] Figure 3 This paper compares the computational cost-performance curves of a visual token pruning method (G-Prune) based on graph information propagation (as described in this invention) with other methods on the TextVQA, DocVQA, POPE, and GQA benchmarks in the LLaVA-NeXT model. The x-axis represents FLOPs Reduction (10^13), indicating the reduction in floating-point computations (in units of 10^13 times), while the y-axis represents Accuracy, indicating the accuracy (or model performance).

[0054] Figure 4 This is a visualization of the different input images pruned by the method of the present invention at different retention ratios in a specific embodiment.

[0055] Figure 5 This is a visualization of the different input images propagating across different iteration numbers t in the method of the present invention in a specific embodiment. Detailed Implementation

[0056] This application provides a visual token pruning method based on graph information propagation for use in Multimodal Large Language Models (MLLMs). This method prunes visual tokens to reduce computational complexity and improve the efficiency of downstream task processing. The overall approach of this technical solution is as follows: This invention introduces a visual token pruning method (G-Prune) based on graph information propagation to improve the efficiency of Multimodal Large Language Models (MLLMs) in processing visual tasks. First, the importance of each visual token is evaluated. Then, based on these evaluations, a graph model is constructed, and the most valuable visual tokens are selected through an iterative information propagation process. Specifically, firstly, for visual tokens extracted by the Vision Transformer (VIT), the information content of each visual token is initialized by calculating its norm. Then, an adjacency matrix is ​​constructed based on the similarity between visual tokens to build graph relationships. Each node (visual token) in the graph is connected to other nodes through edges, and the edge weights reflect the similarity between two nodes. Next, the graph structure is simplified to accurately reflect the important relationships between tokens by iteratively updating the information scores of the nodes in the graph. After multiple iterations, the visual token with the richest information content is selected and retained based on the node's final score. This method improves the computational efficiency of the model, making it particularly suitable for deep learning applications on resource-constrained devices, effectively balancing model performance and computational resource usage.

[0057] To better understand the above technical solution, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0058] Please see Figure 1 and 2 As shown, this invention provides a visual token pruning method based on graph information propagation, the method comprising:

[0059] Step S1: Visual tokens are obtained by visually extracting the input image using a visual encoder. The input image M is first transformed into N visual tokens of dimension d by the visual encoder (VIT). All visual tokens can be represented as a matrix X. v =[v1,v2,…,v N ]∈ N×d The matrix X v It is the encoder output of the VIT model.

[0060] Step S2: Initialize the importance score of each visual token by calculating the L2 norm of each visual token, and obtain the importance scores of all tokens;

[0061] Step S3: Construct a graph structure for visual tokens, treating each visual token as a node, and use importance scores to calculate an adjacency matrix to build connections between visual tokens, thus initializing the graph structure.

[0062] Step S4: Update the adjacency matrix using a preset similarity threshold to obtain the visual token subgraph structure for different regions;

[0063] Step S5: Normalize each row of the adjacency matrix, iteratively propagate the node information according to the preset number of iterations n, and calculate the final score of each visual token by combining the degree of the visual token.

[0064] Step S6: Select the k visual tokens with the highest scores based on the final scores of the visual tokens and project them.

[0065] Step S7: Concatenate the visual token and text token obtained from the projection, input them into the large language model, and obtain the output result of the question answering task.

[0066] Preferably, step S2 specifically involves: processing the N d-dimensional visual token matrix X extracted by the visual encoder (VIT). v =[v1,v2,…,v N ]∈ N×d Using the L2 norm to assign initial information to each visual token, we obtain the importance score vector W∈ for all visual tokens. N×1 ;

[0067] The l2 norm of the visual token is calculated as follows:

[0068]

[0069] W=[||v1||2,||v2||2,…,||vN||2] T

[0070] Among them, v ij This represents the j-th dimension component of the i-th token.

[0071] We initialize the information content of each visual token by calculating its l2 norm, a step that assigns an initial importance score to each token. Specifically, the input set of visual tokens X v ∈ N×d For each visual token Its l2 norm is passed through ||v i ||2 represents the initial information strength used to quantify the tokens, and ultimately the importance score W∈ of all tokens. N×1 .

[0072] Preferably, step S3 specifically involves: defining an adjacency matrix A∈ based on the importance score of the visual token using cosine similarity. N×N :

[0073]

[0074] Among them, v i and v j Let i and j be the vector representations of the i-th and j-th visual tokens, respectively.

[0075] Treating each visual token as a node, we model the interactions between visual tokens in the visual context using an adjacency matrix A, thus obtaining a graph structure about the visual tokens.

[0076] We constructed an adjacency matrix A∈ based on the similarity between visual tokens. N×N Used to construct graph relationships, where A ij Visual token v i and v j The similarity between the tokens is determined by the cosine similarity of their feature vectors. In this way, each node (visual token) in the graph is connected to other nodes through edges, and the weight of the edges reflects the similarity between the two nodes.

[0077] To simplify the graph structure and accurately reflect the important relationships between tokens, we apply a similarity threshold s to the adjacency matrix A. Only when the similarity between tokens exceeds this threshold is a connection between two nodes preserved in the graph; this process is achieved by modifying the adjacency matrix A to obtain A′. This thresholding not only simplifies the graph's complexity but also helps highlight strong connections between tokens within regions, thus laying the foundation for subsequent information propagation pruning.

[0078] Preferably, step S4 specifically comprises:

[0079] Set a similarity threshold s, and update the adjacency matrix through the similarity threshold s, so that only visual tokens with high mutual similarity can be connected, and obtain the visual token subgraph structure of different regions. The update formula of the adjacency matrix is ​​as follows;

[0080]

[0081] Among them, A′ ij This is the updated adjacency matrix.

[0082] This thresholding process not only simplifies the graph structure by eliminating weak connections, but also enhances the differentiation of visual tokens from different regions. In this way, we mitigate the dilution of important information flow between visual tokens, ensuring that the model focuses on processing and disseminating the most relevant data.

[0083] Preferably, step S5 specifically includes:

[0084] To maintain a consistent information flow across each subgraph, the Softmax(·) function is used to normalize each row of the adjacency matrix A':

[0085]

[0086] Where T is a temperature parameter used to control the smoothness of the Softmax(·) function;

[0087] To represent the flow of information in the graph, the normalized adjacency matrix is ​​raised to the power of n:

[0088] A (n) =(A t ) n

[0089] Where n represents the number of iterations, this process is used to model the multi-round iterative propagation of node information;

[0090] Because the number of edges connected to each token (i.e., the degree of the node) varies greatly, it may lead to an imbalance in information propagation. To alleviate this problem, the following processing is performed:

[0091] Calculate the number of edges connected to each visual token, i.e., the degree D∈ 1×N Define it as:

[0092]

[0093]

[0094] The final score of the computational visual token after graph information iteration is as follows:

[0095]

[0096] Where S is the final score vector of the visual token, and W is the importance score vector of the visual token.

[0097] We introduce graph information propagation, a novel approach that models feature aggregations within each subgraph, thereby iteratively flowing the initial information of each visual token to the most representative visual token.

[0098] The graph information propagation pruning stage of this invention is achieved by iteratively updating the information scores of nodes in the graph. We use the Softmax(·) function to normalize the score of each node to ensure the balance of information propagation. Through multiple iterations, the node score S reflects its degree of information concentration and propagation strength in the graph. After multiple iterations, the visual token with the richest information content is selected and retained based on the node's final score.

[0099] This approach improves the computational efficiency of the model, making it particularly suitable for deep learning applications on resource-constrained devices, effectively balancing performance and computational resource usage.

[0100] Preferably, step S6 specifically comprises:

[0101] Select the k visual tokens with the highest scores:

[0102] X g =[v1,v2,…,v k ]

[0103] Among them, each v i From the visual token matrix X v Choose from the options that satisfy:

[0104]

[0105] Among them, v k This indicates that according to ||v i ||2 The visual token located at position k in a list sorted in descending order.

[0106] To further illustrate the method of the present invention, the following explanation is provided in conjunction with specific experimental data:

[0107] Recently, extending large language models (LLMs) to more modalities has become a research hotspot, especially in visual language (VL) tasks. A common paradigm is to directly project extracted visual features into the semantic space of LLMs as input tokens. However, this approach is prone to severe visual illusions and performs poorly when handling fine-grained visual tasks such as TextVQA. To address this issue, an intuitive solution is to increase image resolution, i.e., use more visual tokens. However, this leads to a significant increase in computational cost. For example, the LLaVA-NeXT model uses 2880 visual tokens, requiring approximately 18.52 TFLOPs of computation per inference. To address these issues, this invention proposes a visual token pruning method based on graph information propagation (hereinafter referred to as G-Prune), aiming to reduce the number of visual tokens without affecting model performance, thereby reducing computational complexity. Specifically, we treat visual tokens as nodes in a graph and construct connections based on their feature distances. Then, through an iterative information propagation algorithm, the importance score of each node is updated. Finally, the most important tokens are selected for use by MLLMs. These markers can be foreground or background parts, thus ensuring that the information most valuable to model reasoning is preserved.

[0108] This invention proposes a novel visual tagging pruning method based on graph structure, which has the following outstanding features:

[0109] 1. Traditional pruning methods often focus only on foreground information, neglecting the value of background information. Our G-Prune method, however, considers both foreground and background labels, intelligently selecting the most representative labels through graph structure and information propagation.

[0110] 2. Achieved training-free visual token pruning: The G-Prune method requires no additional training process, is plug-and-play, and can effectively prune visual tokens through graph information flow pruning.

[0111] 3. Suitable for fine-grained visual tasks: In tasks that require fine-grained visual information, such as TextVQA and DocVQA, the G-Prune method performs well and maintains high accuracy even under high pruning rates.

[0112] We compared the three baseline strategies with the state-of-the-art (SOTA) method at different pruning ratios and obtained the following experimental results, as shown in Table 1 (the best results for each pruning ratio are marked in bold):

[0113] Table 1

[0114]

[0115]

[0116] As shown in the table above, the G-Prune method of this invention demonstrates significant technical performance in visual tagging pruning tasks within multimodal language models (MLLMs). Compared to existing pruning methods (such as ToMe, FastV, and random pruning), G-Prune achieves superior performance on multiple benchmarks. For example, in general VQA tasks (such as VQA2.0), G-Prune improves performance by 0.88%–1.58% across various pruning ratios. On MLLM benchmarks such as the MME dataset, this method still achieves a 2.27% performance improvement even when using only 50% visual tags. On the POPE dataset, G-Prune comprehensively outperforms benchmark methods across all pruning ratios. Especially in text-intensive tasks (such as TextVQA), when the pruning ratio reaches 90%, the performance of traditional methods (such as ToMe) drops significantly (TextVQA drops by 40.35%, ChartVQA drops by 26.02%), while G-Prune not only maintains performance but also achieves a performance improvement of 51.99%. Here, Baseline represents no token pruning, which is the performance of the original model without pruning visual tokens; Random represents random token pruning, a very strong and simple consensus-based comparison method; ToMe is a comparison method (from Bolya, D., Fu, CY, Dai, X., Zhang, P., Feichtenhofer, C., & Hoffman, J. Tokenmerging: Your Visible but Faster. arXiv preprint arXiv: 2210.09461.2022.), which uses a "binary soft matching" algorithm and the Key in the Attention module to accurately select the merged visual tokens, thereby accelerating the inference process while maintaining high accuracy; FastV is another comparison method (from Chen L, Zhao H, Liu T, et al. An Image is Worth 1 / 2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language). Models[J].2024.) uses a traditional attention mechanism in the initial layer, while in subsequent layers, visual tokens with lower attention scores are selectively discarded based on the average attention score of the visual tokens, thereby simplifying the computation process and maintaining model performance.

[0117] In addition, such as Figure 2The diagram shows the framework of the G-Prune invention. It mainly includes three steps: (a) visual token extraction, (b) graph connection construction, and (c) graph information propagation. Visual token extraction primarily transforms the input image into a series of visual tokens, which have the same feature dimensions as the text tokens. Graph connection construction utilizes a similarity matrix filtered by a threshold to construct graph connections. This step aims to uncover the similarity relationships between visual tokens while filtering out edges with low similarity to enhance the accuracy of subsequent information propagation. Graph information propagation uses the constructed graph for iterative information propagation. Through iterative propagation, we can find the most representative visual tokens and alleviate redundancy, thereby enhancing the computational efficiency of the model. Figure 2 As shown, even if only three tokens can be retained after using the method of the present invention, the three most representative tokens can be retained, while the content of the background area can be preserved.

[0118] like Figure 3 As shown, the computational cost-performance curves of G-Prune and other methods on the TextVQA, DocVQA, POPE, and GQA benchmarks are compared for the LLaVA-NeXT model. Curves closer to the upper right corner indicate better model performance. From the figure, we can clearly see that the G-Prune method of this invention achieves a better computational cost-performance balance. Our computational cost-performance curve analysis shows that in the initial stage of reducing computational cost, each method has little impact on performance, but as FLOPs are further reduced, model performance begins to decline rapidly. Throughout this process, G-Prune maintains a leading advantage. For example, in the TextVQA task, reducing FLOPs by 1.53 × 10¹³ FLOPs improves performance by 52.05% compared to ToMe; in the DocVQA task, reducing FLOPs by 1.41 × 10¹³ FLOPs still maintains a 46.67% advantage over FastV. These results fully validate G-Prune's ability to maintain high performance while reducing computational cost, highlighting its superior technical performance in multimodal tasks.

[0119] To visualize the effectiveness of our method, we conducted pruning visualization experiments on the LLaVA-NeXT model with retention rates of 50%, 30%, and 10%. Figure 4 As shown, this is a visualization of the pruning of G-Prune at different retention rates. G-Prune is highly effective in preserving the most representative visual tokens. Even at a retention rate of 10%, G-Prune maintains a high level of image quality. These experimental results highlight the capabilities of G-Prune. Specifically, we restored the retained visual tokens to their original positions in the image. From the figure, I can clearly see that the method of this invention can effectively preserve representative visual tokens while discarding a large number of similar, redundant visual tokens.

[0120] We are Figure 5 Further visualization of information propagation was achieved by displaying the information content of visual tokens at different iteration rounds on the original image using a heatmap method. The heatmap represents the scores of the tokens, with tokens in red areas having higher scores. In the initial stage, multiple tokens scored highly for the same object, indicating the presence of much redundant information. As the number of iteration rounds increased, the information from the image tokens gradually converged on the most representative image areas. As information continued to propagate, we observed that information gradually gathered from the foreground and background onto a few tokens. This demonstrates that our G-Prune effectively preserves the tokens of representative objects.

[0121] This invention preserves the most representative visual tokens for each part of a visual image, thereby retaining the overall information of the image and better preserving the contextual capabilities of the multimodal model. This enables advanced pruning effects in various multimodal tasks. While ensuring the processing efficiency of the multimodal model, this invention reduces the consumption of computational resources, allowing the model to run efficiently even in resource-constrained environments. It is particularly suitable for applications requiring real-time processing of large amounts of data, and can improve the processing efficiency and accuracy of downstream visual tasks such as question-answering systems.

[0122] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A visual token pruning method based on graph information propagation, characterized in that: The method includes: Step S1: Visual tokens are obtained by visually extracting the input image using a visual encoder; Step S2: Initialize the importance score of each visual token by calculating the L2 norm of each visual token, and obtain the importance scores of all tokens; Step S3: Construct a graph structure for visual tokens, treating each visual token as a node, and use importance scores to calculate an adjacency matrix to build connections between visual tokens, thus initializing the graph structure. Step S4: Update the adjacency matrix using a preset similarity threshold to obtain the visual token subgraph structure for different regions; Step S5: Normalize each row of the adjacency matrix, iteratively propagate the node information according to the preset number of iterations n, and calculate the final score of each visual token by combining the degree of the visual token. Step S6: Select the k visual tokens with the highest scores based on the final scores of the visual tokens and project them. Step S7: Concatenate the visual token and text token obtained from the projection, input them into the large language model, and obtain the output result of the question answering task.

2. The visual token pruning method based on graph information propagation according to claim 1, characterized in that: Step S2 specifically involves: processing the N-dimensional visual token matrix X extracted by the visual encoder. v =[v1,v2,…,v N ]∈ N×d Using the L2 norm to assign initial information to each visual token, we obtain the importance score vector W∈ for all visual tokens. N×1 ; The l2 norm of the visual token is calculated as follows: W=[||v1||2,||v2||2,…,||v N ||2] T Among them, v ij This represents the j-th dimension component of the i-th token.

3. The visual token pruning method based on graph information propagation according to claim 1, characterized in that: Step S3 specifically involves: based on the importance score of the visual token, defining the adjacency matrix A∈ using cosine similarity. N×N : Among them, v i and v j Let i and j be the vector representations of the i-th and j-th visual tokens, respectively. Treating each visual token as a node, we model the interactions between visual tokens in the visual context using an adjacency matrix A, thus obtaining a graph structure about the visual tokens.

4. The visual token pruning method based on graph information propagation according to claim 1, characterized in that: Step S4 specifically involves: Set a similarity threshold s, and update the adjacency matrix using the similarity threshold s so that only visual tokens with high mutual similarity can be connected, thus obtaining visual token subgraph structures in different regions. The update formula for the adjacency matrix is ​​as follows: Among them, A′ ij This is the updated adjacency matrix.

5. The visual token pruning method based on graph information propagation according to claim 1, characterized in that: Step S5 specifically includes: Use the Softmax() function to normalize each row of the adjacency matrix A': Where T is a temperature parameter used to control the smoothness of the Softmax(·) function; Perform an nth-degree exponentiation operation: A (n) =(A t ) n Where n represents the number of iterations, this process is used to model the multi-round iterative propagation of node information; Calculate the number of edges connected to each visual token, i.e., the degree D∈ 1×N Define it as: The final score of the computational visual token after graph information iteration is as follows: Where S is the final score vector of the visual token, and W is the importance score vector of the visual token.

6. The visual token pruning method based on graph information propagation according to claim 1, characterized in that: Step S6 specifically involves: Select the k visual tokens with the highest scores: X g =[v1,v2,…,v k ] Among them, each v i From the visual token matrix X v Choose from the options that satisfy: Among them, v k This indicates that according to ||v i ||2 The visual token located at position k in a list sorted in descending order.

Images