Visual token pruning method based on graph information propagation

Abstract

The invention provides a visual token pruning method based on graph information propagation. The visual token pruning method comprises the following steps: performing visual extraction on an input image to obtain a visual token; initializing the importance score of the visual token; constructing a graph structure related to the visual tokens, taking each visual token as a node, calculating an adjacent matrix for constructing connection between the visual tokens, and initializing the graph structure; the adjacency matrix is updated through a preset similarity threshold value, and visual token sub-graph structures of different areas are obtained; performing normalization processing on each row of the adjacent matrix, performing iterative propagation on node information, and calculating a final score of each visual token; screening out k visual tokens with the highest scores according to the final scores of the visual tokens for projection; and splicing the visual token obtained by projection with the text token, and inputting the spliced visual token and text token into a large language model to obtain an output result. According to the method, the calculation efficiency of the model can be improved, and the calculation cost is remarkably reduced while the visual task performance is kept.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a visual token pruning method based on graph information propagation. Background Art

[0002] With the rapid development of digital media and artificial intelligence technology, multimodal large language models (MLLMs) have shown unprecedented application potential in the fields of human-computer interaction, automatic content generation, and data analysis. Especially in the intelligent processing of image and video content, MLLMs are gradually promoting the development of technology towards a broader application prospect by leveraging its powerful language understanding and multimodal data processing capabilities.

[0003] In the development of multimodal models, the visual modality is particularly important. In traditional processing methods, the model needs to process a large number of visual tokens to understand and generate content, which not only consumes a lot of computing resources, but also increases the time for model training and reasoning. Especially in tasks that require high-resolution and large-scale visual data processing, the consumption of computing resources has become a major bottleneck limiting the application of models.

[0004] In addition, many visual tokens contain redundant information, which not only increases the computational burden, but also reduces the model processing speed and efficiency. The introduction of pruning technology, especially in the optimization of visual tokens, aims to screen out the tokens that have the greatest impact on the final task, while discarding those tokens that have redundant information or have little impact on model performance. This method can significantly reduce the complexity of the model and improve the efficiency and speed of the model.

[0005] For example, an existing Chinese invention patent (publication number CN116933859A) - a dynamic pruning method for a visual Transformer, discloses: comprising the following steps: Step 1, constructing an image dataset, which will be used for subsequent model training and testing; Step 2, establishing a visual Transformer network model, by stacking multiple layers of encoder blocks after a position embedding module, inserting a token pruning module after a specific encoder block, and embedding a channel sparse module in each layer to perform feature channel sparsification; Step 3, training the visual Transformer network model, and optimizing the weights and structural parameters of the model through a dynamic pruning method during the training process, and the pruned objects include token sequences and channel parameters of fully connected layers; Step 4, selecting the sub-token sequence with the highest global information score and the channel with the largest importance weight from the training process, for reconstructing a lightweight network model to reduce the model's computing resource requirements and improve execution efficiency. And the existing Chinese invention patent (publication number CN115310607A) - a model optimization method based on dynamic pruning technology and visual Transformer, which discloses: in a machine vision reasoning system, firstly, the ViT model is initially trained for several rounds through a data-driven method to generate a complete attention map; then the information entropy of these attention maps is calculated, and the attention heads are pruned according to the obtained information entropy values; then, the weight parameters corresponding to the pruned attention heads are removed to construct a new ViT model; then the parameters of the new ViT model are fine-tuned; by pruning the modules in the multi-head attention mechanism, those feature maps with high uncertainty and their corresponding attention heads are removed, which can reduce the parameters and complexity of the ViT model, reduce the computational complexity and the total number of parameters, thereby effectively reducing the size of the ViT model and achieving lightweight model with minimal performance loss.

[0006] These patents reduce the amount of data processed to a certain extent by introducing pruning technology in image processing tasks, but they still have the following shortcomings and deficiencies: A dynamic pruning method for visual Transformer (publication number CN116933859A), which implements dynamic pruning on the visual Transformer model, learns the importance of features at different levels through training, and thus achieves efficient network operation. However, this method relies on building a high-quality image data set to train and update parameters, resulting in high data costs and training costs. A model optimization method based on dynamic pruning technology and visual Transformer (publication number CN115310607A), which prunes by information entropy based on attention maps, mainly focuses on the significant features of the image, which may lead to the loss of background or contextual information. This insignificant information may be equally important in the multimodal model's understanding of the overall scene, and it also requires fine-tuning to update parameters.

[0007] Therefore, how to effectively prune visual tokens and reduce unnecessary computational overhead while ensuring model performance has become a technical problem that needs to be solved urgently. Summary of the invention

[0008] The technical problem to be solved by the present invention is to provide a visual token pruning method based on graph information propagation, so as to improve the computational efficiency of multimodal large language models and significantly reduce the computational cost while maintaining the performance of visual tasks.

[0009] The present invention is implemented as follows: a visual token pruning method based on graph information propagation, the method comprising:

[0010] Step S1, visually extracting the input image through a visual encoder to obtain a visual token;

[0011] Step S2, by calculating the bi-norm of each visual token, initializing the importance score of each visual token, and obtaining the importance scores of all tokens;

[0012] Step S3, constructing a graph structure about visual tokens, taking each visual token as a node, and calculating an adjacency matrix using importance scores to construct connections between visual tokens and initialize the graph structure;

[0013] Step S4, updating the adjacency matrix by a preset similarity threshold to obtain the visual token subgraph structure of different regions;

[0014] Step S5, normalize each row of the adjacency matrix, iteratively propagate the node information according to a preset number of iterations n, and calculate the final score of each visual token in combination with the degree of the visual token;

[0015] Step S6: select k visual tokens with the highest scores according to the final scores of the visual tokens for projection;

[0016] Step S7: Concatenate the projected visual tokens with the text tokens and input them into the large language model to obtain the output result of the question-answering task.

[0017] Further, the step S2 is specifically: extracting N visual token matrices X of dimension d from the visual encoder v =[v1,v2,…,v N ]∈ N×d , use the two-norm to assign initial information to each visual token, and get the importance score vector W∈ of all visual tokens N×1 ;

[0018] The l2 norm of the visual token is calculated as follows:

[0019]

[0020] W=[||v1||2,||v2||2,…,||v N ||2] T

[0021] Among them, v ij Represents the j-th dimension component of the i-th token.

[0022] Furthermore, the step S3 is specifically as follows: based on the importance score of the visual token, the adjacency matrix A∈ is defined using cosine similarity N×N :

[0023]

[0024] Among them, v i and v j The vector representations of the i-th and j-th visual tokens respectively;

[0025] Each visual token is taken as a node, and the interaction between visual tokens is modeled in the visual context through the adjacency matrix A to obtain the graph structure of the visual token.

[0026] Furthermore, the step S4 is specifically as follows:

[0027] A similarity threshold s is set, and the adjacency matrix is ​​updated by the similarity threshold s so that only visual tokens with high mutual similarity can be connected, and the visual token subgraph structure of different regions is obtained. The updating formula of the adjacency matrix is ​​as follows;

[0028]

[0029] Among them, A′ij is the updated adjacency matrix.

[0030] Furthermore, the step S5 specifically includes:

[0031] Use the Softmax(·) function to normalize each row of the adjacency matrix A':

[0032]

[0033] Where T is the temperature parameter, which is used to control the smoothness of the Softmax(·) function;

[0034] Perform n-power operation:

[0035] A (n) =(A t ) n

[0036] Where n represents the number of iterations, and this process is used to model the multiple rounds of iterative propagation of node information;

[0037] Calculate the number of edges connected to each visual token, that is, the degree D∈ 1×N , which is defined as:

[0038]

[0039]

[0040] The final score of the calculated visual token after graph information iteration is as follows:

[0041]

[0042] Among them, S is the final score vector of visual tokens, and W is the importance score vector of visual tokens.

[0043] Furthermore, the step S6 is specifically as follows:

[0044] Select the k visual tokens with the highest scores:

[0045] X g =[v1,v2,…,v k ]

[0046] Among them, each v i From the visual token matrix X v Select from , and satisfy:

[0047]

[0048] Among them, v k Indicates that according to ||v i||2The visual token at position k in the list sorted in descending order.

[0049] The present invention has the following advantages: by proposing a visual token pruning method based on graph information propagation, a relationship graph between visual tokens is constructed, and the importance scores of tokens are propagated and updated through an iterative algorithm, so as to accurately select the tokens that have the greatest impact on the task, thereby ensuring the processing efficiency of the multimodal model while reducing the consumption of computing resources, so that the model can run efficiently in an environment with limited resources. The pruning method based on graph information propagation of the present invention does not require training and is plug-and-play, and is particularly suitable for application scenarios that require real-time processing of large amounts of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0051] Figure 1 This is an execution flow chart of a visual token pruning method based on graph information propagation according to the present invention.

[0052] Figure 2 A schematic diagram of the framework structure of a visual token pruning method based on graph information propagation according to the present invention.

[0053] Figure 3 Comparison of the computational performance curve of the visual token pruning method based on graph information propagation (i.e., G-Prune) and other methods on the TextVQA, DocVQA, POPE, and GQA benchmarks on the LLaVA-NeXT model. The horizontal axis of the curve is FLOPs Reduction (10^13), which indicates the reduction in the amount of FLOPS floating point calculations (unit: 10^13 times), and the vertical axis is Accuracy, which indicates the accuracy (or model performance).

[0054] Figure 4 Detailed description of the visualization effect of different input images pruned at different retention ratios according to the method of the present invention in a specific embodiment.

[0055] Figure 5 1 is a visualization display of different input images of information propagation across different iteration times t in the method of the present invention in a specific embodiment. DETAILED DESCRIPTION

[0056] The embodiment of the present application provides a visual token pruning method based on graph information propagation, which is used in a multimodal large language model (MLLM) to prune visual tokens to reduce computational complexity and thus improve downstream task processing efficiency. The technical solution in the embodiment of the present application has the following overall idea: The present invention aims to improve the efficiency of the multimodal large language model (MLLM) in processing visual tasks by introducing a visual token pruning method (G-Prune) based on graph information propagation. First, the importance of each visual token is evaluated, and then a graph model is constructed based on these evaluations, and the most valuable visual tokens are selected through an iterative information propagation process. Specifically, first, for the visual tokens extracted by the Vision Transformer (VIT), the information content of the visual tokens is initialized by calculating the norm of each visual token, and then an adjacency matrix is ​​constructed based on the similarity between the visual tokens to construct a graph connection. Each node (visual token) in the graph is connected to other nodes through an edge, and the weight of the edge reflects the similarity between the two nodes. After that, the graph structure is simplified and the important relationship between the tokens is accurately reflected. This is achieved by iteratively updating the information scores of the nodes in the graph. After multiple iterations, the visual token with the richest information is selected according to the final score of the node. This method improves the computational efficiency of the model, which is especially suitable for deep learning applications on resource-constrained devices and can effectively balance model performance and the use of computing resources.

[0057] In order to better understand the above technical solution, the technical solution of the present invention will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0058] See also Figure 1 and 2 As shown, the present invention provides a visual token pruning method based on graph information propagation, the method comprising:

[0059] Step S1: Visually extract the input image through the visual encoder to obtain visual tokens; the input image M is first converted into N visual tokens with dimension d by the visual encoder (VIT). All visual tokens can be represented as a matrix X v =[v1,v2,…,v N ]∈ N×d , the matrix X v is the encoder output of the VIT model.

[0060] Step S2, by calculating the bi-norm of each visual token, initializing the importance score of each visual token, and obtaining the importance scores of all tokens;

[0061] Step S3, constructing a graph structure about visual tokens, taking each visual token as a node, and calculating an adjacency matrix using importance scores to construct connections between visual tokens and initialize the graph structure;

[0062] Step S4, updating the adjacency matrix by a preset similarity threshold to obtain the visual token subgraph structure of different regions;

[0063] Step S5, normalize each row of the adjacency matrix, iteratively propagate the node information according to a preset number of iterations n, and calculate the final score of each visual token in combination with the degree of the visual token;

[0064] Step S6: select k visual tokens with the highest scores according to the final scores of the visual tokens for projection;

[0065] Step S7: Concatenate the projected visual tokens with the text tokens and input them into the large language model to obtain the output result of the question-answering task.

[0066] Preferably, the step S2 is specifically: extracting N visual token matrices X of dimension d from the visual encoder (VIT) v =[v1,v2,…,v N ]∈ N×d , use the two-norm to assign initial information to each visual token, and get the importance score vector W∈ of all visual tokens N×1 ;

[0067] The l2 norm of the visual token is calculated as follows:

[0068]

[0069] W=[||v1||2,||v2||2,…,||vN||2] T

[0070] Among them, v ij Represents the j-th dimension component of the i-th token.

[0071] We initialize the information content of visual tokens by calculating the l2 norm of each visual token. This step gives each token an initial importance score. Specifically, the input set of visual tokens X v ∈ N×d , for each visual token Its l2 norm is given by ||v i ||2 represents the initial information strength of the token, and finally obtains the importance score W∈ of all tokens N×1 .

[0072] Preferably, the step S3 is specifically as follows: based on the importance score of the visual token, the adjacency matrix A∈ is defined using cosine similarity N×N :

[0073]

[0074] Among them, v i and v j The vector representations of the i-th and j-th visual tokens respectively;

[0075] Each visual token is taken as a node, and the interaction between visual tokens is modeled in the visual context through the adjacency matrix A to obtain the graph structure of the visual token.

[0076] We construct an adjacency matrix A∈ based on the similarity between visual tokens N×N To construct a graph relationship, where A ij Represents the visual token v i and v j The similarity between them is determined by the cosine similarity of the feature vectors of the tokens. In this way, each node (visual token) in the graph is connected to other nodes through edges, and the weight of the edge reflects the similarity between the two nodes.

[0077] In order to simplify the graph structure and accurately reflect the important relationships between tokens, we implement a similarity threshold s on the adjacency matrix A. Only when the similarity between tokens is higher than this threshold, the connection between two nodes will be retained in the graph, which is done by modifying the adjacency matrix A to obtain A′. This thresholding process not only simplifies the complexity of the graph, but also helps to highlight the strong connections between tokens in the region, thus laying the foundation for the subsequent information propagation pruning process.

[0078] Preferably, the step S4 is specifically as follows:

[0079] A similarity threshold s is set, and the adjacency matrix is ​​updated by the similarity threshold s so that only visual tokens with high mutual similarity can be connected, and the visual token subgraph structure of different regions is obtained. The updating formula of the adjacency matrix is ​​as follows;

[0080]

[0081] Among them, A′ ij is the updated adjacency matrix.

[0082] This thresholding process not only simplifies the graph structure by eliminating weak connections, but also enhances the distinction between visual tokens from different regions. In this way, we alleviate the dilution of important information flow between visual tokens, ensuring that the model focuses on processing and propagating the most relevant data.

[0083] Preferably, the step S5 specifically includes:

[0084] In order to maintain consistent information flow in each subgraph, each row of the adjacency matrix A' is normalized using the Softmax(·) function:

[0085]

[0086] Where T is the temperature parameter, which is used to control the smoothness of the Softmax(·) function;

[0087] To represent the flow of information in the graph, the normalized adjacency matrix is ​​raised to the power of n:

[0088] A (n) =(A t ) n

[0089] Where n represents the number of iterations, and this process is used to model the multiple rounds of iterative propagation of node information;

[0090] Since the number of edges connected to each token, i.e. the degree of the node, varies greatly, information dissemination may be unbalanced. To alleviate this problem, the following processing is performed:

[0091] Calculate the number of edges connected to each visual token, that is, the degree D∈ 1×N , which is defined as:

[0092]

[0093]

[0094] The final score of the calculated visual token after graph information iteration is as follows:

[0095]

[0096] Among them, S is the final score vector of visual tokens, and W is the importance score vector of visual tokens.

[0097] We introduce graph information propagation, a novel approach that models feature aggregation within each subgraph, so that the initial information of each visual token iteratively flows towards the most representative visual tokens.

[0098] The graph information propagation pruning phase of the present invention is achieved by iteratively updating the information scores of nodes in the graph. We use the Softmax(·) function to normalize the score of each node to ensure the balance of information propagation. Through multiple iterations, the node score S reflects its information concentration and propagation strength in the graph. After multiple iterations, the visual token with the richest information is selected according to the final score of the node.

[0099] With this approach, we improve the computational efficiency of the model, which is particularly suitable for deep learning applications on resource-constrained devices, effectively balancing performance and the use of computing resources.

[0100] Preferably, the step S6 is specifically as follows:

[0101] Select the k visual tokens with the highest scores:

[0102] X g =[v1,v2,…,v k ]

[0103] Among them, each v i From the visual token matrix X v Select from , and satisfy:

[0104]

[0105] Among them, v k Indicates that according to ||v i ||2The visual token at position k in the list sorted in descending order.

[0106] In order to further illustrate the method of the present invention, the following is described in conjunction with specific experimental data:

[0107] Recently, extending large language models (LLMs) to more modalities has become a research hotspot, especially in visual language (VL) tasks. The commonly used paradigm is to directly project the extracted visual features into the semantic space of LLMs as input tags. However, this approach is prone to severe visual illusions and performs poorly when dealing with fine-grained visual tasks such as TextVQA. To address this problem, an intuitive solution is to increase the image resolution, that is, use more visual tags. However, this will lead to a significant increase in the amount of computation and the computational cost is too high. For example, the LLaVA-NeXT model uses 2880 visual tags, and each inference requires about 18.52TFLOPs of computation. To address the above problems, this paper proposes a visual token pruning method based on graph information propagation (hereinafter referred to as G-Prune), which aims to reduce the number of visual tags without affecting the performance of the model, thereby reducing the computational complexity. Specifically, we regard the visual tags as nodes in the graph and build connections based on their feature distances. Then, the importance score of each node is updated through an iterative information propagation algorithm. Finally, the most important tags are selected for use by MLLMs. These markers may be foreground or background parts, ensuring that the most valuable information for model reasoning is preserved.

[0108] This paper proposes a new visual labeling pruning method based on graph structure, which has the following outstanding performances:

[0109] 1. Traditional pruning methods often only focus on the foreground information of the image and ignore the value of background information. Our G-Prune method considers both foreground and background markers, and intelligently selects the most representative markers through graph structure and information propagation.

[0110] 2. Achieved visual token pruning without training: The G-Prune method does not require additional training process and is plug-and-play. It can effectively prune visual markers through graph information flow pruning.

[0111] 3. Suitable for fine-grained visual tasks: In tasks such as TextVQA and DocVQA that require fine-grained visual information, the G-Prune method performs well and maintains high accuracy even at high pruning rates.

[0112] We compare the three baseline strategies with the SOTA method at different pruning ratios and obtain the following experimental results, as shown in Table 1 (the best result for each pruning ratio is marked in bold):

[0113] Table 1

[0114]

[0115]

[0116] As can be seen from the above table, the G-Prune method of the present invention has demonstrated significant technical effects in the visual marker pruning task in multimodal language models (MLLMs). Compared with existing pruning methods (such as ToMe, FastV, and random pruning), G-Prune has achieved excellent performance on multiple benchmarks. For example, in general VQA tasks (such as VQA2.0), G-Prune's performance at each pruning ratio is improved by 0.88%-1.58%. On MLLM benchmarks such as the MME dataset, this method can still improve performance by 2.27% when only 50% of visual markers are used. On the POPE dataset, G-Prune comprehensively surpasses the benchmark methods at all pruning ratios. Especially in text-intensive tasks (such as TextVQA), when the pruning ratio reaches 90%, the performance of traditional methods (such as ToMe) drops significantly (TextVQA drops by 40.35%, ChartVQA drops by 26.02%), while G-Prune not only maintains the performance, but also achieves a 51.99% performance improvement. Among them, Baseline means not cropping tokens, that is, the performance of the original model without cropping visual tokens; Random means randomly cropping tokens, a very strong and simple comparison method based on consensus, ToMe is a comparison method (from Bolya, D., Fu, CY, Dai, X., Zhang, P., Feichtenhofer, C., & Hoffman, J. Token merging: Your vit but faster. arXiv preprint arXiv: 2210.09461.2022.), which uses the "binary soft matching" algorithm and the Key in the Attention module to accurately select the merged visual tokens, thereby accelerating the reasoning process while maintaining high accuracy; FastV is another comparison method (from Chen L, Zhao H, Liu T, et al. An Image is Worth 1 / 2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models[J].2024.), which uses the traditional attention mechanism in the initial layer, and selectively discards those visual tokens with low attention scores in subsequent layers based on the average attention scores of the visual tokens, thereby simplifying the calculation process and maintaining model performance.

[0117] In addition, if Figure 2As shown, the framework diagram of G-Prune of the present invention. It mainly includes three steps: (a) extracting visual tokens, (b) constructing graph connections, and (c) graph information propagation. Extracting visual tokens mainly converts the input image into a series of visual tokens, and these tokens have the same feature dimensions as text tokens. Constructing graph connections uses the similarity matrix after threshold filtering to construct graph connections. This step is to mine the similarity connections between visual tokens, and at the same time filter out edges with too low similarity to enhance the accuracy of subsequent information propagation. Graph information propagation uses the constructed graph for iterative information propagation. Through iterative information propagation, we can find the most representative visual tokens and alleviate redundancy, thereby enhancing the computational efficiency of the model. As shown in Figure 2 As shown, even if only three tokens can be retained after the method of the present invention is used, the three most representative ones can be retained, and the content of the background area can be retained at the same time.

[0118] like Figure 3 As shown in the figure, the computational amount-performance curve of G-Prune and other methods on the TextVQA, DocVQA, POPE and GQA benchmarks is compared on the LLaVA-NeXT model. The closer the curve is to the upper right corner, the better the model performance is. From the figure, we can clearly see that the method G-Prune of the present invention achieves a better computational amount-performance balance. Our computational amount-performance curve analysis shows that in the initial stage of reducing the amount of computation, each method has little effect on the performance, but as the FLOPs are further reduced, the model performance begins to decline rapidly. In this process, G-Prune always maintains a leading advantage. For example, when reducing 1.53×1013FLOPs in the TextVQA task, the performance is improved by 52.05% compared with ToMe; when reducing 1.41×1013FLOPs in the DocVQA task, it still maintains a 46.67% advantage over FastV. These results fully verify G-Prune's ability to maintain high performance while reducing computational costs, highlighting its excellent technical effect in multimodal tasks.

[0119] To visualize the effectiveness of our method, we conducted pruning visualization experiments on the LLaVA-NeXT model with retention ratios of 50%, 30%, and 10%. Figure 4 As shown, it is a visualization of pruning at different retention ratios of G-Prune of the present invention. G-Prune is very effective in retaining the most representative visual markers. Even at a retention rate of 10%, G-Prune is able to maintain a high level of image quality. These experimental results emphasize the capabilities of G-Prune. Specifically, we restore the retained visual tokens to their positions on the original image. From the figure, I can clearly see that the method of the present invention can well retain representative visual tokens and discard a large number of similar redundant visual tokens.

[0120] We are Figure 5 In the figure, we further visualize the information propagation. We display the information content of visual tokens at different iterations on the original image through heat maps. The heat map represents the score of the markers, and the markers in the red area have higher scores. In the initial stage, multiple markers have high scores for the same object, which also indicates that there is a lot of redundant information. As the number of iterations increases, the information of image tokens gradually converges on the most representative image areas. As the information continues to propagate, we can find that the information gradually gathers from the foreground and background to a few markers. This shows that our G-Prune can effectively retain the markers of representative objects.

[0121] The present invention retains the most representative visual token of each part of the visual image, thereby retaining the overall information of the image and better retaining the contextual capabilities of the multimodal model, so as to achieve advanced pruning effects in various multimodal tasks. While ensuring the processing efficiency of the multimodal model, the present invention reduces the consumption of computing resources, so that the model can run efficiently even in a resource-limited environment. It is particularly suitable for application scenarios that require real-time processing of large amounts of data, and can improve the processing efficiency and accuracy of visual tasks such as downstream question-answering systems to a certain extent.

[0122] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A visual token pruning method based on graph information propagation, characterized by: The method comprises: Step S1, visually extracting the input image through a visual encoder to obtain a visual token; Step S2, by calculating the bi-norm of each visual token, initializing the importance score of each visual token, and obtaining the importance scores of all tokens; Step S3, constructing a graph structure about visual tokens, taking each visual token as a node, and calculating an adjacency matrix using importance scores to construct connections between visual tokens and initialize the graph structure; Step S4, updating the adjacency matrix by a preset similarity threshold to obtain the visual token subgraph structure of different regions; Step S5, normalize each row of the adjacency matrix, iterate and propagate the node information according to a preset number of iterations n, and calculate the final score of each visual token in combination with the degree of the visual token; Step S6: select k visual tokens with the highest scores according to the final scores of the visual tokens for projection; Step S7: Concatenate the projected visual tokens with the text tokens and input them into the large language model to obtain the output result of the question-answering task.

2. The visual token pruning method based on graph information propagation according to claim 1, characterized in that: The step S2 is specifically as follows: extracting N visual token matrices X with dimension d from the visual encoder v =[v1,v2,…,v N ]∈ N×d , use the two-norm to assign initial information to each visual token, and get the importance score vector W∈ of all visual tokens N×1 ; The l2 norm of the visual token is calculated as follows: <h2 style=";text-align:left;direction:ltr">W=[||v1||2,||v2||2,…,||v<h2 style=";text-align:left;direction:ltr"> N <h2 style=";text-align:left;direction:ltr"> ||2]<h2 style=";text-align:left;direction:ltr"> T Among them, v ij Represents the j-th dimension component of the i-th token.

3. The visual token pruning method based on graph information propagation according to claim 1, characterized in that: The step S3 is specifically as follows: based on the importance score of the visual token, the adjacency matrix A∈ is defined using cosine similarity. N×N : Among them, v i and v j The vector representations of the i-th and j-th visual tokens respectively; Each visual token is taken as a node, and the interaction between visual tokens is modeled in the visual context through the adjacency matrix A to obtain the graph structure of the visual token.

4. The visual token pruning method based on graph information propagation according to claim 1, characterized in that: The step S4 is specifically as follows: A similarity threshold s is set, and the adjacency matrix is ​​updated by the similarity threshold s so that only visual tokens with high mutual similarity can be connected, and the visual token subgraph structure of different regions is obtained. The updating formula of the adjacency matrix is ​​as follows; Among them, A′ ij is the updated adjacency matrix.

5. The visual token pruning method based on graph information propagation according to claim 1, characterized in that: The step S5 specifically includes: Use the Softmax(·) function to normalize each row of the adjacency matrix A': Where T is the temperature parameter, which is used to control the smoothness of the Softmax(·) function; Perform n-power operation: A (n) =(A t ) n Where n represents the number of iterations, and this process is used to model the multiple rounds of iterative propagation of node information; Calculate the number of edges connected to each visual token, that is, the degree D∈ 1×N , which is defined as: The final score of the calculated visual token after graph information iteration is as follows: Among them, S is the final score vector of visual tokens, and W is the importance score vector of visual tokens.

6. The visual token pruning method based on graph information propagation according to claim 1, characterized in that: The step S6 is specifically as follows: Select the k visual tokens with the highest scores: X g =[v1,v2,…,v k ] Among them, each v i From the visual token matrix X v Select from , and satisfy: Among them, v k Indicates that according to ||v i ||2The visual token at position k in the list sorted in descending order.

Images