Adaptive visual mark pruning method and device based on multi-modal large model

Abstract

The invention discloses an adaptive visual mark pruning method and device based on a multi-modal large model. The method comprises the following steps: calculating two initial distributions of visual marks; initializing an error threshold according to a binary search algorithm; using a greedy algorithm to delete the visual marks which have the minimum influence on the two initial distributions until the error of the two initial distributions reaches an error threshold value, obtaining two groups of candidate deletion marks, taking an intersection, and deleting the visual marks based on the intersection; updating an error threshold boundary according to the model calculation amount after the visual mark is deleted and the target calculation amount; repeating the deleting and updating steps until a stopping condition is met, and calculating the average deleting number of the visual marks in each layer of a batch of data as a deleting strategy; and sorting the visual marks to be deleted and then deleting the visual marks according to a deletion strategy. According to the method, the use of computing resources is optimized by automatically identifying and removing redundant visual marks, the operation cost is reduced on the premise of not sacrificing the performance of the model, and the reasoning speed of the model is increased.

Description

Technical Field

[0001] The present invention relates to a multimodal model, and in particular to an adaptive visual marker pruning method and device based on a multimodal large model. Background Art

[0002] In recent years, multimodal large-scale language models (MLLMs) have made remarkable progress in the fields of natural language processing and computer vision. By combining information from two modalities, text and image, MLLMs have performed well in multimodal tasks such as image description generation and visual question answering. However, as the model parameters and data size continue to increase, the computing resources required for training and inference have also risen sharply.

[0003] One of the main challenges of current MLLMs is the high computational and storage overhead. These models usually contain a large number of tokens, each of which requires complex computational processing, resulting in inefficient reasoning, especially in practical application scenarios. Although existing optimization techniques (such as model pruning and knowledge distillation) can alleviate this problem to a certain extent, they often require additional training data and complex model adjustment processes. In multimodal models, the redundancy problem of tags of the visual modality (i.e., image tokens) is particularly prominent. Images are usually segmented into multiple small blocks, and each block is input into the model as an independent token. There is a lot of redundant information in these visual tokens, resulting in unnecessary computational overhead. For example, in the image description generation task, not all image blocks make important contributions to the generated description.

[0004] Traditional token pruning methods reduce the amount of computation by removing unimportant tokens, but these methods usually require a lot of manual feature engineering or complex training processes. In addition, these methods are less versatile in practical applications because different tasks and datasets may require different pruning strategies.

[0005] In order to improve the inference efficiency of the model and reduce the consumption of computing resources, a general and efficient token pruning method is urgently needed. This method should be able to automatically identify and remove redundant tokens while maintaining model performance, thereby optimizing the use of computing resources. This not only helps to reduce the running cost of the model, but also speeds up the inference speed of the model, allowing it to run efficiently in resource-constrained environments. Summary of the invention

[0006] The purpose of the present invention is to solve the problem that the process of removing visual marks in the prior art is complicated and has poor versatility.

[0007] The technical solution adopted by the present invention to solve the technical problem is: to provide an adaptive visual marker pruning method based on a multimodal large model, comprising the following steps:

[0008] In the initial statistical step, for a batch of data to be pruned, the total internal attention and cross-modal attention of each layer of visual tags after inputting the statistical multimodal large language model are calculated as two initial distributions;

[0009] Initial threshold step, given the target model computation amount, and initialize the error threshold according to the binary search algorithm;

[0010] Deletion step: for each layer of visual markers, use a greedy algorithm to delete the visual markers with the smallest impact on the two initial distributions layer by layer until the errors of the two initial distributions reach the error threshold, and obtain two sets of candidate deletion markers; take the intersection of the two sets of candidate deletion markers as the final visual markers to be deleted, and delete the visual markers;

[0011] An updating step, in which the error threshold boundary is updated according to the model calculation amount after deleting the visual mark and the target calculation amount;

[0012] Stop the judgment step, repeat the deletion step and the update step until the stop condition is reached, and then enter the pruning step;

[0013] In the pruning step, the average number of visual markers deleted in each layer is calculated as the deletion strategy; the visual markers to be deleted are sorted, and the internal attention size and cross-modal attention size of the visual markers are considered during the sorting; the sorted visual markers are deleted according to the deletion strategy to achieve adaptive visual marker pruning.

[0014] Preferably, the multimodality includes a visual modality and a textual modality, and the multimodal large language model includes visual markers and textual markers.

[0015] Preferably, the two initial distributions include the total amount of internal attention and the total amount of cross-modal attention, expressed as:

[0016]

[0017] A ′ =max(A1,A2,…,A h ),

[0018] Among them, D I (l) represents the total amount of internal attention of the visual markers at the lth layer, D C (l) represents the total amount of cross-modal attention of the visual tags in the lth layer, and N represents the total number of visual tags in the lth layer; A ′ represents the maximum value in the attention matrix of each attention head, h represents the number of attention heads; n and m represent the number of visual tags and text tags respectively; f I (t l,i ) represents the visual mark t l,iThe total amount of attention received from other visual markers, f C (t l,i ) represents the visual mark t l,i The total amount of attention received by the text token.

[0019] Preferably, the error threshold is expressed as:

[0020]

[0021] Among them, α min represents the left boundary, α max Indicates the right boundary;

[0022] The initialization error threshold refers to, let α min =0,α max =1.

[0023] Preferably, the deleting step deletes each layer of visual marks separately, and the process of deleting the first layer includes the following steps:

[0024] Initialize two empty sets to store candidate deletion markers with the smallest impact on internal attention and cross-modal attention respectively;

[0025] For each visual marker t l,i , calculate the first impact of deleting the tag on the internal attention distribution and the second impact on the cross-modal attention distribution, which are expressed as:

[0026]

[0027] The visual marker sequence input to the layer is sorted according to the first influence and the second influence respectively to obtain the visual marker sequence, which is expressed as:

[0028] X v,I =sort(X v ,d(D I (l),D I ′ (l)));

[0029] X v,C =sort(X v ,d(D C (l),D ′ C (l)));

[0030] Among them, X v,I represents the visual mark sequence sorted by first impact, X v,C represents the visual mark sequence sorted according to the second influence, X v Represents the visual mark sequence input to this layer, and sort represents the sorting function;

[0031] From the visual marker sequence X v,I and X v,C Find a set of visual marker sequence subsets from the front to the back, store them in two empty sets as two sets of candidate visual markers, expressed as:

[0032]

[0033] Take the set P l,1 and P l,2 The intersection of is used as the deletion set to delete the visual mark of the corresponding layer, and the deletion set is expressed as:

[0034] P l =P l,1 ∩P l,2 .

[0035] Preferably, the set P l,1 and P l,2 The intersection of is used as the deletion set to delete the visual markers of the corresponding layer. During the deletion process, it is ensured that the model calculation amount does not exceed the target model calculation amount. Specifically, when executing the deletion operation, the calculation amount of the entire model after deletion is monitored. If the deletion operation causes the calculation amount to exceed the target calculation amount, the deletion of the next layer of visual markers is stopped.

[0036] Preferably, in the updating step, the error threshold boundary is updated according to the comparison result between the operation amount of the model after deleting the visual mark and the operation amount of the target model, which is expressed as:

[0037]

[0038] Among them, FLOPs is the amount of model operations after deleting the visual labels, X0 is the visual label sequence of the initial input model without deletion, is the set of deletion sets of each layer, and L is the total number of layers in the model; Indicates that given an input visual tag sequence X0 The model operation amount under the condition, s represents the target model operation amount.

[0039] Preferably, the stop condition in the stop judgment step is expressed as:

[0040] α max -α min ≤δ;

[0041] Among them, δ is the preset binary search stop threshold;

[0042] If the stopping condition is met, the pruning step is entered; if it is not met, the error threshold is updated according to the updated error threshold boundary, and the deletion step and the update step are repeated.

[0043] Preferably, the pruning step specifically includes the following steps:

[0044] The average number of visual mark deletions per layer is calculated as the deletion strategy, expressed as:

[0045]

[0046] in, Indicates processing input data b i The visual mark removed at layer l, express The number of, c is the total number of samples;

[0047] The comprehensive importance measure is calculated for the visual markers to be deleted, expressed as:

[0048] f U (t l,i ) = f C (t l,i )·f I (t l,i );

[0049] The visual mark is measured according to the comprehensive importance measure f U (t l,i ) Sort from small to large;

[0050] The sorted visual markers are deleted according to the deletion strategy, which is expressed as:

[0051]

[0052] in, Represents the visual mark t l,i In the comprehensive importance measure f U (t l,i ) in the sort order.

[0053] The present invention also provides an adaptive visual marker pruning device based on a multimodal large model, comprising:

[0054] The initial statistical module calculates the total internal attention and cross-modal attention of each layer of visual tags after inputting the statistical multimodal large language model as two initial distributions for a batch of data to be pruned;

[0055] The initial threshold module is used to initialize the error threshold based on the binary search algorithm given the target model computational load.

[0056] Deletion module, for each layer of visual markers, use a greedy algorithm to delete the visual markers with the smallest impact on the two initial distributions layer by layer until the errors of the two initial distributions reach the error threshold, and obtain two sets of candidate deletion markers; take the intersection of the two sets of candidate deletion markers as the final visual markers to be deleted, and delete the visual markers;

[0057] An updating module updates the error threshold boundary according to the model calculation amount after deleting the visual mark and the target calculation amount;

[0058] Stop the judgment module, repeat the deletion step and the update step until the stop condition is reached, and then enter the pruning step;

[0059] The pruning module calculates the average number of visual markers deleted per layer as a deletion strategy; sorts the visual markers to be deleted, taking into account both the internal attention size and the cross-modal attention size of the visual markers during sorting; and deletes the sorted visual markers according to the deletion strategy to achieve adaptive visual marker pruning.

[0060] The present invention has the following beneficial effects: the present invention proposes an adaptive visual marker pruning method, which optimizes the use of computing resources by automatically identifying and removing redundant visual tokens, thereby reducing operating costs and accelerating the reasoning speed of the model without sacrificing model performance, so that it can run efficiently even in a resource-constrained environment; the method does not require additional training data and complex model adjustment processes, is universal and efficient, and can be widely used in different tasks and data sets.

[0061] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments, but the present invention is not limited to the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 A method step diagram of an embodiment of the present invention;

[0063] Figure 2 A schematic diagram of a process for obtaining a deletion set according to an embodiment of the present invention.

[0064] Figure 3 A schematic diagram of the pruning process of an embodiment of the present invention;

[0065] Figure 4 FIG. 4 is a diagram showing the structure of a device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] See also Figure 1 FIG. 1 is a diagram showing steps of a method according to an embodiment of the present invention, comprising the following steps:

[0067] S101, for a batch of data to be pruned, the total internal attention and the total cross-modal attention of each layer of visual tags after inputting the statistical multimodal large language model are calculated as two initial distributions;

[0068] S102, given the target model computation amount, and initializing the error threshold according to the binary search algorithm;

[0069] S103, for each layer of visual markers, use a greedy algorithm to delete the visual markers with the smallest impact on the two initial distributions layer by layer until the errors of the two initial distributions reach the error threshold, and obtain two groups of candidate deletion markers; take the intersection of the two groups of candidate deletion markers as the final visual markers to be deleted, and delete the visual markers;

[0070] S104, updating the error threshold boundary according to the model calculation amount after deleting the visual mark and the target calculation amount;

[0071] S105, repeat S103 and S104 until the stop condition is reached, and enter the pruning step;

[0072] S106, calculating the average number of visual markers deleted per layer as a deletion strategy; sorting the visual markers to be deleted, considering both the internal attention size and the cross-modal attention size of the visual markers during sorting, deleting the sorted visual markers according to the deletion strategy, and realizing adaptive visual marker pruning.

[0073] Specifically, the process of obtaining the strategy in the embodiment of the present invention requires the use of a batch of data, such as 100; each data includes a picture and a question. Where X υ is the visual mark part, n is the number of visual marks, m is the number of text marks, d is the feature and dimension of the mark, this method is for X υ Pruning. Assuming that the model has L layers, in order to find an optimal strategy, under which the performance loss of the model is minimized and the expected model computation amount s can be achieved.

[0074] The present invention adopts an adaptive visual marker pruning method based on the attention distribution in a multimodal large model. Visual marker pruning is regarded as an attention distribution fitting task. The average attention distribution of the model on a batch of data is calculated, and two distributions are simultaneously fitted according to the defined goal to determine the pruning ratio of each layer of visual markers.

[0075] Specifically, the statistical process of obtaining the deletion set in the embodiment of the present invention is as follows: Figure 2 As shown in Figure 2, the total amount of internal and cross-modal attention for each layer of visual tags in the multimodal large language model is:

[0076]

[0077] A ′ =max(A1,A2,…,A h );

[0078] Where A′ represents the maximum value of the attention matrix of each attention head, n and m represent the number of visual tags and text tags respectively, and f I (t l,i ) represents the visual mark t l,i The total amount of attention received from other visual markers, f C (t l,i ) represents the visual mark t l,i The total amount of attention received by the text token.

[0079] By calculating each layer of the model f I (t l,i ) and f C (t l,i ), we can get the internal and cross-modal attention distribution of each layer of visual tags, which are expressed as:

[0080]

[0081] Where l represents the number of layers in the model and N represents the total number of visual markers in the lth layer.

[0082] Specifically, the target calculation amount of S102 actually needs to be preset, and the initial error threshold is initialized according to the binary search algorithm, and the value range of the threshold is 0 to 1. min =0, right boundary α max =1, the initial value of the error threshold is the mean of the two boundaries:

[0083]

[0084] Specifically, the process of S103 is as follows:

[0085] (1) First, initialize two empty sets P l,1 and P l,2 , l is the number of model layers, which are used to store candidate deletion markers with the least impact on internal and cross-modal attention distributions.

[0086] (2) Calculate the initial internal attention sum of all tags and cross-modal attention sum

[0087] (3) Traverse the small batch of data, and for each sample, process the visual labels of each layer in turn. For each label t l,i, calculate the internal attention distribution D if the mark is deleted I (l) and cross-modal attention distribution D C (l) The impact of each visual mark after deletion on the two distributions is specifically sorted from small to large, expressed as:

[0088] X v,I =sort(X v ,d(D I (l),D I ′ (l)));

[0089] X v,C =sort(X v ,d(D C (l),D ′ C (l)));

[0090] Among them, X v,I represents the visual mark sequence sorted by first impact, X v,C represents the visual mark sequence sorted according to the second influence, X v represents the sequence of visual markers input to the layer, sort represents the sorting function; d is the difference between the distribution before and after the visual markers are deleted, which are respectively expressed as:

[0091]

[0092] According to the two sorted visual marker sequences and, respectively, find a set of visual marker sequence subsets from the front to the back, whose sum is less than the error threshold, expressed as:

[0093]

[0094] So far, we have obtained two sets of candidate visual labels P at the lth layer. l,1 and P l,2 .

[0095] (4) Take the set P l,1 and P l,2 The intersection of is deleted as the deletion set.

[0096] Specifically, S104 is as follows: after traversing the entire network, the boundary of the binary search algorithm is updated according to whether the computation amount of the visual mark reaches the target s:

[0097]

[0098] Among them, FLOPs is the amount of calculation obtained by statistics during the model operation, X0 is the visual mark sequence of the initial input model (complete and not deleted), is the set of deleted sets, and L is the total number of model layers. The meaning is that given the input visual label sequence X0, in the deletion set Under the condition of , the model calculation amount is less than the expected s.

[0099] Specifically, in S105, if the stop condition is met, the process proceeds to S106; otherwise, α is updated and S103 to S104 are repeated; the stop condition is:

[0100] α max -α min ≤δ;

[0101] Wherein, δ defaults to 0.01; the update α is expressed as:

[0102]

[0103] Specifically, the process of obtaining the deletion strategy in S106 is as follows: the average number of tokens deleted in each layer obtained by counting in the previous step is used as the strategy, denoted as E(l), which represents the number of visual markers deleted in the lth layer of the model.

[0104] Specifically, the process of implementing visual mark pruning in S106 is as follows: Figure 3 In the actual reasoning process, according to f U (t l,i ) = f C (t l,i )·f I (t l,i ) is the index to sort the visual marks from small to large, and delete the corresponding number of visual marks according to the pre-obtained deletion strategy E(l). The deleted visual marks are denoted as Q(l):

[0105]

[0106] in, Represents the visual mark t l,i In the comprehensive importance measure f U (t l,i ) position in the sort order; Indicates processing input data b i The visual mark removed at layer l, express The number of

[0107] In order to verify the effectiveness of the method, this method was applied to three models and compared on five picture-text question-answering datasets. The results are shown in Table 1.

[0108] Table 1: Performance comparison of different pruning strategies on multiple multimodal large language models

[0109]

[0110] According to Table 1, the method proposed in the embodiment of the present invention can maintain performance on all three models, while significantly reducing the computational complexity of the model, and achieving impressive results in all data sets. In summary, the visual marker pruning strategy search algorithm proposed in this method can obtain a better visual marker pruning strategy under the premise of a given expected computational complexity of the visual marker, and enable the model to perform well on multiple data sets.

[0111] For details, see Figure 4 FIG. 1 is a diagram showing a structure of a device according to an embodiment of the present invention, comprising:

[0112] The initial statistical module counts the total internal attention and cross-modal attention of each layer of visual tags in the multimodal large language model as two initial distributions;

[0113] The initial threshold module, given the target computation amount of the visual mark, initializes the error threshold according to the binary search algorithm;

[0114] Deletion module: For each layer of visual markers, a greedy algorithm is used to delete the visual markers with the smallest impact on the two initial distributions layer by layer until the errors of the two initial distributions reach the error threshold, and two groups of candidate deletion markers are obtained; the intersection of the two groups of candidate deletion markers is taken as the final visual marker to be deleted, and the visual markers are deleted without exceeding the target calculation amount;

[0115] An update module updates the error threshold boundary according to the model operation amount after deleting the visual mark;

[0116] Stop the judgment module, repeat the deletion step and the update step until the stop condition is reached, and then enter the pruning step;

[0117] The pruning module calculates the average number of visual markers deleted per layer as a deletion strategy; sorts the visual markers to be deleted, taking into account both the internal attention size and the cross-modal attention size of the visual markers during sorting; and deletes the sorted visual markers according to the deletion strategy to achieve adaptive visual marker pruning.

[0118] It can be seen that the method proposed in the present invention significantly reduces the computational complexity, and at the same time shows comparable or better performance than the unpruned model on multiple visual language benchmark tests; and while maintaining or improving the model performance, it effectively reduces the inference cost and improves the model's inference efficiency.

[0119] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An adaptive visual label pruning method based on a multimodal large model, characterized in that: The following steps are involved: In the initial statistical step, for a batch of data to be pruned, the total internal attention and cross-modal attention of each layer of visual tags after inputting the statistical multimodal large language model are calculated as two initial distributions; Initial threshold step, given the target model computation amount, and initialize the error threshold according to the binary search algorithm; Deletion step: for each layer of visual markers, use a greedy algorithm to delete the visual markers with the smallest impact on the two initial distributions layer by layer until the errors of the two initial distributions reach the error threshold, and obtain two sets of candidate deletion markers; take the intersection of the two sets of candidate deletion markers as the final visual markers to be deleted, and delete the visual markers; An updating step, in which the error threshold boundary is updated according to the model calculation amount after deleting the visual mark and the target calculation amount; Stop the judgment step, repeat the deletion step and the update step until the stop condition is reached, and then enter the pruning step; In the pruning step, the average number of visual markers deleted per layer is calculated as the deletion strategy; The visual markers to be deleted are sorted, and the internal attention size and cross-modal attention size of the visual markers are considered during the sorting. The sorted visual markers are deleted according to the deletion strategy to achieve adaptive visual marker pruning.

2. The method for adaptive visual label pruning based on a multimodal large model according to claim 1, characterized in that: The multimodality includes a visual modality and a textual modality, and the multimodal large language model includes visual tags and textual tags.

3. The method for adaptive visual marker pruning based on a multimodal large model according to claim 2 is characterized in that: The two initial distributions include the total amount of internal attention and the total amount of cross-modal attention, expressed as: A ′ =max(A1,A2,…,A h ), Among them, D I (l) represents the total amount of internal attention of the visual markers at the lth layer, D C (l) represents the total amount of cross-modal attention of the visual tags in the lth layer, and N represents the total number of visual tags in the lth layer; A ′ represents the maximum value in the attention matrix of each attention head, h represents the number of attention heads; n and m represent the number of visual tags and text tags respectively; f I (t l,i ) represents the visual mark t l,i The total amount of attention received from other visual markers, f C (t l,i ) represents the visual mark t l,i The total amount of attention received by the text token.

4. The method for adaptive visual marker pruning based on a multimodal large model according to claim 3 is characterized in that: The error threshold is expressed as: Among them, α min represents the left boundary, α max Indicates the right boundary; The initialization error threshold refers to, let α min =0,α max =1.

5. The method for adaptive visual label pruning based on a multimodal large model according to claim 4, characterized in that: The deleting step deletes each layer of visual marks respectively, and the process of deleting the first layer includes the following steps: Initialize two empty sets to store candidate deletion markers with the smallest impact on internal attention and cross-modal attention respectively; For each visual marker t l,i , calculate the first impact of deleting the tag on the internal attention distribution and the second impact on the cross-modal attention distribution, which are expressed as: The visual marker sequence input to the layer is sorted according to the first influence and the second influence respectively to obtain the visual marker sequence, which is expressed as: X v,I =sort(X v ,d(D I (l),D I ′(l))); X v,C =sort(X v ,d(D C (l),D′ C (l))); Among them, X v,I represents the visual mark sequence sorted by first impact, X v,C represents the visual mark sequence sorted according to the second influence, X v Represents the visual mark sequence input to this layer, and sort represents the sorting function; From the visual marker sequence X v,I and X v,C Find a set of visual marker sequence subsets from the front to the back, store them in two empty sets as two sets of candidate visual markers, expressed as: Take the set P l,1 and P l,2 The intersection of is used as the deletion set to delete the visual mark of the corresponding layer, and the deletion set is expressed as: P l =P l,1 ∩P l,2 。 6. The method for adaptive visual label pruning based on multimodal large model according to claim 5, characterized in that: The set P l,1 and P l,2 The intersection of is used as the deletion set to delete the visual markers of the corresponding layer. During the deletion process, it is ensured that the model calculation amount does not exceed the target model calculation amount. Specifically, when executing the deletion operation, the calculation amount of the entire model after deletion is monitored. If the deletion operation causes the calculation amount to exceed the target calculation amount, the deletion of the next layer of visual markers is stopped.

7. The method for adaptive visual label pruning based on multimodal large model according to claim 6, characterized in that: In the updating step, the error threshold boundary is updated according to the comparison result between the operation amount of the model after deleting the visual mark and the operation amount of the target model, which is expressed as: Among them, FLOPs is the amount of model operations after deleting the visual labels, X0 is the visual label sequence of the initial input model without deletion, is the set of deletion sets of each layer, and L is the total number of layers in the model; Indicates that given an input visual tag sequence X0 The model operation amount under the condition, s represents the target model operation amount.

8. The method for adaptive visual marker pruning based on a multimodal large model according to claim 7, characterized in that: The stop condition in the stop judgment step is expressed as: a max -a min ≤δ; Among them, δ is the preset binary search stop threshold; If the stopping condition is met, the pruning step is entered; if it is not met, the error threshold is updated according to the updated error threshold boundary, and the deletion step and the update step are repeated.

9. The method for adaptive visual label pruning based on multimodal large model according to claim 5, characterized in that: The pruning step specifically includes the following steps: The average number of visual mark deletions per layer is calculated as the deletion strategy, expressed as: in, Indicates processing input data b i The visual mark removed at layer l, express The number of, c is the total number of samples; The comprehensive importance measure is calculated for the visual markers to be deleted, expressed as: f U (t l,i )=f C (t l,i )·f I (t l,i ); The visual mark is measured according to the comprehensive importance measure f U (t l，i ) Sort from small to large; The sorted visual markers are deleted according to the deletion strategy, which is expressed as: in, Represents the visual mark t l,i In the comprehensive importance measure f U (t l,i ) in the sort order.

10. An adaptive visual marker pruning device based on a multimodal large model, characterized in that: include: The initial statistical module calculates the total internal attention and cross-modal attention of each layer of visual tags after inputting the statistical multimodal large language model as two initial distributions for a batch of data to be pruned; The initial threshold module is used to initialize the error threshold based on the binary search algorithm given the target model computational load. Deletion module, for each layer of visual markers, use a greedy algorithm to delete the visual markers with the smallest impact on the two initial distributions layer by layer until the errors of the two initial distributions reach the error threshold, and obtain two sets of candidate deletion markers; take the intersection of the two sets of candidate deletion markers as the final visual markers to be deleted, and delete the visual markers; An updating module updates the error threshold boundary according to the model calculation amount after deleting the visual mark and the target calculation amount; Stop the judgment module, repeat the deletion step and the update step until the stop condition is reached, and then enter the pruning step; The pruning module calculates the average number of visual markers deleted per layer as a deletion strategy; The visual markers to be deleted are sorted, and the internal attention size and cross-modal attention size of the visual markers are considered during the sorting. The sorted visual markers are deleted according to the deletion strategy to achieve adaptive visual marker pruning.

Images