A visual language model visual mark selection method based on importance and diversity
By employing a visual tag selection method based on importance and diversity, calculating visual saliency and instruction relevance scores, and combining iterative suppression selection, the performance instability of visual language models during pruning is addressed, achieving efficient visual tag pruning and improved inference speed.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XIAMEN UNIV
- Filing Date
- 2026-02-06
- Publication Date
- 2026-06-23
AI Technical Summary
Existing visual language models struggle to effectively reduce inference overhead while maintaining model performance during visual tagging pruning. Furthermore, existing methods are unstable under high compression rates, which can easily lead to semantic loss or severe redundancy.
A visual marker selection method based on importance and diversity is adopted. By calculating the visual saliency score and instruction relevance score, and combining iterative suppression selection process, key visual markers are selected and similar marker scores are suppressed to generate multimodal input sequences for inference.
It maintains stable inference performance under extreme tag-pruning scenarios, significantly reduces FLOPs and KV cache usage, improves actual inference speed, and provides performance robustness under high compression ratios.
Smart Images
Figure CN122265674A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of visual language processing technology, and specifically to a visual tag selection method based on a visual language model of importance and diversity. Background Technology
[0002] With the rapid development of large-scale visual language models (LVLMs), significant performance improvements have been achieved in tasks such as cross-modal reasoning, visual question answering, and image understanding. However, visual encoders typically encode an image into a large number of visual tokens, which then participate in attention computation and key-value cache storage at each layer of the large language model. As the number of tokens grows linearly, the model's inference overhead increases rapidly in an approximately quadratic manner, leading to a significant increase in FLOPs, memory usage, and end-to-end latency, making it difficult for current LVLMs to meet efficiency requirements in real-world applications. To reduce the inference burden, visual token pruning has become a key technical direction, aiming to reduce the number of visual tokens while maintaining model performance as much as possible.
[0003] However, most existing methods have significant limitations. Importance-based methods typically determine importance through cross-modal attention or CLS attention, but these methods lead to severe redundancy and are more prone to losing key semantics at high compression ratios. Diversity-based tag selection methods are prone to mistakenly deleting high-value tags, resulting in semantic loss and performance degradation. Both types of methods are centered on a single factor for tag selection, thus exhibiting unstable performance at high compression ratios and struggling to balance semantic integrity and redundancy removal. Summary of the Invention
[0004] To address the problems existing in the prior art, the present invention aims to provide a visual tag selection method for visual language models based on importance and diversity, which achieves high semantic preservation and low redundancy in visual tag selection, enabling visual language models to maintain high stability while reducing inference computation.
[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A visual marker selection method based on an importance and diversity visual language model includes the following steps: Step S1: Obtain the input visual data and corresponding text instructions, and extract the visual features and text features respectively; Step S2: Generate multiple visual tags based on the visual features, and calculate the visual salience score for each visual tag; Step S3: Calculate the instruction relevance score between each visual mark and the text instruction; Step S4: Calculate the overall importance score for each visual marker based on the visual saliency score and the instruction relevance score; Step S5: Based on the comprehensive importance score, perform an iterative suppression selection process to select a preset number of key visual markers from all visual markers. In each iteration, select the visual marker with the highest current score and suppress the scores of the remaining visual markers that are semantically similar to the selected markers. Step S6: Concatenate the selected key visual markers and text markers and input them into the large language model to generate the response output.
[0006] The method for calculating the visual saliency score in step S2 includes: visual features Take a projector as input and map it to a sequence of visual labels; Next, the self-attention matrix between visual tags is calculated: ,in, These are the query matrix and the key matrix, respectively. For feature dimensions; Using text features as the query and visual features as the key, we obtain the cross-modal attention score: ,in, For text query matrix, For visual key matrix; Pick The score corresponding to each visual marker is used as its visual saliency. .
[0007] The method for calculating the instruction relevance score in step S3 includes: Calculate the cosine similarity between the feature vector of each visual marker and the feature vector of the text instruction. And all similarities are normalized. , The normalized instruction relevance score is obtained.
[0008] The calculation method for the comprehensive importance score in step S4 is as follows: Multiply the normalized instruction relevance score by the visual saliency score to obtain the overall importance score for each visual marker: .
[0009] The iterative suppression selection process in step S5 specifically includes: The initial selection set is empty, and the candidate set contains all visual markers and their combined importance scores; Repeat the following steps until the number of visual markers in the selected set reaches the preset number: Select the visual marker with the highest overall importance score from the candidate set and add it to the selection set; Calculate the semantic distance between the visual tag and every other visual tag in the candidate set; The overall importance score of each visual marker in the candidate set is updated with suppression based on the semantic distance; After updating, select the next visual marker with the highest score.
[0010] The semantic distance is calculated using cosine distance, specifically: ,in, Visual markers and eigenvectors.
[0011] The suppression update method is as follows: Suppression weights are calculated based on semantic distance. ; And use the weights to assign visual labels to the candidate set. The overall importance score is decayed: ,in, This is an adjustable suppression coefficient.
[0012] The visual data is an image or video frame, and the visual features are extracted by a visual encoder, while the text features are extracted by a text encoder.
[0013] By adopting the above scheme, this invention first calculates the comprehensive importance of each tag based on attention and instruction similarity. Then, after selecting the highest-scoring tag each time, it decays the scores of its similar tags to make subsequent selections more differentiated, thereby significantly reducing redundant regions while preserving key semantic regions. This design enables the model to maintain stable inference performance even in extreme tag pruning scenarios and significantly reduces FLOPs and key-value cache usage, thus improving actual inference speed. The method of this invention can be applied to any visual language model without training and provides stronger performance robustness at high compression rates. Attached Figure Description
[0014] Figure 1 This is a schematic diagram of the principle of the present invention; Figure 2 The results show the average performance comparison between the present invention and existing methods under different marker pruning ratios on nine multimodal benchmark tests. Figure 3 This section compares the inference efficiency of the present invention with that of existing methods. Detailed Implementation
[0015] like Figure 1As shown, this invention discloses a visual label selection method based on importance and diversity in a visual language model. It constructs an importance score of "visual salience × instruction relevance" in a unified manner and introduces a similarity-based inhibition mechanism during the iterative selection process, which effectively avoids the local clustering of high-scoring labels.
[0016] Step 1: Input Acquisition and Feature Extraction.
[0017] It accepts two inputs: image or video data, and text instructions entered by the user.
[0018] First, feature extraction is performed on the two inputs: Step 1.1: Text Feature Extraction; Input the user command into a pre-trained text encoder (such as BERT or CLIP text encoder) for encoding to obtain text features. Simultaneously, the text is segmented into words to obtain a text tag sequence.
[0019] Step 1.2, Visual Feature Extraction: Input the image or video frame into a visual encoder (such as ViT, ResNet, etc.) to extract visual features. The output of the visual encoder can be represented as: (1) in, For category labeling, For the first The feature vector of an image label.
[0020] Step 2: Visual tag generation and attention calculation.
[0021] visual features The input is a projector (usually a linear layer), which maps to a sequence of visual labels. Next, the self-attention matrix between the visual labels is calculated: (2) in, These are the query matrix and the key matrix, respectively. For feature dimensions.
[0022] To evaluate the visual saliency of each visual tag, a cross-modal attention score is further calculated. Using text features as the query and visual features as the key, the cross-modal attention score is obtained: (3) in, For text query matrix, This is the visual key matrix. Take... The score corresponding to each visual marker is used as its visual saliency. .
[0023] Step 3: Calculate instruction dependency.
[0024] To measure the relevance of each visual mark to the text instruction, visual features are calculated. Text features Cosine similarity between them: (4) To standardize the units, cosine similarity is... Perform min-max normalization: (5) Step 4: Calculate the overall importance score.
[0025] By combining visual saliency with instruction relevance, a comprehensive importance score is calculated for each visual marker: (6) This score reflects both the visual prominence of the marker and its semantic relevance to the task instructions.
[0026] Step 5: Iterative suppression selection based on diversity.
[0027] To maximize information diversity within a limited budget, this invention employs an iterative suppression selection mechanism: Initialization: Let the selection set U = The candidate set C contains all visual markers and their importance scores S.
[0028] Iteratively execute the following steps until the number of tags in set U reaches the preset budget K: Step 5.1: Select the highest-scoring marker i from set C, add it to set U, and remove it from set C.
[0029] Step 5.2: Calculate the semantic distance between tag i and each tag j in set C: (7) Step 5.3: Perform a suppression update on the marked scores in set C based on the distance: Calculate inhibition weights based on semantic distance: (8) And use the weights to assign visual labels to the candidate set. The overall importance score is decayed: (9) in, This is the suppression intensity coefficient.
[0030] Step 5.4: Return to step 5.1 and continue making selections.
[0031] Step 6: Multimodal fusion and response generation.
[0032] The visual and textual tags in the final selection set U are concatenated to form a multimodal input sequence, which is then fed into a large visual language model (such as LLaVA, Qwen-VL, etc.) for inference to generate a task response.
[0033] Figure 2 The performance of the method of this invention is compared with that of existing methods under different pruning ratios. Even at a pruning rate as high as 97.2%, the present invention can still maintain more than 90% of the original performance, significantly outperforming methods that rely solely on importance or diversity.
[0034] Figure 3 The improved inference efficiency is demonstrated. This invention achieves a higher speedup ratio while significantly reducing FLOPs and KV cache usage, validating its efficiency advantages in real-world deployments.
[0035] In summary, this invention first calculates the overall importance of each tag based on attention and instruction similarity. Then, after selecting the highest-scoring tag each time, it decays the scores of similar tags to make subsequent selections more differentiated, thereby significantly reducing redundant regions while preserving key semantic regions. This design enables the model to maintain stable inference performance even in extreme tag pruning scenarios and significantly reduces FLOPs and key-value cache usage, thus improving actual inference speed. The method of this invention can be applied to any visual language model without training and provides stronger performance robustness at high compression rates.
[0036] The above description is merely an embodiment of the present invention and does not constitute any limitation on the technical scope of the present invention. Therefore, any minor modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the scope of the technical solution of the present invention.
Claims
1. A visual marker selection method based on an importance and diversity visual language model, characterized in that, Includes the following steps: Step S1: Obtain the input visual data and corresponding text instructions, and extract the visual features and text features respectively; Step S2: Generate multiple visual tags based on the visual features, and calculate the visual salience score for each visual tag; Step S3: Calculate the instruction relevance score between each visual mark and the text instruction; Step S4: Calculate the overall importance score for each visual marker based on the visual saliency score and the instruction relevance score; Step S5: Based on the comprehensive importance score, perform an iterative suppression selection process to select a preset number of key visual markers from all visual markers. In each iteration, select the visual marker with the highest current score and suppress the scores of the remaining visual markers that are semantically similar to the selected markers. Step S6: Concatenate the selected key visual markers and text markers and input them into the large language model to generate the response output.
2. The method according to claim 1, characterized in that, The method for calculating the visual saliency score in step S2 includes: visual features Take a projector as input and map it to a sequence of visual labels; Next, the self-attention matrix between visual tags is calculated: ,in, These are the query matrix and the key matrix, respectively. For feature dimensions; Using text features as the query and visual features as the key, we obtain the cross-modal attention score: ,in, For text query matrix, For visual key matrix; Pick The score corresponding to each visual marker is used as its visual saliency. .
3. The method according to claim 1 or 2, characterized in that, The method for calculating the instruction relevance score in step S3 includes: Calculate the cosine similarity between the feature vector of each visual marker and the feature vector of the text instruction. All similarities are then normalized to obtain the normalized instruction relevance score. ,in, It is a visual feature. It is a text feature.
4. The method according to claim 3, characterized in that, The calculation method for the comprehensive importance score in step S4 is as follows: Multiply the normalized instruction relevance score by the visual saliency score to obtain the overall importance score for each visual marker: .
5. The method according to claim 1, characterized in that, The iterative suppression selection process in step S5 specifically includes: The initial selection set is empty, and the candidate set contains all visual markers and their combined importance scores; Repeat the following steps until the number of visual markers in the selected set reaches the preset number: Select the visual marker with the highest overall importance score from the candidate set and add it to the selection set; Calculate the semantic distance between the visual tag and every other visual tag in the candidate set; The overall importance score of each visual marker in the candidate set is updated with suppression based on the semantic distance; After updating, select the next visual marker with the highest score.
6. The method according to claim 5, characterized in that, The semantic distance is calculated using cosine distance, specifically: ,in, Visual markers and eigenvectors.
7. The method according to claim 5 or 6, characterized in that, The suppression update method is as follows: Suppression weights are calculated based on semantic distance. ; And use the weights to assign visual labels to the candidate set. The overall importance score is decayed: ,in, This is an adjustable suppression coefficient.
8. The method according to claim 1, characterized in that, The visual data is an image or video frame, and the visual features are extracted by a visual encoder, while the text features are extracted by a text encoder.