Efficient token compression method for visual language large model

Abstract

The application discloses an efficient word compression method for a visual language large model, introduces a lightweight word compression module between a visual encoder and a language model, selects a compact word set with information importance and diversity from original visual words without relying on attention information and large-scale word pairwise comparison, thereby effectively reducing the sequence length input to the language model, and through double selection of importance perception and diversity perception on the original word sequence output by the visual encoder, a compressed word sequence is obtained, thereby providing compact and high-quality visual input representation for subsequent large language model reasoning, the method does not need to retrain or fine-tune the original model, has good plug-and-play characteristics, is convenient to integrate into an existing visual-language model system, significantly reduces the deployment threshold, has good engineering application value, and realizes collaborative optimization selection of visual words.

Description

Technical Field

[0001] This invention relates to the field of visual language understanding based on artificial intelligence (AI), and specifically to an efficient lexical compression method for large visual language models. Background Technology

[0002] With the rapid development of multimodal artificial intelligence technology, Vision-Language Models (VLMs), as an important technical approach integrating visual perception and language understanding, have demonstrated excellent performance in tasks such as visual question answering, image description, and multimodal dialogue. These models extract image features through visual encoders and combine them with large language models to achieve cross-modal semantic alignment and reasoning capabilities, and have become one of the core foundational models of current multimodal intelligent systems.

[0003] However, existing vision-language models still face significant efficiency bottlenecks in practical deployments, especially on resource-constrained edge devices or low-power computing platforms. The main reason is that visual encoders generate a large number of visual lexical units; for example, encoders based on the visual Transformer structure often generate hundreds or even thousands of lexical units for a single image. These lexical units, when input into the language model, significantly increase the sequence length and cause the computational complexity of the self-attention mechanism to increase quadratically, resulting in huge computational overhead and inference latency, thus limiting the model's real-time performance and scalability.

[0004] To alleviate the aforementioned problems, visual lexical compression technology has become an important research direction for improving the inference efficiency of vision-language models. This type of method reduces the number of visual lexical units input to the language model while preserving key visual information, thereby reducing computational costs and increasing inference speed.

[0005] In existing technologies, a mainstream approach relies on attention maps generated by attention mechanisms to evaluate the importance of each visual word and discard or merge words with low scores accordingly. These methods can identify visual regions that are more critical to downstream tasks to some extent, but their implementation depends on explicitly computing and storing the attention matrix. However, with the development of efficient attention acceleration frameworks (such as the Flash Attention framework), these frameworks typically avoid explicitly generating complete attention maps, thus creating compatibility issues with the aforementioned attention map-based word selection methods and limiting their application in practical systems.

[0006] Another type of approach attempts to measure redundancy between lexical units through feature similarity and thereby filter diverse visual lexical sets. These methods typically require pairwise similarity calculations between lexical units to identify redundant information. However, this process often introduces high computational overhead, especially when the number of lexical units is large. Pairwise comparison operations incur significant additional overhead, offsetting the efficiency gains from reducing the number of lexical units and hindering large-scale deployment.

[0007] In summary, existing technologies for visual lexical compression generally suffer from problems such as strong reliance on attention mechanisms and high computational complexity. Therefore, it is necessary to propose an attention-independent and computationally lightweight visual lexical compression method to improve the efficiency and deployability of visual-language models in practical applications.

[0008] The existing technology has the following technical problems:

[0009] 1) While existing visual lexical compression methods improve the inference efficiency of visual-language models, there is an irreconcilable contradiction between "attention-independent" and "computationally lightweight". The lack of a unified solution that combines system compatibility and computational efficiency restricts its efficient deployment and large-scale application in real systems.

[0010] 2) Lexical compression methods that rely on attention maps to evaluate lexical importance lack system compatibility and have a mechanism conflict with efficient attention acceleration frameworks such as FlashAttention, making them difficult to apply directly in actual deployments.

[0011] 3) Lexical compression methods that select diverse lexical units based on feature similarity usually require intensive pairwise comparisons of all visual lexical units to measure similarity. The computational complexity increases quadratically with the number of lexical units, which introduces a huge additional computational burden in high-resolution or long video scenarios, weakening the efficiency gains brought by compression. Summary of the Invention

[0012] To address the issues of high computational complexity and strong reliance on attention mechanisms in lexical compression of existing visual-language models, this invention proposes an attention-free, lightweight visual lexical compression method and system for visual-language models. This significantly reduces computational overhead while maintaining multimodal inference performance, thereby improving the model's inference efficiency and deployment adaptability. The basic idea of ​​this invention is to introduce a lightweight lexical compression module between the visual encoder and the language model. This lexical compression method selects a compact set of lexical units from the original visual units that possess both informational importance and diversity, without relying on attention information or large-scale pairwise comparisons, thus effectively reducing the sequence length input to the language model.

[0013] To achieve the above objectives, the present invention provides the following technical solution:

[0014] This invention discloses an efficient lexical compression method for large visual language models, the overall framework of which mainly includes three stages:

[0015] 1) Importance-aware lexical selection: Given a visual lexical sequence The entropy value of each visual word is calculated to measure its information content. The word sequence is then sorted according to the entropy value, and the word with the highest score is selected. Each visual lexical unit serves as a set of important lexical units. The remaining lexical units are denoted as ;

[0016] 2) Lexical selection for diversity perception: for the remaining visual lexical units Feature transformation is performed, and the similarity between lexical units is characterized by measuring the consistency of features before and after the transformation. Interval sampling is then performed based on the ranking to select... Each visual lexical unit, as a diverse set of lexical units ;

[0017] 3) Merging of lexical sets: Merging of important lexical sets With diverse lexical sets The fusion process yields the final compressed set of visual lexical units. This is to achieve efficient compression of word units.

[0018] As a further improvement, the method described in this invention involves inserting a sequence of original lexical units output by the visual encoder between the visual encoder and the large language model. By performing a dual selection based on importance perception and diversity perception, a compressed word sequence is obtained. This provides a compact and high-quality visual input representation for subsequent large language model inference.

[0019] As a further improvement, the given visual lexical sequence described in this invention The entropy value of each visual word is calculated to measure its information content, specifically:

[0020] For the original visual word sequence Each visual lexical element in Entropy values ​​are calculated for its distribution across various feature dimensions. First, for each visual word unit... Normalize its eigenvalues ​​along the channel dimension:

[0021] ,in It is considered as its discrete probability distribution along the feature dimension, and then the second-order Reni entropy of the distribution corresponding to each visual word is calculated: The information richness of each word element is quantitatively evaluated.

[0022] As a further improvement, the present invention sorts the word sequence according to the entropy value and selects the one with the highest score. Each visual lexical unit serves as a set of important lexical units. Specifically:

[0023] For all visual lexical units, based on their second-order Reni entropy value Sort and filter by high and low, prioritizing those with higher entropy values. From visual lexical units, we obtain the set of importance lexical units. .

[0024] As a further improvement, the present invention describes the processing of remaining visual lexical units. Feature transformation is performed, and the similarity between lexical units is characterized by measuring the consistency of features before and after the transformation. Specifically:

[0025] The remaining visual lexical sequence that completes the importance selection By utilizing the degree of feature change of visual lexical units before and after mapping through a fully connected layer, a metric reflecting lexical similarity is constructed. First, for each visual lexical unit... It calculates its value through the last fully connected layer of the visual encoder. Transformed representation: Furthermore, by calculating the cosine similarity of the same lexical unit before and after the transformation, the degree of lexical feature transformation caused by the nonlinear transformation is quantified: This can be used as a metric to reflect word similarity.

[0026] As a further improvement, the present invention performs interval sampling based on sorting to select... Each visual lexical unit, as a diverse set of lexical units Specifically:

[0027] Metrics based on word similarity For all remaining word sequences Sort the data, and then perform a step-size-based sorting operation based on the sorting results. The uniform sampling strategy is selected. Each visual lexical unit yields a diverse set of lexical units. .

[0028] As a further improvement, the merging of the word set described in this invention specifically involves taking the union of the two subsets selected by the importance perception and diversity perception mentioned above: .

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] (1) This invention inserts a lexical compression method between the visual encoder and the large language model of a visual-language model. By performing dual selection of importance perception and diversity perception on the original lexical sequence output by the visual encoder, a compressed lexical sequence is obtained, thereby providing a compact and high-quality visual input representation for subsequent large language model inference. This method does not require retraining or fine-tuning of the original model, has good plug-and-play characteristics, is easy to integrate into the existing visual-language model system, significantly reduces the deployment threshold, and has good engineering application value;

[0031] (2) This invention achieves collaborative optimization selection of visual lexical units by integrating a dual constraint mechanism of importance and diversity. The importance constraint ensures that the retained lexical units possess high information content and semantic contribution, while the diversity constraint further avoids information redundancy and enhances the coverage of the lexical set for global visual content. Compared to screening methods that rely solely on a single indicator, this invention significantly reduces the number of visual lexical units while effectively alleviating the problem of key information loss, maintaining or even improving multimodal task performance, and achieving a balance between efficiency and effectiveness.

[0032] (3) This invention performs importance-aware lexical selection. For a given visual lexical sequence, the entropy value of each visual lexical is calculated to measure its information content. The lexical sequence is sorted according to the entropy value, and the visual lexical with the highest score is selected as the set of important lexicals. Compared with the prior art, this invention does not rely on attention maps for lexical importance evaluation, avoids dependence on explicit attention calculation, and is therefore compatible with existing efficient attention acceleration frameworks, thus improving the overall reasoning efficiency of visual language models.

[0033] (4) This invention performs diversity-aware lexical selection by transforming the features of visual lexical units and characterizing the similarity between lexical units by measuring the consistency of features before and after the transformation. Based on the ranking, interval sampling is performed to select visual lexical units as a diverse lexical set. Compared with the prior art, this invention does not require the introduction of expensive pairwise similarity calculations between lexical units to filter the diverse lexical set, effectively reducing the computational overhead of lexical unit selection and improving the overall reasoning efficiency of the visual language model. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the reasoning of the visual language big model of the insertion lexical compression method in this invention;

[0035] Figure 2 This is a flowchart illustrating an efficient lexical compression method for large-scale visual language models. Detailed Implementation

[0036] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0037] The purpose of this invention is to address the computational efficiency bottleneck caused by the large number of visual lexical units in visual language models. While ensuring no loss of key visual information, this invention implements an efficient lexical compression method to reduce the length of the input sequence and lower inference overhead. Furthermore, addressing the inherent contradiction between "attention independence" and "computational lightweight" in existing methods, and the lack of a unified solution that combines system compatibility and computational efficiency, this paper aims to design an efficient visual lexical compression framework that effectively preserves key visual information without accessing attention maps and avoiding high computational overhead, thereby improving the practical deployment performance of visual language models.

[0038] This section will use the visual language model LLaVA-1.5-7B (whose visual encoder is ViT-L / 14 and language model is Vicuna-7B) as a practical example to explain the above methods step by step, inserting a lexical compression method between the visual encoder and the large language model. Figure 1 As shown), by performing a dual selection of importance perception and diversity perception on the original word sequence output by the visual encoder ( Figure 2 As shown in the figure, a compressed word sequence is obtained, thus providing a compact and high-quality visual input representation for subsequent large language model inference, as detailed below:

[0039] 1) Lexical selection based on importance perception:

[0040] This step aims to prioritize the selection of visual lexical units with higher information content and stronger semantic contribution from the original visual lexical unit sequence. Since the input image generates a large number of visual lexical units after passing through the visual encoder, and the information density of different lexical units varies, inputting all of them into the subsequent language model would lead to significant computational redundancy. Therefore, it is necessary to first measure the information richness of each visual lexical unit.

[0041] Given an input image, features are first extracted using a visual encoder to generate a corresponding set of lengths. Dimension Visual word sequence For each visual lexical unit First, its eigenvalues ​​are normalized along the channel dimension to obtain a discrete probability distribution:

[0042]

[0043] Then, by calculating the second-order Reni entropy of the distribution corresponding to each visual lexical unit: This quantitatively evaluates the information richness of each lexical unit. Second-order Reni entropy is more sensitive to the concentration of probability distributions than Shannon entropy. When lexical units... When the amplitudes on each channel are similar Larger values ​​indicate richer information; conversely, smaller values ​​indicate fewer features concentrated in a few channels. Extremely small indicates a lack of information.

[0044] The set of second-order Reni entropies of all lexical units calculated according to the steps described above. Sort the sequences in descending order. Terminators with higher entropy values ​​carry richer visual information and should be retained in the compressed sequence. Select the highest-scoring term. Each visual lexical unit serves as a set of important lexical units. This corresponds to the high-information-content region of the image.

[0045] This step uses second-order Reni entropy as a metric for lexical information content because entropy calculation only requires statistical analysis of the feature channel dimensions of a single lexical; it also does not rely on an attention matrix, has good compatibility with system-level acceleration frameworks, and is a more lightweight and important metric. It can effectively reduce the redundancy of input lexical units while ensuring sufficient preservation of key semantic information, providing a more compact and discriminative input representation for subsequent multimodal inference in language models.

[0046] 2) Lexical selection based on diversity perception:

[0047] This step aims to further filter out representative and diverse visual lexical units from the remaining visual lexical units after the selection of important lexical units. Since important lexical units primarily retain regional information that contributes significantly to the target semantics, while the remaining lexical units still contain background, contextual structure, and spatial supplementary information, directly discarding them would result in incomplete visual information coverage. Therefore, further filtering of the remaining visual lexical units is necessary.

[0048] This step involves the remaining visual lexical sequence used to complete the importance selection. By utilizing the degree of feature change of visual lexical units before and after mapping through a fully connected layer, a metric reflecting lexical similarity is constructed.

[0049] First, for each visual lexical unit It calculates its value through the last fully connected layer of the visual encoder. Transformed representation: Furthermore, by calculating the cosine similarity of the same word before and after the transformation, the degree of word feature transformation caused by the nonlinear transformation is quantified as a metric reflecting word similarity.

[0050]

[0051] Metrics based on word similarity For all remaining word sequences Sort the data. Based on the sorting results, use a fixed step size. Uniformly spaced sampling, select Each visual lexical unit yields a diverse set of lexical units. .

[0052] This step characterizes the similarity of lexical units based on their consistency before and after transformation. By sampling at uniform intervals, a more diverse set of lexical units is selected, thus ensuring structural coverage of the remaining background information. Compared to methods that calculate vector similarity based on pairwise comparisons, this invention only relies on the sorting operation of scalar metrics, significantly reducing additional computational overhead.

[0053] 3) Merging of word sets:

[0054] This step aims to unify and integrate the high-information lexical units and diverse lexical units obtained from the previous screening to form a compressed visual lexical unit sequence for multimodal reasoning. Importance Lexical Unit Set It primarily preserves regions in the image that have high semantic contribution and rich information content, while retaining diverse word sets. This is then used to supplement the background structure and contextual information in the remaining areas. Therefore, the two are merged:

[0055]

[0056] It can compress sequence length while preserving key semantics and overall visual coverage.

[0057] Will Instead of the original complete visual word sequence, it is transformed by the cross-modal linker and then input into the backend of the language model for inference. The sequence length is greatly reduced to less than 30% of the original input, the computational complexity of self-attention is greatly reduced, and the inference latency and memory usage are significantly reduced.

[0058] It should be noted that this method is a plug-and-play module that requires no training whatsoever. Its operation does not rely on any attention map, nor does it involve pairwise interaction computation between tokens, thus possessing excellent system compatibility and extremely low computational overhead.

[0059] The LLaVA-1.5-7B visual language model was fully validated on the widely used multimodal evaluation dataset MME (the first comprehensive benchmark specifically designed to evaluate the perceptual and cognitive abilities of multimodal large language models). The experiments covered both performance and efficiency dimensions.

[0060] The baseline methods for comparison include the original model baseline and two representative training-free lexical compression schemes: one is to perform lexical compression within the language model, such as SparseVLM and PyramidDrop; the other is to perform lexical compression before entering the language model, such as VisionZip and VisPruner.

[0061] Table 1 shows the performance comparison on the LLaVA-1.5-7B model.

[0062] SparseVLM 95.42% 94.99% 85.48% PyramidDrop 95.09% 88.38% 67.17% VisionZip 94.03% 93.99% 87.99% VisPruner 94.85% 94.80% 89.12% This invention 97.16% 96.24% 91.32%

[0063] To facilitate a unified comparison, the performance of the original model baseline was normalized to 100%, and the relative performance of other methods was adjusted based on this benchmark. Following the settings of previous related studies, experiments were conducted with three word retention percentages of 33%, 22%, and 11%. Table 1 reports the percentage performance comparison results of our proposed method and the baseline methods on the MME dataset.

[0064] The above results demonstrate that, although the present invention does not rely on any attention maps or word pair interactions compared to the comparative schemes, it consistently outperforms all baseline methods in all word compression settings. With a 33% visual word retention setting, the present invention outperforms the best-performing comparative scheme by 1.74% and the worst-performing one by 3.13%; with a 22% visual word retention setting, it outperforms the best-performing comparative scheme by 1.25% and the worst-performing one by 7.86%; and with an 11% visual word retention setting, it outperforms the best-performing comparative scheme by 2.2% and the worst-performing one by 24.15%.

[0065] Table 2 shows the efficiency comparison on the LLaVA-1.5-7B model.

[0066] Table 2 summarizes the computation time and peak memory usage of the proposed method on the LLaVA-1.5-7B model, with 33% of the tokens retained, compared to two other methods that perform token removal before entering the language model. The results show that the proposed method achieves lower efficiency in both metrics compared to the two methods that also perform compression before the language model.

[0067] It should be understood that the above description of the implementation examples is quite detailed and should not be construed as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, all of which fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. An efficient lexical compression method for large-scale visual language models, characterized in that, Its overall framework mainly includes three stages: 1) Importance-aware lexical selection: Given a visual lexical sequence The entropy value of each visual word is calculated to measure its information content. The word sequence is then sorted according to the entropy value, and the word with the highest score is selected. Each visual lexical unit serves as a set of important lexical units. The remaining lexical units are denoted as ; 2) Lexical selection for diversity perception: for the remaining visual lexical units Feature transformation is performed, and the similarity between lexical units is characterized by measuring the consistency of features before and after the transformation. Interval sampling is then performed based on the ranking to select... Each visual lexical unit, as a diverse set of lexical units The remaining visual lexical units described above Feature transformation is performed, and the similarity between lexical units is characterized by measuring the consistency of features before and after the transformation. Specifically: The remaining visual lexical sequence that completes the importance selection By utilizing the degree of feature change of visual lexical units before and after mapping through a fully connected layer, a metric reflecting lexical similarity is constructed. First, for each visual lexical unit... It calculates its value through the last fully connected layer of the visual encoder. Transformed representation: Furthermore, by calculating the cosine similarity of the same lexical unit before and after the transformation, the degree of lexical feature transformation caused by the nonlinear transformation is quantified: This can be used as a metric to reflect word similarity; the aforementioned method of interval sampling based on ranking is used to select... Each visual lexical unit, as a diverse set of lexical units Specifically: Metrics based on word similarity For all remaining word sequences Sort the data, and then perform a step-size-based sorting operation based on the sorting results. The uniform sampling strategy is selected. Each visual lexical unit yields a diverse set of lexical units. ; 3) Merging of lexical sets: Merging of important lexical sets With diverse lexical sets The fusion process yields the final compressed set of visual lexical units. This is to achieve efficient compression of word units.

2. The efficient lexical compression method for large visual language models according to claim 1, characterized in that, The method involves inserting a sequence of original lexical units (TRUs) between the visual encoder and the large language model of a visual language model. By performing a dual selection based on importance perception and diversity perception, a compressed word sequence is obtained. This provides a compact and high-quality visual input representation for subsequent large language model inference.

3. The efficient lexical compression method for large visual language models according to claim 1, characterized in that, The given visual lexical sequence The entropy value of each visual word is calculated to measure its information content, specifically: For the original visual word sequence Each visual lexical element in Entropy values ​​are calculated for its distribution across various feature dimensions. First, for each visual word unit... Normalize its eigenvalues ​​along the channel dimension: ,in It is considered as its discrete probability distribution along the feature dimension, and then the second-order Reni entropy of the distribution corresponding to each visual word is calculated: The information richness of each word element is quantitatively evaluated.

4. The efficient lexical compression method for large visual language models according to claim 3, characterized in that, The process involves sorting the word sequences according to their entropy values ​​and selecting the highest-scoring sequence. Each visual lexical unit serves as a set of important lexical units. Specifically: For all visual lexical units, based on their second-order Reni entropy value Sort and filter by high and low, prioritizing those with higher entropy values. From visual lexical units, we obtain the set of importance lexical units. .

5. The efficient lexical compression method for large visual language models according to claim 4, characterized in that, The merging of the aforementioned lexical sets specifically involves taking the union of the two subsets selected from the importance perception and diversity perception: .

Images