A Visual Token Compression Method Based on Token Segmenter Classification and Applicability Analysis

Abstract

The application discloses a visual token compression method based on word segmenter classification and applicability analysis, and relates to the technical field of image recognition. The application discloses a visual token compression method based on word segmenter classification and applicability analysis, and relates to the technical field of image recognition. The application discloses a visual token compression method based on word segmenter classification and applicability analysis, and relates to the technical field of image recognition. The application discloses a visual token compression method based on word segmenter classification and applicability analysis, and relates to the technical field of image recognition. The application discloses a visual token compression method based on word segmenter classification and applicability analysis, and relates to the technical field of image recognition. The application discloses a visual token compression method based on word segmenter classification and applicability analysis, and relates to the technical field of image recognition. The application discloses a visual token compression method based on word segmenter classification and applicability analysis, and relates to the technical field of image recognition. The application discloses a visual token compression method based on word segmenter classification and applicability analysis, and relates to the technical field of image recognition. The application discloses a visual token compression method based on word segmenter classification and applicability analysis, and relates to the technical field of image recognition. The application discloses a visual token compression method based on word segmenter classification and applicability analysis, and relates to the technical field of image recognition. The application discloses a visual token compression method based on word segmenter classification and applicability analysis, and relates to the technical field of image recognition. The application discloses a visual token compression method based on word segmenter classification and applicability analysis, and relates to the technical field of image recognition. The application discloses a visual token compression method based on word segmenter classification and applicability analysis, and relates to the technical field of image recognition. The application discloses a visual token compression method based on word segmenter classification and applicability analysis, and relates to the technical field of image recognition. The application discloses a visual token compression method based on word segmenter classification and applicability analysis, and relates to the technical field of image recognition. The application discloses a visual token compression method based on word segmenter classification and applicability analysis, and relates to the technical field of image recognition. The application discloses a visual token compression method based

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and specifically to a visual token compression method based on word segmentation classification and applicability analysis. Background Technology

[0002] In the current development of Multimodal Large Language Models (MLLMs), visual token compression (VTC) technology has become a key means to alleviate computational pressure and reduce inference latency. However, existing research and practice still have several obvious limitations. First, the field lacks a unified, training-independent compression evaluation protocol. The compression operators, experimental settings, and evaluation metrics used in different studies vary significantly, making it difficult to make horizontal comparisons and reasonable interpretations of the results, and even more difficult to effectively transfer to real-world applications, seriously hindering technological iteration and consensus formation. Second, most existing work focuses on semantically biased visual segmenters (such as the Transformer-based CLIP-ViT), while there is insufficient research on the systematic behavior of detail-biased (such as the VQ-VAE system) and balanced visual segmenters at different compression stages. This results in conclusions that are obviously one-sided and fragmented, failing to provide a comprehensive and reliable basis for the selection and optimization of a "unified segmenter" architecture. Furthermore, there is a lack of universally applicable rules for choosing compression locations—including the input stage, intermediate stages within the tokenizer, and the final output stage. Different compression strategies exhibit significantly different effects on different types of visual tokenizers. In engineering practice, this often relies on empirical trials or repeated experimentation, which lacks theoretical guidance and greatly increases deployment costs and uncertainties. These shortcomings collectively limit the reliable application of visual token compression technology in high-resolution input, real-time inference, and resource-constrained scenarios, constituting a core problem that urgently needs to be solved in the development of this field. Summary of the Invention

[0003] To enable better application of visual token compression technology in high-resolution input, real-time inference, and resource-constrained scenarios, this invention proposes a visual token compression method based on tokenizer classification and applicability analysis, comprising the following steps:

[0004] S1: Establish an evaluation framework for visual word segmenters under a unified compression evaluation protocol, based on compression operators of uniform strength and consistent performance requirements;

[0005] S2: Based on the architectural features of visual word segmenters, visual word segmenters are classified into three categories: semantically biased, detail-biased, and balanced.

[0006] S3: Based on the visual information processing flow of the visual word segmenter, the compression position is divided into the input stage before the visual word segmenter processing, the intermediate stage between feature layers inside the visual word segmenter, and the final stage after the visual word segmenter output.

[0007] S4: Apply the evaluation framework to insert compression operations at all stages for each of the three types of visual word segmenters, and quantify the performance index value for each combination.

[0008] S5: Based on the performance index values ​​obtained from quantification, analyze the applicable patterns of various visual word segmenters at different compression positions;

[0009] S6: Based on applicable rules, determine the compression positions applicable to different types of visual token segmenters, and execute a visual token compression strategy optimized based on applicable rules accordingly.

[0010] This invention establishes a unified and training-independent compression evaluation protocol and employs parameter-independent plug-and-play compression operators. It constructs a consistent and comparable evaluation system across different visual word segmenter architectures and multi-stage compression locations, fundamentally solving the dilemma of unreproducible and difficult-to-transfer conclusions caused by inconsistent evaluation standards in existing research. Furthermore, this invention systematically covers three representative word segmenter families: semantically biased, detail-biased, and balanced. Through cross-sectional sensitivity comparison experiments, it reveals their essential differences in information preservation, robustness, and compression efficiency, overcoming the cognitive bias problem caused by existing research focusing only on semantically biased visual word segmenters. Most importantly, this invention proposes a universal compression strategy optimized based on applicability principles through systematic compression experiments at the input, intermediate, and final stages.

[0011] Furthermore, in step S2, the semantically biased visual segmenter is a visual segmenter based on the Transformer architecture; the detail-biased visual segmenter is a visual segmenter based on the VQ-VAE or generative encoder architecture; and the balanced visual segmenter is a dual-branch structure visual segmenter that includes semantic branches and detail branches.

[0012] Furthermore, in step S5, the applicable rule is as follows: under the output representation robustness standard, detail-biased visual segmenters are suitable for compression in the input stage, while semantic-biased visual segmenters are suitable for compression in the final stage.

[0013] Furthermore, in step S6, the visual token compression strategy optimized based on applicable rules specifically involves: employing a balanced visual tokenizer, performing compression at the input stage in the detail branch of the balanced visual tokenizer, and performing compression at the final stage in the semantic branch.

[0014] Furthermore, the applicable rules also include: under the output representation robustness standard, visual word segmenters are not used for compression in the intermediate stage.

[0015] Furthermore, in step S1, the compression operation operator is one of average pooling, max pooling, and stride convolution, and the uniform intensity is defined by one of downsampling factor, pooling kernel size, and compression factor.

[0016] Furthermore, the compression in the input stage is achieved by reducing the resolution of the input image or reducing the number of image patches, and the compression in the final stage is achieved by clustering and merging the output visual tokens or by selecting based on importance.

[0017] Furthermore, in step S1, the performance metrics include at least one task performance metric among accuracy, recall, and F1 score, and at least one efficiency metric among inference latency and video memory usage.

[0018] Compared with the prior art, the present invention has at least the following beneficial effects:

[0019] (1) The present invention proposes a visual token compression method based on segmenter classification and applicability analysis. It adopts a parameter-independent plug-and-play compression operator and constructs a consistent and comparable evaluation system at different visual segmenter architectures and multi-stage compression positions. This fundamentally solves the dilemma of unreproducible and difficult-to-transfer conclusions caused by inconsistent evaluation standards in existing studies.

[0020] (2) It covers three representative visual word segmenter families: semantic bias, detail bias and balanced. Through horizontal sensitivity comparison experiments, it obtains the essential differences between them in terms of information preservation, robustness and compression efficiency, and makes up for the cognitive one-sidedness caused by the existing technology only focusing on semantic bias visual word segmenters.

[0021] (3) Based on applicable rules, a differentiated "dual-path compression" strategy based on a balanced visual word segmenter is proposed, which effectively combines the front-end robustness of its detail branches with the end-end stability of its semantic branches, so as to maximize the performance of multimodal understanding and reasoning tasks while reducing computational and storage overhead. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating the steps of a visual token compression method based on word segmenter classification and applicability analysis.

[0023] Figure 2 The performance of various benchmark metrics under input stage compression (1× downsampling vs 2× downsampling).

[0024] Figure 3The benchmark performance under intermediate stage compression (2× average pooling).

[0025] Figure 4 The benchmark performance under the final stage compression (2× average pooling). Detailed Implementation

[0026] The following are specific embodiments of the present invention, which are described in conjunction with the accompanying drawings to further illustrate the technical solutions of the present invention. However, the present invention is not limited to these embodiments.

[0027] In recent years, with the rapid development of multimodal large model technology, visual token compression has become one of the key technologies for reducing computational overhead and improving inference efficiency. However, existing visual token compression methods still have significant limitations: on the one hand, the field lacks unified evaluation standards and protocols, and different studies use significantly different compression operators, experimental settings, and evaluation metrics, making it difficult to reproduce results and conduct cross-sectional comparisons, which seriously restricts the iteration of the technology and the formation of consensus; on the other hand, existing work mainly focuses on semantically biased visual segmenters, lacking systematic research on detail-biased and balanced visual segmenters in various compression scenarios, resulting in one-sided conclusions and a lack of universality. These problems greatly limit the promotion and effectiveness of visual token compression technology in practical applications. Therefore, such as Figure 1 As shown, this invention proposes a visual token compression method based on word segmentation classification and applicability analysis, including the following steps:

[0028] S1: Establish an evaluation framework for visual word segmenters under a unified compression evaluation protocol, based on compression operators of uniform strength and consistent performance requirements;

[0029] S2: Based on the architectural features of visual word segmenters, visual word segmenters are classified into three categories: semantically biased, detail-biased, and balanced.

[0030] S3: Based on the visual information processing flow of the visual word segmenter, the compression position is divided into the input stage before the visual word segmenter processing, the intermediate stage between feature layers inside the visual word segmenter, and the final stage after the visual word segmenter output.

[0031] S4: Apply the evaluation framework to insert compression operations at all compression positions for each of the three types of visual word segmenters, and quantify the performance index value for each combination.

[0032] S5: Based on the performance index values ​​obtained from quantification, analyze the applicable patterns of various word segmenters at different compression positions;

[0033] S6: Based on applicable rules, determine the compression positions applicable to different types of visual token segmenters, and execute a visual token compression strategy optimized based on applicable rules accordingly.

[0034] First, this invention constructs a unified, training-independent compression evaluation protocol to ensure the comparability of results from different architectures of word segmenters at different compression positions. The core of this benchmark is the use of parameter-independent, plug-and-play lightweight compression operators, such as average pooling, max pooling, and stride convolution. This invention also defines compression intensity, such as downsampling factor, pooling kernel size, and compression ratio, and constructs a standardized evaluation process and a comprehensive evaluation metric system. This system includes not only metrics for measuring task performance, such as accuracy for classification tasks, and the F1 score that comprehensively measures model precision and recall, but also metrics for measuring efficiency, such as inference latency and GPU memory usage, thereby enabling a comprehensive evaluation of the overall effectiveness of the compression strategy.

[0035] After establishing a unified compression evaluation protocol, based on the core architectural features and information processing preferences of mainstream visual word segmenters, they are divided into three categories: semantically biased, detail-biased, and balanced. Specifically, semantically biased visual word segmenters are based on the Transformer architecture, with CLIP-ViT as a representative model. It obtains tokens with distinct global semantic features through patch embedding and positional encoding, resulting in a smaller number of tokens but relatively insufficient detail fidelity. Detail-biased visual word segmenters are mostly derived from generative encoders, such as VQ-VAE or VQGAN. They generate a large number of discrete tokens through convolution and quantization mechanisms, preserving rich texture, appearance, and local geometric information, but are relatively weak in high-level semantic abstraction and cross-modal alignment. Balanced visual word segmenters attempt to combine the advantages of the former two, employing a dual-branch structure. For example, a semantic branch based on Transformer is responsible for extracting global features, while a detail branch based on CNN or VQ is responsible for preserving local information. The outputs of both are then fused to form a unified representation. However, its structural complexity makes it more sensitive to input resolution and other conditions.

[0036] To further analyze the impact of compression positions, this invention divides potential compression positions into three distinct stages based on the abstraction process of visual information in the word segmenter: the input stage, the intermediate stage, and the final stage. Input stage compression occurs before image data enters the word segmenter for encoding, typically achieved by directly reducing the input image resolution or the number of image patches, thus directly reducing the amount of information to be processed at the source. Intermediate stage compression is inserted between multiple feature layers within the word segmenter, such as after a convolutional layer or Transformer layer, downsampling or merging intermediate feature maps, which disrupts the continuity of feature abstraction. Final stage compression acts after the visual token sequence output by the word segmenter, but before it is fed into the Large Language Model (LLM). It directly reduces the sequence length by clustering and merging the output tokens (e.g., using k-means) or selecting and pruning based on importance metrics such as attention weights.

[0037] Next, this invention applies the aforementioned evaluation framework to perform a systematic compression sensitivity analysis experiment. For each representative model among the three categories of word segmenters (semantic bias, detail bias, and balanced), this invention inserts predefined compression operations (such as average pooling) into all three compression stages (input, intermediate, and final), and strictly controls the compression intensity (downsampling by 2 times), thus forming a comprehensive "3×3" experimental matrix. For each combination (e.g., downsampling the input stage of the VQGAN detail bias word segmenter by 2 times), this invention runs the model on multiple standard visual-language benchmark tasks (such as image classification, visual question answering, and image caption generation), and strictly records its task performance metrics (such as accuracy and CIDEr) and efficiency metrics (such as latency and memory usage), thereby quantifying the performance degradation in each combination.

[0038] This invention, through experimental research and analysis, obtains the applicable patterns of various word segmenters at different compression stages. Among these patterns, under the output representation robustness criterion, detail-biased visual word segmenters, due to their spatial inductive bias, exhibit significant adaptability to input stage compression, retaining more useful local information even after resolution reduction, with the smallest performance degradation. Conversely, semantic-biased and balanced visual word segmenters, due to their heavy reliance on absolute positional encoding, are extremely sensitive to changes in input resolution, resulting in severe performance degradation under input stage compression. However, in the final stage compression, detail-biased visual word segmenters, lacking a strong semantic abstraction layer to compensate for information loss, almost completely collapse in performance once fine-grained features are discarded; while semantic-biased and balanced visual word segmenters, with their strong semantic abstraction and alignment capabilities, maintain relatively stable performance even after final stage compression. Most importantly, the experimental data consistently show that, regardless of the word segmenter used, inserting compression operations in the intermediate stage severely disrupts the continuity of feature modeling, leading to a precipitous drop in the performance of all models. Therefore, the intermediate stage is generally considered unsuitable for compression.

[0039] Based on the aforementioned applicable rules, this invention proposes and implements a differentiated visual token compression strategy optimized based on these rules—dual-path compression. The optimal implementation platform for this strategy is a balanced visual tokenizer, as it inherently possesses two independent branches: detail and semantic. This invention performs input-stage compression on its detail branch, for example, by reducing the image resolution input to this branch, fully utilizing its robustness to front-end compression to efficiently reduce computational load. Simultaneously, this invention performs final-stage compression on its semantic branch, for example, by performing lightweight clustering and merging of its output semantic tokens, utilizing the stability of its back-end compression to effectively reduce sequence length. This scheme not only avoids compression pitfalls in the intermediate stages but also combines the advantages of both compression paths, ultimately achieving a significant reduction in computational and storage overhead while maximizing the model's semantic understanding and generation performance on complex multimodal tasks.

[0040] In one specific embodiment, to verify the technical concept of the present invention, theoretical verification is performed using a set of specific experimental data. The model is evaluated on nine commonly used visual-language benchmarks: AI2D (Graphic Understanding), CHARTQA (Graphic Question Answering), DOCVQA (Document Visual Question Answering), INFOVQA (Infographic Question Answering), MMBENCH-EN (Multimodal Synthesis Benchmark, English Subset), OK-VQA (Common Sense-Based Question Answering), TEXTVQA (Image Text Question Answering), VIzWIz-VQA (Question Answering Based on Images Taken by Blind Users), and VQAv2 (General Visual Question Answering). Each subset contains 500 examples pre-selected by the LMMS-LAB / LMMS-EVAL-LITE suite.

[0041] like Figure 2 The results show the experimental results for the input stage of this group of experiments. Compression was achieved by reducing the resolution of the input image, and the performance changes of different word segmenters on the standard task were statistically analyzed. Red arrows indicate performance degradation compared to 1×, and cyan arrows indicate performance improvement. The results show that the detail-biased visual word segmenter exhibits a smaller performance degradation under input compression, indicating its better robustness; while the balanced visual word segmenter, due to its sensitivity to positional encoding, shows significant degradation when the input resolution decreases. This experiment reveals the differences in performance between different word segmenters under input compression conditions.

[0042] like Figure 3 The results shown are from the intermediate stage of this experiment. A compression operator was inserted inside the word segmenter to compare the accuracy changes of detail-biased and balanced visual word segmenters. PD represents Post Down, and PM represents Post Mid. The experiment found that both types of word segmenters experienced significant performance degradation in the intermediate stage, with accuracy dropping by more than 70%. This result indicates that compression in the intermediate stage disrupts the feature abstraction process and is generally unsuitable as a compression location.

[0043] like Figure 4 The results show the final stage of the experiment, where compression was performed at the output of the word segmenter, and the performance retention of different word segmenters was compared. BQ represents Before Quant, AD represents After Detail, BC represents Before Combine, and AC represents After Combine. The results show that the detail-biased visual word segmenter almost completely loses performance during final compression, while the balanced visual word segmenter maintains high accuracy due to the presence of semantic branches, demonstrating strong robustness. Therefore, the final stage is more suitable for semantic or balanced visual word segmenters, and not for detail-biased visual word segmenters.

[0044] It is precisely based on the Figures 2 to 4Based on a comprehensive analysis of the experimental results, this invention proposes a differentiated "dual-path compression" strategy: compression is performed at the input end in the detail branch to utilize its spatial inductive bias; and compression is performed at the final stage in the semantic branch to leverage its semantic robustness. This scheme combines the advantages of different word segmenters, effectively reducing computational and storage overhead while ensuring performance.

[0045] Therefore, in practical applications, the method of this invention requires no training or fine-tuning and can be embedded as a plug-and-play module into mainstream multimodal large models (such as Emu-3, Show-o2, LLaVA, etc.). Users can select appropriate compression positions and intensities according to hardware resources and task requirements to achieve a balance between computational efficiency and performance.

[0046] In summary, the visual token compression method proposed in this invention, based on segmenter classification and applicability analysis, establishes a unified and training-independent compression evaluation protocol and employs parameter-independent plug-and-play compression operators. This constructs a consistent and comparable evaluation system across different visual segmenter architectures and multi-stage compression locations, fundamentally solving the dilemma of unreproducible and difficult-to-transfer conclusions caused by inconsistent evaluation standards in existing research.

[0047] Furthermore, this invention systematically covers three representative families of visual word segmenters: semantically biased, detail-biased, and balanced. Through comparative experiments on lateral sensitivity, it reveals their essential differences in information preservation, robustness, and compression efficiency, thus overcoming the cognitive bias caused by existing research focusing only on semantically biased visual word segmenters.

[0048] Based on the above principles, the differentiated "dual-path compression" strategy proposed by the balanced visual tokenizer can effectively combine the front-end robustness of its detail branches with the terminal stability of its semantic branches. This achieves a significant reduction in computational and storage overhead while maximizing the performance of multimodal understanding and reasoning tasks. Ultimately, it forms a visual token compression solution that requires no training, is plug-and-play, and is both efficient and universal. This provides a reliable technical path for the application of multimodal large models in high-resolution input, real-time reasoning, and resource-constrained scenarios.

[0049] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0050] Furthermore, in this invention, descriptions involving terms such as "first," "second," and "a" are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0051] In this invention, unless otherwise explicitly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; it can mean a mechanical connection or an electrical connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0052] Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are feasible for those skilled in the art. If the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

Claims

1. A visual token compression method based on tokenizer classification and suitability analysis, characterized in that, The method comprises the steps of: S1: establishing a unified compression evaluation protocol for a visual tokenizer under the condition of a unified strength of compression operation operators and consistent performance index requirements; S2: classifying visual tokenizers into three categories of semantic bias type, detail bias type and balanced type according to the architecture characteristics of the visual tokenizers; S3: dividing compression positions into an input stage before processing of the visual tokenizer, an intermediate stage between feature layers in the visual tokenizer and a final stage after output of the visual tokenizer according to a processing flow of the visual tokenizer on visual information; S4: applying the evaluation framework, inserting compression operations at all compression positions for each of the three types of visual tokenizers, and quantifying performance index values in each combination; S5: analyzing the application rules of each type of visual tokenizer at different compression positions according to the quantified performance index values; S6: determining the compression positions suitable for different types of visual tokenizers based on the application rules, and executing a visual token compression strategy optimized based on the application rules.

2. The visual token compression method based on tokenizer classification and applicability analysis of claim 1, wherein, In the S2 step, the semantic bias type visual tokenizer is a visual tokenizer based on a Transformer architecture; the detail bias type visual tokenizer is a visual tokenizer based on a VQ-VAE or generative encoder architecture; and the balanced type visual tokenizer is a dual-branch structure visual tokenizer containing a semantic branch and a detail branch.

3. The visual token compression method based on tokenizer classification and applicability analysis of claim 2, wherein, In the S5 step, the application rule is that under the output representation robustness standard, the detail bias type visual tokenizer is suitable for compression at the input stage, and the semantic bias type visual tokenizer is suitable for compression at the final stage.

4. The visual token compression method based on tokenizer classification and applicability analysis of claim 3, wherein, In the S6 step, the visual token compression strategy optimized based on the application rule is specifically that a balanced type visual tokenizer is used, and compression at the input stage is performed on the detail branch of the balanced type visual tokenizer, and compression at the final stage is performed on the semantic branch.

5. The visual token compression method based on tokenizer classification and applicability analysis of claim 3, wherein, The application rule further includes that under the output representation robustness standard, the intermediate stage is not suitable for compression by the visual tokenizer.

6. The method of claim 1, wherein the method further comprises: In the S1 step, the compression operation operator is one of average pooling, maximum pooling and step convolution, and the unified strength is defined by one of downsampling multiple, pooling kernel size and compression multiple.

7. The method of claim 1, wherein the method further comprises: The compression at the input stage is realized by reducing the input image resolution or reducing the number of image blocks, and the compression at the final stage is realized by clustering and merging the output visual tokens or importance-based selection.

8. The method of claim 1, wherein the method further comprises: In the S1 step, the performance index includes at least one of task performance indexes of accuracy, recall rate and F1 score, and at least one of efficiency indexes of inference delay and memory occupation.

Images