A visual token pruning method based on semantic concept driving
By mapping visual tokens to an interpretable semantic concept space and employing a pruning method that maximizes concept coverage, the problems of subjectivity and low computational efficiency in visual token pruning in existing technologies are solved. This achieves efficient and interpretable preservation of semantic information, thereby improving the inference accuracy and performance of the model.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XIDIAN UNIV
- Filing Date
- 2026-03-17
- Publication Date
- 2026-06-26
Smart Images
Figure CN122289889A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a visual token pruning method driven by semantic concepts. Background Technology
[0002] With the continuous development of computer vision and natural language processing technologies, visual language models have become a core tool for parsing and constructing associations between images and text. These models, by combining visual and textual information, demonstrate excellent performance in typical tasks such as image question answering, image description generation, and visual reasoning. However, as image resolution increases, the number of visual tokens generated in the image also increases, leading to a significant increase in computational complexity. Therefore, developing efficient processing techniques for high-resolution visual tokens has become an urgent need in this field.
[0003] To address this issue, token pruning has emerged as a promising solution. This technique aims to remove redundant visual tokens, reducing computational load while accurately preserving semantic information crucial for model inference, thus achieving high-efficiency and high-precision model reasoning. Although researchers have conducted extensive research on token pruning, existing methods still have several significant shortcomings in practical applications: (1) Relying on human experience-based metrics, which are highly subjective; Currently, most visual token pruning methods rely on human experience-based metrics, such as attention weights, similarity scores, or gradient information. These methods usually use the above indicators to evaluate the importance of tokens, but these metrics often lack objectivity and consistency. Due to the different subjective judgments of different researchers, this method is prone to introducing bias and cannot guarantee the rationality of token selection.
[0004] (2) Insufficient interpretability; Existing token pruning methods have shortcomings in interpretability. Most methods rely on simple, human-defined evaluation metrics to complete the token sorting and selection. This not only makes it difficult for users to understand and trust the pruning results, but also fails to provide effective interpretability support for the token selection process of the model in complex task scenarios.
[0005] (3) Insufficient consideration of the multiple semantic information of visual tokens; visual tokens usually have multiple semantic attributes, which have different importance in different tasks and scenarios. However, existing token pruning methods often fail to fully capture and utilize these complex semantic information, which may result in the loss of important details and semantic content during the pruning process, affecting the model's reasoning ability and accuracy.
[0006] (4) Low computational efficiency and inability to effectively handle high-resolution input; a large number of visual tokens are generated during the processing of high-resolution images, and traditional pruning methods are difficult to process such a large set of tokens while ensuring semantic integrity. Although some current methods have adopted dimensionality reduction or selective pruning strategies, there are still problems of excessive computational burden and performance degradation under extremely high compression ratios.
[0007] In summary, existing visual language models face problems such as low computational efficiency in high-resolution image processing, token redundancy, and subjective and poor interpretability in token selection. There is an urgent need for a more efficient and interpretable token pruning method. Summary of the Invention
[0008] To address the aforementioned problems in existing technologies, this invention provides a semantic concept-driven visual token pruning method. The technical problem to be solved by this invention is achieved through the following technical solution: This invention provides a semantic concept-driven visual token pruning method, comprising: Obtain the visual language model The hidden state of the visual token output by the layer decoder; The hidden state of the visual token is encoded and projected onto a dimension of using the sparse autoencoder in the trained visual token pruning model. The interpretable semantic concept space yields sparse feature vectors; Based on the sparse feature vectors corresponding to the visual tokens, a concept-based visual token selection mechanism is used to select the first... The visual tokens of this invention are pruned to obtain the pruned result. The pruning process of this invention can fully preserve semantic information.
[0009] The beneficial effects of this invention are: This invention provides a semantic concept-driven visual token pruning method. On one hand, it utilizes a sparse autoencoder to decouple visual features into an interpretable feature space composed of independent atomic semantic concepts. This process transforms complex visual information into simple, easily interpretable semantic features and ensures the interpretability of the semantic contribution of each token in the subsequent selection process. Unlike existing methods that rely on human experience-based metrics, the proposed method removes human bias and provides a clearly structured and interpretable token pruning approach. On the other hand, by redefining the token pruning task as a problem of maximizing concept coverage, it ensures that during token pruning, not only are the most semantically valuable tokens retained, but all unique semantic concepts in the image are also covered. Unlike traditional methods based on empirical rules, the optimization process of this invention ensures more sufficient preservation of semantic information at high compression ratios, thereby improving the model's inference accuracy and performance.
[0010] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0011] Figure 1 This is a flowchart of a semantic concept-driven visual token pruning method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a semantic concept-driven visual token pruning method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the comparison of visualization results under different pruning ratios provided in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the visualization of the learned interpretable conceptual features provided in an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the visualization of the visual token selection process provided in an embodiment of the present invention. Detailed Implementation
[0012] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.
[0013] Several visual token pruning methods have been proposed in the existing technology. These methods can be roughly divided into the following categories: (1) Attention weight-based token pruning method: This type of method uses the attention weights of visual tokens calculated in the model to evaluate the importance of the tokens. Specifically, the model calculates the attention weight of each visual token, selects those tokens with higher weights, and discards those tokens with lower attention weights. Although this method can reduce the number of redundant tokens to some extent, it still has several problems: First, the attention weight does not reflect the true semantic contribution of the token, and its calculation method relies too much on the internal mechanism of the model and lacks interpretability; second, this method is easily affected by human design, is highly subjective, and cannot guarantee that the pruned tokens can comprehensively cover all the important semantic information of the image.
[0014] (2) Similarity-based token pruning method: This method identifies and removes tokens with semantic repetition or high similarity by calculating the similarity between different visual tokens. The advantage of this method is that it can effectively reduce the number of redundant tokens, but its main disadvantage is that simple similarity calculation often fails to capture the ambiguity of tokens, which may lead to the wrong deletion of some tokens that are crucial to model inference.
[0015] (3) Token pruning method based on gradient or sensitivity analysis; this method evaluates the importance of each token by analyzing the degree of influence of each token on the model output. Specifically, this type of method measures the contribution of a token by calculating gradient or sensitivity, and selects the most important tokens to retain based on this. Although this method can improve the accuracy of pruning to a certain extent, it is inefficient due to its high computational complexity, especially in the case of high-resolution images, and is difficult to meet the needs of real-time applications.
[0016] (4) Clustering or grouping-based token pruning methods: These methods first perform clustering or grouping operations on visual tokens, use the clustering results to classify the semantic categories of tokens, and then select and retain the most representative tokens from each semantic category. This method can effectively remove redundant tokens; however, in semantically complex scenarios, it is still difficult to ensure the high semantic integrity of the pruned model.
[0017] In view of this, the present invention provides a Semantically Comprehensive Token Selection (SCTS) method based on semantic concepts. This method introduces a sparse autoencoder to decouple visual features into an interpretable semantic concept space, thereby pruning visual tokens to ensure that the tokens with the greatest semantic contribution are retained. Unlike traditional methods, the SCTS method proposed in this invention optimizes the token selection process by maximizing concept coverage within the interpretable concept space, thus avoiding human bias, improving interpretability, and maintaining semantic integrity even at high compression ratios.
[0018] Please see Figure 1 and Figure 2 , Figure 1 This is a flowchart of a semantic concept-driven visual token pruning method provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of a semantic concept-driven visual token pruning method provided in an embodiment of the present invention. The semantic concept-driven visual token pruning method provided by the present invention includes: S101, Obtain the visual language model (first step) The hidden state of the visual token output by the layer decoder; Specifically, in this embodiment, as Figure 2 As shown, in the process of processing text and image data using a visual language model, in the first stage of the visual language model... A visual token pruning model is added after the layer decoder to improve the visual language model. The output of the layer decoder The visual tokens are processed to prune redundant visual tokens.
[0019] S102. The hidden state encoding of the visual token is projected onto an interpretable semantic concept space of dimension using the sparse autoencoder in the trained visual token pruning model to obtain a sparse feature vector. Specifically, in this embodiment, the trained visual token pruning model includes an encoder and a decoder, and is obtained by training an initial visual token pruning model using data of a preset category as training samples; wherein, The initial visual token pruning model is trained, including: Acquire large-scale image data as training samples to construct a training dataset; Input the training samples from the training dataset into the first... The visual token pruning model to be trained is then trained, and the training process yields the first... During the training process, the visual language model... The visual features output by the layer decoder are input into the sparse autoencoder for encoding and decoding to obtain the decoded visual features.
[0020] According to the The decoding features of the sparse autoencoder obtained during the training process and the visual language model The visual features output by the layer decoder are used to calculate the reconstruction loss, and this loss is used as the first... Loss during each training session; According to the The loss from the training process is backpropagated to update the loss from the training process. The network parameters of the sparse autoencoder in the visual token pruning model to be trained are obtained. The visual token pruning model to be trained is iterated until the number of training iterations or the degree of convergence meets the preset conditions, and the trained visual token pruning model is obtained.
[0021] Furthermore, this embodiment introduces a sparse semantic projection mechanism to map visual tokens to an interpretable atomic concept space, thereby laying a semantic foundation for subsequent token selection. Specifically, a sparse autoencoder architecture is adopted to deconstruct complex polysemous representations into independent monosemous features; the encoder will... Projecting to a dimension The super-complete potential space is then reconstructed.
[0022] No. The output of this training process includes the sparse feature vector encoded by the sparse autoencoder and the hidden state of the reconstructed visual token output by the decoder; among which, The sparse feature vector output by the encoder is represented as follows: ; in, Represents sparse feature vectors. Represents the real number field. The dimension of the real number field. Indicates the index of the visual token. Indicates the first Layer The hidden state of a visual token , Indicates the hidden dimension. This represents the activation function of the Corrected Linear Unit (ReLU). The hidden state of the reconstructed visual token output by the decoder is represented as follows: ; in, Indicates the reconstructed first... Layer The hidden state of a visual token , These represent different learnable weight matrices. , and These represent different bias vectors.
[0023] Furthermore, calculate the reconstruction loss. Represented as: ; in, Denotes the square of the L2 norm. Indicates the weighting coefficient. express Norm, This ensures the fidelity of the reconstruction. It can explicitly force sparsity by activating the fewest neurons to promote sparsity.
[0024] It should be noted that the loss function of the composite target provided in this embodiment under unsupervised conditions can ensure... Capture the atomic semantics after deconstruction.
[0025] It should be noted that during the inference process, only the encoder output of the sparse autoencoder is used to produce sparse feature vectors.
[0026] This embodiment utilizes a sparse autoencoder to decouple visual features into an interpretable feature space composed of independent atomic semantic concepts. This process transforms complex visual information into simple, easily interpretable semantic features and ensures the interpretability of the semantic contribution of each token in the subsequent selection process. Unlike existing methods that rely on metrics based on human experience, the method proposed in this embodiment can remove human bias and provide a clearly structured and interpretable token pruning method.
[0027] S103. Based on the sparse feature vectors corresponding to the visual tokens, a concept-based visual token selection mechanism is used to prune the visual tokens of the i-th layer, resulting in the pruned result. The pruning process of this invention can fully preserve semantic information.
[0028] Specifically, in this embodiment, the token pruning task is fundamentally redefined based on the deconstructed semantic foundation. Unlike existing methods that rely on simple metrics derived from human experience for token selection, this embodiment refactors the pruning task into a maximum concept coverage problem. Given a set of all visual token indices... In this case, This represents the total number of tokens, and the main goal is to identify an optimal subset. This enables it to fully cover the global semantic space. This perspective prioritizes semantic integrity, ensuring that no unique visual concepts are overlooked.
[0029] Employing a concept-based visual token selection mechanism for the first The visual tokens of each layer are pruned to obtain the pruned result, including: The sparse feature vectors corresponding to visual tokens are preprocessed to obtain concept vectors; each visual token corresponds to a concept vector to construct a concept vector set. According to the preset selection method, select concept vectors from the concept vector set and put them into the selected concept vector set; aggregate all concept vectors in the selected concept vector set to obtain the covered semantic vectors; at the same time, update the concept vector set and construct the remaining concept vectors into a candidate concept vector set. Obtain the marginal semantic gain between each concept vector in the candidate concept vector set and the covered semantic vectors; add the concept vector corresponding to the optimal marginal semantic gain to the selected concept vector set and update the selected concept vector set; aggregate all concept vectors in the updated selected concept vector set to obtain the updated covered semantic vectors; at the same time, update the candidate concept vector set. This process is iterated until the target budget is reached or the optimal marginal semantic gain is zero, resulting in the optimal covered semantic vector.
[0030] This embodiment redefines the token pruning task as a problem of maximizing concept coverage, ensuring that during the token pruning process, not only are the most semantically valuable tokens preserved, but also all unique semantic concepts in the image are covered. Unlike traditional methods that rely on empirical rules for pruning, the optimization process proposed in this embodiment ensures that semantic information is more fully preserved at high compression ratios, thereby improving the model's inference accuracy and performance.
[0031] Furthermore, the sparse feature vectors corresponding to the visual tokens are preprocessed to obtain concept vectors, including: To achieve discrete optimization, the sparse feature vector is binarized to obtain a binary concept vector, represented as: ; in, Represents binary concept vectors, i.e., visual tokens. Discrete representation in the concept space Indicates the first The first visual token The activation values corresponding to each semantic dimension , Each dimension represents an independent atomic semantic concept. Indicates the significance threshold.
[0032] The global semantic space of an image is defined as follows: ; in, It represents the union in set theory, aggregating all unique semantic indices, and represents the semantic space of the entire token set.
[0033] Although theoretically this embodiment is The process is formalized, but the implementation utilizes a sparse autoencoder, which restricts operations within the effective semantic space. This significantly reduces computational and memory overhead.
[0034] Accurately solving the maximum concept coverage problem is an NP-hard combinatorial optimization task. To ensure efficiency, this implementation employs a greedy search strategy based on marginal semantic gain to optimize the task.
[0035] Furthermore, the covered semantic vectors are represented as follows: ; in, This represents the semantic vector that has been covered. , Represents the set of selected concept vectors Index of the concept vector in the middle, This represents an element-wise logical OR operation.
[0036] Furthermore, the marginal semantic gain is expressed as: ; in, This represents the marginal semantic gain, which is the number of new concepts activated. Represents the set of candidate concept vectors The Middle A binary concept vector, , Represents a set of concept vectors. Represents a vector consisting entirely of 1s. It represents the Hadamah accumulation. express Norm, term It acts as a filter, retaining only the dimensions that have not yet been covered.
[0037] The marginal semantic gain mechanism proposed in this embodiment is used to quantify the contribution of each token to uncovered semantic concepts. Through a greedy optimization selection strategy, tokens that provide the maximum semantic gain are prioritized, reducing the selection of redundant tokens. This mechanism effectively ensures that the token set after each pruning step maximizes semantic coverage, guaranteeing that token selection is both efficient and meets semantic integrity requirements.
[0038] Furthermore, it also includes: determining the optimal concept vector by identifying the optimal marginal semantic gain; wherein the optimal concept vector is expressed as: ; in, This represents the optimal concept vector.
[0039] It should be noted that, in order to eliminate ambiguity, if multiple visual tokens provide the same marginal semantic gain, the one with the highest semantic density is selected, giving priority to the visual token with richer information content.
[0040] Furthermore, in the selection Afterwards, through and Update the subset and the cover vector. This iterative process continues until the target budget is reached. Alternatively, the marginal semantic gain can be maximized to zero, ensuring that semantic redundancy is minimized at each step while maximizing visual diversity.
[0041] In this embodiment, the method further includes: if the optimal marginal semantic gain is zero and the target budget for the visual token is not reached, the concept vector with the highest information density in the candidate concept vector set is added to the selected concept vector set to update the covered semantic vectors. This unified strategy ensures that the pruned token subset can guarantee semantic integrity and subsequently maximize information richness; wherein, information density is determined by... norm Quantify, Represents the set of candidate concept vectors The Middle A binary concept vector.
[0042] Furthermore, in identifying semantically complete sets The input sequence is further processed; specifically, the original hidden states are pruned, so that... Effectively reduce the sequence length from Compress to These representative visual tokens are then propagated through the remaining layers. This compression significantly reduces the quadratic complexity of subsequent self-attention operations.
[0043] In summary, the semantic concept-driven visual token pruning method provided by this invention has the following beneficial effects, specifically: First, it eliminates subjective bias and provides an interpretable pruning process. This invention introduces a sparse autoencoder to map visual tokens to an interpretable space composed of independent semantic concepts, thereby eliminating the subjectivity of existing methods that rely on human experience. The pruning process is based on the model's inherent semantic representation, ensuring that each retained visual token has a unique semantic contribution, making the visual token selection process both objective and interpretable.
[0044] Secondly, it comprehensively captures the multiple semantic information of tokens. Based on a concept-based token selection mechanism, this invention ensures that each selected token during pruning covers the unique semantic concepts in the image by maximizing concept coverage. Unlike existing methods, this invention effectively processes the multiple semantic information of visual tokens, ensuring that the model retains key information completely after pruning and does not lose details crucial for inference.
[0045] Third, it improves computational efficiency and reduces computational complexity. This invention significantly reduces the computational complexity of the token pruning process by optimizing the pruning strategy, balancing semantic integrity and computational efficiency. In particular, by utilizing a sparse autoencoder to map visual tokens to a more compact semantic space, it can significantly reduce redundant tokens while preserving key information, thereby improving computational efficiency. Furthermore, the marginal semantic gain mechanism ensures that the token contributing most to semantic integrity is selected under high compression ratios, ensuring that the reduction in computation does not significantly sacrifice inference performance.
[0046] In an optional embodiment of the present invention, the effectiveness of the semantic concept-driven visual token pruning method provided in the above embodiment is verified through simulation experiments, specifically as follows: Please see Figure 3 , Figure 3This is a schematic diagram comparing the visualization results under different pruning ratios provided in this embodiment of the invention. It shows the visualization comparison results between the traditional HoloV method and the method proposed in this invention in a high compression ratio scenario. Experimental results show that the original image and its cropped version have retention rates of 33.3%, 22.2%, and 11.1%, respectively. The bounding boxes mark the regions containing detailed text and objects; the method proposed in this invention can maintain the semantic integrity of this content even under strict pruning, further demonstrating its excellent stability in high compression ratio scenarios.
[0047] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating the visualization of learnable interpretable concept features provided in an embodiment of the present invention. The diagram shows the visualization of interpretable concept semantics obtained after encoding by a sparse autoencoder. Specifically, Figure 4 This indicates that dimensions 1, 674, and 2719 represent the semantic concepts corresponding to motorcycles, pizzas, and skiing, respectively, further proving that each dimension can indeed capture different basic concepts.
[0048] Please see Figure 5 , Figure 5 This is a schematic diagram illustrating the visualization of the visual token selection process provided in an embodiment of the present invention, showing the step-by-step feature selection process at a pruning rate of 77.8%. The visualization of the token selection process at a pruning rate of 77.8% is as follows: The entire selection process is divided into 6 steps (corresponding to...) Figure 5 (a) → (f)). The green box represents the token selected in the current step, while the red box represents the tokens selected in previous steps. As the process progresses, the method proposed in this invention gradually captures unique semantic concepts, while striving to achieve complete semantic preservation while minimizing redundancy. This result proves that the method proposed in this invention effectively reduces redundant information; it also shows that semantic diversity is given priority before optimizing feature accuracy.
[0049] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that an article or device comprising a list of elements includes not only those elements but also other elements not expressly listed. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or device comprising said element. Terms such as "connected" or "linked" are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect. The orientations or positional relationships indicated by terms such as "upper," "lower," "left," and "right" are based on the orientations or positional relationships shown in the accompanying drawings and are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as limiting the invention.
[0050] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.
[0051] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.
Claims
1. A visual token pruning method based on semantic concept-driven approach, characterized in that: Get the The hidden state of the layer visual token; wherein, the first layer visual token... The hidden states of layer visual features are determined by the first layer of the visual language model. Layer decoder output; The hidden states of the visual tokens are projected onto a dimension of using a sparse autoencoder in a pre-trained visual token pruning model. The interpretable semantic concept space yields sparse feature vectors; Based on the sparse feature vector corresponding to the visual token, a concept-based visual token selection mechanism is used to select the first... The visual tokens of the layer are pruned to obtain the pruned result.
2. The semantic concept-driven visual token pruning method according to claim 1, characterized in that, The trained visual token pruning model includes a visual language model and a sparse autoencoder. The sparse autoencoder comprises an encoder and a decoder. During training, the visual language model is frozen, and image data is used as training samples for unsupervised training only on the sparse autoencoder. The initial visual token pruning model is trained, including: Acquire large-scale image data as training samples to construct a training dataset; Input the training samples from the training dataset into the first... The visual token pruning model to be trained is then trained, and the training process yields the first... During the training process, the visual language model... The visual features output by the layer decoder are input into the sparse autoencoder for encoding and decoding to obtain the decoded visual features; According to the The decoding features of the sparse autoencoder obtained during the training process and the visual language model The visual features output by the layer decoder are used to calculate the reconstruction loss, and this loss is used as the first... Loss during each training session; According to the The loss from the training process is backpropagated to update the loss from the training process. The network parameters of the sparse autoencoder in the visual token pruning model to be trained are obtained. The visual token pruning model to be trained is iterated until the number of training iterations or the degree of convergence meets the preset conditions, and the trained visual token pruning model is obtained.
3. The semantic concept-driven visual token pruning method according to claim 2, characterized in that, The first The output of this training process includes the sparse feature vector encoded by the sparse autoencoder and the hidden state of the reconstructed visual token output by the decoder; among which, The sparse feature vector obtained by the sparse autoencoder is represented as follows: ； in, Represents sparse feature vectors. Represents the real number field. The dimension of the real number field. Indicates the index of the visual token. Indicates the first Layer The hidden state of a visual token This represents the modified linear unit activation function; The hidden state of the reconstructed visual token output by the decoder is represented as follows: ； in, Represents the reconstructed first... Layer The hidden state of a visual token , These represent different learnable weight matrices. , and These represent different bias vectors.
4. The semantic concept-driven visual token pruning method according to claim 3, characterized in that, The calculation of reconstruction loss Represented as: ； in, Denotes the square of the L2 norm. Indicates the weighting coefficient. express Norm.
5. The semantic concept-driven visual token pruning method according to claim 1, characterized in that, A concept-based visual token selection mechanism is used for the first... The visual tokens of each layer are pruned to obtain the pruned result, including: The sparse feature vectors corresponding to the visual tokens are preprocessed to obtain concept vectors; wherein each visual token corresponds to a concept vector to construct a concept vector set. According to a preset selection method, the concept vector is selected from the concept vector set and placed into the selected concept vector set; all the concept vectors in the selected concept vector set are aggregated to obtain the covered semantic vectors; at the same time, the concept vector set is updated, and the remaining concept vectors are constructed into a candidate concept vector set. Obtain the marginal semantic gain between each concept vector in the candidate concept vector set and the covered semantic vector; put the concept vector corresponding to the optimal marginal semantic gain into the selected concept vector set, and update the selected concept vector set; aggregate all concept vectors in the updated selected concept vector set to obtain the updated covered semantic vector; and simultaneously update the candidate concept vector set. This process is iterated until the target budget or the optimal marginal semantic gain is zero, resulting in the optimal covered semantic vector.
6. The semantic concept-driven visual token pruning method according to claim 5, characterized in that, The sparse feature vector corresponding to the visual token is preprocessed to obtain a concept vector, including: The sparse feature vector is binarized to obtain a binary concept vector, represented as follows: ； in, Represents a binary concept vector. Indicates the first The first visual token The activation values corresponding to each semantic dimension Indicates the significance threshold.
7. The semantic concept-driven visual token pruning method according to claim 6, characterized in that, The covered semantic vector is represented as follows: ； in, This represents the semantic vector that has been covered. , Represents the set of selected concept vectors The index of the concept vector in the text. This represents an element-wise logical OR operation.
8. The semantic concept-driven visual token pruning method according to claim 7, characterized in that, The marginal semantic gain is expressed as: ； in, This represents the marginal semantic gain, which is the number of new concepts activated. Represents the set of candidate concept vectors The Middle A binary concept vector, , Represents a set of concept vectors. Represents a vector consisting entirely of 1s. Represents the Hadama product. express Norm, term It acts as a filter, retaining only the dimensions that have not yet been covered.
9. The semantic concept-driven visual token pruning method according to claim 8, characterized in that, Also includes: The optimal concept vector is determined by identifying the optimal marginal semantic gain; wherein the optimal concept vector is expressed as: ； in, This represents the optimal concept vector.
10. The semantic concept-driven visual token pruning method according to claim 5, characterized in that, Also includes: If the optimal marginal semantic gain is zero and the target budget for visual tokens is not reached, the concept vector with the highest information density from the candidate concept vector set is added to the already selected concept vector set to update the covered semantic vectors. This unified strategy ensures that the pruned token subset guarantees semantic integrity and subsequently maximizes information richness; where information density is determined by… norm Quantify, Represents the set of candidate concept vectors The Middle A binary concept vector.