ViT semantic segmentation progressive Token pruning method and system based on multi-scale Tsallis entropy and low-level visual feature guidance

Abstract

The invention discloses a ViT semantic segmentation progressive Token pruning method and system based on multi-scale Tsallis entropy and low-level visual feature guidance, and the method comprises the steps: segmenting an input image into a plurality of Patch blocks with a fixed size through Patch Embedding, and converting each Patch block into a feature vector with a fixed dimension; capturing a feature vector which is embedded and output by the Patch Embedding block by using a Transform stacking layer, and carrying out feature extraction so as to generate a Token sequence subjected to feature extraction; performing progressive Token pruning on the Token sequence between every two adjacent Transform layers, wherein the progressive Token pruning comprises first-level entropy value clustering and second-level edge guiding clustering; a Token sequence with complex semantic information is obtained, and the Token sequence continues to be propagated backwards; performing image segmentation operation on the Token sequence of the last Transform layer by using the segmentation head, and outputting a final segmented image; the system comprises a Patch Embedding block embedding module, a Transform stacking layer, a progressive Token trimming module and a segmentation head, and is characterized in that the Patch Embedding block embedding module comprises a Patch Embedding block embedding module, a Transform stacking layer, a progressive Token trimming module and a segmentation head; on the basis of ensuring the performance, the model reasoning is accelerated, and the deployment threshold is reduced.

Description

Technical Field

[0001] This invention belongs to the field of model pruning technology, and specifically relates to a progressive token pruning method and system for ViT semantic segmentation guided by multi-scale Tsallis entropy and low-level visual features. Background Art

[0002] Model pruning is a method for compressing and optimizing deep learning models. It reduces the size and computational cost of the model by removing redundant parameters or structural units, while maintaining the model's performance as much as possible.

[0003] For semantic segmentation tasks based on visual Transformers, the team led by Liu Fagui at South China University of Technology employed a lightweight prediction module to estimate the importance score of each token. However, this required attention masking to prune tokens, necessitating fine-tuning of the model during training. The team led by Mohsen Imani at the University of California innovatively designed a pruning decoder guided by a multimodal large language model (MLLM) to achieve token pruning, achieving good results. However, the introduction of a multimodal large language model makes lightweight inference difficult on complex edge devices. These token pruning methods all require model retraining or tuning, limiting their general application across different scenarios and architectures.

[0004] Many pruning methods require retraining or fine-tuning, which limits their adaptability across different architectures and scenarios, resulting in poor cross-architecture and cross-scenario adaptability. Furthermore, the primary reliance on high-level semantics in pruning neglects the crucial role of low-level image features in depicting object boundaries and segmenting small objects. High-semantic-driven pruning strategies struggle to accurately depict object boundaries and segment small targets, significantly weakening the performance of the pruned model.

[0005] The patent application with publication number CN120106168A discloses a hardware-friendly visual Transformer compression method based on quantization and token pruning techniques. It achieves hardware-friendly visual Transformer compression by combining power-of-two quantization and adaptive token pruning techniques. However, since its pruning strategy relies entirely on high-level semantics and requires fine-tuning training, and the quantization process sacrifices fine-grained accuracy, it suffers from problems such as loss of detailed features, high deployment threshold, and limited quantization accuracy.

[0006] The patent application with publication number CN118761439A discloses a network structure construction method and apparatus based on knowledge distillation. It achieves lightweight visual Transformer compression through a Token Selection Network (TSN) based on knowledge distillation. However, due to its scoring mechanism relying on the prior knowledge of the teacher model and ignoring the underlying visual features, it suffers from problems such as loss of detailed features, limited generalization and high deployment cost. Summary of the Invention

[0007] To overcome the shortcomings of the existing technologies, the present invention aims to provide a progressive token pruning method and system for ViT semantic segmentation based on multi-scale Tsallis entropy and low-level visual features. Through first-level entropy clustering and second-level edge-guided clustering, it obtains semantically complex token sequences without additional training or fine-tuning, making it a plug-and-play solution. While ensuring performance, it accelerates model inference and lowers the deployment threshold.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A progressive token pruning method for ViT semantic segmentation guided by multi-scale Tsallis entropy and low-level visual features includes the following steps:

[0010] Step 1: Use Patch Embedding to segment the input image into multiple fixed-size patch blocks, and convert each patch block into a fixed-dimensional feature vector;

[0011] Step 2: Use the Transformer stacking layer to capture the feature vector of the Patch Embedding block embedding output in Step 1, and perform feature extraction to generate a feature-extracted token sequence.

[0012] Step 3: Perform progressive token pruning on the token sequences extracted in Step 2 between every two adjacent Transformer layers, including first-level entropy clustering and second-level edge-guided clustering; to obtain token sequences with complex semantic information, and continue to propagate backward;

[0013] Step 4: Use the segmentation head to perform image segmentation on the token sequence of the last Transformer layer and output the final segmented image.

[0014] The specific method for step 1 is as follows:

[0015] The Patch Embedding segment the input image into multiple fixed-size patches. Each patch is linearly transformed using a convolution kernel of the same size and stride as the patch, converting it into a fixed-dimensional vector. Position embeddings with shapes corresponding to the number of patches and the embedding dimension are added to each patch. The embedding vectors of all patches are concatenated into a sequence to generate feature vectors that serve as input to the Transformer stacking layers.

[0016] The specific method for step 2 is as follows:

[0017] The Transformer stacked layer consists of multiple Transformer layers stacked together. The Transformer layer embeds the feature vector of the PatchEmbedding block into the output and calculates the attention weights of each position with other positions through a multi-head self-attention mechanism to generate an attention output. The attention output of the multi-head self-attention mechanism is added to the input to form a residual connection. Then, the result after adding the residual connections is normalized. The normalized output is passed through a feedforward neural network, which typically contains two fully connected layers with an activation function in between to perform an independent nonlinear transformation on the features at each position. The output of the feedforward neural network is added to the normalized input to form a residual connection again, and then normalized again to generate a token sequence with extracted features.

[0018] The specific method for step 3 is as follows:

[0019] Progressive token pruning is performed on the token sequence between every two adjacent Transformer layers. K-means clustering is performed on the multi-scale Tsallis entropy of the feature-extracted token sequence using first-level entropy clustering. Tokens discarded by first-level entropy clustering are subjected to second-level edge-guided clustering, while tokens storing edge information are retained.

[0020] The first-level entropy clustering includes: calculating the multi-scale Tsallis entropy of the feature-extracted token sequence, expressed as:

[0021]

[0022] Where, p i Let q1 represent the probability of the i-th value of the input Token, q2 be the feature significance factor and q1 be the detail sensitivity factor, and α and β be dynamically adjustable parameters used to balance the two types of features. These parameters are determined through a dynamic weight adjustment mechanism and are expressed as follows:

[0023]

[0024] Where F is the feature vector matrix of the current token, ||F||2 quantifies the feature activation intensity of the current layer to suppress excessive smoothing, and D is the feature dimension;

[0025] Construct a multi-scale Tsallis entropy weight matrix for the feature vector, as follows:

[0026]

[0027] Among them, W τ It is an entropy-oriented adjustment weight matrix that updates the weights of the tokens, represented as:

[0028]

[0029] Among them, v i The feature value of the token is used to perform K-means clustering operation with cosine similarity as the distance index on the token after updating the weight. All tokens in the cluster represented by the cluster center with low multi-scale Tsallis entropy are passed on to the next level, and all tokens in the cluster represented by the cluster center with high multi-scale Tsallis entropy are subjected to secondary edge-guided clustering.

[0030] The secondary edge-guided clustering includes: using the Sobel operator to compute edge information from the input of the Transformer layer, with the Sobel operator using convolution kernels in the horizontal (x) and vertical (y) directions.

[0031]

[0032] By convolution kernel G x and G y Convolving with image I(x,y) yields gradient components, which highlight edge regions with abrupt changes in intensity; the gradient magnitude is calculated as follows:

[0033] G(x,y)=G x (x,y) 2 +G y (x,y) 2

[0034] Edge information guides assisted token clustering. All tokens in the clusters represented by the cluster centers with high multi-scale Tsallis entropy values ​​in the first-level entropy clustering are clustered again based on edge information. This guides the selection of the tokens that can be retained from those eliminated in the first clustering. The tokens to be finally removed are selected from the second-level edge-guided clustering. The tokens with complex semantic information retained after progressive token pruning are passed to the next Transformer layer.

[0035] The specific method for step 4 is as follows:

[0036] The segmentation head receives a token sequence from the stacked Transformer layers, converts the token sequence into a low-resolution feature map, and then upsamples the low-resolution feature map to restore it to the same resolution as the original input image. Next, the convolutional layer converts the number of feature channels into the number of categories, thereby generating a predicted probability map for each category corresponding to each pixel, and finally obtaining a high-resolution segmentation map for the segmentation task.

[0037] A progressive token pruning system for ViT semantic segmentation based on multi-scale Tsallis entropy and low-level visual features includes: Patch Embedding, Transformer stacking layers, a progressive token pruning module, and a segmentation head; wherein:

[0038] The Patch Embedding is used to segment the input image into multiple fixed-size Patch blocks and convert each Patch block into a fixed-dimensional feature vector.

[0039] The Transformer stack layer is used to capture the feature vector of the Patch Embedding block embedding output and perform feature extraction to generate a feature-extracted token sequence.

[0040] The progressive token pruning module is used to perform progressive token pruning on the token sequence between every two adjacent Transformer layers, including first-level entropy clustering and second-level edge-guided clustering. First-level entropy clustering selects the core tokens that need to be retained, and second-level edge-guided clustering selects the part of the tokens that can be retained from those eliminated by the first clustering based on edge information. The final tokens to be removed are selected from the second-level edge-guided clustering. The token sequence with complex semantic information obtained after two levels of clustering is then propagated to the next layer.

[0041] The segmentation head is used to perform image segmentation on the token sequence of the last Transformer layer and output the final segmented image.

[0042] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0043] 1. This invention uses first-level entropy clustering to retain tokens with complex semantic information and continue to pass them on, thereby improving the inference efficiency of the model; this invention uses second-level edge-guided clustering to retain tokens containing important edge information, thereby achieving a balance between efficiency and performance.

[0044] 2. This invention introduces a dynamic scoring mechanism of multi-scale Tsallis entropy, namely entropy value clustering of first-level clustering, and sets feature saliency factor and detail sensitivity factor. It can strengthen key semantic information in high-probability areas and highlight fine-grained edge and texture features in low-probability areas, so as to achieve accurate evaluation of task-oriented token importance. The proposed simple unsupervised clustering method can evaluate semantic complexity, more effectively guide pruning, and significantly improve the reliability and adaptability of pruning decisions.

[0045] 3. This invention proposes a progressive pruning framework that rapidly removes redundant tokens through entropy-weighted initial clustering, and effectively preserves small targets and complex boundary information by combining it with secondary clustering guided by low-order visual features. This hierarchical pruning strategy maintains high segmentation accuracy while significantly reducing computational cost, achieving an optimal balance between efficiency and performance.

[0046] 4. This invention requires no additional training or fine-tuning; its adaptive dynamic weight adjustment mechanism makes it a plug-and-play solution, lowering the deployment threshold.

[0047] 5. Currently, many pruning methods require retraining, which incurs significant computational overhead for large language models. Furthermore, current ViT pruning methods primarily preserve high-level semantics in semantic segmentation, neglecting the crucial role of low-level image features in depicting object boundaries and segmenting small objects. This not only limits their adaptability across different architectures and scenarios but also reduces pruning efficiency. The purpose of this invention is to provide a plug-and-play pruning plugin for accelerating inference in semantic segmentation models, employing a dynamic weight adjustment mechanism to adaptively balance semantics and details, and a proposed hierarchical pruning strategy.

[0048] In summary, this invention achieves three core advantages by dynamically balancing semantic and detailed features, combining entropy clustering and edge feature preservation mechanisms: plug-and-play adaptability to multiple architectures, reduced computational overhead without training, and precise preservation of boundary details. Attached Figure Description

[0049] Figure 1 This is a model framework diagram of the present invention. DETAILED DESCRIPTION

[0050] The technical solution adopted by the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0051] This invention provides a progressive token pruning method and system for ViT semantic segmentation based on multi-scale Tsallis entropy and low-level visual features. It includes Patch Embedding, Transformer stacking layers, progressive token pruning, and a segmentation head. It can perform image segmentation on the final token sequence and output the final segmented image. By segmenting the image into small blocks and converting them into vector representations, the model can effectively process image data.

[0052] Please see Figure 1 A progressive token pruning method for ViT semantic segmentation based on multi-scale Tsallis entropy and low-level visual features includes the following steps:

[0053] Step 1: Use Patch Embedding to segment the input image into multiple fixed-size patch blocks, and convert each patch block into a fixed-dimensional feature vector;

[0054] The specific method for step 1 is as follows:

[0055] The Patch Embedding segment the input image into multiple fixed-size patches. Each patch is linearly transformed using a convolution kernel of the same size and stride as the patch, converting it into a fixed-dimensional vector. Position embeddings with shapes corresponding to the number of patches and the embedding dimension are added to each patch. The embedding vectors of all patches are concatenated into a sequence to generate feature vectors that serve as input to the Transformer stacking layers.

[0056] Step 2: Use the Transformer stacking layer to capture the feature vector of the Patch Embedding block embedding output in Step 1, and perform feature extraction to generate a feature-extracted token sequence.

[0057] The specific method for step 2 is as follows:

[0058] The Transformer stacked layer consists of multiple stacked Transformer layers. The Transformer layer embeds the feature vector of the PatchEmbedding block into the output and uses a multi-head self-attention mechanism to calculate the attention weights between each position and other positions, generating an attention output. The attention output of the multi-head self-attention mechanism is added to the input to form a residual connection. Then, the result of the sum of the residual connections is normalized. The normalized output is passed through a feedforward neural network, which typically contains two fully connected layers with an activation function in between. This network performs an independent nonlinear transformation on the features at each position. The output of the feedforward neural network is added to the normalized input to form a residual connection again, and then normalized again to generate a feature-extracted token sequence. This captures the global dependencies between elements in the sequence and enhances the expressive power of the model.

[0059] Step 3: Perform progressive token pruning on the token sequences extracted in Step 2 between every two adjacent Transformer layers, including first-level entropy clustering and second-level edge-guided clustering; to obtain token sequences with complex semantic information, and continue to propagate backward;

[0060] The specific method for step 3 is as follows:

[0061] Progressive token pruning is performed on the token sequence between every two adjacent Transformer layers. K-means clustering is performed on the multi-scale Tsallis entropy of the feature-extracted token sequence using first-level entropy clustering. Tokens discarded by first-level entropy clustering are subjected to second-level edge-guided clustering, while tokens storing edge information are retained; thus accelerating model inference.

[0062] The first-level entropy clustering includes: calculating the multi-scale Tsallis entropy of the feature-extracted token sequence, expressed as:

[0063]

[0064] Where, p i Let q1 represent the probability of the i-th value of the input Token, q2 be the feature significance factor and q1 be the detail sensitivity factor, and α and β be dynamically adjustable parameters used to balance the two types of features. These parameters are determined through a dynamic weight adjustment mechanism and are expressed as follows:

[0065]

[0066] Where F is the feature vector matrix of the current token, ||F||2 quantifies the feature activation intensity of the current layer to suppress excessive smoothing, and D is the feature dimension;

[0067] Construct a multi-scale Tsallis entropy weight matrix for the feature vector, as follows:

[0068]

[0069] Among them, W τ It is an entropy-oriented adjustment weight matrix that updates the weights of the tokens, represented as:

[0070]

[0071] Among them, v i The feature value of the token is used to perform K-means clustering operation with cosine similarity as the distance index on the token after updating the weight. All tokens in the cluster represented by the cluster center with low multi-scale Tsallis entropy are passed on to the next level, and all tokens in the cluster represented by the cluster center with high multi-scale Tsallis entropy are subjected to secondary edge-guided clustering to achieve semantic focus in the feature space.

[0072] The secondary edge-guided clustering includes: using the Sobel operator to compute edge information from the input of the Transformer layer, with the Sobel operator using convolution kernels in the horizontal (x) and vertical (y) directions.

[0073]

[0074] By convolution kernel G x and G y Convolving with image I(x,y) yields gradient components, which highlight edge regions with abrupt changes in intensity; the gradient magnitude is calculated as follows:

[0075] G(x,y)=G x (x,y) 2 +G y (x,y) 2

[0076] Edge information guides assisted token clustering. For all tokens in the cluster represented by the cluster center with high multi-scale Tsallis entropy in the first-level entropy clustering, they are clustered again based on edge information. This guides the selection of the tokens that can be retained from those eliminated in the first clustering. The tokens to be finally removed are selected from the second-level edge-guided clustering. The tokens with complex semantic information retained after progressive token pruning are passed to the next Transformer layer, thus achieving the effect of preserving edge semantic information.

[0077] By explicitly incorporating edge features into the decision-making process, ensuring the retention of tokens containing important edge information, the accuracy of segmentation is improved, especially for fine details and small objects that may be overlooked, achieving more precise boundary delineation.

[0078] Step 4: Use the segmentation head to perform image segmentation on the token sequence of the last Transformer layer and output the final segmented image.

[0079] The specific method for step 4 is as follows:

[0080] The segmentation head receives a token sequence from the stacked Transformer layers, converts the token sequence into a low-resolution feature map, and then upsamples the low-resolution feature map to restore it to the same resolution as the original input image. Next, the convolutional layer converts the number of feature channels into the number of categories, thereby generating a predicted probability map for each category corresponding to each pixel, and finally obtaining a high-resolution segmentation map for segmentation tasks, so as to be used for various applications.

[0081] A progressive token pruning system for ViT semantic segmentation based on multi-scale Tsallis entropy and low-level visual features includes: Patch Embedding, Transformer stacking layer, progressive token pruning module and segmentation head;

[0082] The Patch Embedding is used to segment the input image into multiple fixed-size Patch blocks and convert each Patch block into a fixed-dimensional feature vector.

[0083] The Transformer stack layer is used to capture the feature vector of the Patch Embedding block embedding output and perform feature extraction to generate a feature-extracted token sequence.

[0084] The progressive token pruning module is used to perform progressive token pruning on the token sequence between every two adjacent Transformer layers, including first-level entropy clustering and second-level edge-guided clustering. First-level entropy clustering selects the core tokens that need to be retained, and second-level edge-guided clustering selects the part of the tokens that can be retained from those eliminated by the first clustering based on edge information. The final tokens to be removed are selected from the second-level edge-guided clustering. The token sequence with complex semantic information obtained after two levels of clustering is then propagated to the next layer.

[0085] The segmentation head is used to perform image segmentation on the token sequence of the last Transformer layer and output the final segmented image.

[0086] Experimental Analysis

[0087] Experimental conditions

[0088] Model selection: To verify the broad applicability of this invention, the following mainstream Transformer structures were selected: ViT-H (Vision Transformer Huge), ViT-L (Vision Transformer Large), Swin-L (Swin Transformer Large), and Swin-B (Swin Transformer Base). This invention was integrated and tested on ViT-based SAM and Swin-Unet-based Swin-ViT.

[0089] Datasets: To verify the generalization ability of the method in different domain tasks, three cross-domain datasets were selected: RIO Dataset: Multi-scene small target and indoor 3D understanding field;

[0090] COCO-Tasks Dataset: Multi-class dense object detection and segmentation tasks; Massachusetts Road Dataset: Large-scale high-resolution road network segmentation.

[0091] Evaluation metrics: mIoU (mean Intersection over Union), measures semantic segmentation accuracy. GFLOPS (Giga Floating Point Operations per Second), measures computational cost. γ metric (the ratio of computational efficiency to accuracy), defined as:

[0092]

[0093] A higher γ indicates higher computational efficiency while maintaining accuracy.

[0094] Comparison method: Select the representative state-of-the-art token pruning methods in the current semantic segmentation field: CTS, DToP, SViT, and VLTP.

[0095] Experimental content

[0096] Experiment 1: Comparison with the SOTA method

[0097] The GFLOPS reduction rate, mIoU accuracy reduction rate, and γ index of the present invention and existing methods were tested on SAM-ViT-H (pruning at layer 16).

[0098] Experiment 2: Generalization testing with different model structures and datasets

[0099] The ViT-H, ViT-L, Swin-L, and Swin-B models were tested on the RIO, COCO, and Massachusetts Road datasets, respectively. The number of pruned layers was: 16 layers for ViT-H, 12 layers for ViT-L, and 2 layers for both Swin-L and Swin-B.

[0100] Experimental results

[0101] Table 1 shows the results on the SAM-ViT-H model compared to the state-of-the-art (SOTA) methods. It is evident that this invention achieves a 46.2% reduction in GFLOPS without any fine-tuning, while the mIoU accuracy decreases by only 5%, and the γ index is nearly doubled compared to existing methods, demonstrating optimal performance.

[0102] Table 1

[0103] method ΔGFLOPS% ΔmIoU% γ CTS -36.4 -24.7 1.47 DToP -33.4 -24.1 1.39 SViT -35.0 -26.9 1.30 VLTP -40.1 -8.3 4.83 LVTP (This invention) -46.2 -5.0 9.24

[0104] Generalization test results

[0105] Table 2 shows the performance of this invention on different models and datasets. Experimental results show that this invention performs particularly well on large-scale ViT models, and can also reduce computational cost by 20% on lightweight Swin-ViT models while basically maintaining accuracy.

[0106] Table 2

[0107]

[0108]

[0109] Experimental Analysis

[0110] This invention dynamically quantifies token importance through multi-scale Tsallis entropy, enabling pruning to more accurately distinguish between key semantic regions and redundant regions. Simultaneously, secondary clustering guided by low-order Sobel edge features effectively avoids the problems of small target omission and edge blurring found in traditional methods. This invention can be directly deployed as a plug-and-play module on various mainstream Transformer architectures without training or network structure adjustments, greatly reducing the threshold for practical engineering applications, and is particularly suitable for resource-constrained scenarios such as drones, autonomous driving, and mobile devices.

[0111] The γ index is significantly better than the existing technology, proving that the present invention can maximize the reduction of computational overhead while ensuring segmentation accuracy, and provides a feasible solution for the edge application of high-performance semantic segmentation models.

Claims

1. A progressive token pruning method for ViT semantic segmentation guided by multi-scale Tsallis entropy and low-level visual features, characterized in that... The following steps are involved: Step 1: Use Patch Embedding to segment the input image into multiple fixed-size patch blocks, and convert each patch block into a fixed-dimensional feature vector; Step 2: Use the Transformer stacking layer to capture the feature vector of the Patch Embedding block embedding output in Step 1, and perform feature extraction to generate a feature-extracted token sequence. Step 3: Perform progressive token pruning on the token sequences extracted in Step 2 between every two adjacent Transformer layers, including first-level entropy clustering and second-level edge-guided clustering; to obtain token sequences with complex semantic information, and continue to propagate backward; Step 4: Use the segmentation head to perform image segmentation on the token sequence of the last Transformer layer and output the final segmented image.

2. The progressive token pruning method for ViT semantic segmentation based on multi-scale Tsallis entropy and low-level visual features as described in claim 1, characterized in that, The specific method for step 1 is as follows: The Patch Embedding segment the input image into multiple fixed-size patches. Each patch is linearly transformed using a convolution kernel of the same size and stride as the patch, converting it into a fixed-dimensional vector. Position embeddings with shapes corresponding to the number of patches and the embedding dimension are added to each patch. The embedding vectors of all patches are concatenated into a sequence to generate feature vectors that serve as input to the Transformer stacking layers.

3. The progressive token pruning method for ViT semantic segmentation based on multi-scale Tsallis entropy and low-level visual features as described in claim 1, characterized in that... The specific method for step 2 is as follows: The Transformer stacked layer consists of multiple Transformer layers stacked together. The Transformer layer embeds the feature vector of the PatchEmbedding block into the output and calculates the attention weights of each position with other positions through a multi-head self-attention mechanism to generate an attention output. The attention output of the multi-head self-attention mechanism is added to the input to form a residual connection. Then, the result after adding the residual connections is normalized. The normalized output is passed through a feedforward neural network, which typically contains two fully connected layers with an activation function in between to perform an independent nonlinear transformation on the features at each position. The output of the feedforward neural network is added to the normalized input to form a residual connection again, and then normalized again to generate a token sequence with extracted features.

4. The progressive token pruning method for ViT semantic segmentation based on multi-scale Tsallis entropy and low-level visual features as described in claim 1, characterized in that, The specific method for step 3 is as follows: Progressive token pruning is performed on the token sequence between every two adjacent Transformer layers. K-means clustering is performed on the multi-scale Tsallis entropy of the feature-extracted token sequence using first-level entropy clustering. Tokens discarded by first-level entropy clustering are subjected to second-level edge-guided clustering, while tokens storing edge information are retained. The first-level entropy clustering includes: calculating the multi-scale Tsallis entropy of the feature-extracted token sequence, expressed as: Where, p i Let q1 represent the probability of the i-th value of the input Token, q2 be the feature significance factor and q1 be the detail sensitivity factor, and α and β be dynamically adjustable parameters used to balance the two types of features. These parameters are determined through a dynamic weight adjustment mechanism and are expressed as follows: β＝1-α Where F is the feature vector matrix of the current token, ||F||2 quantifies the feature activation intensity of the current layer to suppress excessive smoothing, and D is the feature dimension; Construct a multi-scale Tsallis entropy weight matrix for the feature vector, as follows: Among them, W τ It is an entropy-oriented adjustment weight matrix that updates the weights of the tokens, represented as: Among them, v i The feature value of the token is used to perform K-means clustering operation with cosine similarity as the distance index on the token after updating the weight. All tokens in the cluster represented by the cluster center with low multi-scale Tsallis entropy are passed on to the next level, and all tokens in the cluster represented by the cluster center with high multi-scale Tsallis entropy are subjected to secondary edge-guided clustering. The second-level edge-guided clustering includes: using the Sobel operator to compute edge information from the input of the Transformer layer, with the Sobel operator using convolution kernels in the horizontal (x) and vertical (y) directions. By convolution kernel G x and G y Convolving with image I(x,y) yields gradient components, which highlight edge regions with abrupt changes in intensity; the gradient magnitude is calculated as follows: G(x,y)=G x (x,y) 2 +G y (x,y) 2 Edge information guides assisted token clustering. All tokens in the clusters represented by the cluster centers with high multi-scale Tsallis entropy values ​​in the first-level entropy clustering are clustered again based on edge information. This guides the selection of the tokens that can be retained from those eliminated in the first clustering. The tokens to be finally removed are selected from the second-level edge-guided clustering. The tokens with complex semantic information retained after progressive token pruning are passed to the next Transformer layer.

5. The progressive token pruning method for ViT semantic segmentation based on multi-scale Tsallis entropy and low-level visual features as described in claim 1, characterized in that, The specific method for step 4 is as follows: The segmentation head receives a token sequence from the stacked Transformer layers, converts the token sequence into a low-resolution feature map, and then upsamples the low-resolution feature map to restore it to the same resolution as the original input image. Next, the convolutional layer converts the number of feature channels into the number of categories, thereby generating a predicted probability map for each category corresponding to each pixel, and finally obtaining a high-resolution segmentation map for the segmentation task.

6. A progressive token pruning system for ViT semantic segmentation guided by multi-scale Tsallis entropy and low-level visual features, characterized in that, include: Patch Embedding, Transformer stacking layers, progressive token pruning module, and segmentation header; among which: The Patch Embedding is used to segment the input image into multiple fixed-size Patch blocks and convert each Patch block into a fixed-dimensional feature vector. The Transformer stack layer is used to capture the feature vector of the Patch Embedding block embedding output and perform feature extraction to generate a feature-extracted token sequence. The progressive token pruning module is used to perform progressive token pruning on the token sequence between every two adjacent Transformer layers, including first-level entropy clustering and second-level edge-guided clustering. First-level entropy clustering selects the core tokens that need to be retained, and second-level edge-guided clustering selects the part of the tokens that can be retained from those eliminated by the first clustering based on edge information. The final tokens to be removed are selected from the second-level edge-guided clustering. The token sequence with complex semantic information obtained after two levels of clustering is then propagated to the next layer. The segmentation head is used to perform image segmentation on the token sequence of the last Transformer layer and output the final segmented image.

Images