Grouping token-based vit model self-adaptive sparsification method and device

Abstract

The application provides a packet token-based ViT model adaptive sparsification method and device, and relates to the technical field of deep learning model acceleration. The method comprises: inputting an automatic driving perception image into a trained packet token-based ViT model, and using a token scoring and screening module and a grouping aggregation module integrated in the model to perform end-to-end feature extraction and adaptive sequence compression on the automatic driving perception image. Specifically, low-contribution tokens are screened out according to target attention weights generated by the model at a target network layer; the low-contribution tokens are a preset number of image block tokens with the lowest importance scores; and context information of the low-contribution tokens is soft aggregated into driving scene grouping tokens to output a driving scene recognition result, which includes drivable areas, vehicles, pedestrians and traffic facility categories. In this way, the integrity of feature preservation and the efficiency of computing efficiency are effectively balanced in the automatic driving perception task.

Description

Technical Field

[0001] This invention relates to the field of deep learning model acceleration technology, and in particular to an adaptive sparsity method and apparatus for Vision Transformer (ViT) models based on grouped tokens. Background Technology

[0002] The ViT model, by segmenting images into a fixed number of image patches and converting them into token sequences, has achieved significant performance in computer vision tasks such as image classification. However, the standard ViT model suffers from computational redundancy: a large number of background or low-information image tokens contribute little to the final classification result but consume substantial computational resources. In the ImageNet dataset classification task, over 40% of the image tokens belong to background regions, and the attention weights output by the standard ViT model are significantly lower than those foreground target tokens. With the rapid development of fields such as the Internet of Things (IoT) and autonomous driving, deploying high-performance ViT models on edge devices such as mobile devices, drones, or embedded chips has become an urgent need. However, edge devices are typically limited by limited storage bandwidth and stringent real-time and low-latency requirements. The large number of parameters in the standard ViT model and its computational complexity, which increases quadratically with the number of tokens, make it difficult to deploy on computationally constrained edge devices. Therefore, how to maintain the accuracy of the standard ViT model while eliminating redundant computations to adapt to the extreme load requirements of edge devices has become a key bottleneck for ViT's practical application.

[0003] Existing lightweight methods for ViT models primarily improve efficiency by reducing the number of tokens involved in computation, mainly including dynamic routing and pruning methods and token recombination methods. However, while dynamic routing and pruning methods significantly reduce computation by dynamically evaluating token importance scores in the prediction module and directly eliminating low-scoring tokens, they also lead to the permanent loss of potential contextual information in low-scoring tokens, resulting in a significant decrease in accuracy. Token recombination methods retain some information by merging or recombining tokens, but their recombination strategies rely on complex similarity calculations or attention mechanisms, increasing additional overhead during inference. None of the aforementioned existing methods effectively balance the contradiction between feature preservation and computational efficiency. Summary of the Invention

[0004] The purpose of this invention is to provide an adaptive sparsity method and apparatus for ViT models based on group tokens, which solves the problem that existing methods have failed to effectively balance the contradiction between feature preservation and computational efficiency.

[0005] To address the aforementioned technical problems, the embodiments of the present invention provide the following technical solutions:

[0006] The first aspect of this invention provides an adaptive sparsity reduction method for the ViT model based on group tokens, comprising:

[0007] Acquire autonomous driving perception images from the onboard perception system;

[0008] The autonomous driving perception images are input into a trained token-based ViT model. The token scoring and filtering module and the group aggregation module integrated into the trained ViT model are used to perform end-to-end feature extraction and adaptive sequence compression on the autonomous driving perception images. The end-to-end feature extraction and adaptive sequence compression include:

[0009] Based on the target attention weights generated by the ViT model based on grouped tokens trained in the target network layer, low contribution tokens are selected. The target attention weights represent the degree of correlation between driving scene category tokens and different spatial regions in the autonomous driving perception image. Low contribution tokens are the image patch tokens with the lowest importance scores of a preset number.

[0010] The context information of low contribution tokens is softly aggregated into driving scenario group tokens, and the driving scenario identification results are output. The driving scenario identification results include categories such as drivable area, vehicle, pedestrian and traffic facility.

[0011] Among them, the completed ViT model based on group tokens is a model obtained by end-to-end training based on an autonomous driving scenario dataset, using learnable driving scenario group tokens and a differentiable attention allocation mechanism with Gumbel-Softmax.

[0012] A second aspect of the present invention provides an adaptive sparsity reduction device for the ViT model based on group tokens, comprising:

[0013] The acquisition module is used to acquire autonomous driving perception images from the onboard perception system;

[0014] The extraction and sequence compression module is used to input autonomous driving perception images into a trained token-based ViT model. The token scoring and filtering module and the group aggregation module integrated into the trained token-based ViT model are used to perform end-to-end feature extraction and adaptive sequence compression of the autonomous driving perception images. The end-to-end feature extraction and adaptive sequence compression include: filtering out low-contribution tokens based on the target attention weights calculated by the trained token-based ViT model at the target network layer; soft-aggregating the contextual information of the low-contribution tokens into driving scene group tokens; and outputting the driving scene recognition result, which includes categories such as drivable areas, vehicles, pedestrians, and traffic facilities. The target attention weights characterize the correlation between driving scene category tokens and different spatial regions in the autonomous driving perception image. Low-contribution tokens are a preset number of image patch tokens with the lowest importance scores. The trained token-based ViT model is obtained by end-to-end training based on an autonomous driving scene dataset, using learnable driving scene group tokens and a differentiable attention allocation mechanism with Gumbel-Softmax.

[0015] Compared to existing technologies, the present invention provides an adaptive sparsity method and apparatus for a token-based ViT model, which acquires autonomous driving perception images from an in-vehicle perception system; inputs the autonomous driving perception images into a trained token-based ViT model, and utilizes the token scoring and filtering module and the grouping aggregation module integrated in the trained token-based ViT model to perform end-to-end feature extraction and adaptive sequence compression on the autonomous driving perception images; wherein, the end-to-end feature extraction and adaptive sequence compression includes: filtering out low-contribution tokens based on the target attention weights calculated by the trained token-based ViT model at the target network layer; Attention weights characterize the correlation between driving scene category tokens and different spatial regions in the autonomous driving perception image. Low-contribution tokens are a preset number of image patch tokens with the lowest importance scores. The contextual information of low-contribution tokens is soft-aggregated into driving scene group tokens to output the driving scene recognition result, which includes categories such as drivable areas, vehicles, pedestrians, and traffic facilities. The trained ViT model based on group tokens is obtained through end-to-end training using learnable driving scene group tokens and a differentiable attention allocation mechanism with Gumbel-Softmax, based on the autonomous driving scene dataset. In this way, by filtering low-contribution tokens through importance scores and then fusing the contextual information of low-contribution tokens into driving scene group tokens through differentiable soft aggregation, the global semantics in the autonomous driving perception image are preserved through soft aggregation while significantly compressing the computation sequence. This effectively balances the integrity of feature preservation and the efficiency of computation in autonomous driving perception tasks. Attached Figure Description

[0016] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, with the same or corresponding reference numerals denoteing the same or corresponding parts, wherein:

[0017] Figure 1 A flowchart illustrating the adaptive sparsity method for the ViT model based on group tokens is shown.

[0018] Figure 2 The flowchart of the group aggregation module is shown schematically;

[0019] Figure 3 A schematic diagram of the grouping aggregation module is shown.

[0020] Figure 4 A schematic diagram of the structure of the ViT model adaptive sparsification device based on group tokens is shown. Detailed Implementation

[0021] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the invention and to fully convey the scope of the invention to those skilled in the art.

[0022] It should be noted that, unless otherwise stated, the technical or scientific terms used in this invention should have the ordinary meaning as understood by one of ordinary skill in the art.

[0023] The methods described in the embodiments of the present invention will be explained in detail below.

[0024] Figure 1 A flowchart illustrating an adaptive sparsity method for a token-based ViT model in an embodiment of the present invention is shown schematically. See [link to flowchart illustration]. Figure 1 As shown, the adaptive sparsity method for the ViT model based on group tokens can include:

[0025] S101. Acquire autonomous driving perception images from the onboard perception system.

[0026] The images perceived by autonomous driving can be RGB images captured by the vehicle's forward-facing camera, after distortion correction and standardization processing.

[0027] Specifically, after acquiring autonomous driving perception images from the onboard perception system, the method also includes:

[0028] Step A1: Perform standardization processing on the autonomous driving perception images sequentially to obtain the processed autonomous driving perception images.

[0029] For example, the autonomous driving perception image can be a 224×224 resolution RGB image. First, the pixel values ​​of the autonomous driving perception image are mapped to the [0,1] interval through normalization, and then the mean of the ImageNet dataset for autonomous driving perception images is subtracted. ( The calculated difference is divided by the preset standard deviation. ( The resulting autonomous driving perception image is then processed.

[0030] Step A2: Perform a block embedding operation on the processed autonomous driving perception image, which involves sequentially dividing the processed autonomous driving perception image into blocks and converting it to a preset dimension. Each image block token operation generates an initial token sequence.

[0031] Preset dimensions can be represented as By using block embedding operations, the processed autonomous driving perception images can be divided into multiple fixed-size image blocks, and each image block can be transformed into a block using a convolutional projection layer. Dimensional One image patch token is used to generate an initial token sequence; the number of multiple fixed-size image patches is equal to... The number of image block tokens.

[0032] For example, using a 16×16 block size, the processed autonomous driving perception image is divided into 14×14=196 image blocks of size 16×16 through a block embedding operation. Each image block is transformed into a 384-dimensional image through a convolutional projection layer. A number of image block tokens are generated, forming a shape of [ The initial token sequence of [196,384], where This refers to the batch size.

[0033] Step A3: Concatenate the driving scenario category token and the learnable driving scenario group token into the initial token sequence, and add position encoding to all tokens to form the input sequence.

[0034] The input sequence includes driving scenario category tokens, Each image patch token and a learnable driving scene grouping token are used as input to a trained grouping token-based ViT model.

[0035] The number of input sequences is One driving scenario category token is used to aggregate global features and generate classification predictions. A learnable driving scenario grouping token serves as an aggregation carrier for low-contribution tokens, and feature fusion is achieved through a subsequent grouping aggregation module. Position encoding adopts a learnable parameter form, which is used for the driving scenario category token, Each image block token and driving scene group token is assigned an independent location embedding vector to ensure that the trained ViT model based on group tokens can perceive the spatial positional relationship of the tokens.

[0036] For example, a driving scenario category token and a driving scenario grouping token are introduced into the initial token sequence. Both the driving scenario category token and the driving scenario grouping token are initialized with learnable parameters of shape [1, 1, 384], randomly initialized using a truncated normal distribution, where the standard deviation of the truncated normal distribution is... The cutoff value is 0.02, and the cutoff range is... The location encoding uses learnable parameters. Independent location embedding vectors are assigned to the driving scene category token and the 196 image patch tokens, with a total dimension of [1, 197, 384]. The driving scene group tokens are initialized with separate location embedding vectors (dimension [1, 1, 384]). The driving scene category tokens, the initial token sequence, and the driving scene group tokens are concatenated sequentially, and then their corresponding location codes are added to obtain the input sequence. , Indicates batch size is A three-dimensional set of real numbers with a sequence length of 198 and a feature dimension of 384.

[0037] S102. Input the autonomous driving perception image into the trained ViT model based on group tokens, and use the token scoring and filtering module and group aggregation module integrated in the trained ViT model to perform end-to-end feature extraction and adaptive sequence compression on the autonomous driving perception image.

[0038] The end-to-end feature extraction and adaptive sequence compression include: selecting low-contribution tokens based on the target attention weights generated by the ViT model based on grouped tokens trained in the target network layer; the target attention weights characterize the degree of correlation between driving scene category tokens and different spatial regions in the autonomous driving perception image, and the low-contribution tokens are the image patch tokens with the lowest importance scores of a preset number; the contextual information of the low-contribution tokens is softly aggregated into the driving scene grouped tokens, and the driving scene recognition result is output, which includes the categories of drivable area, vehicle, pedestrian and traffic facility.

[0039] The trained ViT model based on grouping tokens is obtained through end-to-end training using a learnable driving scenario grouping token and a differentiable attention allocation mechanism with Gumbel-Softmax, based on an autonomous driving scenario dataset. The preset number is... indivual.

[0040] Specifically, before filtering out low-contribution tokens by calculating the target attention weights generated in the target network layer based on the trained grouped token-based ViT model, the method also includes:

[0041] Step B1: In the multi-head self-attention module of the target network layer, extract the attention weights of the driving scene category tokens for each image patch token.

[0042] The target network layer is the Transformer encoder of the target layer in the trained ViT model based on group tokens.

[0043] The target layer can be the 4th layer, meaning the target network layer is a 4th layer Transformer encoder. The input sequence... Feature extraction is performed sequentially through the first four Transformer encoder layers. Each Transformer encoder contains a Multi-Head Self-Attention (MHA) module, residual connections and normalization layers, a Multi-Layer Perceptron (MLP) network, and a second residual connection. Each Transformer encoder layer learns a hierarchical representation of the image step by step through the MHA mechanism and the feedforward network, providing a semantic feature foundation for subsequent token scoring.

[0044] Step B2: Average the attention weights extracted by multiple attention heads to obtain the target attention weight corresponding to each image block token.

[0045] Specifically, based on the target attention weights generated by the ViT model based on grouped tokens in the target network layer after training, low-contribution tokens are selected, including:

[0046] Step C1: Apply the target attention weights as the importance scores corresponding to each image patch token.

[0047] The first expression for importance rating for:

[0048] ;

[0049] in, For the number of attention heads, For the first The first attention focus is on the driving scene category token pair. Target attention weights for each image block token For driving scenario category tokens.

[0050] To reduce the computational complexity of hardware platforms such as Field Programmable Gate Arrays (FPGAs), a simplified scoring strategy can be adopted:

[0051] Calculate the L1 norm of the value vector of the image patch tokens using the L1 norm instead of the L2 norm:

[0052] ;

[0053] in, For the first The L1 norm of the value vector of each image patch token. For preset dimensions, For the first The value vector of the image block token in the th image block token at the th Values ​​on a preset dimension.

[0054] By fusing the target attention weights and token value vectors, we obtain the target importance score corresponding to each image patch token, and the second expression for the importance score corresponding to each image patch token. for:

[0055] ;

[0056] in, For driving scenario category tokens to the first The target attention weights for each image block token.

[0057] When determining the importance score corresponding to each image patch token, either the first expression or the second expression can be used. This simplified scoring strategy avoids square root calculations and is compatible with fixed-point quantization and pipelined architectures.

[0058] For example, in the MHA module of the Transformer encoder at layer 4, the target attention weights for the driving scene category token (i.e., position index 0) are extracted for all image patch tokens (i.e., position indices 1 to 196). For each sample in the batch, the attention matrix has a shape of [6, 198, 198] (i.e., 6 attention heads and 198 image patch tokens), and the attention weights for the driving scene category tokens for each image patch token are extracted. :

[0059] ;

[0060] in, For each sample in the batch, extract the attention weights of the driving scene category token to all 196 image patch tokens. For batch size A set of real numbers consisting of 6 attention heads and 196 image block tokens.

[0061] The importance score of each image block token is obtained by averaging the scores from the six attention heads. :

[0062] ;

[0063] in, For the first The target attention weights for each image patch token in the driving scene category token within each attention head. For batch size And a set of real numbers for 196 image block tokens.

[0064] Compared to multi-head models (such as 12-head attention), 6-head attention has lower average computational cost and a larger variance in the attention distribution, making the distinction between high-contribution tokens and low-contribution tokens more significant, which is beneficial for subsequent ranking of tokens at the bottom. Filtering of image block tokens.

[0065] Step C2: Sort the importance scores in ascending order to obtain the sorted results.

[0066] Step C3: Based on the ascending sort result, filter out the items that are at the bottom of the sort order. One image block token, used as a low-contribution token, of which... It is a preset positive integer.

[0067] Specifically, based on the ascending sort result, select the last item in the sorted order. After using image patch tokens as low-contribution tokens, the method also includes: sorting the results in ascending order, and then selecting the first image patch token. The image block token of the bit is identified as a high contribution token.

[0068] The ascending index sequence of scores is obtained through the argsort operation, which is the result of the ascending sort. The last element in the sorted sequence is... Each image block token corresponds to a low contribution token. The first in the order Each image block token corresponds to a high contribution token. ,in, For the last one in the sort The first image block token in the Each corresponds to a low-contribution token. For those listed first The first image block token in the Each corresponds to a high contribution token.

[0069] For example, an ascending sort operation is performed on the importance scores, and the last 99 image patch tokens in the sorted order are selected as low contribution tokens. The top 97 image block tokens are considered high contribution tokens. ,in, For batch size The set consists of 99 low-contribution tokens and a set of real numbers with a feature dimension of 384 for each low-contribution token. For batch size The set consists of 97 high-contribution tokens and a set of real numbers with a feature dimension of 384 for each low-contribution token.

[0070] The choice to fuse 99 image patch tokens (approximately 50.5% of the total number of image patches) is based on the following considerations: Maximizing computational efficiency: In classification tasks, background tokens typically account for 40%-60% of image content. Fusing 99 low-contribution tokens can significantly reduce the computational overhead of subsequent Transformer encoders (i.e., Transformer encoders at layers 5-12); Controllable loss: Retaining 97 high-contribution tokens (approximately 50%) can cover the key foreground regions of the image. The features of low-contribution tokens are fused into driving scene grouping tokens through a soft aggregation mechanism; Avoiding complete information loss and hardware friendliness: Both the number of low-contribution tokens (99) and the number of high-contribution tokens (97) are odd numbers, avoiding storage waste caused by power-of-2 alignment in FPGA implementation.

[0071] Figure 2 The flowchart of the group aggregation module is shown schematically. See [link / reference] Figure 2 As shown, the context information of low-contribution tokens is soft-aggregated into driving scenario group tokens, and the recognition results of driving scenarios are output, including:

[0072] Step D1: Input the low contribution token into the group aggregation module.

[0073] Figure 3 A schematic diagram of the grouping aggregation module is shown below. Figure 3 As shown, the grouping and aggregation module includes a preprocessing cross-attention layer, an assigned attention layer, a feature fusion layer, and a multilayer perceptron output layer. The grouping and aggregation module may also include a normalization preprocessing layer and a token projection layer.

[0074] Step D2: In the group aggregation module, the context information of the low contribution token is merged into the driving scenario group token through a differentiable soft aggregation operation to generate a merged group token.

[0075] Specifically, step D2 includes:

[0076] Step D21: By preprocessing the cross-attention layer, using the driving scenario group token as the query vector and the low contribution token as the key vector and value vector, cross-attention calculation is performed to generate an enhanced group token representation.

[0077] Enhanced block token representation The expression is:

[0078] ;

[0079] in, For query vector, Group tokens for driving scenarios. For low contribution tokens, For value vectors, For key vectors, This is a preprocessing cross-attention function.

[0080] The preprocessing cross-attention operation includes:

[0081] ;

[0082] in, For the output of the attention mechanism, For normalization function, Scaling factor This query is for matrix multiplication between the query vector and the transpose of the key vector.

[0083] This step D21 enables the driving scenario group token to extract global context information from the low contribution token.

[0084] Before step D21, the method further includes: performing normalization preprocessing on the driving scenario group tokens and low contribution tokens respectively to stabilize the feature distribution; and performing dimensional transformation on the normalized preprocessed driving scenario group tokens through the target feedforward network to obtain dimensional transformation tokens to adapt to the subsequent cross-attention calculation requirements.

[0085] During the dimensionality transformation of the normalized preprocessed driving scenario group tokens through the target feedforward network, the hidden layer expansion ratio is 0.5.

[0086] The expression for the token of dimension transformation is:

[0087] ;

[0088] in, For dimensional transformation tokens, For normalized preprocessed driving scene group tokens, It is a lightweight multilayer perceptron output layer. The input dimension is 1, the output dimension is 1, and to keep the number of driving scene grouped tokens after normalized preprocessing unchanged, the hidden layer dimension is 1. In practice, the dimensions of the hidden layer can be adjusted to a fixed dimension to adapt to the hardware.

[0089] Step D22: Utilize the differentiable distributed attention mechanism with Gumbel-Softmax in the distributed attention layer to generate a set of differentiable soft-distribution weights based on the enhanced group token representation and low contribution tokens.

[0090] In this model, the temperature parameter in the Gumbel-Softmax differentiable attention allocation mechanism is gradually reduced from its initial value during the training of the grouped token-based ViT model to optimize the discreteness of token allocation.

[0091] The expression for a set of differentiable soft-assigned weights is as follows:

[0092] ;

[0093] in, For a set of differentiable soft-assigned weights, For enhanced block token representation, The temperature parameter. Gumbel-Softmax is used to generate differentiable soft-assignment probabilities during training, and converts them to discrete assignments through hard assignment during testing, ensuring a balance between gradient propagation and inference efficiency. Temperature parameter The initial value is set to 1.0 during the initial training phase, and then gradually annealed to 0.5 as the training progresses.

[0094] For example, a differentiable attention mechanism with Gumbel-Softmax uses Gumbel-Softmax to implement differentiable hard assignment during training:

[0095] Calculate the original allocation score :

[0096] ;

[0097] in, This is the transpose of the normalized low-contribution token. For batch size The number of driving scenario group tokens is 1, and each low contribution token has a feature dimension of 99, which is a set of real numbers.

[0098] During training, Gumbel-Softmax is applied to generate soft assignment probabilities and temperature parameters. :

[0099] ;

[0100] ;

[0101] ;

[0102] in, For the original allocation of scores Operations with added Gumbel noise, For random noise with a standard Gumbel distribution, The soft-assigned probability weights are generated during training. For the Softmax function, The dimension for the Softmax operation.

[0103] Wherein, the Gumbel noise follows a standard Gumbel distribution:

[0104] ;

[0105] in, To conform to a uniform distribution A random variable.

[0106] During testing, a hard allocation (i.e., a pass-through estimator) is applied:

[0107] ;

[0108] ;

[0109] ;

[0110] in, The index corresponding to the maximum value in the original allocation score. Hard-assigned weights used during testing. For one-hot encoding functions, Assign weights for final use. For soft-assigned probability weights generated during training Perform the separation operation.

[0111] Because 99 image patch tokens are aggregated at once, the dimension of attention allocation (1×99) is relatively large. Gumbel-Softmax can generate soft allocation probabilities with a near-discrete distribution during training, avoiding the vanishing gradient problem. During testing, hard allocation ensures that the driving scene grouping tokens select only the most representative features from the 99 low-contribution tokens for fusion, reducing computational overhead. The differentiable attention allocation mechanism with Gumbel-Softmax achieves end-to-end trainable token allocation, avoiding information loss caused by direct pruning through soft aggregation.

[0112] Step D23: Through the feature fusion layer, the low contribution tokens are weighted and summed according to the soft allocation weights to generate a weighted fusion result, and the weighted fusion result is fused with the enhanced grouped token representation through residual connection.

[0113] Step D24: Perform a nonlinear transformation on the fused result through the output layer of a multilayer perceptron to generate a fused group token.

[0114] The expression for the fused result is:

[0115] ;

[0116] in, The result after fusion For a set of differentiable soft-assigned weights, It is the output layer of a multilayer perceptron.

[0117] The expression for merging group tokens is:

[0118] ;

[0119] in, To merge group tokens, This is a layer normalization operation.

[0120] For example, since the dimension of Assign is [ [,1,99], with After matrix multiplication, the output dimension is [ [1,384], which aggregates 99 image patch token features into a single group token. The hidden layer dimension of the MLP has 384×4=768 channels, consistent with the MLP layer of the standard Transformer editor.

[0121] Step D3: Combine the fusion group token, driving scenario category token, and high contribution token into a compressed token sequence.

[0122] The expression for the compressed token sequence is:

[0123] ;

[0124] in, The compressed token sequence, For those listed first The first high-contribution token sequence from the image block tokens. The length of the compressed token sequence is... Compared to the input sequence, it reduces There are tokens, and the length of the original input sequence is . .

[0125] For example, , For batch size The compressed token sequence contains 99 tokens, and each token has a feature dimension of 384, representing a set of real numbers. The new sequence length is 1 + 97 + 1 = 99, a reduction of 99 tokens compared to the input sequence (198 image patch tokens), achieving a compression ratio of 50%. In the compressed token sequence, the driving scene category token is located at position 0, the 97 low-contribution tokens are located at positions 1 to 97, and the fusion group token is located at position 98. The driving scene group token carries the aggregated features of the 99 low-contribution tokens, serving as a compact representation of background information for subsequent calculations.

[0126] Step D4: Input the compressed token sequence into all Transformer encoders located after the target network layer in the trained ViT model based on grouped tokens to extract features and output the final output sequence.

[0127] The compressed token sequence is then fed into the Transformer editor for layers 5-12 for feature extraction until all layer operations are complete. Because the input sequence length for the subsequent 8 layers (i.e., layers 5-12) is reduced from 198 to 99, computational overhead is decreased.

[0128] Specifically, the computational complexity of each Transformer encoder for:

[0129] ;

[0130] in, The total number of image block tokens, is the dimension of the hidden layer of the model.

[0131] Step D5: Generate the recognition result of the driving scene by passing the final output sequence through a normalization layer and a classification head.

[0132] Recognition results of driving scenarios The expression is:

[0133] ;

[0134] in, For the category tokens in the final output sequence, For batch size A set of real numbers with a total of 1000 categories.

[0135] The method of this invention achieves adaptive token sparsity, significantly improving the inference efficiency of ViT models based on grouped tokens. For example, by inserting grouping aggregation modules into the Transformer encoders at layers 4 and 8, the first fusion of 50 low-contribution tokens (i.e., the 196th to 147th low-contribution tokens) and the second fusion of 30 tokens (i.e., the 147th to 118th low-contribution tokens) can reduce floating-point operations by up to 56%.

[0136] The proposed adaptive sparsity method for ViT models based on grouped tokens introduces learnable driving scene grouped tokens into the standard ViT model. After specifying the Transformer encoder, it scores the importance of image patch tokens based on the attention weights of driving scene category tokens, and then selects the appropriate tokens. Low-contribution tokens are used to fuse their features into driving scenario grouped tokens using a grouping aggregation module, thereby significantly reducing the computational overhead of subsequent layers while preserving key information. Specifically, the grouping aggregation module employs a three-tier structure: a preprocessing cross-attention layer, an allocation attention layer, and a feature fusion layer. Differentiable token allocation is achieved through a differentiable allocation attention mechanism with Gumbel-Softmax, supporting end-to-end training. At the hardware deployment level, the token scoring mechanism can be optimized using L1 norm and fixed-point quantization, avoiding complex floating-point operations and exhibiting hardware friendliness. Simulation experiments demonstrate that this invention achieves a 31% reduction in floating-point computation, a 0.35% improvement in accuracy, and a 30% increase in inference speed in classification tasks. It is particularly suitable for high-efficiency visual inference on edge devices with limited storage bandwidth and extremely high requirements for real-time performance and low latency.

[0137] The simulation experiments of this invention were conducted on the ImageNet-1K dataset, using DeiT-Small and ViT-Base as benchmark models to verify the effectiveness of the proposed method. The experimental process first loads the pre-trained model weights and then fine-tunes the training using the AdamW optimizer and a cosine annealing learning rate strategy. During training, the focus is on configuring the multi-head self-attention grouping module and the feedforward network grouping module, setting different token retention ratios (e.g., 0.7, 0.5) to allow the ViT model based on grouped tokens to learn to adaptively recognize and discard redundant tokens such as background tokens. The experiment will compare mainstream sparsity methods such as Data-efficient Image Transformer-Small (Deit-Small), Dynamic Vision Transformer-S (Dynamic ViT-S), Efficient Vision Transformer-S (EVit-S), Zero-Tprune, and Adaptive Token Compression (ATC), and evaluate them by calculating three core metrics: Top-1 accuracy, floating-point operations (FLOPs), and actual inference throughput.

[0138] Table 1. Comparison of performance metrics for each method

[0139]

[0140] As can be seen from the performance comparison metrics of the various methods in Table 1, the method of this invention demonstrates superior computational optimization capabilities compared to DeiT-Small. While DeiT-Small achieves a top-1 accuracy of 79.8%, its computational burden is heavy (FLOPs are 4.6G). The method of this invention, while significantly reducing FLOPs by 47.8% (from 4.6G to 2.4G), achieves a substantial performance improvement at a negligible precision cost of only 0.6% (79.2%). Specifically, the image throughput of the method of this invention is improved by 65.6%, jumping from 1451 img / s to 2403 img / s. This indicates that the method of this invention, while significantly reducing redundant computation, excellently preserves the model's feature representation capabilities, making it highly valuable for practical deployment.

[0141] Compared to ATC, while ATC increases throughput by 66.6%, it comes at the cost of a 1.4% decrease in accuracy. In contrast, the method of this invention achieves a 71.2% increase in throughput while maintaining 0.8% higher accuracy than ATC. This significant difference verifies that the attention grouping mechanism employed in this invention can more accurately preserve object boundaries and semantic details, avoiding feature loss caused by brute-force aggregation.

[0142] Compared to Dynamic Vit-S and Zero-TPrune, Dynamic Vit-S employs a hard deletion strategy, resulting in a 2.0% drop in accuracy. While Zero-TPrune preserves the structure through zero-padding, its accuracy still decreases by 0.7% compared to the baseline, and its throughput (2351 img / s) is lower than that of the method described in this invention. In contrast, the method described in this invention achieves 1.4% higher accuracy than Dynamic Vit-S and 0.2% higher accuracy than Zero-TPrune, while also maintaining a leading speed. This strongly demonstrates that the soft aggregation strategy of this invention effectively salvages the critical global context information discarded by traditional pruning methods, achieving a dual breakthrough in both speed and accuracy.

[0143] Compared to EVit-S, the method of this invention achieves 0.7% higher accuracy and slightly better throughput while maintaining comparable computational resource consumption. This indicates that the three-level grouping aggregation module proposed in this invention has a more refined recombination mechanism than EVit-S. The differentiable allocation and feature fusion implemented using Gumbel-Softmax can more accurately capture and compress redundant information in low-contribution regions, thus achieving superior feature representation at the same speed and establishing its position as the best performing method among existing approaches.

[0144] This invention employs a soft aggregation mechanism for driving scenario group tokens, fusing feature information from low-contribution tokens into the driving scenario group tokens, thus avoiding information loss caused by direct pruning. By reducing the number of tokens in subsequent layers of the Transformer encoder, this invention achieves reduced FLOPs and improved inference speed. The group aggregation module uses Gumbel-Softmax and a pass-through estimator to achieve differentiable token allocation, eliminating the need for pre-trained teacher models or complex gating networks, resulting in a simple and efficient training process. The token scoring mechanism of this invention can be optimized using L1 norm and fixed-point quantization, avoiding floating-point division and square root operations.

[0145] Based on the above Figure 1As can be seen from the implementation method, the embodiments of the present invention acquire autonomous driving perception images from the vehicle perception system; input the autonomous driving perception images into a trained ViT model based on grouped tokens, and use the token scoring and filtering module and group aggregation module integrated in the trained ViT model to perform end-to-end feature extraction and adaptive sequence compression on the autonomous driving perception images; wherein, the end-to-end feature extraction and adaptive sequence compression includes: filtering out low contribution tokens according to the target attention weights generated by the trained ViT model based on grouped tokens in the target network layer; the target attention weights characterize the degree of correlation between driving scene category tokens and different spatial regions in the autonomous driving perception images, and the low contribution tokens are the image patch tokens with the lowest importance scores of a preset number; softly aggregating the context information of the low contribution tokens into driving scene grouped tokens, and outputting the recognition result of the driving scene, which includes the categories of drivable area, vehicle, pedestrian and traffic facility; wherein, the trained ViT model based on grouped tokens is a model obtained by end-to-end training based on the autonomous driving scene dataset, using learnable driving scene grouped tokens and a differentiable attention allocation mechanism with Gumbel-Softmax. In this way, low-contribution tokens are filtered out by token importance scoring, and then the context information of low-contribution tokens is fused into driving scenario group tokens by differentiable soft aggregation. This achieves significant compression of the computation sequence while preserving the global semantics in autonomous driving perception images through soft aggregation, thus effectively balancing the integrity of feature preservation and the efficiency of computation in autonomous driving perception tasks.

[0146] This invention prioritizes visual features most relevant to determining drivable areas, vehicles, pedestrians, and traffic facilities, even under conditions of severely limited onboard computing resources. This stems from the importance scoring mechanism's direct association of driving scene category tokens with image spatial regions. This allows the trained token-based ViT model to dynamically focus on areas crucial for safety decisions, such as road boundaries and obstacles, while efficiently aggregating semantically consistent background region tokens (e.g., sky, large vehicle bodies) that contribute little to classification decisions. This significantly improves the real-time inference speed of the trained token-based ViT model on in-vehicle embedded platforms, ensuring no loss of critical perception information, and directly meets the core requirements of autonomous driving systems for high-precision perception and low-latency computing.

[0147] Based on the same inventive concept, as an implementation of the above-mentioned adaptive sparsity method for ViT models based on group tokens, this embodiment of the invention also provides an adaptive sparsity device for ViT models based on group tokens. Figure 4 This is a structural diagram of the adaptive sparsity reduction device for the ViT model based on group tokens in an embodiment of the present invention. See also... Figure 4 As shown, the ViT model adaptive sparsity device based on group tokens may include:

[0148] The acquisition module 401 is used to acquire autonomous driving perception images from the on-board perception system;

[0149] The extraction and sequence compression module 402 is used to input the autonomous driving perception image into the trained ViT model based on grouped tokens, so as to perform end-to-end feature extraction and adaptive sequence compression on the autonomous driving perception image using the token scoring and filtering module and the group aggregation module integrated in the trained ViT model. The end-to-end feature extraction and adaptive sequence compression includes: filtering out low contribution tokens based on the target attention weights generated by the trained ViT model in the target network layer; softly aggregating the context information of the low contribution tokens into the driving scene grouped tokens, and outputting the recognition result of the driving scene, which includes the categories of drivable area, vehicle, pedestrian and traffic facility; the target attention weights represent the degree of correlation between the driving scene category tokens and different spatial regions in the autonomous driving perception image, and the low contribution tokens are the image patch tokens with the lowest importance scores of a preset number; the trained ViT model based on grouped tokens is a model obtained by end-to-end training based on the autonomous driving scene dataset, using learnable driving scene grouped tokens and a differentiable attention allocation mechanism with Gumbel-Softmax.

[0150] The device also includes: a standardization processing module, which is used to perform standardization processing on the autonomous driving perception images sequentially after acquiring the autonomous driving perception images from the vehicle perception system to obtain the processed autonomous driving perception images.

[0151] The block embedding module is used to perform block embedding operations on the processed autonomous driving perception images, which involves sequentially dividing the processed autonomous driving perception images into blocks and converting them to a preset dimension. One image block token operation is performed to generate an initial token sequence;

[0152] The concatenation and positional encoding module is used to concatenate driving scenario category tokens and learnable driving scenario grouping tokens from the initial token sequence, and add positional encoding to all tokens to form the input sequence, which includes driving scenario category tokens, Each image patch token and a learnable driving scene grouping token are used as input to a trained grouping token-based ViT model.

[0153] The device also includes: an extraction module for extracting the attention weights of the driving scene category tokens for each image patch token in the multi-head self-attention module of the target network layer before filtering out low-contribution tokens based on the target attention weights generated by the target network layer calculated according to the trained grouped token-based ViT model; the target network layer is the Transformer encoder of the target layer in the trained grouped token-based ViT model.

[0154] The averaging module is used to average the attention weights extracted by multiple attention heads to obtain the target attention weight corresponding to each image block token.

[0155] It should be noted that the above description of the ViT model adaptive sparsity reduction device embodiment based on group tokens is similar to the description of the ViT model adaptive sparsity reduction method embodiment based on group tokens, and has similar beneficial effects. For technical details not disclosed in the embodiments of the ViT model adaptive sparsity reduction device based on group tokens of the present invention, please refer to the description of the ViT model adaptive sparsity reduction method embodiment based on group tokens of the present invention for understanding.

[0156] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An adaptive sparsity reduction method for the ViT model based on group tokens, characterized in that, include: Acquire autonomous driving perception images from the onboard perception system; The autonomous driving perception image is input into a trained token-based ViT model. The token scoring and filtering module and the group aggregation module integrated into the trained token-based ViT model are used to perform end-to-end feature extraction and adaptive sequence compression on the autonomous driving perception image. The end-to-end feature extraction and adaptive sequence compression include: Based on the target attention weights generated by the ViT model based on grouped tokens trained in the target network layer, low contribution tokens are selected; the target attention weights characterize the degree of correlation between driving scene category tokens and different spatial regions in the autonomous driving perception image, and the low contribution tokens are a preset number of image patch tokens with the lowest importance scores. The context information of the low contribution token is softly aggregated into the driving scenario group token, and the driving scenario identification result is output. The driving scenario identification result includes the categories of drivable area, vehicle, pedestrian and traffic facility. The ViT model based on group tokens that has been trained is a model obtained by end-to-end training based on an autonomous driving scenario dataset, using learnable driving scenario group tokens and a differentiable attention allocation mechanism with Gumbel-Softmax. The step of softly aggregating the context information of the low-contribution token into a driving scenario group token and outputting the driving scenario recognition result includes: The low contribution token is input into the group aggregation module; In the grouping aggregation module, the context information of the low contribution token is fused into the driving scenario grouping token through a differentiable soft aggregation operation to generate a fused grouping token; The fusion group token, the driving scenario category token, and the high contribution token are recombined into a compressed token sequence; The compressed token sequence is input into all Transformer encoders located after the target network layer in the trained ViT model based on grouped tokens for feature extraction, so as to output the final output sequence. The final output sequence is then used to generate the recognition result of the driving scene through a normalization layer and a classification head. In the grouping aggregation module, the context information of the low-contribution token is fused into the driving scenario grouping token through a differentiable soft aggregation operation to generate a fused grouping token, including: By preprocessing the cross-attention layer, using the driving scenario group token as the query vector and the low contribution token as the key vector and value vector, cross-attention calculation is performed to generate an enhanced group token representation. Using the differentiable allocation attention mechanism with Gumbel-Softmax in the allocation attention layer, a set of differentiable soft allocation weights is generated based on the enhanced group token representation and the low contribution token. Through the feature fusion layer, the low contribution tokens are weighted and summed according to the soft allocation weights to generate a weighted fusion result, and the weighted fusion result is fused with the enhanced grouped token representation through a residual connection; The fused result is subjected to a nonlinear transformation through the output layer of a multilayer perceptron to generate the fused group token.

2. The adaptive sparsity reduction method for the ViT model based on group tokens according to claim 1, characterized in that, After acquiring autonomous driving perception images from the onboard perception system, the method further includes: The autonomous driving perception images are sequentially standardized to obtain processed autonomous driving perception images; The processed autonomous driving perception image is subjected to a block embedding operation, which involves sequentially dividing the processed autonomous driving perception image into blocks and converting it to a preset dimension. One image block token operation is performed to generate an initial token sequence; The driving scenario category token and the learnable driving scenario grouping token are concatenated into the initial token sequence, and positional encoding is added to all tokens to form an input sequence, which includes the driving scenario category token, the driving scenario grouping token, and the learningable driving scenario grouping token. The input sequence consists of image patch tokens and learnable driving scene grouping tokens, and is used as input to the trained grouping token-based ViT model.

3. The adaptive sparsity reduction method for the ViT model based on group tokens according to claim 2, characterized in that, Before filtering out low-contribution tokens by calculating the target attention weights generated in the target network layer based on the ViT model trained by the grouped tokens, the method further includes: In the multi-head self-attention module of the target network layer, the attention weight of the driving scene category token to each image patch token is extracted. The target network layer is the Transformer encoder of the target layer in the trained ViT model based on grouped tokens. The attention weights extracted by multiple attention heads are averaged to obtain the target attention weight corresponding to each image block token.

4. The adaptive sparsity reduction method for the ViT model based on group tokens according to claim 3, characterized in that, The preset quantity is The step of filtering out low-contribution tokens based on the target attention weights generated in the target network layer by the ViT model based on grouped tokens that has been trained includes: The target attention weight is used as the importance score corresponding to each image block token; The importance scores are sorted in ascending order to obtain the sorted results. Based on the ascending sort result, select the last item in the sorted order. One image block token, as the low contribution token, wherein... It is a preset positive integer.

5. The adaptive sparsity reduction method for the ViT model based on group tokens according to claim 4, characterized in that, Based on the ascending sort result, select the last item in the sorted order. After using image block tokens as the low contribution tokens, the method further includes: In the results of the ascending sort, the first... The image block token of the bit is identified as a high contribution token.

6. The adaptive sparsity reduction method for the ViT model based on group tokens according to claim 1, characterized in that, The temperature parameter value in the differentiable attention allocation mechanism with Gumbel-Softmax is gradually reduced from its initial value during the training of the grouped token-based ViT model to optimize the discreteness of token allocation.

7. A token-based adaptive sparsity reduction device for ViT models, used to implement the token-based adaptive sparsity reduction method for ViT models according to any one of claims 1-6, characterized in that, include: The acquisition module is used to acquire autonomous driving perception images from the onboard perception system; The extraction and sequence compression module is used to input the autonomous driving perception image into a trained ViT model based on grouped tokens, and to perform end-to-end feature extraction and adaptive sequence compression on the autonomous driving perception image using the token scoring and filtering module and group aggregation module integrated in the trained ViT model. The end-to-end feature extraction and adaptive sequence compression includes: filtering out low-contribution tokens based on the target attention weights calculated by the trained ViT model at the target network layer; soft-aggregating the context information of the low-contribution tokens into driving scene grouped tokens, and outputting the recognition result of the driving scene, which includes categories such as drivable area, vehicles, pedestrians, and traffic facilities; the target attention weights characterize the degree of correlation between driving scene category tokens and different spatial regions in the autonomous driving perception image, and the low-contribution tokens are a preset number of image patch tokens with the lowest importance scores; the trained ViT model based on grouped tokens is a model obtained through end-to-end training based on an autonomous driving scene dataset, using learnable driving scene grouped tokens and a differentiable attention allocation mechanism with Gumbel-Softmax.

8. The adaptive sparsity reduction device for the ViT model based on group tokens according to claim 7, characterized in that, The device further includes: The standardization processing module is used to perform standardization processing on the autonomous driving perception images obtained from the vehicle perception system in sequence to obtain the processed autonomous driving perception images. The block embedding module is used to perform block embedding operations on the processed autonomous driving perception images, thereby sequentially dividing the processed autonomous driving perception images into blocks and converting them into preset dimensions. One image block token operation is performed to generate an initial token sequence; The concatenation and position encoding module is used to concatenate the driving scenario category token and the learnable driving scenario grouping token into the initial token sequence, and add position encoding to all tokens to form an input sequence, the input sequence including the driving scenario category token, the driving scenario grouping token, and the learningable driving scenario grouping token. The input sequence consists of image patch tokens and learnable driving scene grouping tokens, and is used as input to the trained grouping token-based ViT model.

Images