An Efficient Fine-Tuning Method and System for Visual Transformer Parameters Based on Side-Rank Tuners

Abstract

This application relates to the field of autonomous driving perception technology, specifically a method and system for efficient fine-tuning of visual Transformer parameters based on side-level low-rank tuners. By inserting lightweight low-rank tuners in parallel on the sides of the frozen pre-trained backbone network, detection accuracy comparable to full-parameter fine-tuning is achieved with only a minimal number of trainable parameters. This significantly reduces GPU memory usage and storage overhead during training, making scenario-based adaptation of large models feasible on consumer-grade hardware. The frozen main path fully preserves pre-trained visual knowledge and achieves lossless transfer, while the side branches specifically learn task-specific transformation residuals. A zero-initialization strategy ensures strict alignment between the training starting point and the pre-trained state, effectively bridging the gap in cross-task feature space while fully protecting general knowledge. Multi-granularity collaborative adaptation of spatial interaction, channel transformation, and overall distribution is achieved.

Description

Technical Field

[0001] This application relates to the field of autonomous driving perception technology, specifically to a method and system for efficient fine-tuning of visual Transformer parameters based on a side-low-rank tuner. Background Technology

[0002] With the rapid development of autonomous driving technology, 3D object detection based on multi-view cameras has become a core perception technology for Level 4 autonomous driving systems. Vision Transformer (ViT), with its excellent global modeling capabilities and strong scalability, has gradually become the mainstream backbone network for autonomous driving perception systems. From ViT-Base (approximately 88 million parameters) to ViT-Large (approximately 307 million parameters) and then to ViT-Huge (approximately 637 million parameters), the increase in model size has significantly improved the accuracy of 3D object detection.

[0003] However, existing technical solutions generally employ Full Fine-Tuning (FFT) to adapt pre-trained ViT models to 3D detection tasks, i.e., updating all parameters of the backbone network. This approach consumes significant GPU memory and other storage resources, and lightweight fine-tuning methods such as LoRA and Adapter methods result in poor detection accuracy for perceptual tasks. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide an efficient method and system for fine-tuning visual Transformer parameters based on a side-low-rank tuner, so as to solve the problems in the background art.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] This application presents an efficient method for fine-tuning visual Transformer parameters based on a side-low-rank tuner, which is applied to the fine-tuning of autonomous driving perception models, including:

[0007] Acquire fine-tuned training samples and labels from vehicle sensors for a 3D perception task adapted to autonomous driving, and obtain a pre-trained Transformer model, wherein the pre-trained Transformer model is pre-trained from multi-task general 2D image data to retain general visual representation capabilities.

[0008] A low-rank tuner with zero initialization policy is inserted on the side of the target layer of the pre-trained Transformer model to obtain a side-adaptor enhanced visual Transformer model, wherein the low-rank tuner is a lightweight trainable side-path adapter module.

[0009] Freeze all parameters in the backbone network of the side adapter-enhanced visual Transformer model except for the bias term, and fine-tune the side adapter-enhanced visual Transformer model based on the fine-tuning training samples to obtain the fine-tuned visual Transformer model.

[0010] In one embodiment of this application, the mathematical expression for the low-rank tuner is:

[0011]

[0012] In the formula, Indicates input features, This represents the result of the low-rank tuner processing the input features. Representation layer normalization, For the dimension-reduced projection matrix, Indicates nonlinear activation. For the up-dimensional projection matrix, is the scaling factor, where the up-dimensional projection matrix processed by the zero-initialization strategy is... and , Indicates uniform distribution. For the input channel dimension, It is a low-rank intermediate dimension.

[0013] In one embodiment of this application, the output of the low-rank tuner is connected to the target layer of the backbone network via residuals. The target layer includes a multi-head self-attention submodule, a feedforward network submodule, and a Transformer block.

[0014] In one embodiment of this application, when the target layer is a multi-head self-attention submodule, the intermediate variable output by the multi-head self-attention submodule with a low-rank tuner inserted on the side is... The mathematical expression is:

[0015]

[0016] In the formula, Representation layer normalization, Indicates the first Output features of each Transformer block This represents the raw output of multi-head self-attention. This represents the output of the low-rank tuner in the multi-head self-attention submodule.

[0017] In one embodiment of this application, when the target layer is a feedforward network submodule, the intermediate variable output by the feedforward network submodule with a low-rank tuner inserted on the side is... The mathematical expression is:

[0018]

[0019] In the formula, Intermediate variables output by the multi-head self-attention submodule that inserts a low-rank tuner on the side. Representation layer normalization, This represents the raw output of the feedforward network. This represents the output of the low-rank tuner on the side of the feedforward network submodule.

[0020] In one embodiment of this application, when the target layer is a Transformer block, an intermediate variable output by a low-rank tuner Transformer block is inserted on the side. The mathematical expression is:

[0021]

[0022] In the formula, The intermediate variable output by the feedforward network submodule with the low-rank tuner inserted on the side. This represents the output of the low-rank tuner on the side of the Transformer block.

[0023] In one embodiment of this application, the side adapter-enhanced visual Transformer model is fine-tuned based on the fine-tuning training samples to obtain a fine-tuned visual Transformer model, including:

[0024] S1, divide all fine-tuned training samples into training set and validation set;

[0025] S2, input the fine-tuned training samples into the side adapter-enhanced visual Transformer model after freezing the backbone network to obtain the recognition result;

[0026] S3, calculate the combined loss of the recognition result and the labels of the fine-tuned training samples based on the pre-constructed loss function, wherein the loss function includes classification loss and regression loss;

[0027] S4. Based on the backpropagation of the comprehensive loss, update the bias term of the backbone network, the dimension reduction projection matrix of the low-rank tuner, the dimension increase projection matrix of the low-rank tuner, and the scaling factor of the low-rank tuner in the side adapter enhanced vision Transformer model.

[0028] S5. Repeat steps S2-S4 until the conditions for completing fine-tuning training are met.

[0029] In one embodiment of this application, the mathematical expression of the loss function is:

[0030]

[0031]

[0032]

[0033] In the formula, To account for the overall losses, The weights for the classification loss, For classifying losses, For the weights of the regression loss, To regress the loss, As a balance factor, This represents the probability that the model predicts the true class. To focus parameters, For the parameter dimension index of the prediction box, The number of parameter dimensions of the predicted bounding box. The first prediction box Predicted values ​​of parameters in each dimension. The first prediction box The true values ​​of parameters in each dimension.

[0034] In one embodiment of this application, the condition for completing fine-tuning training is:

[0035] The maximum number of training rounds is reached, or the performance metrics of the validation set do not show an improvement exceeding the preset threshold for five consecutive rounds.

[0036] This application also provides a system for efficient fine-tuning of visual Transformer parameters based on a side-low-rank tuner, including:

[0037] The acquisition module is used to acquire fine-tuned training samples and labels from vehicle sensors for 3D perception tasks adapted to autonomous driving, and to acquire a pre-trained Transformer model, wherein the pre-trained Transformer model is pre-trained from multi-task general 2D image data to retain general visual representation capabilities.

[0038] A side adapter enhancement module is used to insert a low-rank tuner with zero initialization policy processing on the side of the target layer of the pre-trained Transformer model to obtain a side adapter enhanced visual Transformer model, wherein the low-rank tuner is a lightweight trainable side-channel adapter module.

[0039] The fine-tuning module is used to freeze all parameters in the backbone network of the side adapter-enhanced visual Transformer model except for the bias term, and to fine-tune the side adapter-enhanced visual Transformer model based on the fine-tuning training samples to obtain the fine-tuned visual Transformer model.

[0040] The beneficial effects of this application are as follows: The efficient fine-tuning method and system for visual Transformer parameters based on side-level low-rank tuners achieves detection accuracy comparable to full-parameter fine-tuning with only a minimal number of trainable parameters by inserting lightweight low-rank tuners in parallel on the sides of the frozen pre-trained backbone network. This significantly reduces GPU memory usage and storage overhead during training, making scenario-based adaptation of large models feasible on consumer-grade hardware. The core mechanism of this method lies in its side-level residual connection design: the frozen main path fully preserves pre-trained visual knowledge and achieves lossless transfer, while the side branches specifically learn task-specific transformation residuals. A zero-initialization strategy ensures strict alignment between the training starting point and the pre-trained state, effectively bridging the gap in cross-task feature space while fully protecting general knowledge. By inserting tuners in parallel at three levels—the feedforward network, the attention module, and the overall block output—and allowing them to learn independently, this method achieves multi-granularity collaborative adaptation to spatial interactions, channel transformations, and overall distribution. Furthermore, this method follows the standard visual Transformer architecture, is compatible with a variety of mainstream 3D detectors, and remains effective on backbone networks of different sizes, exhibiting excellent plug-and-play characteristics and scalability. Attached Figure Description

[0041] The present application will be further described below with reference to the accompanying drawings and embodiments:

[0042] Figure 1 This is a flowchart illustrating an efficient method for fine-tuning visual Transformer parameters based on a side-low-rank tuner, as shown in one embodiment of this application.

[0043] Figure 2 This is a structural diagram of a low-rank tuner in one embodiment of this application;

[0044] Figure 3 This is an overall model structure diagram of a specific embodiment of this application;

[0045] Figure 4 This is a logic structure diagram of a visual Transformer parameter high-efficiency fine-tuning system based on a side low-rank tuner, as shown in one embodiment of this application. Detailed Implementation

[0046] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0047] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the layers related to this application and are not drawn according to the actual number, shape and size of the layers in the actual implementation. In the actual implementation, the form, number and proportion of each layer can be arbitrarily changed, and the layer layout may also be more complex.

[0048] Numerous details are explored in the following description to provide a more thorough explanation of embodiments of this application; however, it will be apparent to those skilled in the art that embodiments of this application may be practiced without these specific details.

[0049] Existing technical solutions generally employ Full Fine-Tuning (FFT) to adapt pre-trained ViT models to 3D detection tasks, i.e., updating all parameters of the backbone network. This approach has the following technical drawbacks:

[0050] (1) GPU memory consumption is huge. Each training sample for autonomous driving 3D detection contains multi-view images from 6 to 8 cameras, and usually uses high-resolution input (640×1600 and above) and fuses multi-frame temporal information. Under these conditions, updating all ViT parameters will quickly exhaust GPU memory, causing large models at the ViT-Huge level to experience memory overflow (Out of Memory) on A100 GPUs with 80GB of memory, making it impossible to train on conventional hardware.

[0051] (2) High storage costs. Actual autonomous driving perception systems need to be adapted to different operating cities, road environments, sensor configurations and deployment scenarios. If a complete copy of the backbone network (637 million parameters) needs to be stored for each adaptation, the storage overhead will increase dramatically, seriously affecting the mass production deployment and iteration efficiency of the system.

[0052] (3) Existing efficient parameter fine-tuning methods have poor portability. Various efficient parameter fine-tuning methods (such as Adapter, Prefix Tuning, LoRA, etc.) have been developed in the fields of natural language processing and 2D vision, but these methods are mainly aimed at downstream tasks with representation spaces similar to those of the pre-training task. However, there is a huge knowledge gap between 2D image pre-training and 3D object detection. Directly porting these methods will lead to a significant performance drop (NDS decrease of 5% to 10% or more), which cannot meet the high requirements of autonomous driving for perception accuracy.

[0053] (4) Limitations of the LoRA method: LoRA achieves efficient parameter fine-tuning by injecting a trainable low-rank decomposition matrix into the Transformer weight matrix. The core idea is to decompose the weight update into the product of two low-rank matrices (Delta W = B*A). However, LoRA has the following limitations: First, LoRA usually only operates on the Query and Value projection matrices of the Attention module and does not cover the FFN module, which limits the feature transformation and expression capabilities; Second, LoRA adopts a sequential insertion design (W_prime = W + B*A), which modifies the main path weight representation and may destroy the pre-trained knowledge; Third, on the nuScenes multi-view 3D detection task, LoRA (13.8M parameters) only achieves 50.4% NDS, which is 5.2 percentage points lower than full parameter fine-tuning (55.6% NDS), and cannot meet the high-precision perception requirements of autonomous driving.

[0054] (5) Limitations of the Adapter method: The Adapter achieves task adaptation by inserting a lightweight bottleneck layer between Transformer layers. Although the number of parameters is small (10.2M), the performance degrades significantly in multi-view 3D detection, with NDS only reaching 45.2%, a decrease of 10.4 percentage points compared to full parameter fine-tuning. The bottleneck layer structure of the Adapter severely limits the channel capacity of feature transformation, resulting in insufficient mapping of the 2D to 3D feature space. In addition, the sequential insertion method of the Adapter interrupts the original feature flow, creating an information bottleneck in deep networks.

[0055] (6) Limitations of the Prefix Tuning method: Prefix Tuning achieves task adaptation by adding learnable prefix tokens to the input sequence, and the number of trainable parameters is extremely small (1.5M). However, Prefix Tuning performs the worst in multi-view 3D detection, with an NDS of only 46.1%, a decrease of 9.5 percentage points. Prefix Tuning only adapts at the input layer and cannot effectively transform intermediate layer features, making it difficult to handle complex geometric mapping relationships from 2D to 3D.

[0056] (7) Limitations of the BitFit method: BitFit only fine-tunes the model bias terms, with a very small number of parameters (less than 0.1% of the total parameters), which is comparable to full parameter fine-tuning in small dataset scenarios in NLP. However, BitFit still has a significant performance gap in multi-view 3D detection. Adjusting only the bias terms cannot achieve a full conversion from 2D image features to 3D BEV features because the bias terms can only provide translation of feature activations and cannot learn complex cross-dimensional feature mappings. This invention compensates for the lack of expressive power of BitFit by using bias fine-tuning as an auxiliary component in conjunction with SLRT.

[0057] (8) Common shortcomings of existing PEFT methods: In summary, existing PEFT methods have three common shortcomings: First, they do not consider the knowledge gap between 2D image pre-training and 3D object detection—these methods are designed based on the assumption that the representation spaces of the downstream task and the pre-training task are similar, while 3D detection requires inferring depth, scale, and motion from 2D images, resulting in huge differences in representation spaces; Second, the sequential insertion design interrupts the frozen main path, which may destroy the pre-training knowledge; Third, they do not design a dedicated feature conversion mechanism for BEV perception tasks. The side-rank tuner proposed in this invention systematically solves the above problems through side residual connection design, multi-level parallel insertion, and zero initialization strategy.

[0058] Therefore, there is an urgent need for a parameter fine-tuning method specifically designed for multi-view 3D object detection tasks, which can significantly reduce the number of trainable parameters and GPU memory usage while maintaining detection accuracy comparable to full parameter fine-tuning, and effectively bridge the feature space differences between 2D pre-training and 3D detection.

[0059] To address the aforementioned shortcomings of existing technologies, the present invention aims to provide an efficient method for fine-tuning visual Transformer parameters based on a side-rank tuner. This method achieves its technical objective through the following design:

[0060] First, by freezing more than 90% of the pre-trained parameters of the ViT backbone network and only introducing lightweight side low-rank tuners (SLRT) and bias fine-tuning, the number of trainable parameters is reduced from 637 million to 63.8 million (about 10%), thereby significantly reducing the GPU memory usage during the training phase. This allows large models at the ViT-Huge level to be trained on consumer-grade GPUs with 24GB of VRAM.

[0061] Second, through the lateral residual connection design of SLRT, while keeping the pre-trained visual path completely frozen, it learns the task-specific transformation from the two-dimensional image feature space to the three-dimensional bird's eye view (BEV) feature space, effectively bridging the knowledge gap between two-dimensional pre-training and three-dimensional detection.

[0062] Third, by inserting SLRT in parallel at three levels of the Transformer—Attention, Feed-Forward Network (FFN), and Block—cooperative adaptation of multi-granularity feature representations is achieved, ensuring that the transformed features can meet the requirements of the sparse-aware Decoder for high-quality multi-scale image features.

[0063] Figure 1 This is a flowchart of an efficient fine-tuning method for visual Transformer parameters based on a side-low-rank tuner, as described in one embodiment of this application. Figure 1 As shown, the efficient fine-tuning method for visual Transformer parameters based on a side-low-rank tuner in this application mainly includes the following steps:

[0064] S110, acquire fine-tuned training samples and labels from vehicle sensors for a 3D perception task adapted to autonomous driving, and acquire a pre-trained Transformer model, wherein the pre-trained Transformer model is pre-trained from multi-task general two-dimensional image data to retain general visual representation capabilities.

[0065] In this context, the fine-tuning training samples and labels for the 3D perception task of autonomous driving come from vehicle-mounted cameras and vehicle-mounted radar. The image data collected by the cameras is used as training samples, and the ranging data collected by the vehicle-mounted radar is used as labels.

[0066] S120, a low-rank tuner with zero initialization policy is inserted on the side of the target layer of the pre-trained Transformer model to obtain a side adapter enhanced visual Transformer model, wherein the low-rank tuner is a lightweight trainable side-channel adapter module.

[0067] The parameter-efficient fine-tuning method proposed in this application—BEV Efficient Fine-Tuning (BEFT)—is applied to a multi-view 3D target detection system based on the Sparse4D architecture and includes the following core components:

[0068] (1) Pre-trained ViT backbone network. The VisionTransformer (ViT-B / L / H) pre-trained with large-scale two-dimensional image data is used as the feature extractor, and all its weight parameters are frozen in the fine-tuning stage of the downstream three-dimensional detection task.

[0069] (2) Side Low-Rank Tuner (SLRT). In each Transformer block of ViT, SLRT side branches are inserted in parallel with the frozen main path at three positions: the Attention module, the FFN module, and the overall Block output, to learn the feature transformation residuals from two dimensions to three dimensions.

[0070] (3) Bias Tuning. While freezing all weight matrices, only the bias terms of each layer are fine-tuned, and lightweight adaptation is achieved by shifting the feature activation.

[0071] (4) Sparse4D Decoder. Maintains a fully trainable state, receives multi-scale image features adapted by BEFT, and performs sparse four-dimensional object detection.

[0072] The SLRT is the core innovative component of this invention. To facilitate understanding of the technical essence of this invention, its mathematical basis will first be analyzed from the perspective of first principles.

[0073] The mathematical principle of low-rank adaptation:

[0074] In standard full-parameter fine-tuning, for the pre-trained weight matrix FFT learns an incremental update matrix Forward propagation is represented as:

[0075]

[0076] in All All parameters are trainable. For ViT-Huge, a single-layer FFN... That is, the number reaches tens of millions.

[0077] The core insight of low-rank adaptation lies in the fact that the parameter updates of pre-trained models in downstream tasks exhibit an inherent low-rank property—that is... It can be decomposed into the product of two low-rank matrices:

[0078]

[0079] At this point, the number of trainable parameters is from Down to Compression ratio is .when , At that time, the compression ratio is approximately .

[0080] Unlike existing methods such as LoRA, the SLRT proposed in this invention adopts a side residual connection design. Figure 2 This is a structural diagram of a low-rank tuner in one embodiment of this application, as shown below. Figure 2 As shown, for input features (in For the number of tokens, (For channel dimensions), the mathematical form of SLRT is:

[0081]

[0082] in:

[0083] This is a down-projection matrix. ;

[0084] For up-projection;

[0085] It is a learnable scaling factor.

[0086] Layer Normalization;

[0087] It is a non-linear activation function.

[0088] SLRT adds the residual connection to the frozen main path output:

[0089]

[0090]

[0091] Zero initialization strategy:

[0092] Dimensionality reduction matrix Kaiming uniform initialization is used:

[0093]

[0094] Always Indicates uniform distribution. For the input channel dimension, It is a low-rank intermediate dimension.

[0095] Upgraded Dimension Matrix Zero initialization:

[0096]

[0097] The mathematical significance of this design lies in: the initial training phase This means that the side-channel output is zero, and the model starts to optimize from the original pre-trained state, avoiding the destruction of the learned pre-trained representation in the early stage of training.

[0098] Trainable parameter count analysis: For ViT-Huge ( (32 layers in total) The number of trainable parameters for SLRT is:

[0099]

[0100] With the addition of bias fine-tuning (approximately 0.8M) and the SPC module (approximately 0.6M), the total number of trainable parameters is approximately 64M, which is only about 10% of the total number of parameters in ViT-Huge (637M).

[0101] Specifically, this application further proposes to insert SLRT in parallel at three different functional levels within the Transformer block:

[0102] (1) Attention-level SLRT. Inserted on the side of the multi-head self-attention module to adjust the spatial interaction relationship between image tokens:

[0103]

[0104] In the formula, Representation layer normalization, Indicates the first Output features of each Transformer block This represents the raw output of multi-head self-attention. This represents the output of the low-rank tuner in the multi-head self-attention submodule.

[0105] (2) FFN-level SLRT. Inserted on the side of the feedforward network module to adjust the feature transformation of the channel dimension:

[0106]

[0107] In the formula, Intermediate variables output by the multi-head self-attention submodule that inserts a low-rank tuner on the side. Representation layer normalization, This represents the raw output of the feedforward network. This represents the output of the low-rank tuner on the side of the feedforward network submodule.

[0108] (3) Block-level SLRT. Inserted on the side of the output end of the entire Transformer block:

[0109]

[0110] In the formula, The intermediate variable output by the feedforward network submodule with the low-rank tuner inserted on the side. This represents the output of the low-rank tuner on the side of the Transformer block.

[0111] The three levels of SLRT parameters are learned independently, capturing 2D to 3D transformation information at different levels of abstraction.

[0112] S130, freeze all parameters in the backbone network of the side adapter-enhanced visual Transformer model except for the bias term, and fine-tune the side adapter-enhanced visual Transformer model based on the fine-tuning training samples to obtain the fine-tuned visual Transformer model.

[0113] Building upon SLRT, this application further introduces a bias fine-tuning mechanism. The number of parameters in each layer related to the bias term is typically less than 0.1% of the total number of parameters, and its update method is as follows:

[0114]

[0115] That is, weight Keep frozen, bias only Participate in gradient updates.

[0116] The training strategy of this application is as follows:

[0117] (1) Parameter freezing strategy. All parameters in the ViT backbone network except for the bias term are frozen and do not participate in gradient update.

[0118] (2) Trainable parameters. Only the projection matrix of SLRT ( , scaling factor And the bias terms of each layer are trainable parameters, so the Sparse4D Decoder remains fully trainable.

[0119] (3) Optimizer configuration. The AdamW optimizer is used, with a base learning rate of... Cosine annealing learning rate scheduling strategy.

[0120] The training process based on the above strategy is as follows:

[0121] S1, divide all fine-tuned training samples into training set and validation set;

[0122] S2, input the fine-tuned training samples into the side adapter-enhanced visual Transformer model after freezing the backbone network to obtain the recognition result;

[0123] S3, calculate the combined loss of the recognition result and the labels of the fine-tuned training samples based on the pre-constructed loss function, wherein the loss function includes classification loss and regression loss;

[0124] The mathematical expression for the loss function is:

[0125]

[0126]

[0127]

[0128] In the formula, To account for the overall losses, The weights for the classification loss, For classifying losses, For the weights of the regression loss, To regress the loss, As a balance factor, This represents the probability that the model predicts the true class. To focus parameters, For the parameter dimension index of the prediction box, The number of parameter dimensions of the predicted bounding box. The first prediction box Predicted values ​​of parameters in each dimension. The first prediction box The true values ​​of parameters in each dimension.

[0129] S4. Based on the backpropagation of the comprehensive loss, update the bias term of the backbone network, the dimension reduction projection matrix of the low-rank tuner, the dimension increase projection matrix of the low-rank tuner, and the scaling factor of the low-rank tuner in the side adapter enhanced vision Transformer model.

[0130] S5, Repeat steps S2-S4 until the conditions for completing fine-tuning training are met. The conditions for completing fine-tuning training are:

[0131] The maximum number of training rounds is reached, or the performance metrics of the validation set do not show an improvement exceeding the preset threshold for five consecutive rounds.

[0132] Figure 3 This is an overall model structure diagram of a specific embodiment of this application. Please refer to it. Figure 3 The specific embodiments described below are for understanding purposes, and include:

[0133] This embodiment uses ViT-Huge (637 million parameters) as the backbone network, Sparse4Dv3 as the detector, and the nuScenes dataset as the benchmark. The specific implementation steps are as follows:

[0134] 1. Multi-view data input and spatial prior calibration;

[0135] First, images from six cameras (front-view-2, side-view-2, rear-view-1, and blind spot-1-3) from the vehicle surround view system are loaded to form a multi-view input tensor. To adapt to the high-resolution input of downstream 3D detection tasks, the original ViT position embedding is replaced with a Spatial Prior Calibration (SPC) module. Specifically, the pre-trained position embedding is bicubic interpolated according to the current input resolution, and the weights of the interpolated position embedding are frozen. Only the 1x1 Conv1D channel calibrator inside the SPC is trained. This calibrator performs channel-dimensional response calibration on the frozen interpolated position embedding to eliminate the position domain shift caused by resolution changes.

[0136] 2. Construction and parameter freezing of the BEFT-enhanced ViT backbone;

[0137] A backbone network consisting of stacked L layers (L=32 in ViT-Huge) of Transformer blocks is constructed. After loading ViT-Huge weights pre-trained on large-scale 2D image data (such as ImageNet-21K), all weight matrices in the backbone network except for the bias term (including the Q / K / V projection matrices in MSA and the fully connected layers in FFN) are frozen. This step ensures that more than 90% of the parameters (approximately 637 million parameters) remain constant during subsequent training, with only the bias term set to a trainable state.

[0138] 3. Three-stage parallel insertion of side-low rank tuners (SLRTs);

[0139] Within each frozen Transformer block, SLRT modules are injected in parallel according to a three-level insertion strategy, forming a side adapter-enhanced structure.

[0140] Attention layer: SLRT_attn is inserted on the side of the multi-head self-attention (MSA) submodule. Input features x flow in parallel into the frozen main path (LN->MSA) and the side path (SLRT_attn), and the outputs are summed by residuals. It is used to adjust the spatial interaction relationship between image tokens.

[0141] FFN Level: Insert SLRT_ffn on the side of the feedforward network (FFN) submodule. Continuing from the intermediate variable x' in the previous step, the output formula is: , used to adjust the feature transformation of the channel dimension.

[0142] Block level: Insert SLRT_blocks on the side of the output of the entire Transformer block, i.e. This is used to align the entire block output distribution with 3D detection requirements.

[0143] 4. The internal structure and initialization strategy of the SLRT module;

[0144] Each SLRT module strictly follows a lightweight bottleneck structure of "dimensionality reduction-activation-dimensionality increase". Let the input features be... (N is the number of tokens, C is the number of channels, and C=1280 in ViT-H):

[0145] Dimensionality reduction projection (W_down): Compresses features from C dimensions to a low-rank intermediate dimension R=256. Matrix Kaiming uniform initialization is used to ensure that the signal variance is stable before ReLU activation.

[0146] Nonlinear activation: The ReLU activation function is applied to introduce nonlinear mapping capabilities.

[0147] Upscaling projection (W_up): Restores features from R dimensions to C dimensions. (Matrix) Forced zero initialization ensures that the side-channel outputs are zero during the initial training phase, allowing the model to start from a perfect pre-trained state.

[0148] Learnable scaling factor (S): Introduce a trainable scalar S (initial value set to 15), which is added to the frozen main path through residual connections to balance the contribution ratio of frozen features and task-specific modified features.

[0149] 5. Multi-scale feature pyramid fusion;

[0150] After processing by the L-layer BEFT-enhanced Transformer block, the output multi-level token features are reshaped into a two-dimensional spatial feature map. Multi-scale fusion is then performed through the FPN Neck (Feature Pyramid Network) to generate three feature levels with different resolutions—P3 (high resolution, fine granularity), P4 (medium resolution), and P5 (low resolution, strong semantics), providing rich multi-scale image features for subsequent detection heads.

[0151] 6. Sparse4D decoder and physics prior query;

[0152] The P3, P4, and P5 features output from the FPN are input into the Sparse4D Decoder. This detector head remains fully trainable, and its internal mechanism is as follows:

[0153] Query initialization: Initialize sparse 3D anchor point queries based on physical scene priors (such as vehicle position and lane spatial distribution).

[0154] Self-Attention: Queries interact with each other through self-attention.

[0155] Deformable 4D Aggregation: Combines temporal information (historical frames) to perform deformable sampling and 4D feature aggregation on features P3~P5.

[0156] The Refine module iteratively optimizes the position and size of the prediction box through a cascading approach.

[0157] 7. Training strategy and parameter updates;

[0158] The AdamW optimizer was used, with a base learning rate of 4×10⁻ for the backbone bias term and the SLRT module. 4 The decoder part uses a learning rate of 1×10⁻ 4 The training process employs a cosine annealing scheduling strategy. The total training epochs are set to 48. The loss function is a detection-specific set prediction loss, weighted by Focal Loss (for classification, addressing imbalance between positive and negative samples) and L1 Loss (7-DOF 3D bounding box regression, including position, size, yaw angle, and velocity), and labeled using the Hungarian algorithm.

[0159] 8. Backpropagation and gradient cutoff;

[0160] During backpropagation, since the backbone weights are set to require_grad=False, the gradients only flow to and update the trainable groups: W_down, W_up, scaling factor S of SLRT, bias terms of each layer, 1x1 Conv1D of the SPC module, and the fully trainable Sparse4D Decoder. The frozen backbone weights remain unchanged at their initial pre-trained values.

[0161] 9. Fine-tuning completion judgment and output (corresponding flowchart: Output head -> 3D detection / multi-target tracking / velocity estimation)

[0162] At the end of each epoch, the nuScenes Detection Score (NDS) is calculated on the validation set. If the NDS on the validation set does not improve for five consecutive epochs, early stopping is triggered, and the optimal checkpoint is restored. After training, the model output head directly predicts the 3D bounding boxes of objects in the scene, multi-object tracking trajectories, and instantaneous velocity estimates, completing the full inference from 2D image input to 3D bird's-eye view perception.

[0163] The experimental results of this application are shown in Table 1:

[0164] Table 1. Key Experimental Results

[0165]

[0166] A comparison of this application with other PEFT methods is shown in Table 2:

[0167] Table 2. Comparison with other PEFT methods

[0168]

[0169] Compared with the prior art, the present invention has the following outstanding features and beneficial effects:

[0170] (1) Extremely high parameter efficiency. Only 10% to 17% of the trainable parameters are needed to achieve detection accuracy comparable to or even better than full parameter fine-tuning (the difference between NDS and NDS is within 0.2%), which significantly reduces the GPU memory usage during the training phase (saving 12% to 34%), making it possible to train a large model with 637 million parameters on a consumer-grade 24GB GPU.

[0171] (2) Strong 2D to 3D feature conversion capability. Compared with the general PEFT method, the SLRT of this invention specifically learns the task-specific conversion from 2D image features to 3D BEV features through side residual connections, resulting in significant performance advantages. Comparative experiments on the ViT-B and nuScenes datasets show that: the NDS of the Adapter (10.2M parameters) is 45.2%, a decrease of 10.4 percentage points compared to FFT; the NDS of the Prefix Tuning (1.5M parameters) is 46.1%, a decrease of 9.5 percentage points; the NDS of LoRA (13.8M parameters) is 50.4%, a decrease of 5.2 percentage points; while the NDS of the BEFT of this invention (15.0M parameters) is 54.4%, a decrease of only 1.2 percentage points. The performance loss of this invention is 23% of LoRA (1.2 / 5.2) and 12% of the Adapter (1.2 / 10.4). This significant advantage stems from the lateral residual design of SLRT—the lateral branches learn task-specific 2D-to-3D transformation residuals while freezing the main path to protect pre-trained visual knowledge, achieving the best balance between knowledge protection and task adaptation.

[0172] (3) Sufficient protection of pre-trained knowledge. The side design of SLRT does not interrupt the frozen main path, and zero initialization ensures that training starts from the pre-trained state, thus protecting the general visual knowledge contained in the large-scale pre-trained model to the greatest extent. Ablation experiments verified the effectiveness of knowledge protection: when the main branch is completely frozen (without adaptation modules), the NDS is 41.5%, indicating that directly reusing 2D pre-trained features is insufficient for 3D detection; after introducing SLRT, the NDS increases to 53.2% (+11.7%), verifying the core conversion capability of the side branch; after changing SLRT to a sequential insertion design, the NDS drops sharply to 46.5% (-6.7%), verifying the key role of the side design in protecting pre-trained knowledge. The sequential design destroys the frozen path, resulting in information loss, while the side design maintains the lossless transmission of the frozen path through residual connections. At the same time, the side branch learns task-specific residuals, achieving the optimal balance between pre-trained knowledge protection and downstream task adaptation.

[0173] (4) Multi-granularity collaborative adaptation. By inserting SLRT in parallel at the Attention, FFN, and Block levels, collaborative adaptation of features at different levels of abstraction was achieved. Ablation experiments (ViT-B, nuScenes val) show that the NDS is 52.4% when SLRT is inserted only at the FFN level, 52.0% when inserted only at the Attention level, 53.5% when inserted at both the FFN and Attention levels, and 54.4% when inserted at all three levels. The collaborative adaptation at the three levels captures 2D-to-3D transformation information at different levels of abstraction: the Attention level SLRT adjusts the spatial interaction between image tokens, the FFN level SLRT adjusts the feature transformation of the channel dimension, and the Block level SLRT aligns the output distribution of the entire Transformer block with the 3D detection requirements. The collaboration of the three achieves comprehensive adaptation of multi-granularity feature representations, enabling the transformed features to meet the requirements of the Sparse4D Decoder for high-quality multi-scale image features.

[0174] (5) Plug and play, highly versatile. This invention does not depend on a specific detector architecture and can be used in conjunction with various mainstream multi-view 3D detectors such as Sparse4D, BEVFormer, and PETR. Experiments have verified its versatility across detectors: on BEVFormer, the NDS of BEFT (15.0M parameters) is 40.8%, slightly higher than FFT (88.9M parameters) at 40.6% by 0.2 percentage points; on PETR, the NDS of BEFT is 37.8%, lower than FFT's 39.2% by 1.4 percentage points; on StreamPETR, the NDS of BEFT is 54.4%, lower than FFT's 55.6% by 1.2 percentage points. On all detectors, BEFT achieves performance close to that of FFT with approximately 16.85% of trainable parameters. Furthermore, BEFT is effective on three different scale backbone networks: ViT-B (88M parameters), ViT-L (307M parameters), and ViT-H (637M parameters), validating its scalability from basic models to ultra-large models. BEFT's design follows the vanilla ViT architecture, without modifying the Transformer's internal architecture, and can be plug-and-play integrated into any ViT-based multi-view detection system. Combined with the SPC position embedding calibration module and bias fine-tuning mechanism, BEFT provides a complete and efficient parameter fine-tuning solution for autonomous driving perception systems, making the training, storage, and iteration of large-scale vision base models more efficient in practical deployments.

[0175] like Figure 4As shown, this application also provides an efficient fine-tuning system for visual Transformer parameters based on a side-low-rank tuner, including:

[0176] The acquisition module is used to acquire fine-tuned training samples and labels from vehicle sensors for 3D perception tasks adapted to autonomous driving, and to acquire a pre-trained Transformer model, wherein the pre-trained Transformer model is pre-trained from multi-task general 2D image data to retain general visual representation capabilities.

[0177] A side adapter enhancement module is used to insert a low-rank tuner with zero initialization policy processing on the side of the target layer of the pre-trained Transformer model to obtain a side adapter enhanced visual Transformer model, wherein the low-rank tuner is a lightweight trainable side-channel adapter module.

[0178] The fine-tuning module is used to freeze all parameters in the backbone network of the side adapter-enhanced visual Transformer model except for the bias term, and to fine-tune the side adapter-enhanced visual Transformer model based on the fine-tuning training samples to obtain the fine-tuned visual Transformer model.

[0179] This application presents a method and system for efficient fine-tuning of visual Transformer parameters based on side-level low-rank tuners. By inserting lightweight low-rank tuners in parallel along the sides of the frozen pre-trained backbone network, it achieves detection accuracy comparable to full-parameter fine-tuning with only a minimal number of trainable parameters. This significantly reduces GPU memory usage and storage overhead during training, making scene adaptation of large models feasible on consumer-grade hardware. The core mechanism of this method lies in its side-level residual connection design: the frozen main path fully preserves pre-trained visual knowledge and achieves lossless transfer, while the side branches specifically learn task-specific transformation residuals. A zero-initialization strategy ensures strict alignment between the training starting point and the pre-trained state, effectively bridging the gap in cross-task feature space while fully protecting general knowledge. By inserting tuners in parallel at three levels—the feedforward network, the attention module, and the overall block output—and allowing them to learn independently, this method achieves multi-granularity collaborative adaptation to spatial interactions, channel transformations, and overall distribution. Furthermore, this method follows the standard visual Transformer architecture, is compatible with a variety of mainstream 3D detectors, and remains effective on backbone networks of different sizes, exhibiting excellent plug-and-play characteristics and scalability.

[0180] This embodiment also provides an electronic terminal, including: a processor and a memory;

[0181] The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory so that the terminal performs any of the methods in this embodiment.

[0182] As will be understood by those skilled in the art, the computer-readable storage medium described in this embodiment allows for the implementation of all or part of the steps in the above method embodiments by computer program-related hardware. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0183] The electronic terminal provided in this embodiment includes a processor, a memory, a transceiver, and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication between them. The memory is used to store computer programs, the communication interface is used to perform communication, and the processor and the transceiver are used to run the computer programs, so that the electronic terminal performs the steps of the above method.

[0184] In this embodiment, the memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0185] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0186] In the above embodiments, although the present application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. The embodiments of the present application are intended to cover all such substitutions, modifications, and variations falling within the broad scope of the appended claims.

[0187] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A highly efficient fine-tuning method for visual Transformer parameters based on a side-rank tuner, applied to the fine-tuning of autonomous driving perception models, characterized in that... include: Acquire fine-tuned training samples and labels from vehicle sensors for a 3D perception task adapted to autonomous driving, and obtain a pre-trained Transformer model, wherein the pre-trained Transformer model is pre-trained from multi-task general 2D image data to retain general visual representation capabilities. A low-rank tuner with zero initialization policy is inserted on the side of the target layer of the pre-trained Transformer model to obtain a side-adaptor enhanced visual Transformer model, wherein the low-rank tuner is a lightweight trainable side-path adapter module. Freeze all parameters in the backbone network of the side adapter-enhanced visual Transformer model except for the bias term, and fine-tune the side adapter-enhanced visual Transformer model based on the fine-tuning training samples to obtain the fine-tuned visual Transformer model.

2. The efficient fine-tuning method for visual Transformer parameters based on a side-low-rank tuner according to claim 1, characterized in that, The mathematical expression for the low-rank tuner is: In the formula, Indicates input features, This represents the result of the low-rank tuner processing the input features. Representation layer normalization, For the dimension-reduced projection matrix, Indicates nonlinear activation. For the up-dimensional projection matrix, is the scaling factor, where the up-dimensional projection matrix processed by the zero-initialization strategy is... ,and , Indicates uniform distribution. For the input channel dimension, It is a low-rank intermediate dimension.

3. The efficient fine-tuning method for visual Transformer parameters based on a side-low-rank tuner according to claim 1, characterized in that, The output of the low-rank tuner is connected to the target layer of the backbone network via residuals. The target layer includes a multi-head self-attention submodule, a feedforward network submodule, and a Transformer block.

4. The efficient fine-tuning method for visual Transformer parameters based on a side-low-rank tuner according to claim 3, characterized in that, When the target layer is a multi-head self-attention submodule, the intermediate variable output by the multi-head self-attention submodule with a low-rank tuner inserted on the side. The mathematical expression is: In the formula, Representation layer normalization, Indicates the first Output features of each Transformer block This represents the raw output of multi-head self-attention. This represents the output of the low-rank tuner in the multi-head self-attention submodule.

5. The efficient fine-tuning method for visual Transformer parameters based on a side-low-rank tuner according to claim 3, characterized in that, When the target layer is a feedforward network submodule, the intermediate variable output by the feedforward network submodule with a low-rank tuner inserted on the side. The mathematical expression is: In the formula, Intermediate variables output by the multi-head self-attention submodule that inserts a low-rank tuner on the side. Representation layer normalization, This represents the raw output of the feedforward network. This represents the output of the low-rank tuner on the side of the feedforward network submodule.

6. The efficient fine-tuning method for visual Transformer parameters based on a side-low-rank tuner according to claim 3, characterized in that, When the target layer is a Transformer block, the intermediate variable output by the Transformer block with a low-rank tuner is inserted on the side. The mathematical expression is: In the formula, The intermediate variable output by the feedforward network submodule with the low-rank tuner inserted on the side. This represents the output of the low-rank tuner on the side of the Transformer block.

7. The efficient fine-tuning method for visual Transformer parameters based on a side-low-rank tuner according to claim 2, characterized in that, The side adapter-enhanced visual Transformer model is fine-tuned based on the fine-tuned training samples to obtain the fine-tuned visual Transformer model, including: S1, divide all fine-tuned training samples into training set and validation set; S2, input the fine-tuned training samples into the side adapter-enhanced visual Transformer model after freezing the backbone network to obtain the recognition result; S3, calculate the combined loss of the recognition result and the labels of the fine-tuned training samples based on the pre-constructed loss function, wherein the loss function includes classification loss and regression loss; S4. Based on the backpropagation of the comprehensive loss, update the bias term of the backbone network, the dimension reduction projection matrix of the low-rank tuner, the dimension increase projection matrix of the low-rank tuner, and the scaling factor of the low-rank tuner in the side adapter enhanced vision Transformer model. S5. Repeat steps S2-S4 until the conditions for completing fine-tuning training are met.

8. The efficient fine-tuning method for visual Transformer parameters based on a side-low-rank tuner according to claim 7, characterized in that, The mathematical expression for the loss function is: In the formula, To account for the overall losses, The weights for the classification loss, For classifying losses, For the weights of the regression loss, To regress the loss, As a balance factor, This represents the probability that the model predicts the true class. To focus parameters, For the parameter dimension index of the prediction box, The number of parameter dimensions of the predicted bounding box. The first prediction box Predicted values ​​of parameters in each dimension. The first prediction box The true values ​​of parameters in each dimension.

9. The efficient fine-tuning method for visual Transformer parameters based on a side-low-rank tuner according to claim 7, characterized in that, The conditions for completing fine-tuning training are: The maximum number of training rounds is reached, or the performance metrics of the validation set do not show an improvement exceeding the preset threshold for five consecutive rounds.

10. A high-efficiency fine-tuning system for visual Transformer parameters based on a side-rank tuner, characterized in that, include: The acquisition module is used to acquire fine-tuned training samples and labels from vehicle sensors for 3D perception tasks adapted to autonomous driving, and to acquire a pre-trained Transformer model, wherein the pre-trained Transformer model is pre-trained from multi-task general 2D image data to retain general visual representation capabilities. A side adapter enhancement module is used to insert a low-rank tuner with zero initialization policy processing on the side of the target layer of the pre-trained Transformer model to obtain a side adapter enhanced visual Transformer model, wherein the low-rank tuner is a lightweight trainable side-channel adapter module. The fine-tuning module is used to freeze all parameters in the backbone network of the side adapter-enhanced visual Transformer model except for the bias term, and to fine-tune the side adapter-enhanced visual Transformer model based on the fine-tuning training samples to obtain the fine-tuned visual Transformer model.

Images