Automatic driving model based on hybrid low-rank experts and multi-domain adaptation fine-tuning method

Abstract

This application relates to the field of image data processing technology, and particularly to an autonomous driving model based on hybrid low-rank experts and a multi-domain adaptation fine-tuning method. The model includes: a general model base in a frozen state and a multi-domain adaptation component in a trainable state. The general model base includes: an image encoder, a measurement information encoder, and a trajectory planner. The multi-domain adaptation component includes: a hybrid low-rank expert group and a domain-aware router. The hybrid low-rank expert group is used to calculate differential correction signals based on input features. The domain-aware router is used to receive shared image features extracted by the image encoder, output probability vectors, and dynamically activate the corresponding experts in the hybrid low-rank expert group according to the feature distribution of the input data. This application achieves highly efficient "plug-and-play" adaptation of a single general model to multiple downstream domains with extremely low computational and storage costs, perfectly solving the pain points of traditional multi-domain adaptation methods.

Description

Technical Field

[0001] This application relates to the field of image data processing technology, and in particular to an autonomous driving model based on hybrid low-rank experts and a multi-domain adaptive fine-tuning method. Background Technology

[0002] As end-to-end autonomous driving models are gradually applied to real-world traffic environments, their biggest challenge lies in adapting to the complex and ever-changing driving domain. Differences in traffic rules and culture between different cities (geographic domain), environmental changes under different weather and lighting conditions (environment domain), and different interaction logics between highways and urban areas (scenario domain) can all cause significant domain drift, leading to a severe performance degradation of the model when deployed across domains.

[0003] To address this challenge, the current mainstream technical approach is to train a single general-purpose model, attempting to learn a comprehensive general policy by aggregating data from multiple sources and domains. However, this approach is highly susceptible to the pattern averaging problem: the model's driving policy is a trade-off between behaviors across all different domains, rather than the optimal solution for any specific domain, which fundamentally limits the model's performance ceiling. A direct approach to solving the pattern averaging problem is to train and maintain a specialized, complete model separately for each specific domain. However, this strategy is prohibitively expensive in engineering deployment. On the one hand, it leads to a dramatic increase in computational, storage, and maintenance costs; on the other hand, attempting to accumulate multi-domain capabilities through sequential fine-tuning on a single model can trigger severe "catastrophic forgetting."

[0004] Therefore, there is an urgent need for a new, lightweight multi-domain adaptation paradigm that can build exclusive and specialized knowledge for different domains to break through the performance bottleneck of a single model, and fundamentally avoid the high training and deployment costs, thus becoming a core challenge for promoting the large-scale and refined implementation of autonomous driving technology. Summary of the Invention

[0005] This application provides an autonomous driving model based on hybrid low-rank experts and a multi-domain adaptation fine-tuning method, which realizes a single general model to be efficiently adapted to multiple downstream domains in a "plug-and-play" manner, and perfectly solves the pain points of traditional multi-domain adaptation methods.

[0006] To address the aforementioned technical problems, in a first aspect, this application provides an autonomous driving model based on hybrid low-rank experts, comprising: a general model base in a frozen state and a multi-domain adaptation component in a trainable state; the general model base includes: an image encoder, a measurement information encoder, and a trajectory planner; the image encoder receives RGB images captured by a vehicle-mounted single-view camera, performs convolution operations, and outputs a high-dimensional image feature vector; the measurement information encoder outputs a measurement feature vector; the trajectory planner receives the fused features obtained by concatenating the image feature vector and the measurement feature vector, and predicts the coordinates of vehicle trajectory points at a preset time step in an autoregressive manner; the multi-domain adaptation component includes: a hybrid low-rank expert group and a domain perception router; the hybrid low-rank expert group calculates a differential correction signal based on the input features; the domain perception router receives shared image features extracted by the image encoder, outputs a probability vector, and dynamically activates the corresponding experts in the hybrid low-rank expert group according to the feature distribution of the input data.

[0007] In some exemplary embodiments, the general model pedestal is loaded with weight parameters pre-trained on the full set of general data and is configured to keep the parameters frozen during fine-tuning.

[0008] In some exemplary embodiments, the image encoder employs a ResNet-34 network architecture.

[0009] In some exemplary embodiments, the measurement information encoder employs a multilayer perceptron architecture; the input data of the measurement information encoder includes a vehicle speed scalar, discretized high-level navigation commands, and local target point coordinates.

[0010] In some exemplary embodiments, the trajectory planner employs a gated cyclic cell network.

[0011] In some exemplary embodiments, the hybrid low-rank expert group includes multiple independent plug-in low-rank adapter experts, each corresponding one-to-one with a target driving domain to be adapted; the low-rank adapter experts are configured to calculate differential correction signals based on input features.

[0012] In some exemplary embodiments, the domain-aware router employs a multi-layer perceptron architecture.

[0013] Secondly, this application also provides an autonomous driving multi-domain adaptive fine-tuning method based on hybrid low-rank experts. This method is implemented based on the autonomous driving model based on hybrid low-rank experts described in the above embodiments. The method includes: an integrated general model strategy training stage and a hybrid low-rank expert multi-domain adaptive fine-tuning stage. The integrated general model strategy training stage is used to train the autonomous driving model based on hybrid low-rank experts. The hybrid low-rank expert multi-domain adaptive fine-tuning stage is used to optimize the decision of the general model in a specific domain by using plug-in hybrid low-rank adapter experts and outputting differential correction signals.

[0014] In some exemplary embodiments, the hybrid low-rank expert multi-domain adaptation fine-tuning stage includes: freezing general model parameters; constructing a pluggable low-rank adapter expert group; designing a supervised domain-aware router; fine-tuning based on differential correction; and parameter-efficient loss update.

[0015] In some exemplary embodiments, fine-tuning based on differential correction includes: when data is input from a specific domain, a frozen encoder is responsible for extracting general, shared scene features; a trainable domain-aware router receives the features and dynamically selects and activates the corresponding low-rank expert; a frozen decision head outputs a general, baseline planned trajectory based on the shared features; the activated low-rank expert receives the shared features and outputs a differential correction signal for the specific domain; the trajectory of the general model is added to or fused with the differential correction signal of the low-rank expert to generate the final, domain-optimized high-precision trajectory.

[0016] The technical solution provided in this application has at least the following advantages:

[0017] This application provides an autonomous driving model based on hybrid low-rank experts and a multi-domain adaptation fine-tuning method. The model includes: a general model base in a frozen state and a multi-domain adaptation component in a trainable state. The general model base includes: an image encoder, a measurement information encoder, and a trajectory planner. The image encoder receives RGB images captured by a vehicle-mounted single-view camera, performs convolution operations, and outputs a high-dimensional image feature vector. The measurement information encoder outputs a measurement feature vector. The trajectory planner receives the fused features obtained by concatenating the image feature vector and the measurement feature vector, and predicts the coordinates of vehicle trajectory points at a preset time step using an autoregressive approach. The multi-domain adaptation component includes: a hybrid low-rank expert group and a domain perception router. The hybrid low-rank expert group calculates a differential correction signal based on input features. The domain perception router receives shared image features extracted by the image encoder, outputs a probability vector, and dynamically activates the corresponding experts in the hybrid low-rank expert group based on the feature distribution of the input data. This application provides an autonomous driving model based on hybrid low-rank experts and a parameter-efficient model fine-tuning method, utilizing pluggable hybrid low-rank adapter experts to achieve rapid and flexible adaptation of a single general driving model to multiple different driving domains. Moreover, this application achieves "plug-and-play" efficient adaptation of a single general model to multiple downstream domains with extremely low computational and storage costs, perfectly solving the pain points of traditional multi-domain adaptation methods. Attached Figure Description

[0018] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations do not constitute a limitation on the embodiments, and unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0019] Figure 1 The flowchart illustrates the specific architecture of the autonomous driving model and multi-domain adaptive fine-tuning method based on hybrid low-rank experts provided in this application embodiment. Detailed Implementation

[0020] As the background technology shows, the current mainstream approach is to train a single general-purpose model, attempting to learn a comprehensive and universal strategy by aggregating data from multiple sources and domains. However, this method is highly susceptible to the pattern averaging problem.

[0021] Therefore, to solve this technical problem, this application provides an autonomous driving model based on hybrid low-rank experts and a multi-domain adaptation fine-tuning method. The model includes: a general model base in a frozen state and a multi-domain adaptation component in a trainable state; the general model base includes: an image encoder, a measurement information encoder, and a trajectory planner; the image encoder is used to receive RGB images captured by a vehicle-mounted single-view camera, and outputs a high-dimensional image feature vector after convolution operation; the measurement information encoder is used to output a measurement feature vector; the trajectory planner is used to receive the fused features after concatenating the image feature vector and the measurement feature vector, and predicts the coordinates of the vehicle trajectory points at a preset time step in an autoregressive manner; the multi-domain adaptation component includes: a hybrid low-rank expert group and a domain perception router; the hybrid low-rank expert group is used to calculate a differential correction signal based on the input features; the domain perception router is used to receive shared image features extracted by the image encoder, output a probability vector, and dynamically activate the corresponding experts in the hybrid low-rank expert group according to the feature distribution of the input data. This application provides an autonomous driving model based on hybrid low-rank experts and a multi-domain adaptation fine-tuning method, which realizes a single general model to be efficiently adapted to multiple downstream domains in a "plug-and-play" manner, and perfectly solves the pain points of traditional multi-domain adaptation methods.

[0022] The embodiments of this application will now be described in detail with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the embodiments of this application to facilitate a better understanding of the application. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments.

[0023] See Figure 1 To address the aforementioned technical problems, in a first aspect, embodiments of this application provide an autonomous driving model based on hybrid low-rank experts, comprising: a general model base in a frozen state and a multi-domain adaptation component in a trainable state; the general model base includes: an image encoder, a measurement information encoder, and a trajectory planner; the image encoder receives RGB images captured by a vehicle-mounted single-view camera, performs convolution operations, and outputs a high-dimensional image feature vector; the measurement information encoder outputs a measurement feature vector; the trajectory planner receives the fused features obtained by concatenating the image feature vector and the measurement feature vector, and predicts the coordinates of vehicle trajectory points at a preset time step in an autoregressive manner; the multi-domain adaptation component includes: a hybrid low-rank expert group and a domain perception router; the hybrid low-rank expert group calculates a differential correction signal based on input features; the domain perception router receives shared image features extracted by the image encoder, outputs a probability vector, and dynamically activates the corresponding experts in the hybrid low-rank expert group according to the feature distribution of the input data.

[0024] In some embodiments, the general model pedestal is loaded with weight parameters pre-trained on the full general data and configured to keep the parameters frozen during fine-tuning.

[0025] In some embodiments, the image encoder employs a ResNet-34 network structure.

[0026] In some embodiments, the measurement information encoder employs a multilayer perceptron structure; the input data of the measurement information encoder includes a vehicle speed scalar, discretized high-level navigation commands, and local target point coordinates.

[0027] In some embodiments, the trajectory planner employs a gated cyclic cell network.

[0028] In some embodiments, the hybrid low-rank expert group includes multiple independent plug-in low-rank adapter experts, each corresponding to a target driving domain to be adapted; the low-rank adapter experts are configured to calculate differential correction signals based on input features.

[0029] In some embodiments, the domain-aware router employs a multilayer perceptron architecture.

[0030] Furthermore, this application also provides an autonomous driving multi-domain adaptive fine-tuning method based on hybrid low-rank experts. This method is implemented based on the autonomous driving model based on hybrid low-rank experts described in the above embodiments. The method includes two stages: an integrated general model strategy training stage and a hybrid low-rank expert multi-domain adaptive fine-tuning stage. The integrated general model strategy training stage is used to train the autonomous driving model based on hybrid low-rank experts. The hybrid low-rank expert multi-domain adaptive fine-tuning stage is used to optimize the decision of the general model in a specific domain by using plug-in hybrid low-rank adapter experts and outputting differential correction signals.

[0031] The multi-domain adaptive fine-tuning method based on hybrid low-rank experts provided in this application is based on the core idea of ​​freezing the pre-trained general model and introducing a set of lightweight, pluggable, trainable low-rank adapter experts and a domain-aware router. The method optimizes the decision of the general model in a specific domain by outputting differential correction signals.

[0032] Specifically, this invention is divided into two stages, with its core innovation lying in the second stage. The specific implementation steps are as follows:

[0033] Phase 1: Training of the integrated general model strategy.

[0034] First, a robust end-to-end general autonomous driving model is trained on a large-scale, diverse full-scale driving dataset. This model consists of an encoder and a decision head. After training, it possesses robust general planning capabilities, which will serve as an immutable capability foundation for subsequent lightweight adaptation.

[0035] Phase Two: Hybrid Low-Rank Expert Multi-Domain Adaptation Fine-Tuning.

[0036] This stage is the core innovation of this invention, and its specific steps are as follows:

[0037] 1. Freeze general model parameters: Load the general model trained in Phase 1 and freeze all its parameters completely, keeping them unchanged throughout the fine-tuning process. This aims to fully preserve the model's general knowledge and fundamentally avoid catastrophic forgetting.

[0038] 2. Construct a pluggable low-rank adapter expert group: Design a group of pluggable low-rank adapter experts corresponding to different driving domains. Each low-rank expert is an independent, lightweight network with a very small number of parameters.

[0039] 3. Design a supervised domain-aware router: Design and introduce a lightweight, trainable domain-aware router. The router's function is to determine the domain to which the input data belongs and activate the corresponding low-rank expert.

[0040] 4. Implement a fine-tuning process based on differential correction.

[0041] a. When data is input from a specific domain (e.g., "domain N"), the frozen encoder is responsible for extracting common, shared scene features.

[0042] b. A trainable domain-aware router receives this feature and dynamically selects and activates the corresponding low-rank expert N.

[0043] c. The frozen decision head outputs a general, benchmark planning trajectory based on shared features.

[0044] d. The activated low-rank expert N receives the shared features and outputs a differential correction signal for the "domain N".

[0045] e. Add or fuse the trajectory of the general model with the differential correction signal of the low-rank expert to generate the final high-precision trajectory optimized in a specific domain.

[0046] 5. Efficient Loss Update: Throughout the fine-tuning process, the computed loss function is used only to update the parameters of the domain-aware router and all low-rank adapter experts. The parameters of the general model remain frozen.

[0047] To enable those skilled in the art to better understand the present application, the technical solution of the present application will be described in detail below through specific embodiments. Please continue reading. Figure 1This application provides an end-to-end autonomous driving multi-domain adaptation fine-tuning method based on hybrid low-rank experts, including an integrated general model strategy training stage and a hybrid low-rank expert multi-domain adaptation fine-tuning stage.

[0048] Specifically, the embodiments of the present invention mainly include three parts: model architecture construction, multi-domain adaptation problem, and inference deployment.

[0049] First, build the model architecture.

[0050] In this embodiment, the following is constructed: Figure 1 The end-to-end autonomous driving network architecture shown includes a general model base in a frozen state and a multi-domain adaptation component in a trainable state.

[0051] 1. General model base.

[0052] The general model base is loaded with weight parameters pre-trained on the full set of general data and configured to keep the parameters frozen during fine-tuning. Specifically, the base includes:

[0053] Image encoder: Employs a ResNet-34 network structure. It is configured to receive RGB images captured by a vehicle-mounted monocular forward-looking camera, perform convolution operations, and output a high-dimensional shared image feature vector.

[0054] Measurement information encoder: Employs a multilayer perceptron structure. Its input data includes vehicle speed scalar, discretized high-level navigation commands, and local target point coordinates; the output data is a measurement feature vector.

[0055] Trajectory Planner: Employs a Gated Recurrent Unit (GRU) network. This module receives the fused features from the concatenation of image and measurement feature vectors and predicts the coordinates of vehicle trajectory points at a preset time step using an autoregressive approach. During the fine-tuning phase, the pre-trained weight matrix of this module... W 0 remains unchanged.

[0056] 2. Multi-domain adaptation components.

[0057] The multi-domain adaptation component is a trainable module in this embodiment, specifically including:

[0058] Hybrid low-rank expert group: includes N Each independent plug-in low-rank adapter expert corresponds to... N Each low-rank adapter expert does not contain a complete neural network layer, but rather consists of two low-rank matrices. A and B It is composed of parallel connections. Among them, A For a dimension reduction matrix, B For an increased-dimensional matrix, the rank of both is...r Much smaller than the pre-trained weight matrix W The dimension is 0. The low-rank adapter expert is configured to compute the differential correction signal based on the input features, using the following formula: .

[0059] Domain-aware router: Employs a multi-layer perceptron architecture. This router receives shared image features extracted by an image encoder and outputs a domain-aware router with a specific dimension. N The probability vector is used to dynamically activate the corresponding experts in the hybrid low-rank expert group based on the feature distribution of the input data.

[0060] Phase Two: Implementation Process of Multi-Domain Adaptive Fine-tuning

[0061] After constructing the above architecture, the following steps are performed to adapt the model to multiple driving domains:

[0062] Step 1: Multi-domain data input and feature extraction.

[0063] The system receives training sample data from a specific driving domain. The training sample data is input into a frozen general model base, and a shared feature vector containing environmental semantics and vehicle state is extracted by an image encoder and a measurement information encoder.

[0064] Step 2: Dynamic Routing and Expert Activation.

[0065] The domain-aware router receives the shared feature vector and calculates the probability value of the feature vector belonging to each preset driving domain. Based on the probability distribution results, the router generates an activation signal and dynamically selects a low-rank adapter expert from the hybrid low-rank expert group that matches the domain to which the current input data belongs.

[0066] Step 3: Decision-making on the integration of benchmark and correction.

[0067] This step involves parallel benchmark and correction calculations:

[0068] Baseline trajectory generation: The frozen trajectory planner performs inference based on the shared feature vectors, utilizing a pre-trained weight matrix. W 0 Output universal reference trajectory.

[0069] Differential correction signal generation: Activated low-rank adapter experts receive the same shared feature vector and perform low-rank matrix operations. The differential correction signal for the current specific driving domain is calculated.

[0070] Final trajectory synthesis: The general reference trajectory is added element by element to the differential correction signal to obtain the optimized final planned trajectory.

[0071] Step 4: Efficient parameter update.

[0072] Calculate the loss function value between the final planned trajectory and the real expert trajectory. During backpropagation, lock all parameters of the general model base, and only process the parameters of the domain-aware router and the matrices in the activated low-rank adapter experts. A sum matrix B Perform gradient updates.

[0073] Phase 3: Reasoning and Deployment.

[0074] After fine-tuning the training, the model is deployed on an onboard computing platform. During vehicle operation, onboard sensors collect environmental data in real time; the domain perception router automatically identifies the driving domain to which the current driving environment belongs based on the feature distribution of the real-time data and activates the corresponding low-rank adapter expert; the activated expert outputs a differential correction signal in real time to compensate for the baseline trajectory output by the general model base, thereby generating a planned trajectory adapted to the current driving domain.

[0075] This embodiment, through the above architecture and method, achieves flexible adaptation of the model to changing driving environments with a very small number of trainable parameters, while ensuring that general driving capabilities are not degraded, and significantly reducing the computation and storage costs of multi-domain adaptation.

[0076] Based on the above technical solutions, this application provides an autonomous driving model based on hybrid low-rank experts and a multi-domain adaptation fine-tuning method. This method includes two stages: an integrated general model strategy training stage and a hybrid low-rank expert multi-domain adaptation fine-tuning stage. The integrated general model strategy training stage is used to train the autonomous driving model based on hybrid low-rank experts. The hybrid low-rank expert multi-domain adaptation fine-tuning stage utilizes plug-in hybrid low-rank adapter experts to optimize the decisions of the general model in specific domains by outputting differential correction signals. This application provides a parameter-efficient model fine-tuning method that utilizes plug-in hybrid low-rank adapter experts to achieve rapid and flexible adaptation of a single general driving model to multiple different driving domains. Moreover, this application achieves "plug-and-play" efficient adaptation of a single general model to multiple downstream domains with extremely low computational and storage costs, perfectly solving the pain points of traditional multi-domain adaptation methods.

[0077] Those skilled in the art will understand that the above-described embodiments are specific examples of implementing this application, and in practical applications, various changes in form and detail may be made without departing from the spirit and scope of this application. Any person skilled in the art can make their own modifications and alterations without departing from the spirit and scope of this application; therefore, the scope of protection of this application should be determined by the scope defined in the claims.

Claims

1. An automatic driving multi-domain adaptation fine-tuning method based on mixed low-rank experts, characterized in that, This method includes: an integrated general model strategy training phase and a hybrid low-rank expert multi-domain adaptation fine-tuning phase; The integrated general model strategy training phase is used to train an autonomous driving model based on hybrid low-rank experts; The hybrid low-rank expert multi-domain adaptive fine-tuning stage is used to optimize the decision of the general model in a specific domain by using plug-in hybrid low-rank adapter experts and outputting differential correction signals. The autonomous driving model based on hybrid low-rank experts includes: a general model base in a frozen state and a multi-domain adaptation component in a trainable state. The general model base includes: an image encoder, a measurement information encoder, and a trajectory planner; The image encoder is used to receive RGB images captured by a vehicle-mounted single-view camera, and outputs a high-dimensional image feature vector after convolution operation; the measurement information encoder is used to output a measurement feature vector. The trajectory planner is used to receive the fused features of the image feature vector and the measurement feature vector, and predict the coordinates of the vehicle trajectory points in the future preset time step in an autoregressive manner. The multi-domain adaptation component includes: a hybrid low-rank expert group and a domain-aware router; The hybrid low-rank expert group is used to calculate the differential correction signal based on the input features; the domain-aware router is used to receive the shared image features extracted by the image encoder, output the probability vector, and dynamically activate the corresponding experts in the hybrid low-rank expert group according to the feature distribution of the input data. The general model base is loaded with weight parameters that have been pre-trained on the full set of general data and is configured to keep the parameters frozen during fine-tuning. The mixed low-rank expert group includes a plurality of mutually independent plug-in low-rank adapter experts, and the low-rank adapter experts correspond one-to-one to target driving domains to be adapted; each low-rank adapter expert is composed of two low-rank matrices A and B in parallel; wherein A is a dimension reduction matrix, B is a dimension increase matrix, and the ranks r of the two are less than the dimension of the pre-trained weight matrix W0; the low-rank adapter expert is configured to calculate a differential correction signal based on an input feature, and the calculation formula is .

2. The hybrid low-rank expert-based multi-domain adaptation fine-tuning method for autonomous driving according to claim 1, wherein, The image encoder uses a ResNet-34 network structure. 3.The hybrid low-rank expert-based multi-domain adaptation fine-tuning method for autonomous driving according to claim 1, wherein, The measurement information encoder adopts a multilayer perceptron structure; The input data of the measurement information encoder includes vehicle speed scalar, discretized high-level navigation commands, and local target point coordinates. 4.The automatic driving multi-domain adaptation fine-tuning method for mixed low-rank experts of claim 1, wherein, The trajectory planner employs a gated cyclic unit network.

5. The method for multi-domain adaptive fine-tuning of autonomous driving based on hybrid low-rank experts according to claim 1, characterized in that, The domain-aware router adopts a multi-layer sensor architecture.

6. The autonomous driving multi-domain adaptive fine-tuning method based on hybrid low-rank experts according to claim 1, characterized in that, The hybrid low-rank expert multi-domain adaptation fine-tuning stage includes: Freeze the general model parameters; Build an expert group for plug-in low-rank adapters; Design a supervised domain-aware router; Fine-tuning is performed based on differential correction; Efficient loss update of parameters.

7. The autonomous driving multi-domain adaptive fine-tuning method based on hybrid low-rank experts according to claim 1, characterized in that, Based on the differential correction, fine-tuning is performed, including: When data is input from a specific domain, the frozen encoder is responsible for extracting common, shared scene features; A trainable domain-aware router receives this feature and dynamically selects and activates the corresponding low-rank expert. The frozen decision head, based on shared features, outputs a general, benchmark planning trajectory; The activated low-rank expert receives shared features and outputs a differential correction signal for a specific domain; The trajectory of the general model is added or fused with the differential correction signal of the low-rank expert to generate the final high-precision trajectory optimized in a specific domain.

Images