Automatic driving model based on mixed low-rank experts and multi-domain adaptive fine tuning method

Abstract

The invention relates to the technical field of image data processing, in particular to an automatic driving model based on a mixed low-rank expert and a multi-domain adaptive fine tuning method, and the model comprises a general model base in a frozen state and a multi-domain adaptive component in a trainable state; the universal model base comprises an image encoder, a measurement information encoder and a trajectory planner; the multi-domain adaptation component comprises a mixed low-rank expert group and a domain sensing router; the mixed low-rank expert group is used for calculating a difference correction signal according to the input characteristics; and the domain sensing router is used for receiving the shared image features extracted by the image encoder, outputting probability vectors and dynamically activating corresponding experts in the mixed low-rank expert group according to feature distribution of input data. According to the method, plug-and-play type efficient adaptation of a single universal model to multiple downstream domains is achieved with extremely low calculation and storage cost, and the pain point of a traditional multi-domain adaptation method is perfectly solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image data processing, in particular to an automatic driving model based on a hybrid low-rank expert and a multi-domain adaptation fine-tuning method. BACKGROUND

[0002] With the gradual application of end-to-end automatic driving models in real traffic environments, the biggest challenge they face is how to adapt to complex and variable driving domains. Differences in traffic rules and culture (geographical domain), environmental changes under different weather and light conditions (environmental domain), and different interaction logics such as highways and urban areas (scene domain) will all cause significant domain drift, resulting in a serious decline in model performance when deployed across domains.

[0003] To address this challenge, the current mainstream technical route is to train a single general model by aggregating multi-source, multi-domain data, trying to learn a general strategy that is big and complete. However, this method is prone to mode averaging problems: the driving strategy of the model is a compromise of behaviors in all different domains, rather than the optimal solution for any specific domain, which fundamentally limits the upper limit of the model's performance. To solve the mode averaging problem, one direct idea is to train and maintain a specialized complete model for each specific domain. However, this strategy is costly in engineering deployment, on the one hand, it will lead to a sharp increase in computing, storage and maintenance costs, on the other hand, if you try to accumulate multi-domain capabilities through sequential fine-tuning on a single model, it will cause serious "catastrophic forgetting".

[0004] Therefore, there is an urgent need for a new, lightweight multi-domain adaptation paradigm that can build exclusive, specialized knowledge for different domains to break through the performance bottleneck of a single model, and fundamentally avoid high training and deployment costs, becoming the core problem of promoting the large-scale and fine-grained landing of automatic driving technology. SUMMARY

[0005] The embodiments of the present application provide an automatic driving model based on a hybrid low-rank expert and a multi-domain adaptation fine-tuning method, which realizes the "plug and play" efficient adaptation of a single general model to multiple downstream domains, and perfectly solves the pain points of traditional multi-domain adaptation methods.

[0006] To solve the above technical problems, the first aspect of the present application provides an automatic driving model based on a hybrid low-rank expert, comprising: a general model base in a frozen state and a multi-domain adaptation component in a trainable state; the general model base comprises: an image encoder, a measurement information encoder, and a trajectory planner; the image encoder is used to receive an RGB image collected by a vehicle-mounted single front-view camera, and outputs a high-dimensional image feature vector after convolution operation; the measurement information encoder is used to output a measurement feature vector; the trajectory planner is used to receive a fused feature after splicing of the image feature vector and the measurement feature vector, and predict vehicle trajectory point coordinates in a future preset time step in a self-recurrent manner; the multi-domain adaptation component comprises: a hybrid low-rank expert group and a domain-aware router; the hybrid low-rank expert group is used to calculate a differential correction signal according to input features; the domain-aware router is used to receive shared image features extracted by the image encoder, output a probability vector, and dynamically activate corresponding experts in the hybrid low-rank expert group according to the feature distribution of input data.

[0007] In some example embodiments, the general model base is loaded with pre-trained weight parameters on full-quantity general data, and is configured to keep parameter freezing in a fine-tuning process.

[0008] In some example embodiments, the image encoder adopts a ResNet-34 network structure.

[0009] In some example embodiments, the measurement information encoder adopts a multi-layer perception machine structure; input data of the measurement information encoder includes a vehicle speed scalar, a discretized high-level navigation instruction, and a local target point coordinate.

[0010] In some example embodiments, the trajectory planner adopts a gated recurrent unit network.

[0011] In some example embodiments, the hybrid low-rank expert group comprises a plurality of plug-in low-rank adapter experts independent of each other, and the low-rank adapter experts correspond one-to-one to target driving domains to be adapted; the low-rank adapter experts are configured to calculate a differential correction signal based on input features.

[0012] In some example embodiments, the domain-aware router adopts a multi-layer perception machine structure.

[0013] In a second aspect, the embodiments of the present application also provide an automatic driving multi-domain adaptation fine-tuning method based on hybrid low-rank experts. The method is based on the automatic driving model implementation based on hybrid low-rank experts described in the above embodiments. The method comprises: an integrated general model strategy training stage and a hybrid low-rank expert multi-domain adaptation fine-tuning stage; the integrated general model strategy training stage is used to train the automatic driving model based on hybrid low-rank experts; and the hybrid low-rank expert multi-domain adaptation fine-tuning stage is used to utilize the plug-in hybrid low-rank adapter experts to output a difference correction signal to optimize the decision of the general model in a specific domain.

[0014] In some example embodiments, the hybrid low-rank expert multi-domain adaptation fine-tuning stage comprises: freezing the general model parameters; constructing a plug-in low-rank adapter expert group; designing a supervised domain-aware router; fine-tuning based on the difference correction; and parameter-efficient loss updating.

[0015] In some example embodiments, the fine-tuning based on the difference correction comprises: when data from a specific domain is input, the frozen encoder is responsible for extracting general and shared scene features; the trainable domain-aware router receives the features and dynamically selects and activates the corresponding low-rank expert; the frozen decision head outputs a general and benchmark planning trajectory based on the shared features; the activated low-rank expert receives the shared features and outputs a difference correction signal for the specific domain; and the trajectory of the general model and the difference correction signal of the low-rank expert are added or fused to generate a final and specific domain optimized high-precision trajectory.

[0016] The technical solutions provided by the embodiments of the present application have at least the following advantages: The embodiment of the application provides an automatic driving model based on a mixed low-rank expert and a multi-domain adaptive fine-tuning method, the model comprises: a general model base in a frozen state and a multi-domain adaptive component in a trainable state; the general model base comprises: an image encoder, a measurement information encoder and a trajectory planner; the image encoder is used for receiving an RGB image collected by a vehicle-mounted single front-view camera, and outputs a high-dimensional image feature vector after convolution operation; the measurement information encoder is used for outputting a measurement feature vector; the trajectory planner is used for receiving fusion features after splicing of the image feature vector and the measurement feature vector, and predicting vehicle trajectory point coordinates in a future preset time step in a self-recurrence manner; the multi-domain adaptive component comprises: a mixed low-rank expert group and a domain perception router; the mixed low-rank expert group is used for calculating a differential correction signal according to input features; the domain perception router is used for receiving shared image features extracted by the image encoder, outputting a probability vector, and dynamically activating corresponding experts in the mixed low-rank expert group according to the feature distribution of input data. The application provides an automatic driving model based on a mixed low-rank expert and a parameter-efficient model fine-tuning method, uses a plug-in mixed low-rank adapter expert, realizes rapid and flexible adaptation of a single general driving model to multiple different driving domains. Moreover, the application realizes "plug and play" efficient adaptation of a single general model to multiple downstream domains at extremely low computing and storage costs, and perfectly solves the pain points of traditional multi-domain adaptation methods. BRIEF DESCRIPTION OF DRAWINGS

[0017] One or more embodiments are illustrated by way of example in the figures that form a part of this patent document, and in which like reference numbers typically indicate similar components in the figures, and which execution is not to be construed as limiting the embodiments. The figures of the accompanying drawings are intended to further assist in the understanding of the embodiments, and are not intended to be to scale unless specifically indicated otherwise.

[0018] Figure 1 A specific architecture flowchart of the automatic driving model based on the mixed low-rank expert and the multi-domain adaptive fine-tuning method provided by the embodiment of the application is provided. DETAILED DESCRIPTION

[0019] As known from the background, the current mainstream technical route is to train a single general model, and try to learn a general strategy by aggregating multi-source and multi-domain data. However, this method is prone to cause mode average problems.

[0020] Therefore, in order to solve the technical problem, the embodiment of the present application provides an automatic driving model based on a mixed low-rank expert and a multi-domain adaptive fine-tuning method. The model comprises: a general model base in a frozen state and a multi-domain adaptive component in a trainable state; the general model base comprises: an image encoder, a measurement information encoder, and a trajectory planner; the image encoder is used to receive an RGB image collected by a vehicle-mounted single front-view camera, and outputs a high-dimensional image feature vector after convolution operation; the measurement information encoder is used to output a measurement feature vector; the trajectory planner is used to receive a fused feature after splicing of the image feature vector and the measurement feature vector, and predict vehicle trajectory point coordinates in a future preset time step in a self-recurrent manner; the multi-domain adaptive component comprises: a mixed low-rank expert group and a domain-aware router; the mixed low-rank expert group is used to calculate a differential correction signal according to input features; and the domain-aware router is used to receive shared image features extracted by the image encoder, output a probability vector, and dynamically activate corresponding experts in the mixed low-rank expert group according to the feature distribution of input data. The embodiment of the present application provides an automatic driving model based on a mixed low-rank expert and a multi-domain adaptive fine-tuning method, realizes efficient adaptation of a single general model to multiple downstream domains in a "plug and play" manner, and perfectly solves the pain points of traditional multi-domain adaptive methods.

[0021] The embodiments of the present application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art can understand that in the embodiments of the present application, many technical details are proposed in order to enable the reader to better understand the present application. However, the technical solutions claimed by the present application can be realized even without these technical details and various changes and modifications based on the following embodiments.

[0022] Referring to Figure 1 In order to solve the above technical problem, the embodiment of the first aspect of the present application provides an automatic driving model based on a mixed low-rank expert, comprising: a general model base in a frozen state and a multi-domain adaptive component in a trainable state; the general model base comprises: an image encoder, a measurement information encoder, and a trajectory planner; the image encoder is used to receive an RGB image collected by a vehicle-mounted single front-view camera, and outputs a high-dimensional image feature vector after convolution operation; the measurement information encoder is used to output a measurement feature vector; the trajectory planner is used to receive a fused feature after splicing of the image feature vector and the measurement feature vector, and predict vehicle trajectory point coordinates in a future preset time step in a self-recurrent manner; the multi-domain adaptive component comprises: a mixed low-rank expert group and a domain-aware router; the mixed low-rank expert group is used to calculate a differential correction signal according to input features; and the domain-aware router is used to receive shared image features extracted by the image encoder, output a probability vector, and dynamically activate corresponding experts in the mixed low-rank expert group according to the feature distribution of input data.

[0023] In some embodiments, the general model base is loaded with pre-trained weight parameters on full general data, and is configured to keep parameter freezing during the fine-tuning process.

[0024] In some embodiments, the image encoder adopts a ResNet-34 network structure.

[0025] In some embodiments, the measurement information encoder adopts a multi-layer perception structure; the input data of the measurement information encoder includes a vehicle speed scalar, a discretized high-level navigation instruction, and a local target point coordinate.

[0026] In some embodiments, the trajectory planner adopts a gated recurrent unit network.

[0027] In some embodiments, the hybrid low-rank expert group includes a plurality of mutually independent plug-in low-rank adapter experts, and the low-rank adapter experts correspond one-to-one to the target driving domains to be adapted; the low-rank adapter experts are configured to calculate a differential correction signal based on the input features.

[0028] In some embodiments, the domain-aware router adopts a multi-layer perception structure.

[0029] In addition, the embodiment of the present application also provides an automatic driving multi-domain adaptation fine-tuning method based on a hybrid low-rank expert, which is based on the automatic driving model based on the hybrid low-rank expert described in the above embodiment. The method includes two stages, namely: an integrated general model strategy training stage and a hybrid low-rank expert multi-domain adaptation fine-tuning stage; the integrated general model strategy training stage is used to train the automatic driving model based on the hybrid low-rank expert; the hybrid low-rank expert multi-domain adaptation fine-tuning stage is used to utilize the plug-in hybrid low-rank adapter expert to output a differential correction signal to optimize the decision of the general model in a specific domain.

[0030] The multi-domain adaptation fine-tuning method based on the hybrid low-rank expert provided by the present application has the core idea of freezing the pre-trained general model and introducing a group of lightweight, plug-in, trainable low-rank adapter experts and a domain-aware router to output a differential correction signal to optimize the decision of the general model in a specific domain.

[0031] Specifically, the present application is divided into two stages, and the core innovation is in the second stage, and the specific implementation steps are as follows: Stage one: integrated general model strategy training.

[0032] First, a powerful end-to-end automatic driving general model is trained on a large-scale, diversified full driving data set. The model is composed of an encoder and a decision head, and after training, it has stable general planning ability and will serve as an immutable ability base for subsequent lightweight adaptation.

[0033] Phase two: mixed low-rank expert multi-domain adaptation fine-tuning. This phase is the core innovation of the present application, and the specific steps are as follows: 1. Freeze the general model parameters: load the general model trained in phase one, and completely freeze all its parameters, which remain unchanged throughout the fine-tuning process. This is done to completely retain the general knowledge of the model and fundamentally avoid catastrophic forgetting.

[0034] 2. Construct a plug-in low-rank adapter expert group: design a group of plug-in low-rank adapter experts corresponding to different driving domains. Each low-rank expert is an independent, parameter-lightweight network.

[0035] 3. Design a supervised domain-aware router: design and introduce a domain-aware router that is also lightweight and trainable. The role of this router is to determine the domain to which the input data belongs and activate the corresponding low-rank expert.

[0036] 4. Implement a fine-tuning process based on differential correction.

[0037] a. When data from a specific domain (e.g., "domain N") is input, the frozen encoder is responsible for extracting general, shared scene features. b. The trainable domain-aware router receives the features and dynamically selects and activates the corresponding low-rank expert N. c. The frozen decision head outputs a general, baseline planning trajectory based on the shared features.

[0038] d. The activated low-rank expert N receives the shared features and outputs a differential correction signal for "domain N".

[0039] e. Add or fuse the trajectory of the general model with the differential correction signal of the low-rank expert to generate the final, domain-optimized high-precision trajectory.

[0040] 5. Parameter-efficient loss update: Throughout the fine-tuning process, the calculated loss function is only used to update the parameters of the domain-aware router and all low-rank adapter experts. The parameters of the general model remain frozen.

[0041] In order to enable those skilled in the art to better understand the present application, the technical solutions of the present application will be described in detail below through specific embodiments. Please continue to read Figure 1 The embodiment of the present application provides an end-to-end automatic driving multi-domain adaptation fine-tuning method based on mixed low-rank experts, which includes an integrated general model strategy training phase and a mixed low-rank expert multi-domain adaptation fine-tuning phase.

[0042] Specifically, the embodiments of the present application mainly include model architecture construction, multi-domain adaptation problem and reasoning deployment three parts: First, the model architecture is constructed.

[0043] In this embodiment, an end-to-end autonomous driving network architecture as shown in Figure 1 is constructed, which includes a general model base in a frozen state and a multi-domain adaptation component in a trainable state.

[0044] 1. General model base.

[0045] The general model base is loaded with pre-trained weight parameters on full general data and is configured to keep the parameters frozen during the fine-tuning process. The base specifically includes: Image encoder: adopts ResNet-34 network structure. It is configured to receive the RGB image collected by the vehicle-mounted single camera, and after convolution operation, it outputs a high-dimensional shared image feature vector.

[0046] Measurement information encoder: adopts a multi-layer perception structure. Its input data includes vehicle speed scalar, discretized high-level navigation instructions and local target point coordinates, and the output data is a measurement feature vector.

[0047] Trajectory planner: adopts a gated recurrent unit (GRU) network. This module receives the fused features after splicing the image feature vector and the measurement feature vector, and predicts the vehicle trajectory point coordinates in the future preset time step in a self-recurrent manner. In the fine-tuning stage, the pre-training weight matrix W 0 of this module remains unchanged.

[0048] 2. Multi-domain adaptation component.

[0049] The multi-domain adaptation component is a trainable module in this embodiment, specifically including: Mixed low-rank expert group: contains N an independent plug-in low-rank adapter expert, respectively corresponding N to a target driving domain to be adapted. Each low-rank adapter expert does not contain a complete neural network layer, but is composed of two low-rank matrices A and B A is a dimension reduction matrix, B is a dimension increasing matrix, and the ranks r of the two are much smaller than the dimension of the pre-training weight matrix W 0. The low-rank adapter expert is configured to calculate a differential correction signal based on the input feature, and the calculation formula is .

[0050] ​Domain-aware router: adopts a multi-layer perceptron structure. The router receives shared image features extracted by the image encoder and outputs a probability vector with a dimension of N , which is used to dynamically activate the corresponding expert in the mixed low-rank expert group according to the feature distribution of the input data.

[0051] Phase II: Implementation process of multi-domain adaptive fine-tuning.

[0052] After building the above architecture, the following steps are performed to adapt the model to multiple driving domains: Step 1: Multi-domain data input and feature extraction.

[0053] The system receives training sample data from a specific driving domain. The training sample data is input into the frozen general model base, and shared feature vectors containing environmental semantics and vehicle states are extracted by the image encoder and measurement information encoder.

[0054] Step 2: Dynamic routing and expert activation. The domain-aware router receives the shared feature vectors and calculates the probability values of the feature vectors belonging to each pre-set driving domain. The router generates an activation signal based on the probability distribution results, dynamically selects the low-rank adapter expert from the mixed low-rank expert group that matches the domain to which the current input data belongs.

[0055] Step 3: Fusion decision of benchmark and correction. This step includes parallel execution of benchmark calculation and correction calculation: Benchmark trajectory generation: the frozen trajectory planner performs inference based on the shared feature vectors, using the pre-trained weight matrix W 0 to output a general benchmark trajectory.

[0056] Difference correction signal generation: the activated low-rank adapter expert receives the same shared feature vectors and calculates the difference correction signal for the current specific driving domain through low-rank matrix operation .

[0057] Final trajectory synthesis: element-wise addition of the general benchmark trajectory and the difference correction signal to obtain the optimized final planning trajectory.

[0058] Step 4: Parameter efficient update.

[0059] Calculate the loss function value between the final planning trajectory and the real expert trajectory. During backpropagation, lock all parameters of the general model base, and only update the parameters of the domain-aware router and the matrices A and B in the activated low-rank adapter expert.

[0060] Stage three: reasoning deployment.

[0061] After the fine-tuning training is completed, the model is deployed on the vehicle-mounted computing platform. During the driving of the vehicle, the vehicle-mounted sensor collects environmental data in real time; the domain-aware router automatically identifies the driving domain to which the current driving environment belongs according to the feature distribution of the real-time data, and activates the corresponding low-rank adapter expert; the activated expert outputs a differential correction signal in real time to compensate for the reference trajectory output by the general model base, thereby generating a planning trajectory adapted to the current driving domain.

[0062] The above-mentioned architecture and method are used to realize flexible adaptation of the model to the variable driving environment with a small amount of trainable parameters, while ensuring that the general driving ability is not degraded, and significantly reducing the calculation and storage cost of multi-domain adaptation.

[0063] According to the above technical solution, the embodiment of the present application provides an automatic driving model based on a hybrid low-rank expert and a multi-domain adaptation fine-tuning method, which includes two stages, namely: an integrated general model strategy training stage and a hybrid low-rank expert multi-domain adaptation fine-tuning stage; the integrated general model strategy training stage is used to train an automatic driving model based on a hybrid low-rank expert; the hybrid low-rank expert multi-domain adaptation fine-tuning stage is used to use a plug-in hybrid low-rank adapter expert to optimize the decision of the general model in a specific domain by outputting a differential correction signal. The present application provides a parameter-efficient model fine-tuning method, which uses a plug-in hybrid low-rank adapter expert to realize fast and flexible adaptation of a single general driving model to multiple different driving domains. Moreover, the present application realizes the "plug and play" efficient adaptation of a single general model to multiple downstream domains with very low calculation and storage cost, perfectly solving the pain points of traditional multi-domain adaptation methods.

[0064] Those skilled in the art can understand that the above-mentioned embodiments are specific embodiments for implementing the present application, and in actual application, various changes can be made in form and details without departing from the spirit and scope of the present application. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application, therefore the protection scope of the present application should be limited by the scope defined in the claims.

Claims

1. An autonomous driving model based on hybrid low-rank experts, characterized in that, include: A general model base in a frozen state and a multi-domain adaptation component in a trainable state; The general model base includes: an image encoder, a measurement information encoder, and a trajectory planner; The image encoder is used to receive RGB images captured by a vehicle-mounted single-view camera, and outputs a high-dimensional image feature vector after convolution operation; the measurement information encoder is used to output a measurement feature vector. The trajectory planner is used to receive the fused features of the image feature vector and the measurement feature vector, and predict the coordinates of the vehicle trajectory points in the future preset time step in an autoregressive manner. The multi-domain adaptation component includes: a hybrid low-rank expert group and a domain-aware router; The hybrid low-rank expert group is used to calculate the differential correction signal based on the input features; the domain-aware router is used to receive the shared image features extracted by the image encoder, output the probability vector, and dynamically activate the corresponding experts in the hybrid low-rank expert group according to the feature distribution of the input data.

2. The autonomous driving model based on hybrid low-rank experts according to claim 1, characterized in that, The general model base is loaded with weight parameters that have been pre-trained on the full set of general data and is configured to keep the parameters frozen during fine-tuning.

3. The autonomous driving model based on hybrid low-rank experts according to claim 1, characterized in that, The image encoder uses a ResNet-34 network structure.

4. The autonomous driving model based on hybrid low-rank experts according to claim 1, characterized in that, The measurement information encoder adopts a multilayer perceptron structure; The input data of the measurement information encoder includes vehicle speed scalar, discretized high-level navigation commands, and local target point coordinates.

5. The autonomous driving model based on hybrid low-rank experts according to claim 1, characterized in that, The trajectory planner employs a gated cyclic unit network.

6. The autonomous driving model based on hybrid low-rank experts according to claim 1, characterized in that, The hybrid low-rank expert group includes multiple independent plug-in low-rank adapter experts, each corresponding to a target driving domain to be adapted; the low-rank adapter experts are configured to calculate differential correction signals based on input features.

7. The autonomous driving model based on hybrid low-rank experts according to claim 1, characterized in that, The domain-aware router adopts a multi-layer sensor architecture.

8. A method for multi-domain adaptive fine-tuning of autonomous driving based on hybrid low-rank experts, wherein the method is implemented based on the autonomous driving model based on hybrid low-rank experts as described in any one of claims 1 to 7, characterized in that, This method includes: an integrated general model strategy training phase and a hybrid low-rank expert multi-domain adaptation fine-tuning phase; The integrated general model strategy training phase is used to train an autonomous driving model based on hybrid low-rank experts; The hybrid low-rank expert multi-domain adaptation fine-tuning stage is used to optimize the decision of the general model in a specific domain by using plug-in hybrid low-rank adapter experts and outputting differential correction signals.

9. The autonomous driving multi-domain adaptive fine-tuning method based on hybrid low-rank experts according to claim 8, characterized in that, The hybrid low-rank expert multi-domain adaptation fine-tuning stage includes: Freeze the general model parameters; Build an expert group for plug-in low-rank adapters; Design a supervised domain-aware router; Fine-tuning is performed based on differential correction; Efficient loss update of parameters.

10. The autonomous driving multi-domain adaptive fine-tuning method based on hybrid low-rank experts according to claim 8, characterized in that, Based on the differential correction, fine-tuning is performed, including: When data is input from a specific domain, the frozen encoder is responsible for extracting common, shared scene features; A trainable domain-aware router receives this feature and dynamically selects and activates the corresponding low-rank expert. The frozen decision head, based on shared features, outputs a general, benchmark planning trajectory; The activated low-rank expert receives shared features and outputs a differential correction signal for a specific domain; The trajectory of the general model is added or fused with the differential correction signal of the low-rank expert to generate the final high-precision trajectory optimized in a specific domain.

Images