Automatic driving trajectory planning and control method and system based on parameter space skill combination and storage medium

Abstract

The application provides an automatic driving trajectory planning and control method and system based on parameter space skill combination, and a storage medium, solves the problems of large calculation resource consumption, long research and development period, and easy loss of basic environment perception ability for long tail scene in existing automatic driving model fine tuning. The method takes the diffusion model of the frozen parameters after training as a basic model, introduces a LoRA module, trains for new skills, freezes the LoRA module parameters after training, adds them to the expert library as experts, dynamically selects and activates K LoRA modules to synthesize the execution parameters of the cross attention mechanism of the basic model by using the mixed expert model. When learning new skills, only a small amount of parameters need to be fine tuned, the calculation resource consumption and reasoning delay of the vehicle end are reduced, the model is adapted to the complex traffic environment, a safe and high-quality trajectory is output, the lateral and longitudinal control instructions of the vehicle are generated based on the trajectory, and precise control of the automatic driving vehicle can be realized.

Description

Technical Field

[0001] This disclosure relates to the technical fields of autonomous driving and generative artificial intelligence, and in particular to an autonomous driving trajectory planning and control method, system and storage medium based on parameter space skill combination. Background Technology

[0002] As autonomous driving expands from closed highway scenarios to complex open urban areas, the high dynamism and uncertainty of the environment give rise to long-tail scenarios (long-tail scenarios refer to edge cases with low probability of occurrence in specific domains (such as autonomous driving), which are usually difficult to predict and easily missed in datasets). In-vehicle systems not only need to process complex multimodal perception information in real time, but also need to output safe and accurate trajectory planning instructions under extremely low latency conditions. Therefore, how to balance complex scene data processing and trajectory prediction accuracy with limited computing resources on the vehicle is the core challenge currently facing advanced autonomous driving. Currently, the Transformer-based conditional diffusion architecture has demonstrated outstanding performance in trajectory generation tasks. By leveraging the spatiotemporal sequence modeling capabilities of Transformer and integrating the advantages of diffusion models in multimodal action distribution generation, it can effectively predict high-quality driving trajectories in dynamic interactive environments. However, when learning new scene features, many current solutions adopt a full-scale fine-tuning training mode, which has obvious limitations: on the one hand, if multi-branch networks or multiple large models are used to deal with different scenarios, the computing resources and memory consumption required by the vehicle during inference will increase exponentially, which cannot meet the millisecond-level low-latency control requirements of the underlying actuators; on the other hand, existing conventional parameter efficient fine-tuning techniques (such as static or single-load LoRA) are mostly used for driving style switching or static adaptation across urban areas, which cannot meet the stringent requirements of real-time and dynamic combination of multiple driving skills under the ever-changing road conditions of autonomous driving. Summary of the Invention

[0003] To address the aforementioned problems in existing technologies, this disclosure proposes an autonomous driving trajectory planning and control method based on parameter space skill combination. This method utilizes linear combination of LoRA modules in a diffusion model decoder and introduces a hybrid expert model (MoE) during the LoRA module selection process. By modifying only a small number of parameters, it can rapidly adjust to different complex road conditions, generating a noise-removed trajectory. Based on this noise-removed trajectory, lateral and longitudinal control commands for the vehicle are generated, enabling precise control of the autonomous vehicle.

[0004] Firstly, this disclosure proposes an autonomous driving trajectory planning and control method based on parameter space skill combinations. The steps include: using a diffusion model with frozen parameters after training as a base model; for each new skill learning, introducing a LoRA module and the cross-attention mechanism of the base model in parallel, adjusting the parameters of the LoRA module for training, and adding the LoRA module with frozen parameters after training as an expert to the expert library of a hybrid expert network; after the expert library is constructed, using the hybrid expert model to dynamically select and activate K LoRA modules to synthesize the execution parameters of the cross-attention mechanism of the base model, and using the base model with the synthesized execution parameters as the trajectory planning model based on skill combinations; using the trained trajectory planning model based on skill combinations, outputting the planned trajectory based on the real-time perceived environmental features, and realizing driving control based on the planned trajectory.

[0005] In one embodiment of the above technical solution, the rank of the dimensionality reduction matrix and the dimensionality increase matrix of the LoRA module is preferably 8.

[0006] In one embodiment of the above technical solution, the execution parameters of the cross-attention mechanism of the basic model are dynamically selected and activated using a hybrid expert model. The steps include: converting the feature vector output by the multilayer perceptron into a single vector. As input to the hybrid expert network, the gated decision network linearly maps it to a weight matrix of a dimension and a frozen cross-attention mechanism. Consistent vectors : , Here is the weight matrix of the gated decision network. , The encoder of the basic model outputs keys and values ​​based on environmental features C. The features extracted by the multilayer perceptron based on noisy trajectories are used as queries; normalization is performed using the Softmax function. , This represents the probability distribution vector of LoRA expert modules being activated in each specific scenario under the current traffic condition; a Top-K sparse routing strategy is adopted to select the K LoRA weights with the highest probabilities for parameter synthesis. fusion weight parameters , The weight matrix represents the frozen cross-attention mechanism. This represents the LoRA weight matrix corresponding to the selected k-th expert; the weight parameters of the cross-attention mechanism of the base model are updated using the fused weight parameters.

[0007] In one embodiment of the above technical solution, the trajectory planning model based on skill combinations includes calculating a loss function during training: In the formula: This indicates the environmental information provided by multiple currently active LoRA expert modules. and time step Below, the predicted trajectory with noise removed is inferred from the base model; This represents real safe driving trajectories derived from a globally mixed training dataset. This represents the diffusion time step for all samples in the globally mixed training dataset. Characteristics of surrounding traffic environment and the corresponding noise-adding trajectory The mathematical expectation of the joint distribution is obtained; based on the loss, the parameters of the gated decision network in the hybrid expert network are updated according to the gradient descent method.

[0008] Secondly, this disclosure proposes an autonomous driving trajectory planning and control system based on parameter space skill combinations, comprising: a basic model construction module, configured to use a pre-trained diffusion model with frozen parameters as the basic model; an expert library construction module, configured to, for each new skill learning, introduce a LoRA module and the cross-attention mechanism of the basic model in parallel, adjust the parameters of the LoRA module for training, and add the LoRA module with frozen parameters after training as an expert to the expert library of the hybrid expert network; a skill combination-based trajectory planning model construction module, configured to, after the expert library is constructed, use the hybrid expert model to dynamically select and activate K LoRA modules to synthesize the execution parameters of the cross-attention mechanism of the basic model, and use the basic model using the synthesized execution parameters as the skill combination-based trajectory planning model; and an application module, configured to, with the help of the trained skill combination-based trajectory planning model, output the planned trajectory based on the real-time perceived environmental features, and realize driving control based on the planned trajectory.

[0009] Thirdly, this disclosure proposes a computer-readable storage medium storing a computer program that can be loaded by a processor and execute any of the methods or systems of this disclosure.

[0010] The beneficial technical effects of this disclosure are as follows: Through phased training, skills for different complex road conditions are learned without compromising basic environmental perception capabilities, achieving physical decoupling of complex driving skills. In each skill learning process, only a very small number of parameters need to be fine-tuned to acquire the skill to handle a particular complex road condition, significantly reducing the computational resource requirements for skill learning. A hybrid expert network with experts possessing different skills is integrated into the diffusion model. Through training, different experts are dynamically selected to participate in decision-making, enabling the final trained model to accurately and efficiently adapt to complex and changing real-world traffic conditions, outputting high-quality trajectories with safety. Deploying the trained model on the vehicle enables precise control of autonomous vehicles. Using this solution, even when the model is deployed on the vehicle, learning new skills does not require significant computational resources, lowering the hardware threshold for vehicles and accelerating mass production. Furthermore, since data does not leave the vehicle, privacy protection is enhanced, and dependence on the cloud is reduced. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of a model in one implementation method. Detailed Implementation

[0013] Terminology Explanation: LoRA (Low-Rank Adaptation) is an efficient technique for fine-tuning large pre-trained models, such as large language models.

[0014] MoE (Mixture of Experts) is a neural network architecture design that breaks down a large problem into multiple specialized "small expert" models, with a "gating network" determining which expert(s) should handle each problem.

[0015] Skills: refers to driving strategies, that is, the distribution patterns of trajectory generation data in response to specific complex traffic conditions, such as emergency avoidance, lane changing and overtaking, and unprotected left turns.

[0016] When fine-tuning existing autonomous driving models to cope with complex road conditions, as mentioned in the background technology, all parameters usually need to be updated. This operation not only consumes a lot of computing power and prolongs the development cycle, but also easily causes the model to lose its original basic environmental perception ability in long-tail scenarios.

[0017] To address this, this disclosure proposes an autonomous driving trajectory planning and control method based on parameter space skill combinations. This method freezes the weights of a pre-trained Transformer-based conditional diffusion model, and then, for each driving skill, introduces a LoRA module into the model, linearly paralleling it with the attention mechanism of the original decoder. The trained LoRA module serves as the expert skill database for constructing the expert skill library. In subsequent stages, only the weights of the LoRA module need to be adjusted. Fine-tuning and updates are performed. This significantly reduces the size of trainable parameters, thereby greatly improving computational efficiency and reducing computational resource overhead. During actual vehicle operation, the system employs a Top-K sparse routing strategy, dynamically selecting and activating the weights of the most suitable LoRA modules based on real-time road conditions. The final execution parameters are synthesized. This scheme is based on the dynamic combination of parameters in the parameter space, without the need for fine-tuning all parameters, thus avoiding the high cost of full-scale fine-tuning, improving computational efficiency, and achieving efficient, low-latency generation of autonomous driving trajectories under massive data and adaptive capabilities under complex working conditions. The following description, in conjunction with the accompanying drawings, clearly and completely describes how the technical solution of this case is implemented. Obviously, the described embodiments are only a part of the embodiments of this case, and not all of them. Based on the embodiments in this case, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application.

[0018] First, a Transformer-based conditional diffusion model is constructed, consisting of an encoder and a decoder. The encoder includes a multilayer perceptron and a self-attention mechanism, while the decoder integrates key components such as a multilayer perceptron (MLP), a self-attention mechanism, and a feedforward neural network (FFN). The Transformer-based conditional diffusion model is then used to analyze noisy trajectories. Perform iterative denoising and output the denoised trajectory. .

[0019] To avoid unconstrained, blind generation, environmental conditions are used as constraints on the model. Specifically, the encoder is configured to process environmental information. As input, query the obtained sequence features ,key ,value With the encoder's weight matrix Multiply to obtain the encoded query vector. Key vector Sum value vector , , , Then, the encoder extracts... and Injecting a decoder is key to enabling condition-driven trajectory generation in the model.

[0020] The environmental information includes static environment, dynamic targets, environmental conditions, semantics, and prior knowledge. Static environment includes road structure and static objects, such as lane markings (lane type, lane curvature, lane width, number of lanes), road boundaries (curb, guardrail, median, shoulder, road edge), road markings (straight / turn / U-turn arrows, bus-only / time-limited text, stop lines, zebra crossings, guide lines), road topology (intersection structure, ramp entrances and exits, lane merging / branching, roundabouts), road surface conditions (dry / slippery / waterlogged / snowy / icy, potholes, speed bumps), traffic facilities, infrastructure, buildings, and temporary facilities. Dynamic targets include motor vehicles, non-motor vehicles, pedestrians, and animals. Environmental conditions include lighting, weather, time, and temperature. Semantics and priors include map information (lane-level navigation, speed limit areas, school / hospital areas), traffic rules (speed limits, no-entry rules, yield rules, and traffic light timing priors for the current road segment), regional attributes (school areas, residential areas, commercial areas, construction areas, highways / urban / rural roads), and historical experience (common congestion times, accident-prone areas, and typical driving behavior patterns).

[0021] In the decoder, the noisy trajectory of the current diffusion step is... diffusion time step and the key vector output by the encoder. Sum value vector As input. Since the noise removal process of the diffusion model is a time-step process... During the change process, the model needs to simultaneously perceive the current trajectory state and the noise removal stage. Therefore, the input information first enters the MLP layer for feature extraction, obtaining trajectory features and time step embedding features, which are then fused into a feature vector representing the current integrated noisy state. Then the feature vectors The input is fed into the cross-attention layer as the query object of the cross-attention mechanism. It interacts globally with the environmental features extracted by the encoder and is fused together. Then, the nonlinear mapping of the FFN layer completes the deep filtering and refinement of the core features. Finally, the noise-removed result of the current time step is predicted and output.

[0022] See Figure 1 The weight matrix learned during the training phase by the cross-attention mechanism inside the decoder is denoted as... Weight matrix From the query weight matrix Key weight matrix and value weight matrix The decoder receives the key vector output by the encoder. Sum value vector Multiply by the key weight matrix in the cross-attention mechanism respectively. Sum weight matrix Perform a linear mapping to generate a key vector. Sum value vector The expression is: , . The feature vector With query weight matrix Multiply and perform a linear mapping to generate the query vector required by the attention mechanism. ,Right now .

[0023] A well-trained Transformer-based conditional diffusion model can be used for trajectory prediction and generation in common traffic scenarios. However, when learning new scene features, due to the large number of parameters in the Transformer decoder, full fine-tuning not only incurs huge memory and computational overhead, but may also cause the model to lose the generalization and environmental awareness capabilities learned during the pre-training stage.

[0024] To address this, this disclosure presents a method for adapting a model to complex traffic environments. This method uses a pre-trained diffusion model with frozen parameters as the base model, introduces a LoRA module, trains for new skills, freezes the LoRA module parameters after training, adds them to an expert database as experts, and uses a hybrid expert model to dynamically select and activate K LoRA modules to synthesize the execution parameters of the base model's cross-attention mechanism. When learning new skills, this invention only requires fine-tuning a small number of parameters, reducing the computational resource consumption and inference latency on the vehicle side, enabling the model to adapt to complex traffic environments and output safe, high-quality trajectories. The following description uses the parallel connection of the introduced LoRA module and the decoder's cross-attention mechanism as an example.

[0025] (a) Conditional diffusion model with LoRA module Existing technologies that incorporate LoRA into cross-attention mechanisms are primarily limited to personalized fine-tuning of driving styles or static adaptation across urban areas. The core focus of this invention, however, is how advanced autonomous driving systems can achieve dynamic self-adaptation of driving strategies and completely avoid "catastrophic forgetting" when facing complex, long-tail scenarios.

[0026] Since physical environment information is relatively fixed, using conventional fine-tuning to learn new features of complex long-tail scenarios in autonomous driving can easily damage the model's already acquired basic capabilities. Therefore, this invention freezes the model parameters after the Transformer-based conditional diffusion model is trained. These frozen parameters remain unchanged during the parameter space skill combination and fine-tuning phase for specific scenarios, including the encoder's weight parameters. Weight parameters of the cross-attention mechanism with decoder This is to prevent the basic environmental perception capabilities from being compromised during subsequent training and to prevent the model from suffering catastrophic forgetting.

[0027] Next, the diffusion model with frozen parameters after training is used as the base model. For a new skill in a specific scenario, an independent LoRA module is introduced into the base model, and the LoRA module is linearly combined with the cross-attention mechanism. This design achieves physical decoupling of complex driving skills without compromising basic environmental perception and underlying control capabilities. By fine-tuning a small number of parameters, the model can adapt to a specific scenario and learn a new skill.

[0028] The specific scenarios cover various situations faced by autonomous driving in complex open roads, including high-frequency interaction and long-tail conditions, including but not limited to: (1) Sudden danger avoidance scenarios: emergency braking and avoidance scenarios of vehicles / pedestrians in front, and emergency avoidance scenarios of sudden crossing of blind spots ("ghost peek"); (2) Complex intersection scenarios: unprotected left / right turn scenarios, game-like passage scenarios at intersections without traffic lights, and merging and exiting scenarios at complex roundabouts; (3) Dynamic interaction scenarios: highly dynamic lane changing and overtaking scenarios, smooth following scenarios to prevent cutting in on congested road sections, and two-way passing scenarios on narrow road sections; (4) Rule compliance and compliance scenarios: compliant detour scenarios on construction sections (such as cone placement areas), and merging and exiting scenarios on highway ramps.

[0029] The base model incorporating the LoRA module is trained using a fine-tuning method, adjusting only the parameters of the LoRA module. The weight parameters of the LoRA module are denoted as... The LoRA module approximates the parameter update by introducing two trainable low-rank matrices. Specifically, constructing a dimensionality reduction matrix. With an upgraded matrix , where matrix dimension and The rank is uniquely determined by the hidden layer input and output dimensions of the corresponding linear layer in the Transformer decoder (e.g., both are set to 256); The size of the training data for the new scenario is set within the integer range of [4, 64]. Represents the real number field. Preferably, in small-sample fine-tuning scenarios, the rank r is set to 8. If the rank is too large, it can easily lead to overfitting in the model with limited data, thus causing model performance degradation; while limiting the rank r to 8 can accurately approximate the update amount of the original weights under conditions of very low parameter count, achieving the best fine-tuning effect. During parameter initialization, the matrix... Initialize using random Gaussian matrix It is then initialized as a zero matrix. This initialization strategy ensures that at the start of training... This ensures that the output of the model in its initial state remains consistent with the original pre-trained model, maintaining stability in the early stages of training.

[0030] It should be clarified that this invention introduces a separate LoRA module for a single, specific scenario. During training, a model consisting of a LoRA module and a base model is trained. Specifically, for a LoRA module corresponding to a particular skill, its parameters... Update the matrix according to the corresponding query Key update matrix and value update matrix constitute.

[0031] For a specific scenario, such as emergency avoidance, the decoder obtains... , and After processing them simultaneously through the decoder's cross-attention mechanism and the newly added LoRA module, the query vector is linearly superimposed. Key vector Sum value vector Its mathematical expression is: , , .Will , , After the core features are deeply filtered and refined by the FFN layer, the noise-removed trajectory at the current time step is output. This trajectory is the safe driving path planned by the autonomous vehicle in the current specific scenario.

[0032] Subsequently, the vehicle control module performs vehicle cooperative control: extracting physical spatial information such as position, speed, and heading angle from the noise-removed trajectory and inputting it to the lower-level controller. The lower-level controller uses this noise-removed trajectory as the sole reference benchmark, and through dynamic analysis, transforms it into joint control commands including steering wheel angle, brake pedal opening, and drive motor torque. These commands are then used to coordinate and schedule various vehicle subsystems, driving the autonomous vehicle to strictly execute the path and speed planned based on the noise-removed trajectory, thus completing the intended driving maneuvers.

[0033] By using the above method, simply updating the parameters of the LoRA module can guide the model to quickly adapt to a complex traffic scenario. This significantly reduces the computational resource requirements for training while ensuring efficient convergence of the conditional diffusion model in trajectory prediction tasks.

[0034] (II) Dynamic parameter reconstruction and adaptive fitting based on MoE routing Based on the above analysis, this invention, for each specific traffic scenario, introduces a new scenario into the base model and trains a LoRA module. Under this premise, when multiple specific scenarios exist, multiple independent LoRA modules are introduced. Each independent LoRA module collaborates with the base model to process a specific traffic scenario, thus raising the question of how to adaptively load specific LoRA modules for different scenarios. To address this, this invention innovatively introduces MoE (Modal of Evolution) to adapt to complex and ever-changing real-world traffic scenarios. Specifically, after training, a LoRA module for a specific scenario will be obtained. Since it corresponds to a driving strategy or skill, it will be added as an expert to the expert library of the hybrid expert model. LoRA modules corresponding to multiple specific scenarios, because the parameters of each expert in this skill library are independent and do not interfere with each other, fundamentally avoid the catastrophic forgetting problem in the multi-skill learning process.

[0035] After the expert database is constructed, a hybrid expert network (MoE) based on the LoRA expert database is obtained. The hybrid expert model dynamically selects and activates K LoRA modules to synthesize the execution parameters of the cross-attention mechanism of the base model. Specifically, the gating network of the hybrid expert network achieves millisecond-level dynamic skill routing. Based on real-time perceived environmental information, the probability of each expert and the weights of the corresponding activated LoRA modules are dynamically calculated and assigned. After weighted combination, it is then linearly merged with the decoder's base weight matrix in real time. The parameters of the i-th LoRA module are denoted as... For parameters of multiple LoRA modules in the expert database, please refer to [link / reference]. Figure 1 illustrative Each Update the matrix according to the corresponding query Key update matrix and value update matrix constitute, This represents the total number of experts in the expert database.

[0036] like Figure 1 The skill-based trajectory planning model shown will determine the noisy trajectory of the current diffusion step. diffusion time step and environmental information extracted by the encoder. As input. Environmental information. The encoder processes the key vector. Sum value vector Because the denoising process of the diffusion model varies with the time step... The trajectory is dynamically changing, and the model needs to simultaneously perceive the current trajectory state and the denoising stage it is in. Therefore, the noisy trajectory and time step first enter the MLP layer of the decoder for feature extraction and fusion to obtain a query feature vector representing the overall noisy state at the current moment. The training scenario for the skill combination-based trajectory planning model is the same as that for the conditional diffusion model that incorporates the LoRA module; the training samples can be the same or different.

[0037] Subsequently, the gated decision network of the hybrid expert network receives... , and As input, the gating decision network first linearly maps the input vector to a weight matrix of a dimension and a cross-attention mechanism frozen by the decoder. A consistent vector Z. This vector Z represents the unnormalized activation score of experts for each specific scenario in the expert database, and its mathematical expression is: , This is the weight matrix of the gated decision network; then, it is normalized using the Softmax function. , This represents the probability distribution vector of LoRA expert modules being activated under the current traffic conditions for each specific scenario. The activation probabilities of each expert are obtained... Then, the weights of all LoRA experts are weighted according to this probability to form the instantaneous forward inference weights.

[0038] During training, the weight matrix of the gated decision network The initial values ​​are random. The goal of training is to calculate the loss value of the diffusion model for noise removal by comparing the predicted, noise-removed trajectory of the model's final output with the labels of the true trajectories. Its loss function is defined as follows.

[0039]

[0040] In the above formula, Indicates the encoder at time step Extracted features of the surrounding traffic environment (including surrounding vehicles, lane topology, etc.). This represents the real safe driving trajectory (i.e., the target noiseless label) derived from the global mixed training dataset. This indicates the environmental information provided by multiple currently active LoRA expert modules. and time step Below, based on noisy trajectories The inferred noise-removed trajectory; This represents the trajectory prediction error (loss value) of the autonomous driving diffusion model during the end-to-end training process. This error reflects the deviation between the noise-removed trajectory generated by the fused expert skills and the real safe trajectory. This represents the diffusion time step for all samples in the globally mixed training dataset. Surrounding traffic information and the corresponding noise-adding trajectory The mathematical expectation of the joint distribution.

[0041] Using the backpropagation algorithm, calculate the loss function pair The gradient is then updated according to the gradient descent method. Parameters: , The learning rate, representing the parameter update of the gating decision network, determines the step size for adjusting the routing policy. The typical range is: ; This represents the gradient vector of the prediction error after noise removal with respect to the gating weight matrix. This gradient indicates to the gating network how to optimize the probability distribution of each driving skill so that the final generated autonomous driving trajectory best matches the real intention of safe driving.

[0042] The above-described loss calculation, backpropagation, and parameter update process iterates continuously until the model meets preset convergence conditions. These convergence conditions include, but are not limited to, the loss value. Decrease to less than the preset minimum error threshold (This threshold is set empirically by those skilled in the art based on the specific dataset size and environmental complexity), or the number of iterations in the training system reaches the preset maximum threshold, or during multiple consecutive iterations... The rate of change approaches zero. Once the convergence condition is met, the gated weight matrix... The parameters will be fixed and will stop updating, marking the completion of the gating decision network training.

[0043] Furthermore, to ensure the inference efficiency of the autonomous driving system with extremely low latency, this invention employs a Top-K sparse routing strategy, selecting only the top K values ​​from P to participate in the final parameter synthesis. The instantaneous execution weight at the current moment. The calculation formula is defined as follows.

[0044]

[0045] in, The weight matrix representing the cross-attention mechanism of the frozen decoder; This represents the number of expert modules that best match the current road conditions, selected using the Top-K strategy; its value is preset. Indicates the selected first Normalized routing weights for core expert skills; Indicates the selected first The LoRA weight matrix corresponding to each expert; This represents the instantaneous execution weight matrix obtained through dynamic weighted combination, used for generating the trajectory of the autonomous vehicle at the current moment. .

[0046] Merged query vector Key vector Sum value vector The calculation is as follows.

[0047]

[0048]

[0049]

[0050] After fusion , , The input is fed to the FFN layer of the decoder, and the output is the noise-removed trajectory at the current time step. This trajectory is the safe driving path planned by the autonomous vehicle under the current complex conditions.

[0051] Based on the final noise-removed trajectory, the vehicle control module performs cooperative vehicle control. Specifically, it extracts physical spatial information such as position, speed, and heading angle from the noise-removed trajectory and inputs it to the lower-level controller. The lower-level controller uses this noise-removed trajectory as the sole reference and, through dynamic analysis, transforms it into joint control commands including steering wheel angle, brake pedal opening, and drive motor torque. These commands are then used to coordinate and schedule various vehicle subsystems, driving the autonomous vehicle to strictly execute the path and speed planned by the noise-removed trajectory, thus completing the intended driving maneuvers.

[0052] As can be seen from the above, this solution does not require direct fusion of a massive global weight matrix. Instead, it enables the model to effectively handle complex driving scenarios by fine-tuning only a small number of modular parameters. Furthermore, to address the complex and ever-changing road conditions for autonomous driving, the system introduces a hybrid expert routing mechanism. This mechanism dynamically activates the most suitable LoRA weight combination based on real-time environmental input. This design ensures that the model parameters can accurately and adaptively fit the current real-world traffic conditions. Deploying the pre-trained hybrid expert network (MoE) based on the LoRA expert database on the vehicle significantly reduces the computational resource consumption and inference latency when handling complex driving scenarios. This enables efficient generation of high-quality driving trajectories and high-real-time closed-loop vehicle control, significantly improving driving safety.

[0053] (III) The complete multi-step training process of the model Step S1: Build the basic model.

[0054] By utilizing a massive general-purpose autonomous driving trajectory dataset, a Transformer-based conditional diffusion model is globally pre-trained, enabling it to possess basic environmental perception and trajectory generation capabilities. This provides a fundamental control reference for the smooth daily driving of the vehicle's underlying actuators. After training, the parameters of the diffusion model are frozen, including the encoder parameters responsible for extracting environmental features and the decoder parameters.

[0055] Step S2: Build an expert database.

[0056] For different complex traffic scenarios (such as emergency avoidance, lane changing and overtaking, and unprotected left turns involving the joint coordination of steering and braking systems), corresponding scenario subsets are constructed. By training the model using a subset of scenario datasets, the model acquires a skill or driving strategy.

[0057] For a given scenario, a LoRA module is introduced into the base model. One implementation involves setting up several LoRA modules in the decoder bypass, activating the corresponding LoRA module for the i-th scenario. After training of the LoRA module, the first LoRA module... The parameters of each LoRA module are frozen and they do not participate in the learning and training of other scenarios. This process is repeated to complete the training for all scenarios, thereby obtaining several independent LoRA modules. Each LoRA module is added as an expert to the expert library, and each LoRA module is then frozen. Each expert in this expert library essentially encapsulates a vehicle cooperative control strategy for a specific type of scenario.

[0058] Step S3: Construct a trajectory planning model module based on skill combinations.

[0059] At this point, a global training dataset containing various complex mixed operating conditions is introduced to train the MoE gating decision network, enabling it to learn how to dynamically allocate and fuse control commands from different vehicle subsystems under varying road conditions. The trajectory of noise is then considered. Input network to obtain features Through the aforementioned and Calculate the activation probability of each expert. At this point, the instantaneous weights of the forward propagation are weighted and synthesized from the LoRA modules.

[0060] Step S4: Train and apply the skill combination-based trajectory planning model.

[0061] The system compares the noise-removed predicted trajectory generated by the above-mentioned fusion weights with the true trajectory labels. Compare and calculate the diffusion denoising loss. Using the backpropagation algorithm, the loss function is calculated with respect to the gate weight matrix. The gradient is calculated and updated using the gradient descent method. The parameters. This iterative process continues until the loss value is reached. Convergence conditions.

[0062] Different driving skills may be trained using the aforementioned framework simultaneously. However, through the decoupled step-by-step training described above, since the LoRA module corresponding to each driving skill is frozen after training, gradient conflicts during joint training of different driving skills can be effectively avoided, ensuring the purity of each skill and endowing the gating network with precise routing capabilities. The trained model will be deployed as a core algorithm module in the driving control system of road vehicles. Through the real-time output of high-quality denoised trajectories, it will directly guide the vehicle's lower-level controller to perform joint and coordinated control of multiple subsystems such as the drive motor, braking system, and steering system. The joint training involves embedding the MoE (Modal of the Engine) in the decoder of the diffusion model to learn and dynamically select experts to participate in decision-making.

[0063] During inference, the trained trajectory planning model based on skill combinations is used directly to output the planned trajectory based on the real-time perceived environmental features, and driving control can be achieved based on the planned trajectory.

[0064] In summary, the present invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the present invention.

[0065] For example, an autonomous driving trajectory planning and control method based on parameter space skill combinations includes the following steps: using a diffusion model with frozen parameters after training as a base model; for each new skill learned, introducing a LoRA module in parallel with the cross-attention mechanism of the base model, adjusting the parameters of the LoRA module for training, and adding the LoRA module with frozen parameters after training as an expert to the expert library of a hybrid expert network; after the expert library is constructed, using the hybrid expert model to dynamically select and activate K LoRA modules to synthesize the execution parameters of the cross-attention mechanism of the base model, and using the base model with the synthesized execution parameters as the trajectory planning model based on skill combinations; using the trained trajectory planning model based on skill combinations, outputting the planned trajectory according to the real-time perceived environmental features, and realizing driving control based on the planned trajectory.

[0066] For example, an autonomous driving trajectory planning and control system based on parameter space skill combinations includes: a basic model construction module, configured to use a pre-trained diffusion model with frozen parameters as the basic model; an expert library construction module, configured to, for each new skill learning, introduce a LoRA module in parallel with the cross-attention mechanism of the basic model, adjust the parameters of the LoRA module for training, and add the LoRA module with frozen parameters after training as an expert to the expert library of a hybrid expert network; a skill combination-based trajectory planning model construction module, configured to, after the expert library is constructed, use a hybrid expert model to dynamically select and activate K LoRA modules to synthesize the execution parameters of the cross-attention mechanism of the basic model, and use the basic model using the synthesized execution parameters as the skill combination-based trajectory planning model; and an application module, configured to, with the help of the trained skill combination-based trajectory planning model, output a planned trajectory based on real-time perceived environmental features, and realize driving control based on the planned trajectory.

[0067] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0068] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0069] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0070] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. An automatic driving trajectory planning and control method based on parameter space skill combination, characterized by the steps of include: The diffusion model with frozen parameters after training is used as the base model. The diffusion model includes an encoder and a decoder. The decoder includes a multilayer perceptron, a cross-attention mechanism, and a feedforward neural network. For each new skill learned, a LoRA module is introduced and connected in parallel with the cross-attention mechanism of the base model. The parameters of the LoRA module are adjusted for training. The LoRA module with its parameters frozen after training is added to the expert library of the hybrid expert network as an expert, where the skill is a driving strategy. After the expert database is constructed, a hybrid expert network is integrated into the diffusion model. The hybrid expert model dynamically selects and activates K LoRA modules to synthesize the execution parameters of the base model's cross-attention mechanism. The base model using these synthesized execution parameters is then used as a trajectory planning model based on skill combinations. Training is performed using the same scenario as the diffusion model with the LoRA modules. The training steps include: The noisy trajectory of the current diffusion step diffusion time step and environmental information extracted by the encoder. As input to the diffusion model, the feature vector output by the multilayer perceptron is obtained. This input is used as the input to a hybrid expert network, whose gated decision network linearly maps this input to a weight matrix of a dimension and a frozen cross-attention mechanism. Consistent vectors : , Here is the weight matrix of the gated decision network. , The encoder of the basic model outputs keys and values ​​based on environmental features C. Features extracted by the multilayer perceptron based on noise trajectories and used as queries; Normalization is performed using the Softmax function. , This represents the probability distribution vector of LoRA expert modules being activated in each specific scenario under the current traffic conditions; A Top-K sparse routing strategy is adopted, selecting the K LoRA weights with the highest probability for parameter synthesis. fusion weight parameters , The weight matrix represents the frozen cross-attention mechanism. This represents the LoRA weight matrix corresponding to the selected k-th expert; The weight parameters of the cross-attention mechanism in the base model are updated using the fused weight parameters. The query vector, key vector, and value vector obtained based on the updated weights are then input into the feedforward neural network of the decoder of the diffusion model, which outputs the denoised trajectory at the current time step. The current loss is calculated based on the denoised trajectory at the current time step. In the formula: This indicates the environmental information provided by multiple currently active LoRA expert modules. and time step The predicted trajectory with noise removed is inferred from the base model; This represents real safe driving trajectories derived from a globally mixed training dataset. This represents the diffusion time step for all samples in the globally mixed training dataset. Characteristics of surrounding traffic environment and the corresponding noise trajectory The mathematical expectation of the joint distribution; Based on the loss function and using gradient descent, the weight matrix of the gated decision network in the hybrid expert network is analyzed. Update; however, once the convergence condition is met, the weight matrix of the gated decision network will be updated. Fix and stop updating to complete training; Using a skill-based trajectory planning model that has been trained, the system outputs a planned trajectory based on real-time perceived environmental features, and driving control can be achieved based on the planned trajectory.

2. The autonomous driving trajectory planning and control method according to claim 1, characterized in that, The preferred rank of the reduced-dimensional matrix and the increased-dimensional matrix of the LoRA module is 8.

3. An autonomous driving trajectory planning and control system based on parameter space skill combination, characterized in that, include: The base model module is configured to use a pre-trained diffusion model with frozen parameters as the base model. The diffusion model includes an encoder and a decoder. The decoder includes a multilayer perceptron, a cross-attention mechanism, and a feedforward neural network. The expert library module is configured to learn each new skill. A LoRA module is introduced and connected in parallel with the cross-attention mechanism of the base model. The parameters of the LoRA module are adjusted for training. The LoRA module with its parameters frozen after training is added to the expert library of the hybrid expert network as an expert. A skill-based trajectory planning model module is constructed and configured to, after the expert database is built, integrate a hybrid expert network into the diffusion model. The hybrid expert model dynamically selects and activates K LoRA modules to synthesize the execution parameters of the base model's cross-attention mechanism. The base model using these synthesized execution parameters is then used as the skill-based trajectory planning model. Training is performed using the same scenario as the diffusion model incorporating the LoRA module. The training steps include: The noisy trajectory of the current diffusion step diffusion time step and environmental information extracted by the encoder. As input to the diffusion model, the feature vector output by the multilayer perceptron is obtained. This input is used as the input to a hybrid expert network, whose gated decision network linearly maps this input to a weight matrix of a dimension and a frozen cross-attention mechanism. Consistent vectors : , Here is the weight matrix of the gated decision network. , The encoder of the basic model outputs keys and values ​​based on environmental features C. Features extracted by the multilayer perceptron based on noise trajectories and used as queries; Normalization is performed using the Softmax function. , This represents the probability distribution vector of LoRA expert modules being activated in each specific scenario under the current traffic conditions; A Top-K sparse routing strategy is adopted, selecting the K LoRA weights with the highest probability for parameter synthesis. fusion weight parameters , The weight matrix represents the frozen cross-attention mechanism. This represents the LoRA weight matrix corresponding to the selected k-th expert; The weight parameters of the cross-attention mechanism in the base model are updated using the fused weight parameters. The query vector, key vector, and value vector obtained based on the updated weights are then input into the feedforward neural network of the decoder of the diffusion model, which outputs the denoised trajectory at the current time step. The current loss is calculated based on the denoised trajectory at the current time step. In the formula: This indicates the environmental information provided by multiple currently active LoRA expert modules. and time step The predicted trajectory with noise removed is inferred from the base model; This represents real safe driving trajectories derived from a globally mixed training dataset. This represents the diffusion time step for all samples in the globally mixed training dataset. Characteristics of surrounding traffic environment and the corresponding noise trajectory The mathematical expectation of the joint distribution; Based on the loss function and using gradient descent, the weight matrix of the gated decision network in the hybrid expert network is analyzed. Update; however, once the convergence condition is met, the weight matrix of the gated decision network will be updated. Fix and stop updating to complete training; The application module is configured to use a trained trajectory planning model based on skill combinations to output a planned trajectory based on real-time perceived environmental features, and to achieve driving control based on the planned trajectory.

4. A computer-readable storage medium, characterized in that: The computer program is stored that can be loaded by a processor and executed as described in any one of claims 1 to 2.

Images