An end-to-end automatic driving decision method and device based on a hybrid expert architecture

Abstract

The application discloses an end-to-end automatic driving decision method and equipment based on a mixed expert architecture, and belongs to the field of automatic driving deep learning multi-modal perception decision. In view of the weak scene adaptation ability of the existing end-to-end single backbone network, the conventional MoE is prone to modal averaging and expert collapse, fine-tuning is prone to losing pre-training knowledge, perception decision is fragmented, and the overall robustness is poor. The application builds a layered decoupling MoE perception architecture and a double perception MoE decision architecture, and fuses a LoRAMoE plug-in and a knowledge-driven diffusion strategy. The perception end relies on heterogeneous visual experts and a self-attention router to complete multi-modal BEV feature fusion. The decision end switches general or scene experts according to scene uncertainty, and outputs smooth driving actions with the help of a sparse diffusion network. The application simultaneously improves the multi-scene perception accuracy and decision-making ability of automatic driving, and the sparse expert activation mechanism takes into account the computing efficiency of the vehicle-mounted equipment and the real-time performance of the vehicle-level reasoning, and is suitable for an end-to-end perception decision module of an automatic driving vehicle and a test platform.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving technology, and in particular relates to a layered decoupled hybrid expert end-to-end perception and decision-making integrated method and device. Background Technology

[0002] With the deep integration of artificial intelligence technology and the automotive industry, autonomous driving technology has entered a new stage of innovative development based on end-to-end architecture, gradually replacing the traditional divide-and-conquer modular architecture and becoming the core technology direction for the implementation of autonomous driving. End-to-end autonomous driving technology uses a unified deep learning model to directly map the raw data collected by multiple sensors such as cameras and LiDAR into control commands for vehicle driving. It eliminates the need for manually designing intermediate modules and interfaces, effectively solving problems such as information transmission loss, error accumulation, inefficient multi-module collaboration, and computational latency in traditional modular architectures. At the same time, it has stronger generalization capabilities and can better cope with complex and ever-changing long-tail driving scenarios.

[0003] While end-to-end autonomous driving technology has made significant progress, with commercial products achieving end-to-end control from sensor input to control command output, and domestic OEMs and technology companies actively investing in related R&D, existing end-to-end models still face numerous technical bottlenecks, making it difficult to meet practical application needs. Specifically, on the one hand, existing end-to-end models mostly adopt a single backbone network architecture, which limits their modeling capabilities and cannot simultaneously adapt to the perception and decision-making needs of different driving scenarios such as lane changing, overtaking, and emergency braking. This results in insufficient perception accuracy, poor decision-making rationality, and unstable performance in complex scenarios. On the other hand, end-to-end models need to simultaneously process massive amounts of raw data generated by multimodal sensors such as cameras and LiDAR, leading to a sharp increase in model parameter scale. This, in turn, causes problems such as excessive computational load and increased inference latency, seriously affecting the real-time performance and safety of autonomous driving systems, making it difficult to meet the stringent standards of automotive-grade applications. Furthermore, existing hybrid expert architecture applications are prone to expert collapse and load imbalance, and the basic model is prone to forgetting world knowledge during training, further limiting the technology's practical application.

[0004] To address the technical bottlenecks of the aforementioned end-to-end autonomous driving models, the Mixture of Experts (MoE) architecture, as a highly efficient distributed neural network architecture, is being gradually applied in the field of autonomous driving technology. This architecture, by setting up multiple specialized "expert networks" and corresponding routing mechanisms, can achieve targeted processing for different tasks or scenarios. It activates only the expert networks matching the current input data for inference, significantly reducing computational load and improving inference efficiency while ensuring model performance, providing an effective technical path for optimizing end-to-end autonomous driving models. Simultaneously, the Low-Rank Adapter (LoRA) technique can effectively reduce model training parameters, and the diffusion model can improve the temporal coherence of decision sequences, providing technical support for further optimizing end-to-end perception and decision-making performance.

[0005] However, the application of hybrid expert architecture, LoRA technology, and diffusion models in the field of autonomous driving still has significant limitations: existing technologies mostly focus on optimizing single perception tasks (such as object detection and semantic segmentation) or single decision tasks (such as path planning and vehicle speed control) in hybrid expert architecture, and have not yet formed an end-to-end integrated solution that combines a layered and decoupled hybrid expert perception architecture with a dual-perception hybrid expert decision architecture; LoRA technology has not been deeply integrated with hybrid expert architecture to form a plug-in module, making it impossible to achieve efficient training and knowledge retention while freezing the backbone network of the basic model; the application of diffusion models is not combined with driving knowledge-driven approaches, making it difficult to generate temporally coherent action sequences that conform to driving logic; at the same time, existing routing mechanisms lack dual adaptation to scene perception and uncertainty perception, making it impossible to achieve a balance between decision adaptability and robustness, and are prone to expert collapse problems. This single-task-oriented approach, lacking sufficient technology integration, cannot fully leverage the synergistic advantages of hybrid expert architecture, LoRA technology, and diffusion models. It struggles to meet the comprehensive requirements of autonomous driving for perception accuracy, decision-making speed, and environmental adaptability, thus limiting the large-scale application of end-to-end autonomous driving technology in complex real-world scenarios and failing to fundamentally address the technical bottlenecks of existing end-to-end models.

[0006] In summary, existing end-to-end autonomous driving technologies suffer from problems such as poor scenario adaptability, low computational efficiency, insufficient perception and decision-making coordination, easy collapse of experts, easy forgetting of basic model knowledge, and poor coherence of decision sequences. The existing application methods of hybrid expert architecture, LoRA technology, and diffusion models cannot effectively solve the above problems. Therefore, there is an urgent need for a technical solution that can break through the limitations of existing technologies, achieve integrated collaborative optimization of perception and decision-making, and integrate layered decoupled hybrid experts, LoRA plug-ins, and knowledge-driven diffusion strategies to improve the perception accuracy, decision rationality, operational efficiency, and scenario adaptability of autonomous driving. Summary of the Invention

[0007] This invention aims to address the shortcomings of existing end-to-end autonomous driving models, such as poor scenario adaptability, low computational efficiency, insufficient perception and decision-making coordination, and the limitations of hybrid expert (MoE) architectures in single-task applications, which cannot achieve integrated perception and decision-making and are prone to expert collapse, easy forgetting of basic model knowledge, and poor coherence of decision sequences. It provides an end-to-end autonomous driving decision-making method and device based on a hybrid expert architecture. This method constructs a layered and decoupled hybrid expert perception architecture and a dual-perception hybrid expert decision-making architecture, integrating a low-rank adapter hybrid expert (LoRAMoE) plug-in module and a knowledge-driven diffusion strategy to achieve integrated collaborative optimization of perception and decision-making across the entire chain. It balances perception accuracy, decision rationality, operational efficiency, and scenario adaptability, solving problems such as expert collapse, knowledge forgetting, and insufficient coherence of decision sequences. This overcomes existing technical bottlenecks, meets automotive-grade application requirements, and promotes the large-scale deployment of end-to-end autonomous driving technology.

[0008] The technical solution adopted by the decision-making method of the present invention is as follows:

[0009] Step 1: Construct a multi-source sensor data preprocessing module

[0010] The multi-source sensor data preprocessing module is used to standardize the raw data from various types of sensors on autonomous vehicles, providing high-quality data support for subsequent perception and decision-making operations. The specific steps are as follows:

[0011] (1) Data collection: Collect raw data from cameras, lidar and millimeter-wave radar on autonomous vehicles. The cameras collect image data of the road environment and traffic participants, the lidar collects three-dimensional point cloud data of objects around the vehicle, and the millimeter-wave radar collects distance and relative speed data between the vehicle and surrounding objects. At the same time, environmental parameter data such as ambient temperature, humidity and visibility are collected simultaneously.

[0012] (2) Data synchronization and alignment: Based on timestamps and spatial calibration parameters, the original data of the above multi-source sensors are processed for time synchronization and spatial alignment to eliminate time deviations (such as data acquisition delays) and spatial deviations (such as coordinate deviations caused by differences in installation positions) between different sensors, and ensure the spatiotemporal consistency of multi-source data.

[0013] (3) Data purification and enhancement: Gaussian filtering algorithm is used to filter noise in the original data, removing isolated points in point cloud data, salt-and-pepper noise in image data and interference signals in millimeter-wave radar data; normalization is used to map various types of data to the same numerical range to eliminate the influence of dimensions; outlier detection algorithm (such as 3σ criterion) is used to remove abnormal data that appear during the acquisition process; feature enhancement is performed through image enhancement (such as histogram equalization) and point cloud resampling to improve the data's recognizability and lay the foundation for subsequent feature extraction;

[0014] (4) Data fusion output: The multi-source data processed above are fused at the feature level to generate standardized multi-source fusion input data, which is then transmitted to the shared input layer of the hybrid expert perception-decision integrated network.

[0015] Step 2: Construct a hybrid expert perception-decision integrated network

[0016] The hybrid expert perception-decision integrated network is the core execution unit of this invention, used to realize the integrated collaborative processing of perception and decision-making tasks. This network adopts a hybrid expert architecture design, including a shared input layer, multiple specialized expert networks, and a shared output layer. The structure and function of each part are as follows:

[0017] 1. Shared Input Layer: Adopting a structure combining Convolutional Neural Network (CNN) and Transformer, it receives standardized fused data output from the multi-source sensor data preprocessing module, extracts local features of the data through convolutional layers, and extracts global correlation features of the data through the Transformer encoder, thereby completing the unification of data dimensions and the initial extraction of basic features. The extracted basic features are then synchronously transmitted to the routing allocation module and each expert network.

[0018] 2. Expert Networks: Specialized divisions are made according to driving scenarios and task requirements, including at least urban road perception-decision experts, highway perception-decision experts, severe weather perception-decision experts, and emergency decision-making experts for sudden scenarios. Each expert network adopts an integrated structure of "perception branch + decision branch," and the parameters of each expert network are trained independently without interference. Specifically, the perception branch uses a concatenated structure of an improved YOLO network and a semantic segmentation network, responsible for perception tasks such as target detection (e.g., vehicles, pedestrians, traffic signs), semantic segmentation (e.g., road areas, sidewalks, obstacles), and scene recognition (e.g., intersections, tunnels, roundabouts) in the corresponding scenario, outputting scene perception results and feature confidence scores. The decision branch uses a reinforcement learning network (DQN), based on the scene perception results output by the perception branch, combined with the vehicle's current driving status (vehicle speed, steering angle, remaining battery power), to complete decision-making tasks such as path planning, speed control, obstacle avoidance decisions, and following distance adjustment, outputting intermediate decision results and decision confidence scores.

[0019] 3. Shared Output Layer: Adopting an attention mechanism fusion structure, it receives the perception results, intermediate decision results and corresponding confidence scores of each activated expert network. Through the attention weight allocation algorithm, it performs weighted fusion on the output results of different expert networks, eliminates abnormal outputs with confidence scores below the threshold, and generates unified and coherent driving control commands, including steering angle, vehicle speed commands, braking signals, gear shifting signals, etc., which are then transmitted to the control command output module.

[0020] Step 3: Build the route allocation module

[0021] The routing allocation module is used to implement dynamic activation and scheduling of expert networks, reducing computational load while ensuring model performance. The specific steps are as follows:

[0022] (1) Scene feature analysis: Receive the basic features output by the shared input layer, analyze the current driving scene type, environmental complexity and task priority through the scene classification network, and extract key scene features (such as road type, traffic flow, weather conditions, and whether there are sudden obstacles).

[0023] (2) Matching degree calculation: Based on the key features of the scene, the cosine similarity algorithm is used to calculate the matching degree between each expert network and the current input data. The scene weight coefficient is introduced in the matching degree calculation process, and special scenes such as severe weather and sudden scenes are given higher weights to improve the matching accuracy.

[0024] (3) Dynamic activation strategy: Set a matching degree threshold (which can be dynamically adjusted according to the actual application scenario), activate one or more expert networks with a matching degree higher than the threshold, and stop the inference operation of the unactivated expert networks, only maintaining a basic standby state, thereby reducing the overall computational load of the model;

[0025] (4) Dynamic optimization and adjustment: Real-time reception of perception results, decision intermediate results and confidence levels output by each expert network, dynamic adjustment of matching degree calculation parameters and thresholds through feedback adjustment algorithm, optimization of expert network activation strategy, ensuring that the appropriate expert network can be quickly activated when switching scenes, and improving the flexibility of the model in scene adaptation.

[0026] Step 4: Construct the control command output module

[0027] The control command output module is used to verify, optimize, and execute the driving control commands output by the hybrid expert perception-decision integrated network to ensure the safety and rationality of the control commands. The specific steps are as follows:

[0028] (1) Legality verification of commands: Based on the vehicle's physical characteristics (such as maximum steering angle, maximum speed, and braking distance) and automotive-grade safety standards, legality verification of received driving control commands is performed, and abnormal commands that exceed the safety range (such as oversteering and speeding commands) are eliminated.

[0029] (2) Command smoothing: The moving average algorithm is used to smooth the legal control commands, correct command abrupt changes (such as sudden changes in steering angle and vehicle speed), avoid vehicle shaking and loss of control due to command abrupt changes, and improve driving comfort and safety.

[0030] (3) Command execution and feedback: The control commands after verification and smoothing are transmitted to the vehicle actuators (steering system, power system, braking system) to realize the autonomous driving control of the vehicle; at the same time, the feedback data of the vehicle actuators (such as actual steering angle and actual vehicle speed) are collected in real time and transmitted to the routing allocation module and the model training optimization unit to provide data support for the adjustment of model parameters.

[0031] (4) Data storage: The control commands, corresponding perception results, decision results and vehicle feedback data are encrypted and stored to form a driving log, which provides data basis for subsequent model iteration optimization and fault diagnosis.

[0032] Step 5: Model Training and Optimization

[0033] To ensure that the model performance meets the requirements of automotive-grade applications, end-to-end training and iterative optimization are performed on the hybrid expert perception-decision integrated network. The specific steps are as follows:

[0034] (1) Dataset construction: Construct an autonomous driving dataset containing multiple scenarios and working conditions, covering long-tail scenarios such as urban roads, highways, severe weather such as rain, snow and fog, and sudden obstacles. Each scenario contains raw data from multiple sensor sources, corresponding scenario labels, perception labels, and optimal decision labels.

[0035] (2) Model initialization: Initialize the parameters of the hybrid expert perception-decision integrated network and the routing allocation module, and set hyperparameters such as the number of expert networks, matching degree threshold, and training learning rate;

[0036] (3) End-to-end training: Based on the constructed dataset, the model is trained end-to-end using a joint loss function. The joint loss function includes perception task loss (object detection loss, semantic segmentation loss) and decision task loss (path deviation loss, vehicle speed control loss). The parameters of the perception branch and the decision branch are optimized synchronously through the backpropagation algorithm to achieve collaborative optimization of perception and decision.

[0037] (4) Parameter optimization: During training, the parameter weights and routing strategies of each expert network are dynamically adjusted based on the output confidence and error of each expert network to improve the model’s scene adaptability and generalization ability; an early stopping strategy is adopted to avoid model overfitting.

[0038] (5) Testing and iteration: After training, the model performance is verified through real-world scenario testing and simulation testing, with a focus on testing scenario adaptability, computational efficiency, perception accuracy and decision rationality; for problems that arise during testing, the model parameters, expert network partitioning and routing strategies are iteratively optimized until the model performance meets automotive-grade application standards.

[0039] The beneficial effects of this invention are:

[0040] This invention proposes an end-to-end integrated perception and decision-making solution for autonomous driving based on a multi-type hybrid expert (MoE) architecture. It integrates four core modules: layered decoupled MoE perception, dual-perception routing decision-making, knowledge-driven diffusion strategy, and the LoRAMoE lightweight plugin. It systematically optimizes existing end-to-end autonomous driving systems to address pain points such as fragmented perception and decision-making, weak scenario adaptability, high computational overhead, knowledge forgetting, and insufficient decision robustness. Specific beneficial effects are as follows:

[0041] (1) Achieve integrated collaboration of perception and decision-making across the entire chain and improve overall task performance: This invention deeply integrates the layered decoupled MoE architecture and the dual-perception MoE decision architecture to construct an end-to-end network integrating perception and decision-making. All expert networks uniformly design perception branches and decision branches to achieve lossless flow of sensor raw data from feature extraction, cross-modal fusion, target perception to driving decision-making across the entire chain. This breaks the error accumulation problem of traditional modular architecture and solves the shortcomings of traditional autonomous driving MoE schemes, such as the separation of perception and decision-making modules, feature transmission loss, weak module linkage, and difficulty in global joint optimization. This effectively improves perception detection accuracy and optimizes driving decision-making logic.

[0042] (2) Dynamic expert routing combined with specialized expert division enhances the ability to adapt to all scenarios: This invention is designed for differentiated driving scenarios, builds a heterogeneous specialized expert cluster, deploys four types of heterogeneous visual experts at the perception end to adapt to different dimensions of visual features, and distinguishes between general experts and scenario-specific experts at the decision end. Combined with the dynamic routing mechanism, it adaptively matches the optimal expert combination, which solves the shortcomings of the traditional end-to-end autonomous driving model in terms of fixed structural parameters, insufficient cross-scenario feature modeling ability, and precipitous drop in model performance under extreme and long-tail road conditions. It significantly reduces the fluctuation of model performance in complex scenarios and improves the stability of driving in all domains.

[0043] (3) Introducing a sparse activation strategy to balance performance and computing power, and meet automotive-grade real-time requirements: This invention deploys a sparse expert activation strategy in all modules of perception, decision-making and LoRAMoE, and activates corresponding expert branches and low-rank adapters in a hierarchical manner as needed. It does not require all experts to participate in reasoning, which significantly reduces the computational load and memory usage of the model and reduces the reasoning latency. It solves the shortcomings of traditional large-scale autonomous driving models and conventional MoE architectures, such as high inference computing power consumption, high network latency, difficulty in adapting to vehicle edge hardware, and high difficulty in mass production.

[0044] (4) Multi-dimensional optimization of model capabilities to ensure the safety of autonomous driving operation in all aspects: This invention achieves differentiated division of labor of expert capabilities through joint constraint training of multiple loss functions, and improves the generalization ability of the model in unknown scenarios; at the same time, it introduces a knowledge-driven diffusion strategy to optimize decision output, and relies on LoRAMoE to freeze the backbone network to avoid forgetting model knowledge, which solves the defects of traditional autonomous driving models such as poor generalization, easy mutation of decision instructions, easy loss of original knowledge in the fine-tuning process, and insufficient overall stability and safety of the system, and comprehensively improves the operation stability and driving safety of the autonomous driving system.

[0045] (5) Lightweight plug-in design to reduce deployment and iteration costs: This invention builds a LoRAMoE lightweight plug-in, uses a low-rank adapter as an expert module and freezes the pre-trained backbone network, compressing the number of trainable parameters to about 0.57% of the original model. This not only avoids knowledge forgetting, but also significantly reduces the training cost and hardware computing power threshold for model iteration optimization and adaptation to new scenarios. It solves the problems of large number of parameters for full fine-tuning of traditional autonomous driving models, high training costs, slow model iteration and high difficulty in migration and deployment. It effectively improves the model engineering feasibility and functional scalability, and facilitates subsequent scenario expansion and rapid functional iteration. Attached Figure Description

[0046] Figure 1 The overall flowchart of the method of this invention. Detailed Implementation

[0047] The invention will now be further described with reference to the accompanying drawings.

[0048] The process of the end-to-end autonomous driving decision-making method based on a hybrid expert architecture in this invention is as follows: Figure 1 As shown, by constructing an integrated end-to-end network for perception and decision-making, the problems of poor scenario adaptability, low computational efficiency, insufficient perception-decision collaboration, expert collapse, and model knowledge forgetting in traditional end-to-end models are effectively solved. This comprehensively improves the perception accuracy, decision rationality, multi-scenario adaptability, and real-time performance of autonomous driving systems. The specific implementation steps are as follows:

[0049] (1) This invention uses publicly available autonomous driving datasets combined with real-vehicle collected data to jointly construct a sample library, covering conventional scenarios such as urban roads, highways, rain and fog, nighttime, and sudden obstacles, as well as long-tail extreme scenarios. The sample library is randomly divided according to the ratio of training set: validation set: test set = 7:2:1 to ensure that the samples of each scenario are evenly distributed in the three subsets and to avoid sample bias. For the low proportion of long-tail samples such as severe weather and sudden emergencies, an oversampling strategy is adopted to increase the proportion of samples and enhance the model's ability to learn from extreme conditions.

[0050] (2) Build a multi-source sensor data preprocessing module. This module is used to complete the acquisition, synchronization, purification and standardization of multi-source sensor data, and at the same time complete the dataset construction, sample division and label association to provide high-quality input data for the back-end network. It distinguishes between two types of processing logic: model training stage and real vehicle inference stage. The specific implementation steps are as follows:

[0051] 1. Multi-source data acquisition. Raw data is simultaneously acquired from three types of sensors mounted on the vehicle: cameras, LiDAR, and millimeter-wave radar. Specifically, cameras capture images to characterize the road environment, traffic participants, and traffic signs; LiDAR outputs LiDAR point clouds to acquire 3D spatial information about surrounding obstacles; and millimeter-wave radar outputs data such as target distance and relative speed, while simultaneously acquiring environmental parameters such as temperature, humidity, and visibility.

[0052] 2. Spatiotemporal Synchronization and Label Association. Based on the sensor's preset timestamps and vehicle body spatial calibration parameters, multi-channel data is synchronized in time and aligned in coordinate space. This eliminates data misalignment issues caused by acquisition delays from different sensors and installation position deviations, ensuring the spatiotemporal consistency of image, point cloud, and radar data. Additional operations during the training phase: The synchronized multi-source sensor data is bound one-to-one with manually labeled perception labels, decision labels, and scene labels to form "raw data-label" paired samples. During the inference phase, only the data alignment process is retained, and label association is not required.

[0053] 3. Data Cleaning, Enhancement, and Normalization. Gaussian filtering is used to remove salt-and-pepper noise in images, isolated noise in point clouds, and millimeter-wave radar interference signals; the 3σ criterion is used to remove outliers; histogram equalization and point cloud resampling are used to enhance data features; all data are normalized to unify numerical ranges and eliminate dimensional differences. Sample augmentation strategies: offline augmentation is performed on image data, including random cropping, random flipping, brightness perturbation, and fog effect simulation; random rotation and translation augmentation are performed on point cloud data to further expand sample diversity and improve model generalization ability; sample augmentation is turned off during the inference phase to ensure that the input data is consistent with the real driving environment.

[0054] 4. Feature-level fusion output. The processed image, point cloud, and millimeter-wave data are fused at the feature level to generate standardized fused data, which is then transmitted to the shared input layer of the hybrid expert perception-decision integrated network.

[0055] (3) Construct a hybrid expert perception-decision integrated network. This network is the core execution unit of this invention and includes a shared input layer, a perception layer, a decision layer, and a shared output layer.

[0056] 1. Construct a shared input layer. A CNN+Transformer composite structure is adopted to receive preprocessed fused data: the convolutional layer extracts local texture and contour features of the data, the Transformer encoder mines long-range global correlation features, unifies the dimension and feature format of all input data, and synchronously distributes the basic features to the routing allocation module and each expert network.

[0057] 2. Construct the perception layer processing module. The perception layer adopts a layered, decoupled, hybrid expert architecture, including a heterogeneous expert set, a self-attention router (SAR), a soft-weighted feature fusion unit, a sparse expert activation unit, a load balancing regularization unit, and a BEV cross-modal fusion unit. The specific implementation steps are as follows:

[0058] 2.1 Constructing a Heterogeneous Expert Set. A visual expert set is constructed, consisting of four heterogeneous networks: Swin Transformer, ResNet, ConvNeXt, and Pyramid Visual Transformer (PVT), with a total of four experts (…). The four types of experts have different roles, respectively adapting to global structural features, local texture features, multi-scale target features, and lightweight visual features; an adapter layer is added at the end of each expert to force the spatial resolution and channel dimension of all expert output features to be unified, ensuring the effectiveness of subsequent feature fusion.

[0059] 2.2 Self-Attention Router (SAR) Operations.

[0060] 2.2.1 Hierarchical Feature Extraction: 3×3 convolution and 2×2 max pooling operations are sequentially performed on the input camera feature map, with the number of channels progressively increasing to 32 and 64. The final output channels equal the attention embedding dimension. ;

[0061] 2.2.2 Self-attention encoding: The two-dimensional feature map is reshaped into a one-dimensional token sequence. Multi-head self-attention (MHA) is used to capture the global spatial dependencies of the image. Global average pooling is performed on all token features to generate image-level embedding features with a dimension of 128.

[0062] 2.2.3 MLP Expert Scoring: Image-level embedded features are input into a 3-layer MLP network to generate expert log odds (Logits). After normalization by the Softmax function, the output is a 4-dimensional image-level expert routing probability. .

[0063] 2.3 Soft-weighted feature fusion. Using the routing probability output by SAR as the weight, the original features of the four heterogeneous experts are weighted and summed to obtain the fused camera visual features.

[0064] 2.4 Sparse Expert Activation. Following the sparse activation strategy, only the Top-1 expert with the highest routing probability is activated during inference, while the calculations of other experts are stopped, which significantly reduces computational overhead while ensuring perception accuracy.

[0065] 2.5 Load Balancing Regularization. Calculate the average activation rate and cumulative routing load of each expert in each training round to construct the load balancing loss. The joint loss formula for the perception modules is:

[0066] (1)

[0067] Among them, samples The corresponding route allocation probability is Average activation rate of experts Cumulative routing load of each expert N is the total number of training samples in a single round, and the perceptual loss weight hyperparameter is... , The focus loss for 3D object detection is the main loss for the perception task, used to supervise object detection accuracy; the load balancing loss at the perception end. subscript Representing the A perceptual heterogeneous vision expert, summation upper limit The total number of visual experts in the perception layer, in the examples of this invention. This loss constraint ensures that all experts are called upon in a balanced manner, preventing expert collapse in the perception module.

[0068] 2.6 BEV Cross-Modal Fusion and Perception Output. The weighted fused camera features are projected onto the bird's-eye view (BEV) space, and the coordinate transformation of the LiDAR point cloud features is performed to complete the fusion of camera features and LiDAR features. Based on the fused BEV features, 3D object detection and semantic segmentation tasks are performed, and global scene perception results such as obstacle category, location, size, and road topology are output and transmitted to the downstream decision layer.

[0069] 3. Establish a decision-making layer processing module. The decision-making layer receives BEV scene features output from the perception layer and consists of a dual-perception hybrid expert submodule and a knowledge-driven diffusion strategy (KDP) submodule to achieve scene-adaptive decision-making and sequential action generation.

[0070] 3.1 Dual-sensory hybrid expert submodule.

[0071] 3.1.1 Expert Cluster Division. Two types of networks are set up: global experts and scenario-adaptive expert groups. Global experts are trained on the full-scale autonomous driving dataset and have general decision-making capabilities for all scenarios. Scenario-adaptive expert groups are divided into five categories of experts: lane changing, overtaking, emergency braking, yielding, and traffic sign recognition. Each type of expert receives specialized training only in the corresponding subset of scenarios.

[0072] 3.1.2 Dual-Sensory Router Operation. The dual-sensory router consists of two parts: scene-aware routing and uncertainty-aware routing. Scene-aware routing is trained based on cross-entropy loss, performs scene classification based on input scene features, and initially matches corresponding scene experts. Uncertainty-aware routing calculates the normalized information entropy of the route probability distribution. Quantify the degree of uncertainty in the current scenario. The formula is as follows:

[0073] (2)

[0074] in, This indicates that when the input scene features are At that time, scene-aware routing is assigned to the first Normalized probabilities of scene-adaptive experts, all satisfy , representing the current scene matching the first Confidence level of experts in similar scenarios A higher value indicates a more complex scenario and greater uncertainty. A value close to 0 indicates a clear and stable scene.

[0075] 3.1.3 Branch Decision Logic. Preset Uncertainty Threshold. As an expert, switch judgment conditions: when Activate the scene-aware adaptive expert with the highest route matching degree to achieve refined optimal decision-making; when Activating global experts in a timely manner ensures driving robustness, thereby balancing the adaptability and safety of decision-making.

[0076] 3.2 Constructing the Knowledge-Driven Diffusion Strategy (KDP) submodule. This submodule combines the diffusion model with a sparse hybrid expert architecture, modeling autonomous driving decision-making as a conditional sequence generation task. The specific implementation steps are as follows:

[0077] 3.2.1 Forward Diffusion Process. A Markov chain is constructed using a cosine-squared noise scheduling method. Gaussian noise is progressively superimposed on the standard driving action sequence labeled by the human driver. The formula is as follows:

[0078] (3)

[0079] (4)

[0080] in, For the first The diffused-step, noisy driving action sequence vector contains three types of control variables: accelerator, brake, and steering. The action vector of the previous diffusion step. Represents the original, noise-free standard driving action sequence, a unit diagonal matrix. The dimension is consistent with the dimension of the driving action vector, and the conditional transition probability distribution of the diffuse Markov chain. Indicates a given first Step movement At that time, the first generation is generated. Step noise action The probability of a multivariate Gaussian distribution The first item The second term is the distribution mean. The distributed covariance matrix represents the distribution pattern of the superimposed noise in the current step, and the time-step noise variance coefficients are... Standard Gaussian noise Finally, the clean action sequence is transformed into a noisy sequence that approximates a standard Gaussian distribution, which is then used as the training target.

[0081] 3.2.2 Constructing an inverse denoising network. Using vehicle status, LiDAR point cloud, and navigation signal as conditions, a sparse MoE denoising network is constructed as the core of the driving strategy. Through iterative inverse denoising, Gaussian noise is gradually restored into a time-coherent, continuous action sequence that conforms to driving logic.

[0082] 3.2.3 Sparse Expert Activation. The denoising network internally incorporates multiple types of driving knowledge experts, corresponding to capabilities such as longitudinal speed adjustment, traffic interaction processing, and lateral steering negotiation; a Top-K router is configured (…). The two optimal expert subsets are dynamically activated and fused to output the denoising result.

[0083] 3.2.4 Regularization Constraint Training. The training process introduces load balancing loss and mutual information regularization. On the one hand, this ensures that the load of all experts is balanced, and on the other hand, it promotes the formation of differentiated knowledge division among experts, thereby improving the level of expert specialization.

[0084] 4. Establish a shared output layer. Adopting an attention fusion structure, it receives the intermediate decision results and confidence levels of each activated expert in the decision layer, performs weighted fusion of multiple results according to attention weights, eliminates abnormal outputs with confidence levels below a preset threshold, and generates preliminary driving control commands.

[0085] (4) Build the LoRAMoE lightweight plugin module. This module is a globally embedded plugin that covers the perception layer and decision layer, freezes the backbone network of the basic model, and uses a low-rank adapter as the expert to achieve lightweight training and suppress knowledge forgetting. The specific implementation steps are as follows:

[0086] 1. Network Layer Replacement. Replace all feedforward neural network (FFN) layers in the perception and decision backbone networks with LoRAMoE layers; each LoRAMoE layer contains several LoRA low-rank experts, with single expert weight increments. Satisfy the formula

[0087] (5)

[0088] Among them, the input-side low-rank matrix Output side low-rank matrix The low-rank dimension has a fixed value. , Input dimension for layer For the layer output dimension, the original weight matrix of the backbone network The process is frozen and does not participate in parameter updates.

[0089] 2. LoRAMoE Layer Output Calculation. The output of the LoRAMoE layer is the superposition of the backbone network output and the weighted output of each LoRA expert, as shown in the formula:

[0090] (6)

[0091] in, The original weight matrix of the backbone network of the basic model. The layer input feature vector, Let be the weight coefficient assigned by the router to the i-th LoRA expert. Scaling the hyperparameters to the total number of LoRA experts This is used to scale the LoRA expert output to the same order of magnitude as the backbone network output, thus avoiding training oscillations.

[0092] 3. Local Balance Constraints. The training data is divided into two categories: world knowledge tasks and downstream autonomous driving tasks. LoRA expert functions are allocated through soft constraint rules: some experts focus on calling up and retaining the model's original world knowledge, while the rest focus on optimizing downstream driving tasks, while ensuring that the load of experts of the same type is balanced.

[0093] 4. Joint Loss Training. The joint loss function of the LoRAMoE plugin is shown below:

[0094] (7)

[0095] in, The main loss for downstream tasks includes token prediction loss and driving action regression loss, which monitors the overall task and decision output accuracy. To balance the loss in a localized manner, the LoRA experts' division of labor and load balancing are constrained, while preserving the model's original world knowledge and preventing knowledge forgetting; hyperparameters are also balanced. During the training phase, only the LoRA low-rank matrix and router parameters are updated, while the backbone network parameters are completely frozen. The number of trainable parameters is only 0.57% of the original model's full fine-tuning, thus fundamentally avoiding the problem of forgetting world knowledge after model fine-tuning.

[0096] (5) Establish a routing allocation module. This module runs through the entire perception and decision-making process, realizing dynamic activation of the expert network and dynamic optimization of parameters. The specific implementation steps are as follows:

[0097] 1. Scene Feature Analysis. Receives basic features from the shared input layer and uses a scene classification network to analyze key scene information such as current road type, traffic flow, weather conditions, and unexpected obstacles to determine task priority.

[0098] 2. Expert Matching Degree Calculation. The cosine similarity algorithm is used to calculate the matching degree between the current scene features and each expert network. For high-risk scenarios such as severe weather and sudden emergencies, additional scene weight coefficients are added to improve the expert matching accuracy in special scenarios.

[0099] 3. Dynamic activation strategy. A preset matching threshold is used to activate only one or more expert networks with a matching degree higher than the threshold. Unactivated experts enter a standby state to reduce invalid computations.

[0100] 4. Dynamic parameter optimization. Real-time acquisition of expert output results and confidence levels, and dynamic adjustment of matching degree calculation parameters and activation thresholds through feedback adjustment algorithms, to achieve seamless scheduling of experts when rapidly switching scenarios.

[0101] (6) Build a control command output module. This module performs safety verification, smooth optimization, and execution on the control commands output by the decision-making level. The specific implementation steps are as follows:

[0102] 1. Command validity verification. Combining vehicle physical parameters, including maximum steering angle, maximum speed, braking distance, and automotive safety standards, the control commands are verified, and abnormal commands that exceed the safe range, such as speeding and excessively large steering angles, are directly rejected.

[0103] 2. Command Smoothing Processing. A moving average algorithm is used to smooth legal commands, suppressing abrupt changes in steering, speed, and braking commands, avoiding vehicle vibration and loss of control, and improving driving comfort and safety.

[0104] 3. Command Execution and Status Feedback. Optimized commands are sent to actuators such as the vehicle's steering system, power system, and braking system; real-time data on the vehicle's actual driving status is collected and transmitted back to the routing module and training unit, providing a basis for model iteration.

[0105] 4. Data storage. Sensing results, decision commands, and vehicle operation logs are encrypted and stored for subsequent fault diagnosis and model iteration optimization.

[0106] (7) Construct the model training and optimization unit. This unit is used to complete the end-to-end training, testing, and iteration of the entire network, ensuring that the model meets the requirements for automotive-grade applications. The specific implementation steps are as follows:

[0107] 1. Dataset Construction. A multi-scenario autonomous driving dataset is constructed, covering long-tail conditions such as urban roads, highways, severe weather (rain, snow, fog), and sudden obstacles. The dataset includes multi-source sensor data, scene labels, perception labels, and optimal decision labels.

[0108] 2. Parameter Initialization. Initialize the network parameters of the hybrid expert network, router, and LoRAMoE plugin, setting all hyperparameters such as the number of experts, activation threshold, learning rate, and batch size.

[0109] 3. End-to-end joint training. A multi-task joint loss function is adopted, fusing perception task loss and decision task loss. The perception task loss includes 3D detection loss and semantic segmentation loss, while the decision task loss includes path deviation loss and vehicle speed control loss. The perception branch, decision branch, LoRA parameters, and router parameters are simultaneously optimized through the backpropagation algorithm to achieve collaborative optimization of perception and decision.

[0110] 4. Overfitting Suppression and Parameter Tuning. An early stopping strategy is employed during training to prevent model overfitting; expert weights and routing strategies are dynamically adjusted based on the output errors of each expert to improve the model's generalization ability.

[0111] 5. Testing and Iterative Optimization. After training, performance tests are conducted on a simulation platform and real roads, focusing on perception accuracy, inference latency, scene adaptability, and decision-making safety. Expert division, routing strategies, and network parameters are iteratively optimized to address test deficiencies until all indicators meet automotive-grade standards.

[0112] (8) Implementation of auxiliary perception tasks. Based on the BEV fusion features output by this invention, two typical autonomous driving perception tasks are extended. The specific implementation steps are as follows:

[0113] 1. 3D Object Detection. Employing a BEV detection framework, the system inputs fused features into the detection head and combines a multi-scale feature fusion strategy with a CIoU loss function to achieve accurate 3D detection of targets such as vehicles, pedestrians, and obstacles.

[0114] 2. Semantic Segmentation. The DeepLabv3+ model is selected. BEV features are input into the ASPP module to extract multi-scale contextual information. Shallow and deep features are fused through skip connections. The loss function combines cross-entropy loss and Dice loss to solve the class imbalance problem of roads, obstacles, sidewalks, etc., and improve the semantic segmentation accuracy.

[0115] Based on the aforementioned end-to-end autonomous driving decision-making method based on a hybrid expert architecture, this invention also proposes a device for autonomous driving, such as an onboard domain controller or onboard computing unit for an autonomous vehicle. This device can execute the end-to-end autonomous driving decision-making method described in this invention, completing the entire process including multi-source sensor data preprocessing, hybrid expert network feature computation, dynamic expert routing, driving decision generation, control command verification and output, etc.

[0116] In summary, this invention deeply integrates a layered, decoupled hybrid expert architecture, dual-perception routing, a knowledge-driven diffusion strategy, and the LoRAMoE lightweight plugin, achieving integrated collaborative processing across the entire perception and decision-making chain for autonomous driving. This invention effectively solves problems such as poor scenario adaptability, low computational efficiency, expert collapse, model knowledge forgetting, and insufficient decision-making temporal coherence in traditional end-to-end models. In practical applications, it exhibits excellent perception accuracy, decision rationality, environmental robustness, and real-time performance, fully meeting the requirements for automotive-grade autonomous driving and possessing strong engineering implementation capabilities and broad application value.

[0117] The detailed descriptions listed above are merely specific descriptions of feasible embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. All equivalent methods or modifications that do not depart from the technology of the present invention should be included within the scope of protection of the present invention.

Claims

1. An end-to-end autonomous driving decision-making method based on a hybrid expert architecture, characterized in that, include: Step S1: Construct a multi-source sensor data preprocessing module to standardize the raw data from various types of sensors on autonomous vehicles. Step S2: Construct a hybrid expert perception-decision integrated network to realize the integrated collaborative processing of perception tasks and decision-making tasks. This network adopts a hybrid expert architecture. Step S3: Construct a routing allocation module to realize the dynamic activation and scheduling of the expert network, thereby reducing the computational load while ensuring the performance of the network model. Step S4: Construct a control command output module to verify, optimize, and execute the driving control commands output by the hybrid expert perception-decision integrated network, ensuring the safety and rationality of the control commands; Step S5: Perform end-to-end training and iterative optimization of the hybrid expert perception-decision integrated network, and use the trained network model for autonomous driving perception and decision-making.

2. The end-to-end autonomous driving decision-making method based on a hybrid expert architecture according to claim 1, characterized in that, In S2, the hybrid expert perception-decision integrated network includes a shared input layer, a perception layer, a decision layer, and a shared output layer; The shared input layer adopts a structure combining convolutional neural networks and Transformers. It receives standardized fused data output by the multi-source sensor data preprocessing module, extracts local features of the data through the convolutional layer, and extracts global correlation features of the data through the Transformer encoder. This completes the unification of data dimensions and the initial extraction of basic features. The extracted basic features are then synchronously transmitted to the routing allocation module and each expert network. The perception layer achieves multimodal bird's-eye view perception based on a hierarchical decoupled hybrid expert architecture; The decision layer receives BEV scene features output by the perception layer and combines dual-perception hybrid experts and knowledge-driven diffusion strategies to complete end-to-end decision generation. The shared output layer adopts an attention fusion structure, receives the intermediate decision results and confidence levels of each activated expert output from the decision layer, performs weighted fusion of multiple results according to attention weights, eliminates abnormal outputs with confidence levels below a preset threshold, and generates preliminary driving control commands.

3. The end-to-end autonomous driving decision-making method based on a hybrid expert architecture according to claim 2, characterized in that, The perception layer includes a heterogeneous expert set, a self-attention router (SAR), a soft-weighted feature fusion unit, a sparse expert activation unit, a load balancing regularization unit, and a BEV cross-modal fusion unit, which are specifically implemented as follows: Step 2.1 Construct a heterogeneous expert set consisting of Swin Transformer, ResNet, ConvNeXt and Pyramid Vision Transformer. Each expert independently processes multi-view camera image features. The adapter layer ensures that the spatial resolution and channel dimension of the output features of all experts are consistent. Step 2.2 Construct a self-attention router (SAR) to extract the input camera features through hierarchical convolution, self-attention encoding, and MLP expert scoring to generate image-level expert route probabilities; Step 2.3 Construct a soft-weighted feature fusion unit, and perform weighted fusion of the output features of each expert according to the routing probability to obtain the fused camera features; Step 2.4 Construct sparse expert activation units, and only activate the Top-1 experts with the highest routing probability during the inference phase to reduce computational overhead; Step 2.5 Construct a load balancing regularization unit. By calculating the average activation rate of experts and the cumulative routing load, a load balancing loss is introduced to avoid expert collapse and ensure that all experts are used evenly during training. Step 2.6 Construct a BEV cross-modal fusion unit, project the fused camera features onto the BEV space, fuse them with LiDAR features, complete perception tasks such as 3D target detection, and transmit them to the decision layer.

4. The end-to-end autonomous driving decision-making method based on a hybrid expert architecture according to claim 3, characterized in that, The specific steps of step 2.2, the Self-Attention Router (SAR), are as follows: Step 2.2.1 Hierarchical Feature Extraction: Perform 3×3 convolution + 2×2 max pooling sequentially on the input camera feature map to complete feature dimensionality reduction and channel dimensionality increase, with the number of channels being 32 and 64 respectively. ,in Embedded dimension for attention; Step 2.2.2 Self-attention encoding: The convolutional features are reshaped into a token sequence, and the global spatial dependency is captured by multi-head self-attention (MHA). The average of the token sequence is then used to obtain the image-level embedding features. Step 2.2.3 MLP Expert Scoring: Input the image-level embedded features into a 3-layer MLP to generate expert log odds, and obtain the image-level expert routing probability through Softmax. ,in The total number of visual experts in the perception layer is 4.

5. The end-to-end autonomous driving decision-making method based on a hybrid expert architecture according to claim 4, characterized in that, The specific steps of load balancing regularization in step 2.5 include: Step 2.5.1 Calculate the average activation rate of experts , where N is the number of samples; Step 2.5.2 Calculate the cumulative routing load for each expert. ; Step 2.5.3 Calculate load balancing losses It is trained jointly with the perception task loss, and the joint loss of the perception module is: , For 3D object detection focus loss, This is a hyperparameter.

6. The end-to-end autonomous driving decision-making method based on a hybrid expert architecture according to claim 5, characterized in that, The dual-perception hybrid expert submodule in the decision-making layer integrates global experts, scene-adaptive expert groups, and dual-perception routers, and is specifically implemented as follows: Step 3.1 The global expert is trained on the full dataset and has robust decision-making capabilities across all scenarios; Step 3.2 The scenario adaptation expert group is divided into five categories of experts according to driving scenarios: lane changing, overtaking, emergency braking, yielding, and traffic signs. Each expert is trained on the corresponding scenario subset. Step 3.3 The dual-sensory router includes scene-aware routing and uncertainty-aware routing: scene-aware routing is trained through cross-entropy loss to achieve accurate routing from input features to the corresponding scene experts; uncertainty-aware routing quantifies scene uncertainty by calculating the normalized information entropy of the routing probability distribution. Step 3.4 Preset uncertainty threshold When the scene uncertainty is below a preset threshold, the scene adaptive expert with the highest scene-aware routing score is activated. When the uncertainty of the scenario exceeds a preset threshold, global experts are activated to achieve a balance between the adaptability and robustness of the decision.

7. The end-to-end autonomous driving decision-making method based on a hybrid expert architecture according to claim 6, characterized in that, The normalized information entropy calculation for uncertainty-aware routing in step 3.3 is as follows: （1） Where M represents the number of scene-adaptive experts. The larger the value, the higher the uncertainty of the scenario; the closer the value is to 0, the higher the certainty of the scenario.

8. The end-to-end autonomous driving decision-making method based on a hybrid expert architecture according to claim 7, characterized in that, The knowledge-driven diffusion strategy submodule in the decision-making layer combines the diffusion model with a sparse hybrid expert architecture, and is specifically implemented as follows: Step 4.1 Model autonomous driving decision-making as a conditional sequence generation problem. Gaussian noise is added to the action sequence demonstrated by the expert through a forward diffusion process to construct the training objective. Step 4.2 The inverse denoising process is used as a driving strategy. Based on the vehicle status, LiDAR point cloud, and navigation signal, the Gaussian noise is iteratively refined into a time-coherent action sequence. Step 4.3 introduces a sparse hybrid expert architecture as a denoising network, where each expert is an abstract driving knowledge unit, covering driving capabilities such as longitudinal adjustment, interactive processing, and lateral negotiation. Step 4.4 Build a Top-K router, dynamically activate the optimal expert subset according to the driving scenario, fuse the denoised outputs of each expert, and complete the action sequence generation; Step 4.5 introduces load balancing loss and mutual information regularization to ensure balanced utilization of experts and promote the specialization of expert knowledge.

9. The end-to-end autonomous driving decision-making method based on a hybrid expert architecture according to claim 2, characterized in that, Step S2 also includes: building a LoRAMoE lightweight plug-in module, which freezes the backbone network of the autonomous driving basic model, uses low-rank adapters as experts, integrates various LoRA experts through a router network, and introduces local equilibrium constraints. The specific steps are as follows: Step 9.1 Replace all feedforward neural network layers in the backbone network of the perception layer and decision layer with LoRAMoE layers. Each LoRAMoE layer contains multiple LoRA experts, and the expert parameters are represented by a low-rank matrix, as shown in the following formula: （4） in, Represents the individual expert weight increment, input-side low-rank matrix Output side low-rank matrix , For input dimensions, For output dimension, low-rank dimension ; Step 9.2 The router network encodes the input features and generates weight assignments for each LoRA expert to achieve dynamic collaboration among experts; Step 9.3 Local balance constraint: The training data is divided into two categories: world knowledge-related tasks and downstream tasks. In the soft constraint part, LoRA experts focus on the utilization of world knowledge, while other experts focus on improving the performance of downstream tasks, while ensuring the load balance of experts of the same type. Step 9.4 Combines the next token prediction loss and the local equilibrium constraint loss for joint training, updating only the LoRA expert and router parameters to avoid forgetting the world knowledge of the base model. The joint loss function of the LoRAMoE plugin is shown below: （5） in, The main losses for downstream tasks include token prediction loss and driving action regression loss. For local equilibrium constraint loss, To balance the hyperparameters, the weight of the local equilibrium constraint loss in the total loss is controlled.

10. An autonomous driving device or vehicle, characterized in that, The device or vehicle is capable of running the end-to-end autonomous driving decision-making method based on a hybrid expert architecture as described in claim 1, and completing multi-source sensor data preprocessing, hybrid expert network feature calculation, dynamic expert routing, driving decision generation, control command verification and output.

Images