An end-to-end automatic driving method and system based on multi-task expert fine-tuning

Abstract

The application discloses an end-to-end automatic driving method and system based on multi-task expert fine-tuning, and belongs to the technical field of computer vision image analysis and processing. The method comprises the following steps: collecting a multi-view surround view image sequence around a vehicle; performing feature extraction on the multi-view surround view image sequence through a backbone network to obtain multi-scale image features; taking the multi-scale image features, a query after position enhancement and a previous frame BEV feature as inputs, inputting the inputs into a BEV rendering module to obtain BEV feature encoding; inputting the BEV feature encoding into a mixed expert perception and prediction module, performing dynamic feature distribution between perception and prediction tasks based on the BEV feature encoding, obtaining task-specific specialized representation, and generating a future driving track of the vehicle through a planning module based on the specialized representation, and outputting the future driving track in the form of a waypoint. The application effectively improves the perception and prediction accuracy of multi-task by exploring the feature transmission of an intermediate module, and plans a safe and feasible driving track for the vehicle.

Description

Technical Field

[0001] This invention relates to the field of computer vision image analysis and processing technology, specifically to an end-to-end autonomous driving method and system based on multi-task expert fine-tuning. Background Technology

[0002] The explosive growth of artificial intelligence has driven profound iterations in autonomous driving technology, significantly reducing the risk of human-caused accidents while achieving a leapfrog improvement in traffic efficiency. At the system architecture level, autonomous driving typically integrates three core modules: perception, prediction, and planning. Through close collaboration between these modules, it lays the foundation for reliable navigation in complex environments. Inspired by the scalability of a unified architecture, autonomous driving systems are rapidly evolving towards end-to-end solutions oriented towards multi-task learning and planning. The former processes multiple sub-tasks in parallel through a combination of shared encoders and specific task heads, encompassing multiple stages such as perception integration and perception-prediction integration. However, when handling weakly correlated and highly heterogeneous tasks such as object detection and online mapping, this architecture often struggles to balance multi-objective optimization and is highly susceptible to negative transfer effects, thus limiting the overall performance of the system.

[0003] On the other hand, end-to-end autonomous driving directly maps raw observation data to the vehicle's planned trajectory or underlying control signals (such as steering angle and speed). This approach breaks the limitations of local tasks, enabling the entire network to collaboratively optimize around a global objective. Through gradient propagation throughout the process and massive data-driven approaches, it effectively suppresses cascading errors in traditional modular architectures, significantly improving the system's performance ceiling. Based on whether structured intermediate representations are retained, existing research can be divided into two main paradigms: one-stage end-to-end and modular joint end-to-end. The former simplifies the process into a single black box to pursue extreme computational efficiency, but its limited interpretability weakens safety and robustness in complex scenarios. In contrast, the modular joint approach retains intermediate representations such as perception and prediction within a differentiable framework and promotes more reliable trajectory planning learning through auxiliary task supervision. Nevertheless, this paradigm still lacks sufficient feature propagation and allocation mechanisms between modules. This ambiguity easily leads to poor multi-task collaboration, ultimately compromising driving performance, which is crucial for safety.

[0004] To address this challenge, recent research has introduced Mixture-of-Expert (MoE) architectures, aiming to achieve scalable model capacity and adaptive adaptation of dynamic features. This paradigm was first explored in large language models, driving trillion-parameter models such as Qwen3-Max, and subsequently extended to the field of computer vision. This successful practice has inspired extensive exploration of integrating MoE into autonomous driving systems, covering various methods from object detection to the entire end-to-end pipeline. For example, CBDES-Moe introduces structurally variable experts into the detection backbone network, while ARTEMIS embeds sparse hybrid experts into the planning stage of an autoregressive framework. However, existing research often limits MoE to the feature extraction stage of single-task solutions or applies it in isolation to specific modules of the end-to-end pipeline (such as the prediction module). These localized designs ignore the inherent multi-task nature of autonomous driving and fail to fully activate the enormous potential of expert modules in terms of specialization and collaboration. Furthermore, vision-centric end-to-end solutions typically rely on forward or backward perspective transformations to construct a shared bird's-eye view (BEV) representation. However, due to the uncertainty of monocular depth estimation, the viewpoint conversion process inevitably introduces geometric distortion, that is, visual features are stretched along the camera ray direction, producing striped artifacts, which in turn reduces the perception accuracy and the quality of subsequent trajectory planning.

[0005] To address the aforementioned issues, there is an urgent need for an end-to-end autonomous driving method and system based on multi-task expert fine-tuning, which can solve the problems existing in traditional methods. Summary of the Invention

[0006] The purpose of this invention is to provide an end-to-end autonomous driving method and system based on multi-task expert fine-tuning, which effectively improves the perception and prediction accuracy of multi-tasks by exploring the feature transfer of intermediate modules, and plans a safe and feasible driving trajectory for the autonomous vehicle.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: An end-to-end autonomous driving method based on multi-task expert fine-tuning includes: Step 1: Acquire a sequence of multi-view surround view images around the vehicle; Step 2: Extract features from the multi-view panoramic image sequence using the backbone network to obtain multi-scale image features; Step 3: Input the multi-scale image features, the location-enhanced query, and the previous frame BEV features into the BEV rendering module to obtain the BEV feature encoding. The BEV rendering module includes a temporal self-attention module, a spatial cross-attention module, and an implicit rendering module. The implicit rendering module includes a differentiable ray projection component, a termination probability calculation component, and a feature aggregation component. Step 4: Input the BEV feature encoding into the hybrid expert perception and prediction module. Based on the hybrid expert perception and prediction module, perform dynamic feature allocation between perception and prediction tasks to obtain task-specific professional representations. Step 5: Based on the specialized representation, the planning module generates the vehicle's future driving trajectory and outputs it in the form of waypoints.

[0008] Furthermore, the micro-light projection component is centered on the BEV origin. Projecting onto the target grid coordinates from the center. ray, Indicates spatial resolution, and by interval Perform uniform sampling, and then... The path point along the first path point The position of a ray is represented as: ; In the formula, Represents pointing to the The unit direction vector of each BEV grid, This represents the sampling depth.

[0009] Furthermore, the calculation process of the termination probability calculation component is as follows: The first The first ray of light Termination probability at individual element Defined as the product of cumulative transmittance and opacity, it is: ; In the formula, Representing the number The probability of non-occlusion in a step, This indicates element-wise operations. This represents the occupancy of the current voxel where the light has stopped. Indicates the use of bilinear interpolation From the geometric volume The opacity obtained from it It is obtained by passing through the temporal self-attention module and the spatial cross-attention module. Obtained through linear mapping, This represents the initial BEV representation constructed after spatiotemporal attention operations, and It is the number of feature channels. Indicates height dimension, It is an indicator function, which only applies when... The value is 1 at time. Representing the The coordinates of the multiple BEV grids through which the ray passes.

[0010] Furthermore, the feature aggregation component, based on the termination probability, weights and integrates diverse mesh features along the ray direction to obtain ray features, wherein a bottleneck linear layer is employed. The overall process of aggregating all contexts and reducing computational burden is as follows: ; In the formula, Represents the extracted first ray characteristics, Representational functions, only when When it is 1, Represents the unit direction vector The coordinates; The obtained ray features are passed through an upsampling linear layer Map back to the original number of feature channels ,Right now ; The geometrically enhanced discriminative BEV representation is obtained by element-wise modulation with the learned termination probability. ,for: In the formula, Representative to Striped light characteristics The resulting multidimensional matrix is ​​upsampled and integrated, and then divided along the channel dimension. Group, i.e. Represents the occupancy matrix The result obtained by reshaping the spatial grid, and satisfying The BEV characterization after multiplication by broadcast is obtained. .

[0011] Further, in step 4, the hybrid expert perception and prediction module includes a perception module and a prediction module. The perception module includes a tracking task module and a mapping task module. The tracking task module includes a tracking encoder and a tracking decoder. The mapping task module includes a mapping encoder and a mapping decoder. The prediction module includes a motion task module and an occupancy task module. The motion task module is a motion decoder, and the occupancy task module is an occupancy decoder.

[0012] Furthermore, the tracking encoder is composed of temporal self-attention, spatial cross-attention, and task-specific adaptation operators; the mapping encoder is composed of multi-scale self-attention and task-specific adaptation operators; and both the tracking decoder and the mapping decoder are composed of multi-head self-attention, deformable cross-attention, and task-specific adaptation operators.

[0013] Furthermore, the motion decoder includes two parallel BEV interaction modules and a task interaction module, both of which are composed of multi-layered deformable cross-attention and task-specific adaptation operators; the occupancy decoder is composed of temporal self-attention, cross-BEV interaction modules, and task-specific adaptation operators.

[0014] Furthermore, the task-specific adaptation operator is constructed based on the MoE mechanism. The task-specific adaptation operator includes a sparse branch and a parallel shared expert network branch. The sparse branch includes a routing mechanism and multiple sparse expert networks. Task-related tokens are selectively assigned to specific expert groups to obtain diverse representations across samples and obtain sparse features. The shared expert network branch is used to extract general representations. The general representations and sparse features are added and fused element-wise to form task-specific specialized representations.

[0015] This invention also provides an end-to-end autonomous driving system based on multi-task expert fine-tuning, applied to the aforementioned end-to-end autonomous driving method based on multi-task expert fine-tuning, comprising: Multi-view surround view image acquisition unit, used to acquire multi-view surround view image sequences around the vehicle; The bird's-eye view BEV construction unit includes a backbone network and a BEV rendering module. The backbone network is used to extract features from multi-view panoramic image sequences to obtain multi-scale image features. The BEV rendering module is used to generate BEV feature codes based on the multi-scale image features. The hybrid expert perception and prediction module is used to receive BEV feature encoding and perform dynamic feature allocation between perception and prediction tasks to obtain task-specific specialized representations. The planning module is used to generate the vehicle's future driving trajectory based on specialized representations and output it in the form of waypoints.

[0016] Furthermore, the BEV rendering module includes a temporal self-attention module, a spatial cross-attention module, and an implicit rendering module. The implicit rendering module includes a differentiable ray casting component, a termination probability calculation component, and a feature aggregation component. The BEV construction unit takes multi-scale image features, location-enhanced queries, and previous frame BEV features as inputs and outputs BEV feature codes.

[0017] In summary, the present invention has at least one of the following beneficial technical effects: 1. Improved geometric perception consistency: By introducing a bird's-eye view (BEV) implicit rendering module based on self-supervised pre-training, geometric cues are injected into the latent representation using differentiable rays and modulation operators. This effectively alleviates ray artifacts caused by monocular depth ambiguity, enhances the geometric consistency of BEV features, and thus improves the perception accuracy of the surrounding environment and the quality of subsequent trajectory planning.

[0018] 2. Achieved efficient inter-task collaboration: A set of task-specific adaptation operators were proposed, and a dual-branch design of generalist and specialist collaboration (shared branch and route-guided sparse expert network) was adopted. The hybrid expert (MoE) was reconstructed to adapt to different driving tasks, realizing the effective transmission of shared information within tasks and the accurate allocation of task perception features. This solved the problem of insufficient feature propagation and allocation mechanism in the modular end-to-end paradigm, promoted the effect of multi-task collaboration, and improved the overall driving performance.

[0019] 3. Enhanced multi-task specialization capabilities: Task-specific adaptation operators are seamlessly integrated into various functional modules, instantiated as tracking task module, map module, motion task module, and occupancy task module. Dynamic feature allocation and specialized representation are achieved in perception and prediction tasks, respectively, enhancing the model's adaptability to complex driving scenarios and enabling the system to achieve better performance in sub-tasks such as target tracking, semantic mapping, motion prediction, and occupancy prediction.

[0020] 4. Ensures the safety and rationality of planning: By training through a multi-task joint optimization method and combining perception, prediction and planning losses, the model achieves stable convergence, effectively suppresses the cascading error in the traditional modular architecture, and ultimately plans a safer and more reasonable future driving trajectory for the autonomous vehicle, significantly improving the reliability and safety of the end-to-end autonomous driving system. Attached Figure Description

[0021] Figure 1 This is a flowchart of the present invention; Figure 2 A schematic diagram of the overall architecture of an end-to-end autonomous driving expert model and system based on multi-task expert fine-tuning provided for embodiments of this application; Figure 3 A schematic diagram of an implicit rendering architecture for an end-to-end autonomous driving expert model and system based on multi-task expert fine-tuning, provided for embodiments of this application; Figure 4 This is a schematic diagram of a task-specific adaptation operator for an end-to-end autonomous driving expert model and system based on multi-task expert fine-tuning, provided in an embodiment of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0023] like Figure 1As shown, this invention provides an end-to-end autonomous driving method based on multi-task expert fine-tuning, comprising: Step 1: Acquire a sequence of multi-view surround view images around the vehicle; Step 2: Extract features from the multi-view panoramic image sequence using the backbone network to obtain multi-scale image features; Step 3: Using multi-scale image features, location-enhanced queries, and bird's-eyeview (BEV) features from the preceding frame as input, input them into the BEV rendering module to obtain BEV feature encoding. The BEV rendering module includes a temporal self-attention module, a spatial cross-attention module, and an implicit rendering module. The implicit rendering module includes a differentiable ray casting component, a termination probability calculation component, and a feature aggregation component. Step 4: Input the BEV feature encoding into the hybrid expert perception and prediction module. Based on the hybrid expert perception and prediction module, perform dynamic feature allocation between perception and prediction tasks to obtain task-specific professional representations. Step 5: Based on the specialized representation, the planning module generates the vehicle's future driving trajectory and outputs it in the form of waypoints.

[0024] First, it should be noted that this invention is based on the classic modular end-to-end autonomous driving paradigm UniAD, and the overall architecture of the proposed end-to-end autonomous driving expert model is as follows: Figure 2 As shown, it is equipped with carefully designed BEV rendering and task-specific adaptation operators. Next, combined with the overall architecture of the model, the method is summarized and explained, given a multi-view visual sequence acquired by a surround-view camera. The system performs BEV feature encoding for geometric perception through implicit rendering, then uses a hybrid expert perception and prediction module to dynamically allocate features between perception and prediction tasks, and finally generates the future trajectory through a planning module, which is formalized as a set of waypoints. .in, These represent the number of input and output frames, respectively. This refers to the number of monocular cameras positioned around the vehicle; the default is 6 fields of view. Refers to the resolution of a monocular image.

[0025] Multi-scale image features extracted from the backbone network As input, BEV rendering aims to achieve this through query-based... The attention mechanism performs viewpoint transformations to generate discriminative scene representations. In the formula Indicates spatial resolution. This represents the corresponding number of channels. Furthermore, in terms of perception and prediction, a task-specific adaptation operator with both general and specialized capabilities is constructed. This operator reconstructs a group of hybrid experts and applies them to various tasks to form a series of specialized modules, such as tracking, mapping, motion, and occupancy modules. This design aims to achieve the transmission of shared task information (generalist) and the allocation of task perception features (specialist), thereby promoting safer and more efficient autonomous driving.

[0026] The methodology section will now be explained in detail: In step 2, feature extraction is performed on the multi-view panoramic image sequence using the backbone network to obtain multi-scale image features, specifically: Feature extraction is performed on multi-view panoramic image sequences using a backbone network (e.g., ResNet50) to obtain multi-frame, multi-scale 2D image features. Subsequently, a Feature Pyramid Network (FPN) is used for multi-scale feature fusion. Through top-down paths and lateral connections, low-resolution, strong semantic features are combined with high-resolution, weak semantic features, so that each level in multiple time series has rich feature representations.

[0027] like Figure 3 As shown, in step 3, multi-scale image features are used. Location-enhanced queries and preceding frame BEV features The input is fed into the BEV rendering module to obtain the BEV feature encoding. The BEV rendering module includes a temporal self-attention module, a spatial cross-attention module, and an implicit rendering module. Considering the huge computational and memory overhead required, the rendering module is only applied in the intermediate layer, and a low-rank bottleneck design is adopted to achieve cost control. Specifically, the implicit rendering module includes a differentiable ray casting component, a termination probability calculation component, and a feature aggregation component. The three components are described in detail below: 1. Micro-light projection component The micro-light projection component is centered on the BEV origin. Centered on the target grid coordinates projection ray, and through intervals Perform uniform sampling, and then... The path point along the first path point The position of a ray is represented as: ; In the formula, Represents pointing to the The unit direction vector of each BEV grid, This represents the sampling depth.

[0028] 2. Termination Probability Calculation Component The calculation process of the termination probability calculation component is as follows: The first The first ray of light Termination probability at individual element Defined as the product of cumulative transmittance and opacity, it is: ; In the formula, Representing the number The probability of non-occlusion in a step, This indicates element-wise operations. This represents the occupancy of the current voxel where the light has stopped. Indicates the use of bilinear interpolation From the geometric volume The opacity obtained from it It is obtained by passing through the temporal self-attention module and the spatial cross-attention module. Obtained through linear mapping, Indicates height dimension, It is an indicator function, which only applies when... The value is 1 at time. This represents the initial BEV representation constructed after spatiotemporal attention operations, and It is the number of feature channels. Representing the The coordinates of the multiple BEV grids through which the ray passes.

[0029] 3. Feature aggregation component The feature aggregation component, based on termination probability, weights and integrates diverse mesh features along the ray direction to obtain ray features, wherein a bottleneck linear layer is employed. The overall process of aggregating all contexts and alleviating computational burden is as follows: ; In the formula, Represents the extracted first ray characteristics, Representational functions, only when When it is 1, Represents the unit direction vector The coordinates; The obtained ray features are passed through an upsampling linear layer Map back to the original number of feature channels ,Right now ; Finally, the geometrically enhanced discriminative BEV representation is obtained by element-wise modulation with the learned termination probability. ,for: In the formula, Representative to Striped light characteristics The resulting multidimensional matrix is ​​upsampled and integrated, and then divided along the channel dimension. Group, i.e. Represents the occupancy matrix The result obtained by reshaping the spatial grid, and satisfying Finally, the BEV characterization after multiplication by broadcast is obtained. ; It is worth noting that, considering that self-supervised learning can abstract rich and general driving knowledge from large-scale unlabeled data, a model pre-trained on an image-point cloud prediction agent task is used here as the initial weights for this method to facilitate better scene understanding.

[0030] like Figure 4 As shown, in step 4, the BEV feature encoding is input into the hybrid expert perception and prediction module. Based on the dynamic feature allocation between the perception and prediction tasks performed by the hybrid expert perception and prediction module, a task-specific specialized representation is obtained, specifically: The hybrid expert perception and prediction module includes a perception module and a prediction module. The perception module includes a tracking task module and a mapping task module. The tracking task module includes a tracking encoder and a tracking decoder. The mapping task module includes a mapping encoder and a mapping decoder. The prediction module includes a motion task module and an occupancy task module. The motion task module is a motion decoder, and the occupancy task module is an occupancy decoder. The tracking encoder consists of temporal self-attention, spatial cross-attention, and task-specific adaptation operators; the mapping encoder consists of multi-scale self-attention and task-specific adaptation operators; and both the tracking decoder and the mapping decoder consist of multi-head self-attention, deformable cross-attention, and task-specific adaptation operators. The motion decoder includes two parallel BEV interaction modules and a task interaction module. Both the BEV interaction module and the task interaction module are composed of multi-layered deformable cross-attention and task-specific adaptation operators. The occupancy decoder is composed of temporal self-attention, cross-BEV interaction modules, and task-specific adaptation operators. Task-specific adaptation operators are introduced into the encoders or decoders of the perception and prediction modules to construct corresponding task modules. The architecture of the adaptation operators in the four task modules is completely identical, only the embedding position is different: 1. Tracking and Mapping Task Modules: In their respective decoding modules, the original standard feedforward neural networks (FFNs) are replaced with the proposed adaptation operator. Notably, to avoid compromising the scene perception and understanding capabilities crucial for downstream tasks, the method does not modify the perceptual encoder used to extract key environmental information. Furthermore, since the tracking task module adopts a "detection-tracking" paradigm, it can act as a static detection module even without temporal cue input.

[0031] 2. Motion and Mapping Task Modules: In the motion prediction and occupancy prediction task modules, the proposed adaptation operators are used to replace the original FFNs layers in the decoder. Since the input to the task interaction module in motion prediction is the optimized tracking and mapping query vectors, to save computational resources, the proposed method is only inserted in the parallel BEV interaction module to capture fine-grained motion details. The introduction of these adaptation operators not only enables scalable model capacity but also promotes feature decoupling between tasks and information interaction within tasks.

[0032] A detailed introduction to task-specific adaptation operators: Modular end-to-end autonomous driving paradigms typically link multiple functional modules together in a unified optimization manner. However, this leads to ambiguity in knowledge transfer and utilization between tasks. To alleviate this difficulty, a task-specific adaptation operator is constructed based on the MoE mechanism. This task-specific adaptation operator includes a sparse branch and a parallel shared expert network branch. The sparse branch includes a routing mechanism and multiple sparse expert networks, where task-related tokens are selectively assigned to specific expert groups to obtain diverse representations across samples, resulting in sparse features. The shared expert network branch is used to extract general representations. The general representations and sparse features are then fused element-wise to form task-specific specialized representations, which will be further explained below. Task-related tokens (tokens) are first flattened into a feature matrix that can adapt to the sparse branch input dimension. In the formula Representing batch size and number of... To improve the training stability of the adaptation operator, a random noise term was injected into the input for each task, which controls the number of tokens. The formula is expressed as: ; In the formula, Represents uniform sampling, router Responsible for distributing tokens to experts, among whom... Parameterized gating networks Responsible for determining the expert network pool of Activation status: ; in, Preservation vector middle Find the largest element and set the remaining items to zero. By weighted aggregation of contributions from routing experts with domain-specific knowledge, the final sparse features are obtained. It can be formalized as: ; in, This represents a sparse expert network composed of multilayer perceptrons (MLPs), which processes matching task tokens based on gating probability values, and is parameterized as follows: Additionally, in the shared branch, input... They are fed into a shared expert network, similar to a sparse network, to generate general representations. Finally, these two paths of explicitly learning general knowledge and task-specific knowledge are added and merged element by element to form a task-specific specialized representation.

[0033] Next, the training method of the end-to-end autonomous driving expert model of the present invention will be described, specifically as follows: Its training method follows a multi-task joint optimization scheme, integrating data from perception (tracking)... Map building ) and prediction (movement) Occupy Auxiliary losses and final planning losses By combining these methods, the model achieves stable convergence, and the total loss... The formula is as follows: ; In the formula, Combined with classification Loss and positioning loss: ,and , ; Combined with methods for prospect regression loss and GIoU loss And Dice loss used for segmentation and classification and loss : ,and All other assets have a weight of 2; Includes methods for minimizing displacement error (FDE) Norm loss, used to supervise the negative log-likelihood loss of Gaussian distribution of trajectories. and multimodal classification scoring term loss : ,in , ; Dice loss used for instance-level occupancy assessment With binary cross-entropy loss (BCE) combined: ,and ; right Normative terms With variable safety distance Multi-boundary collision penalty Summation: Among them, distance-weight pairs Set as Furthermore, the model employs a two-stage training strategy: first, it performs joint optimization on the tracking-mapping task, and finally, it fine-tunes the perception-prediction-planning task.

[0034] This invention also provides an end-to-end autonomous driving system based on multi-task expert fine-tuning, applied to the aforementioned end-to-end autonomous driving method based on multi-task expert fine-tuning, comprising: Multi-view surround view image acquisition unit, used to acquire multi-view surround view image sequences around the vehicle; The bird's-eye view BEV construction unit includes a backbone network and a BEV rendering module. The backbone network is used to extract features from multi-view panoramic image sequences to obtain multi-scale image features. The BEV rendering module is used to generate BEV feature codes based on the multi-scale image features. The hybrid expert perception and prediction module is used to receive BEV feature encoding and perform dynamic feature allocation between perception and prediction tasks to obtain task-specific specialized representations. The planning module is used to generate the vehicle's future driving trajectory based on specialized representations and output it in the form of waypoints.

[0035] The BEV rendering module includes a temporal self-attention module, a spatial cross-attention module, and an implicit rendering module. The implicit rendering module includes a differentiable ray casting component, a termination probability calculation component, and a feature aggregation component. The BEV construction unit takes multi-scale image features, location-enhanced queries, and previous frame BEV features as inputs and outputs BEV feature codes.

[0036] Embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0037] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0038] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0039] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0040] Contents not described in detail in this specification are prior art known to those skilled in the art. It is hereby indicated that the above description is intended to help those skilled in the art understand this invention, but does not limit the scope of protection of this invention. Any equivalent substitutions, modifications, improvements, or simplifications of the above descriptions that do not depart from the essential content of this invention fall within the scope of protection of this invention.

Claims

1. An end-to-end autonomous driving method based on multi-task expert fine-tuning, characterized in that, include: Step 1: Acquire a sequence of multi-view surround view images around the vehicle; Step 2: Extract features from the multi-view panoramic image sequence using the backbone network to obtain multi-scale image features; Step 3: Input the multi-scale image features, the location-enhanced query, and the previous frame BEV features into the BEV rendering module to obtain the BEV feature encoding. The BEV rendering module includes a temporal self-attention module, a spatial cross-attention module, and an implicit rendering module. The implicit rendering module includes a differentiable ray projection component, a termination probability calculation component, and a feature aggregation component. Step 4: Input the BEV feature encoding into the hybrid expert perception and prediction module. Based on the hybrid expert perception and prediction module, perform dynamic feature allocation between perception and prediction tasks to obtain task-specific professional representations. Step 5: Based on specialized representations, the planning module generates the vehicle's future driving trajectory and outputs it in the form of waypoints; In step 4, the hybrid expert perception and prediction module includes a perception module and a prediction module. The perception module includes a tracking task module and a mapping task module. The tracking task module includes a tracking encoder and a tracking decoder. The mapping task module includes a mapping encoder and a mapping decoder. The prediction module includes a motion task module and an occupancy task module. The motion task module is a motion decoder, and the occupancy task module is an occupancy decoder. The tracking encoder consists of temporal self-attention, spatial cross-attention, and task-specific adaptation operators; the mapping encoder consists of multi-scale self-attention and task-specific adaptation operators; and both the tracking decoder and the mapping decoder consist of multi-head self-attention, deformable cross-attention, and task-specific adaptation operators. The motion decoder includes two parallel BEV interaction modules and a task interaction module. Both the BEV interaction module and the task interaction module are composed of multi-layered deformable cross-attention and task-specific adaptation operators. The occupancy decoder is composed of temporal self-attention, cross-BEV interaction modules, and task-specific adaptation operators. The task-specific adaptation operator is constructed based on the MoE mechanism. The task-specific adaptation operator includes a sparse branch and a parallel shared expert network branch. The sparse branch includes a routing mechanism and multiple sparse expert networks. Task-related tokens are selectively assigned to specific expert groups to obtain diverse representations across samples and obtain sparse features. The shared expert network branch is used to extract general representations. The general representations and sparse features are added and fused element-wise to form task-specific specialized representations.

2. The end-to-end autonomous driving method based on multi-task expert fine-tuning according to claim 1, characterized in that, The micro-light projection component is centered on the BEV origin. Projecting onto the target grid coordinates from the center. ray, Indicates spatial resolution, and by interval Perform uniform sampling, and then... The path point along the first path point The position of a ray is represented as: ； In the formula, Represents pointing to the The unit direction vector of each BEV grid, This represents the sampling depth.

3. The end-to-end autonomous driving method based on multi-task expert fine-tuning according to claim 2, characterized in that, The calculation process of the termination probability calculation component is as follows: The first The first ray of light Termination probability at each path sampling point Defined as the product of cumulative transmittance and opacity, it is: ； In the formula, Representing the number The probability of non-occlusion in a step, This indicates element-wise operations. This represents the occupancy of the current voxel where the light has stopped. Indicates the use of bilinear interpolation From the geometric volume The opacity obtained from it It is obtained by passing through the temporal self-attention module and the spatial cross-attention module. Obtained through linear mapping, This represents the initial BEV representation constructed after spatiotemporal attention operations, and It is the number of feature channels. Indicates height dimension, It is an indicator function, which only applies when... The value is 1 at time. Representing the The coordinates of the multiple BEV grids through which the ray passes.

4. The end-to-end autonomous driving method based on multi-task expert fine-tuning according to claim 3, characterized in that, The feature aggregation component, based on termination probability, weights and integrates diverse mesh features along the ray direction to obtain ray features, wherein a bottleneck linear layer is employed. The overall process of aggregating all contexts and reducing computational burden is as follows: ； In the formula, Represents the extracted first ray characteristics, Representational functions, only when When it is 1, Represents the unit direction vector The coordinates; The obtained ray features are passed through an upsampling linear layer Map back to the original number of feature channels ,Right now ; The geometrically enhanced discriminative BEV representation is obtained by element-wise modulation with the learned termination probability. ,for: In the formula, Representative to Striped light characteristics The resulting multidimensional matrix is ​​upsampled and integrated, and then divided along the channel dimension. Group, i.e. Represents the occupancy matrix The result obtained by reshaping the spatial grid, and satisfying The BEV characterization after multiplication by broadcast is obtained. .

5. An end-to-end autonomous driving system based on multi-task expert fine-tuning, applied to the end-to-end autonomous driving method based on multi-task expert fine-tuning as described in any one of claims 1-4, characterized in that, include: Multi-view surround view image acquisition unit, used to acquire multi-view surround view image sequences around the vehicle; The bird's-eye view BEV construction unit includes a backbone network and a BEV rendering module. The backbone network is used to extract features from multi-view panoramic image sequences to obtain multi-scale image features. The BEV rendering module is used to generate BEV feature codes based on the multi-scale image features. The hybrid expert perception and prediction module is used to receive BEV feature encoding and perform dynamic feature allocation between perception and prediction tasks to obtain task-specific specialized representations. The planning module is used to generate the vehicle's future driving trajectory based on specialized representations and output it in the form of waypoints.

6. An end-to-end autonomous driving system based on multi-task expert fine-tuning according to claim 5, characterized in that, The BEV rendering module includes a temporal self-attention module, a spatial cross-attention module, and an implicit rendering module. The implicit rendering module includes a differentiable ray casting component, a termination probability calculation component, and a feature aggregation component. The BEV construction unit takes multi-scale image features, location-enhanced queries, and previous frame BEV features as inputs and outputs BEV feature codes.

Images