Method and device for training end-to-end model of automatic driving based on weakly labeled data

Abstract

Embodiments of the present application disclose a method and device for training an end-to-end model of automatic driving based on weakly-labeled data, which comprises: using a weakly-labeled data set to train a model; using weakly-labeled true values and a hybrid expert model to supervise the training results respectively to obtain loss values; calculating parameter feedback gradient values to perform parameter optimization until the overall loss value of the model training meets a preset condition. According to the scheme provided by the embodiments of the present application, a high-quality model can be trained using a weakly-labeled data set, which greatly reduces the data threshold of end-to-end automatic driving and promotes the development of end-to-end models in various fields of automatic driving.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a method and apparatus for training an end-to-end autonomous driving model based on weakly labeled data. Background Technology

[0002] Traditional autonomous driving algorithms employ a modular, layered architecture. This architecture achieves autonomous vehicle operation through the collaborative division of labor among perception, localization, planning, and control. Its key features are modular design, clear division of labor among modules, simple engineering implementation, and good interpretability. However, this modular, layered architecture has significant drawbacks. For example, it suffers from insufficient algorithm generalization ability, inability to handle complex scenarios such as traffic congestion and inclement weather, complex system structure (especially the expert rule-based control module), low development efficiency, and slow algorithm iteration.

[0003] To address these challenges, the industry has proposed end-to-end autonomous driving technology solutions. End-to-end autonomous driving algorithms are an innovative technology whose core concept lies in achieving a direct mapping from raw sensor data to vehicle control commands. This algorithm abandons the explicit division of perception, prediction, planning, and control modules in traditional autonomous driving systems, instead integrating these functions through deep learning models to form a unified, end-to-end processing flow. The technical advantages of end-to-end algorithms are significant: they simplify system architecture by integrating perception, prediction, planning, and control functions into a single model, reducing information transfer losses and computational complexity between modules. Completely data-driven, end-to-end autonomous driving algorithms rely entirely on data for decision-making and control, enabling them to automatically adapt to different driving scenarios and conditions, improving the system's generalization ability. They reduce intermediate processing steps, optimize computational efficiency, and improve real-time performance. End-to-end autonomous driving algorithms can handle diverse and complex driving scenarios, improving the system's robustness in dynamic environments.

[0004] Currently, mainstream end-to-end autonomous driving algorithms in the industry include UniAD, GenAD, SparseDrive, GraphAD, GaussianAD, FusionAD, DiffAD, and others. End-to-end autonomous driving algorithms are data-driven, therefore, a common feature of these algorithms is that they are all trained on high-quality datasets. If the annotation information is incomplete or the ground truth values ​​are significantly biased, the accuracy of the trained model will be very low.

[0005] However, the cost of data collection and annotation for a single scene in high-quality datasets is in the thousands of yuan. The most widely used high-quality dataset for autonomous driving in academia is the nuScenes dataset, which contains data from 1,000 scenes. In practical industrial applications of end-to-end algorithms, to improve model stability, leading domestic and international OEMs, such as Tesla, Li Auto, and Xiaomi, have high-quality datasets containing over 10 million scene data points. The cost of such datasets is enormous, which severely limits the scope and speed of end-to-end model deployment and development. Summary of the Invention

[0006] The purpose of this invention is to provide a method and apparatus for training an end-to-end model for autonomous driving based on weakly labeled data, so as to at least partially solve the above-mentioned problems of the prior art.

[0007] To achieve the above objectives, one aspect of the present invention provides a method for training an end-to-end autonomous driving model based on weakly labeled data, comprising:

[0008] Model training is performed using a weakly labeled dataset.

[0009] The training results were supervised using both weakly labeled ground truth and hybrid expert models, resulting in the following loss values:

[0010] Loss_total = Loss_WGT * (1-w1) + Loss_MOE * w1

[0011] Where Loss_total is the overall loss result of model training, Loss_WGT is the loss of weakly labeled ground truth supervision, Loss_MOE is the loss value of hybrid expert model supervision, w1 is the weight of hybrid expert model, and 0 <w1<1;

[0012] Calculate the parameter feedback gradient value and optimize the parameters until Loss_total meets the preset condition;

[0013] Among these, model training using weakly labeled datasets includes:

[0014] Optimize low-quality datasets;

[0015] Use a low-quality dataset that has been optimized for data processing to train the model;

[0016] Data optimization includes:

[0017] Data filtering processing is used to filter data where the vehicle's speed is below a threshold;

[0018] Data framing is used to divide the filtered data into frames based on the sampling time. One frame of data includes a point cloud and image data from multiple panoramic views.

[0019] Feature extraction processing is used to perform feature extraction and feature matching for single-frame point cloud and each image data separately;

[0020] The extrinsic parameter optimization process is used to optimize the extrinsic parameters of each image data so that the error between the point cloud feature at the image projection position and the image feature position meets the preset conditions; the extrinsic parameter optimization includes lidar2camera extrinsic parameter optimization.

[0021] In the preferred scheme, the use of a hybrid expert model to supervise the training results includes:

[0022] Based on the type of low-quality dataset, selectively enable one or more corresponding expert models;

[0023] The supervision results are calculated separately using the enabled expert models;

[0024] Loss_MOE is obtained by fusing multiple supervision results through an expert fusion network.

[0025] In a preferred embodiment, the hybrid expert model includes one or more of the following models: the Dino-X expert model, the Focalformer expert model, and the LATR expert model.

[0026] In the preferred scheme, multiple supervision results are fused through an expert fusion network, including:

[0027] If the output elements of different expert models are of different categories, then zeros are used to fill in the missing output results.

[0028] In a preferred embodiment, the hybrid expert model includes expert models for different scenarios, including an expert model for point cloud 3D detection and an expert model for image traffic element perception.

[0029] Depending on the type of the low-quality dataset, selectively enable one or more corresponding expert models, including enabling one or more corresponding expert models based on the scenario type of the low-quality dataset.

[0030] The method of fusing multiple supervision results through an expert fusion network includes: assigning different weights to different expert models based on the degree of matching between the expert models and the scene types of the low-quality dataset; and comprehensively calculating the supervision results output by different expert models based on the weights.

[0031] Another aspect of the present invention provides an apparatus for training an end-to-end autonomous driving model based on weakly labeled data, comprising:

[0032] The model training module is used to train models using weakly labeled datasets.

[0033] The supervision module is used to supervise the training results using both weakly labeled ground truth and a hybrid expert model, resulting in the following loss value:

[0034] Loss_total = Loss_WGT * (1-w1) + Loss_MOE * w1

[0035] Where Loss_total is the overall loss result of model training, Loss_WGT is the loss of weakly labeled ground truth supervision, Loss_MOE is the loss value of hybrid expert model supervision, w1 is the weight of hybrid expert model, and 0 <w1<1;

[0036] The optimization module is used to calculate the parameter feedback gradient value and optimize the parameters until Loss_total meets the preset conditions.

[0037] The model training module includes:

[0038] The data optimization submodule is used to optimize low-quality datasets.

[0039] The training submodule is used to train the model using a low-quality dataset that has been optimized for data processing.

[0040] Data optimization includes:

[0041] Data filtering processing is used to filter data where the vehicle's speed is below a threshold;

[0042] Data framing is used to divide the filtered data into frames based on the sampling time. One frame of data includes a point cloud and image data from multiple panoramic views.

[0043] Feature extraction processing is used to perform feature extraction and feature matching for single-frame point cloud and each image data separately;

[0044] The extrinsic parameter optimization process is used to optimize the extrinsic parameters of each image data so that the error between the point cloud feature at the image projection position and the image feature position meets the preset conditions; the extrinsic parameter optimization includes lidar2camera extrinsic parameter optimization.

[0045] In the preferred embodiment, the supervision module is used for:

[0046] Based on the type of low-quality dataset, selectively enable one or more corresponding expert models;

[0047] The supervision results are calculated separately using the enabled expert models;

[0048] Loss_MOE is obtained by fusing multiple supervision results through an expert fusion network.

[0049] In a preferred embodiment, the hybrid expert model includes expert models for different scenarios, including an expert model for point cloud 3D detection and an expert model for image traffic element perception.

[0050] The supervision module is used for:

[0051] Based on the scenario type of the low-quality dataset, enable one or more corresponding expert models;

[0052] Based on the degree of matching between the expert model and the scene type of the low-quality dataset, different weights are set for different expert models; the supervision results output by different expert models are calculated in combination based on the weights.

[0053] Compared with the prior art, the present invention has at least the following advantages:

[0054] Using weakly labeled datasets can train high-quality models, greatly reducing the data threshold for end-to-end autonomous driving and promoting the development of end-to-end models in various fields of autonomous driving. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating a method for training an end-to-end autonomous driving model based on weakly labeled data, as provided in Embodiment 1 of the present invention.

[0056] Figure 2 This diagram illustrates a schematic block diagram of supervising training results in Embodiment 1 of the present invention.

[0057] Figure 3 A schematic block diagram of the hybrid expert model in Embodiment 1 of the present invention is shown.

[0058] Figure 4 The experimental results of the model trained using the method of this invention are shown in the figure.

[0059] Figure 5 This is a schematic diagram of the structure of an end-to-end model training device for autonomous driving based on weakly labeled data, provided in Embodiment 2 of the present invention. Detailed Implementation

[0060] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0061] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be used interchangeably where appropriate to understand the embodiments of the invention described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a product or device comprising a series of units is not necessarily limited to those explicitly listed, but may include other units not explicitly listed or inherent to such product or device.

[0062] In this invention, the terms "upper," "lower," "left," "right," "front," "rear," "top," "bottom," "inner," "outer," "middle," "vertical," "horizontal," "lateral," and "longitudinal" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are primarily for the purpose of better describing the invention and its embodiments, and are not intended to limit the indicated devices, elements, or components to having a specific orientation, or to be constructed and operated in a specific orientation.

[0063] Furthermore, some of the aforementioned terms, besides indicating direction or positional relationships, may also have other meanings. For example, the term "above" may, in certain circumstances, indicate a dependency or connection. Those skilled in the art can understand the specific meaning of these terms in this invention based on the specific circumstances.

[0064] Furthermore, the terms "installation," "setup," "equipped with," "connection," "linking," and "socketing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral structure; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium, or an internal connection between two devices, components, or parts. Those skilled in the art can understand the specific meaning of these terms in this invention based on the specific circumstances.

[0065] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0066] Example 1

[0067] Embodiment 1 of the present invention provides a method for training an end-to-end autonomous driving model based on weakly labeled data. First, the dataset is processed to improve its quality. Then, the model is trained based on the processed dataset. In addition to using ground truth, a hybrid expert model is introduced during model training to integrate the results of weakly labeled ground truth and hybrid expert model for supervision. Figure 1 A flowchart illustrating this method is shown, as follows: Figure 1 As shown, the method includes:

[0068] Step 101: Train the model using a weakly labeled dataset.

[0069] Step 102: The training results are supervised using both weakly labeled ground truth and a hybrid expert model, resulting in the following loss values:

[0070] Loss_total = Loss_WGT * (1-w1) + Loss_MOE * w1

[0071] Where Loss_total is the overall loss result of model training, Loss_WGT is the loss of weakly labeled ground truth supervision, Loss_MOE is the loss value of hybrid expert model supervision, w1 is the weight of hybrid expert model, and 0 <w1<1。

[0072] Step 103: Calculate the parameter feedback gradient value and optimize the parameters until Loss_total meets the preset conditions.

[0073] In a preferred embodiment, step 101, training the model using a weakly labeled dataset, includes:

[0074] Data optimization for low-quality datasets; and

[0075] Use a low-quality dataset that has been optimized for data processing to train the model.

[0076] Data optimization includes:

[0077] Data filtering processing is used to filter data where the vehicle's speed is below a threshold;

[0078] Data framing is used to divide the filtered data into frames based on the sampling time. One frame of data includes a point cloud and image data from multiple panoramic views.

[0079] Feature extraction processing is used to perform feature extraction and feature matching for each single-frame point cloud and each image data separately; and

[0080] The extrinsic parameter optimization process is used to optimize the extrinsic parameters of each image data so that the error between the point cloud feature at the image projection position and the image feature position meets the preset conditions.

[0081] In one example, extrinsic parameter optimization includes lidar2camera extrinsic parameter optimization. The extrinsic parameters between lidar and cameras are crucial for fusing information from lidar, cameras, and other sensors, especially in end-to-end autonomous driving algorithms based on sensor-front fusion. However, over time, sensor positions may deviate slightly, leading to inaccurate extrinsic parameters and reduced algorithm accuracy. Traditional lidar and camera calibration uses offline calibration algorithms, requiring data collection and recalculation of extrinsic parameters in specific scenarios and on specific calibration equipment. This is not only time-consuming and labor-intensive but also fails to address the issue of inaccurate extrinsic parameters from existing data. This invention employs an online extrinsic parameter optimization algorithm. Its key feature is that it does not require re-collecting data or calibration references such as calibration boards. It performs registration based on image and point cloud computing features from existing data, automatically optimizing extrinsic parameters and improving sensor fusion accuracy.

[0082] The specific steps for lidar2camera extrinsic parameter correction in this implementation plan include data filtering, data framing, feature extraction, and extrinsic parameter optimization, as detailed below:

[0083] Data filtering: Filter data with low vehicle speed, especially no rotation, to eliminate the impact of asynchronous acquisition by LiDAR and camera.

[0084] Data framing: Based on the sampling time, the filtered data is divided into frames. One frame of data includes a point cloud and image data from multiple panoramic views. The sampling time of these data is basically the same.

[0085] Feature extraction: Feature extraction and feature matching are performed for a single frame of point cloud and an image dataset, respectively. Features are generally planar features; in this implementation, the image planar feature extraction algorithm uses an image semantic segmentation algorithm, such as the SAM2 algorithm. The point cloud planar detection algorithm can use the feature extraction method from the LIO-SAM algorithm.

[0086] Extrinsic parameter optimization: Existing algorithms can be used to optimize extrinsic parameters, minimizing the error between point cloud features at image projection positions and image feature positions. The preferred extrinsic parameter optimization method is a region-to-region coarse matching algorithm. However, single-frame point cloud and image features are limited, and extrinsic parameter optimization based on single-frame information cannot achieve accurate results. To improve the robustness of the algorithm in practical applications, this implementation modifies the optimization function to fuse information from multiple scenes. The aim is to increase the number of matching features, optimize across multiple scenes, and obtain more robust and accurate results. This extrinsic parameter optimization scheme can obtain more accurate lidar2camera extrinsic parameters, improving the accuracy of subsequent algorithms.

[0087] In the preferred embodiment, step 102, which uses a hybrid expert model to supervise the training results, includes:

[0088] Based on the type of low-quality dataset, selectively enable one or more corresponding expert models;

[0089] The supervision results are calculated separately using the enabled expert model; and

[0090] Loss_MOE is obtained by fusing multiple supervision results through an expert fusion network.

[0091] In the preferred scheme, multiple supervision results are fused through an expert fusion network, including:

[0092] If the output elements of different expert models are of different categories, then zeros are used to fill in the missing output results.

[0093] In a preferred embodiment, the hybrid expert model includes expert models for different scenarios, including an expert model for point cloud 3D detection and an expert model for image traffic element perception.

[0094] Depending on the type of the low-quality dataset, one or more corresponding expert models can be selectively enabled, including enabling one or more corresponding expert models based on the scenario type of the low-quality dataset.

[0095] The method of fusing multiple supervision results through an expert fusion network includes: assigning different weights to different expert models based on the degree of matching between the expert models and the scene types of the low-quality dataset; and comprehensively calculating the supervision results output by different expert models based on the weights.

[0096] Figure 2 This diagram illustrates a schematic block diagram of the supervision of training results in this invention. Figure 3 A schematic block diagram of a hybrid expert model is shown. (Reference) Figure 2 and Figure 3 As shown, the approach to supervising the training results in this invention is to use a hybrid expert model to obtain supervisory information for each sample, thus compensating for the deficiencies of weak labels. For example... Figure 2 As shown, the original data, after data optimization, enters the training process. This training process includes forward inference based on the initial training parameters to obtain the inference result. Then, the inference result is supervised by both ground truth supervision and hybrid expert model supervision, and the output evaluations are fused to obtain Loss_total. The model parameters are optimized based on the fused evaluation result, and then inference is performed again using the optimized model parameters to obtain the inference result. This process of ground truth supervision and hybrid expert model supervision is repeated until Loss_total meets the preset conditions.

[0097] In a hybrid expert model, there can be multiple expert models. These can be selectively enabled, for example, through a routing network. The enabled expert models will calculate the supervision results, and then be fused through an expert fusion network to obtain the Loss_MOE in Formula 1.

[0098] In the preferred embodiment, the hybrid expert model includes one or more of the following models: the Dino-X expert model, the Focalformer expert model, and the LATR expert model. The Dino-X and LATR expert models are expert algorithms based on single-view images. The Dino-X expert model is a large vision-based model with very high accuracy in detecting traffic elements in 2D images. Traffic elements output during the end-to-end training of the autonomous driving model are projected onto 2D images, and the Dino-X output results are used for supervision. LATR is a 3D lane detection algorithm based on the autonomous driving forward view image. In this embodiment, its results are transformed to the BEV (Bird's Eye View) coordinate system for weak supervision of BEV lane detection. Focalformer also detects traffic participants. Unlike Dino-X, Focalformer can fuse 2D multi-view images and LiDAR to generate 3D BEV detection results without requiring data space coordinate system transformation.

[0099] Since the inputs to each expert model are different, the data in this invention is split and input to each expert according to their needs. Since the output element categories of each expert are not entirely the same, in this implementation, results without output are padded with 0. Furthermore, each expert model's output also corresponds to a mask[0,..,n-1] array: mask[i]=1 indicates the result is valid; mask[i]=0 indicates the result is invalid; n represents the number of expert models, preferably 9.

[0100] Although each expert model in the hybrid expert model adopts a large cloud model and has high perception accuracy, the information obtained by the hybrid expert model is not the true value of the data and cannot be used as hard-labeling. In this invention, it is used as a fuzzy label and used in conjunction with weakly labeled true values.

[0101] The operation of routing networks and expert fusion networks may include:

[0102] Routing Network: In this solution, each expert model has its own area of ​​expertise. For example, some expert models specialize in point cloud 3D detection, while others excel at image traffic element perception. Therefore, for different scenarios and different perceived elements, a routing network is needed to calculate the weights of each expert and rank the weights. For instance, the weights of the three expert models with the highest correlation are not 0, while the weights of other expert models are 0. For subsequent steps, if the weight = 0, that expert is turned off; if the weight > 0, that expert is turned on. The expert fusion network in this solution uses a 6-layer MLP network, and the network parameters are entirely obtained through training.

[0103] Expert Fusion Network: Each activated expert provides supervision input, which is then fused by the expert fusion network to form the input of the hybrid expert model. In this scheme, the expert fusion network can employ a 5-layer MLP network, with network parameters obtained through training. (Reference) Figure 3 The illustrated process of the expert fusion network involves obtaining the expert models corresponding to the original data through a routing network, where N represents the number of expert models. The original data corresponding to the original data undergoes supervised processing. The expert fusion network then merges the supervised results output by different expert models and outputs the fused result. To facilitate the fusion process, the outputs of different expert models can be set to a uniform format.

[0104] Compared with existing technologies, the present invention does not require high-quality datasets. It can train high-quality models using weakly labeled datasets, which greatly reduces the data threshold for end-to-end autonomous driving and promotes the development of end-to-end models in various fields of autonomous driving.

[0105] Specifically, based on the SparseDrive end-to-end model, on the weakly labeled dataset corresponding to this invention, the model accuracy comparison between this algorithm and conventional model training algorithms is shown in Table 1 below:

[0106]

[0107] Table 1

[0108] In this example, the training set is a weakly labeled dataset containing 560 scene information items, and the test set is a finely labeled dataset containing 100 scene information items. 3D Object Detection is the accuracy of traffic participant detection; a higher value is better, and the corresponding unit in the table is mAP. Boundary detection is the accuracy of traffic sign detection; a higher value is better, and the corresponding unit in the table is mAP. minADE is the error of the predicted trajectory; a lower value is better, and the corresponding unit in the table is meters (m).

[0109] Figure 4 The experimental results of the model trained using the method of this invention are shown in the figure. Figure 4 The left side shows the camera input image and the projection of the detection result, while the right side shows the BEV result of the model inference, which is relatively accurate.

[0110] Example 2

[0111] Based on the same technical concept as Embodiment 1, Embodiment 2 of the present invention provides a device for training an end-to-end model of autonomous driving based on weakly labeled data. The specific implementation of this embodiment can refer to Embodiment 1 above. Figure 5 This diagram illustrates the structure of the device for training an end-to-end autonomous driving model based on weakly labeled data, as provided in Embodiment 2. Figure 5 As shown, the device includes:

[0112] Model training module 51 is used to train models using weakly labeled datasets;

[0113] Supervision module 52 is used to supervise the training results using both weakly labeled ground truth and a hybrid expert model, resulting in the following loss value:

[0114] Loss_total = Loss_WGT * (1-w1) + Loss_MOE * w1

[0115] Where Loss_total is the overall loss result of model training, Loss_WGT is the loss of weakly labeled ground truth supervision, Loss_MOE is the loss value of hybrid expert model supervision, w1 is the weight of hybrid expert model, and 0 <w1<1;

[0116] Optimization module 53 is used to calculate the parameter feedback gradient value and optimize the parameters until Loss_total meets the preset conditions;

[0117] The model training module includes:

[0118] The data optimization submodule is used to optimize low-quality datasets.

[0119] The training submodule is used to train the model using a low-quality dataset that has been optimized for data processing.

[0120] Data optimization includes:

[0121] Data filtering processing is used to filter data where the vehicle's speed is below a threshold;

[0122] Data framing is used to divide the filtered data into frames based on the sampling time. One frame of data includes a point cloud and image data from multiple panoramic views.

[0123] Feature extraction processing is used to perform feature extraction and feature matching for single-frame point cloud and each image data separately;

[0124] The extrinsic parameter optimization process is used to optimize the extrinsic parameters of each image data so that the error between the point cloud feature at the image projection position and the image feature position meets the preset conditions; the extrinsic parameter optimization includes lidar2camera extrinsic parameter optimization.

[0125] In the preferred embodiment, the supervision module is used for:

[0126] Based on the type of low-quality dataset, selectively enable one or more corresponding expert models;

[0127] The supervision results are calculated separately using the enabled expert models;

[0128] Loss_MOE is obtained by fusing multiple supervision results through an expert fusion network.

[0129] In a preferred embodiment, the hybrid expert model includes expert models for different scenarios, including an expert model for point cloud 3D detection and an expert model for image traffic element perception.

[0130] The supervision module is used for:

[0131] Based on the scenario type of the low-quality dataset, enable one or more corresponding expert models;

[0132] Based on the degree of matching between the expert model and the scene type of the low-quality dataset, different weights are set for different expert models; the supervision results output by different expert models are calculated in combination based on the weights.

[0133] The solution provided in this embodiment can train a high-quality model using a weakly labeled dataset, which greatly reduces the data threshold for end-to-end autonomous driving and promotes the development of end-to-end models in various fields of autonomous driving.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Those skilled in the art should understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training an end-to-end autonomous driving model based on weakly labeled data, characterized in that, include: Model training is performed using a weakly labeled dataset. The training results were supervised using both weakly labeled ground truth and hybrid expert models, resulting in the following loss values: Loss_total = Loss_WGT * (1-w1) + Loss_MOE * w1 Where Loss_total is the overall loss result of model training, Loss_WGT is the loss of weakly labeled ground truth supervision, Loss_MOE is the loss value of hybrid expert model supervision, w1 is the weight of hybrid expert model, and 0 <w1<1; Calculate the parameter feedback gradient value and optimize the parameters until Loss_total meets the preset condition; Among these, model training using weakly labeled datasets includes: Optimize low-quality datasets; Use a low-quality dataset that has been optimized for data processing to train the model; Data optimization includes: Data filtering processing is used to filter data where the vehicle's speed is below a threshold; Data framing is used to divide the filtered data into frames based on the sampling time. One frame of data includes a point cloud and image data from multiple panoramic views. Feature extraction processing is used to perform feature extraction and feature matching for single-frame point cloud and each image data separately; The extrinsic parameter optimization process is used to optimize the extrinsic parameters of each image data so that the error between the point cloud feature at the image projection position and the image feature position meets the preset conditions; the extrinsic parameter optimization includes lidar2camera extrinsic parameter optimization.

2. The method for training an end-to-end autonomous driving model based on weakly labeled data according to claim 1, characterized in that, Supervising the training results using a hybrid expert model includes: Based on the type of low-quality dataset, selectively enable one or more corresponding expert models; The supervision results are calculated separately using the enabled expert models; Loss_MOE is obtained by fusing multiple supervision results through an expert fusion network.

3. The method for training an end-to-end autonomous driving model based on weakly labeled data according to claim 2, characterized in that, Hybrid expert models include one or more of the following models: Dino-X expert model, Focalformer expert model, and LATR expert model.

4. The method for training an end-to-end autonomous driving model based on weakly labeled data according to claim 2 or 3, characterized in that, Multiple supervisory results are integrated through an expert fusion network, including: If the output elements of different expert models are of different categories, then zeros are used to fill in the missing output results.

5. The method for training an end-to-end autonomous driving model based on weakly labeled data according to claim 2 or 3, characterized in that, Hybrid expert models include expert models for different scenarios, including expert models for point cloud 3D detection and expert models for image traffic element perception. Depending on the type of the low-quality dataset, selectively enable one or more corresponding expert models, including enabling one or more corresponding expert models based on the scenario type of the low-quality dataset. The method of fusing multiple supervision results through an expert fusion network includes: assigning different weights to different expert models based on the degree of matching between the expert models and the scene types of the low-quality dataset; and comprehensively calculating the supervision results output by different expert models based on the weights.

6. A device for training an end-to-end autonomous driving model based on weakly labeled data, characterized in that, include: The model training module is used to train models using weakly labeled datasets. The supervision module is used to supervise the training results using both weakly labeled ground truth and a hybrid expert model, resulting in the following loss value: Loss_total = Loss_WGT * (1-w1) + Loss_MOE * w1 Where Loss_total is the overall loss result of model training, Loss_WGT is the loss of weakly labeled ground truth supervision, Loss_MOE is the loss value of hybrid expert model supervision, w1 is the weight of hybrid expert model, and 0 <w1<1; The optimization module is used to calculate the parameter feedback gradient value and optimize the parameters until Loss_total meets the preset conditions. The model training module includes: The data optimization submodule is used to optimize low-quality datasets. The training submodule is used to train the model using a low-quality dataset that has been optimized for data processing. Data optimization includes: Data filtering processing is used to filter data where the vehicle's speed is below a threshold; Data framing is used to divide the filtered data into frames based on the sampling time. One frame of data includes a point cloud and image data from multiple panoramic views. Feature extraction processing is used to perform feature extraction and feature matching for single-frame point cloud and each image data separately; The extrinsic parameter optimization process is used to optimize the extrinsic parameters of each image data so that the error between the point cloud feature at the image projection position and the image feature position meets the preset conditions; the extrinsic parameter optimization includes lidar2camera extrinsic parameter optimization.

7. The apparatus for training an end-to-end autonomous driving model based on weakly labeled data according to claim 6, characterized in that, The supervision module is used for: Based on the type of low-quality dataset, selectively enable one or more corresponding expert models; The supervision results are calculated separately using the enabled expert models; Loss_MOE is obtained by fusing multiple supervision results through an expert fusion network.

8. The apparatus for training an end-to-end autonomous driving model based on weakly labeled data according to claim 7, characterized in that, Hybrid expert models include expert models for different scenarios, including expert models for point cloud 3D detection and expert models for image traffic element perception. The supervision module is used for: Based on the scenario type of the low-quality dataset, enable one or more corresponding expert models; Different weights are assigned to different expert models based on the degree of matching between the expert model and the scene type of the low-quality dataset. The supervision results of different expert models are calculated by combining the weights.

Images