Automatic driving trajectory prediction method based on multi-modal large model and model training thereof

Abstract

The application provides a multi-modal large model-based automatic driving trajectory prediction method and model training. The method first acquires training samples, determines splicing features according to sample real-time images, sample radar images and preset semantic guidance codes. Then, the splicing features are input into a multi-modal large model for processing, so that the model simultaneously learns visual texture and radar spatial features, improving the accuracy of the multi-modal large model and obtaining image prediction trajectories and confidence, radar prediction trajectories and confidence, and fused prediction trajectories and confidence. Then, the image prediction trajectories and confidence, the radar prediction trajectories and confidence, and the fused prediction trajectories and confidence are used to determine deviation trajectories and reference trajectories. Finally, the multi-modal large model is trained based on the deviation trajectories and the reference trajectories, obtaining a trained trajectory prediction model, which accurately corrects the prediction deviation in a single modal failure scenario and enhances the prediction accuracy and generalization ability of the model in complex scenarios.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and in particular to an autonomous driving trajectory prediction method based on a multimodal large model and its model training. Background Technology

[0002] Autonomous driving technology faces the dual challenges of multimodal perception and decision-making in complex and dynamic traffic environments. Vehicle trajectory prediction requires deep fusion of multimodal data to dynamically adjust perception and decision-making logic, ensuring that the system remains robust even when sensor data fails or becomes abnormal.

[0003] In related technologies, multimodal large models have limited ability to understand 3D sensors and model 3D sensor data, making it difficult to extract their geometric structure and semantic features. Furthermore, when one type of sensor data is abnormal, it is impossible to predict vehicle trajectories using data from another sensor.

[0004] Therefore, improving the accuracy of vehicle trajectory prediction has become an urgent technical problem to be solved. Summary of the Invention

[0005] This application provides an autonomous driving trajectory prediction method and its model training based on a multimodal large model to solve the problem of low accuracy in vehicle trajectory prediction in related technologies.

[0006] In a first aspect, embodiments of this application provide a training method for an autonomous driving trajectory prediction model based on a multimodal large model, comprising:

[0007] Acquire training samples, which include real-time images and radar images from the vehicle's historical driving process;

[0008] Based on the sample real-time image and sample radar image together with the preset semantic guided coding, the splicing features of the sample real-time image and sample radar image are determined.

[0009] The spliced ​​features are input into a multimodal large model for processing to obtain the image predicted trajectory and image trajectory confidence, radar predicted trajectory and radar trajectory confidence, and fusion predicted trajectory and fusion trajectory confidence.

[0010] Based on the image-predicted trajectory, radar-predicted trajectory, and fused predicted trajectory, the deviation trajectory is determined, and based on the confidence scores of the image trajectory, radar trajectory, and fused trajectory, the reference trajectory is determined.

[0011] Based on the deviation trajectory and the reference trajectory, a multimodal large model is trained to obtain a well-trained trajectory prediction model.

[0012] In one possible implementation, determining the deviation trajectory based on the image-predicted trajectory, the radar-predicted trajectory, and the fused predicted trajectory includes:

[0013] The average trajectory is determined based on the image-predicted trajectory, radar-predicted trajectory, and fused predicted trajectory.

[0014] Calculate the first deviation between the fused predicted trajectory and the average trajectory, the second deviation between the image predicted trajectory and the average trajectory, and the third deviation between the radar predicted trajectory and the average trajectory, respectively.

[0015] The predicted trajectory corresponding to the maximum value among the first, second, and third deviation values ​​is determined as the deviation trajectory.

[0016] In one possible implementation, a reference trajectory is determined based on image trajectory confidence, radar trajectory confidence, and fused trajectory confidence, including:

[0017] Calculate the maximum value among the image trajectory confidence score, radar trajectory confidence score, and fused trajectory confidence score;

[0018] The target trajectory corresponding to the maximum value is determined as the reference trajectory; the target trajectory includes any one of the image prediction trajectory, radar prediction trajectory, and fusion prediction trajectory.

[0019] In one possible implementation, a multimodal large model is trained based on the deviation trajectory and the reference trajectory to obtain a trained trajectory prediction model, including:

[0020] The predicted trajectory corresponding to the maximum value among the image trajectory confidence, radar trajectory confidence, and fused trajectory confidence is taken as the true trajectory value.

[0021] The features corresponding to the deviation from the trajectory are input into a multimodal large model to obtain the trajectory to be optimized;

[0022] Calculate the loss value between the trajectory to be optimized and the true value of the trajectory;

[0023] The loss value is backpropagated to the multimodal large model, and the parameters of the multimodal large model are updated until the loss function corresponding to the loss value converges, thus obtaining the trained trajectory prediction model.

[0024] In one possible implementation, the stitching features of the sample real-time image and the sample radar image are determined together with a preset semantic guidance code, including:

[0025] The real-time image and radar image of the sample are encoded separately to obtain image code and radar image code;

[0026] The image encoding and radar image encoding are concatenated together with the preset semantic guided encoding to obtain the concatenated features of the sample real-time image and the sample radar image.

[0027] Secondly, embodiments of this application provide an autonomous driving trajectory prediction method based on a multimodal large model, including:

[0028] Acquire real-time images, lidar images, and millimeter-wave radar images during vehicle operation;

[0029] The real-time image, lidar image, and millimeter-wave radar image are encoded separately to obtain image code, lidar image code, and millimeter-wave radar image code;

[0030] The image encoding, lidar image encoding, and millimeter-wave radar image encoding are concatenated together with the preset semantic-guided encoding to obtain the concatenated features of real-time image, lidar, and millimeter-wave radar.

[0031] The spliced ​​features are input into the trajectory prediction model for prediction to obtain the vehicle's predicted trajectory; the trajectory prediction model is trained according to any of the methods in the first aspect.

[0032] In one possible implementation, before acquiring real-time images, lidar images, and millimeter-wave radar images during vehicle operation, the method further includes:

[0033] Acquire real-time lidar data and real-time millimeter-wave radar data during vehicle operation;

[0034] Real-time lidar data and real-time millimeter-wave radar data are visualized to obtain lidar images and millimeter-wave radar images.

[0035] Thirdly, embodiments of this application provide a training device for an autonomous driving trajectory prediction model based on a multimodal large model, comprising:

[0036] The acquisition module is used to acquire training samples, which include real-time images and radar images of samples from the vehicle's historical driving process.

[0037] The determination module is used to determine the stitching features of the sample real-time image and the sample radar image together with the preset semantic guidance code.

[0038] The processing module is used to input the spliced ​​features into a multimodal large model for processing, to obtain the image predicted trajectory and image trajectory confidence, radar predicted trajectory and radar trajectory confidence, and fusion predicted trajectory and fusion trajectory confidence;

[0039] The determination module is also used to determine the deviation trajectory based on the image predicted trajectory, radar predicted trajectory, and fused predicted trajectory, and to determine the reference trajectory based on the confidence scores of the image trajectory, radar trajectory, and fused trajectory.

[0040] The training module is used to train a multimodal large model based on the deviation trajectory and the reference trajectory, so as to obtain a trained multimodal large model.

[0041] Fourthly, embodiments of this application provide an autonomous driving trajectory prediction device based on a multimodal large model, comprising:

[0042] The acquisition module is used to acquire real-time images, lidar images, and millimeter-wave radar images during the vehicle's driving process;

[0043] The encoding module is used to encode real-time images, lidar images, and millimeter-wave radar images respectively, to obtain image codes, lidar image codes, and millimeter-wave radar image codes.

[0044] The stitching module is used to stitch together the image encoding, LiDAR image encoding, and millimeter-wave radar image encoding with the preset semantic guidance encoding to obtain the stitched features of real-time image, LiDAR, and millimeter-wave radar.

[0045] The prediction module is used to input the spliced ​​features into the trajectory prediction model for prediction to obtain the vehicle's predicted trajectory; the trajectory prediction model is trained according to any of the methods in the first aspect.

[0046] Fifthly, embodiments of this application provide an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0047] The memory stores the instructions that the computer executes;

[0048] The processor executes computer-executable instructions stored in memory to implement the methods described in the first aspect, the second aspect, or any of the above.

[0049] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods described in the first aspect, the second aspect, or any of the above-described approaches.

[0050] Fifthly, embodiments of this application provide a computer program. The computer program product includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, it can implement the methods described in the first aspect, the second aspect, or any of the above-described methods.

[0051] The autonomous driving trajectory prediction method and its model training based on a multimodal large model provided in this application embodiment first acquires training samples. Based on the sample real-time images and sample radar images, and using a preset semantic guidance code, the splicing features of the sample real-time images and sample radar images are determined. Then, the splicing features are input into the multimodal large model for processing, allowing the model to simultaneously learn visual texture and radar spatial features, improving the accuracy of the multimodal large model, and obtaining the image predicted trajectory and image trajectory confidence, radar predicted trajectory and radar trajectory confidence, and fused predicted trajectory and fused trajectory confidence. Next, based on the image predicted trajectory, radar predicted trajectory, and fused predicted trajectory, the deviation trajectory is determined, and based on the image trajectory confidence, radar trajectory confidence, and fused trajectory confidence, a reference trajectory is determined. Finally, based on the deviation trajectory and reference trajectory, the multimodal large model is trained to obtain a trained trajectory prediction model, accurately correcting prediction biases in single-modal failure scenarios, and enhancing the model's prediction accuracy and generalization ability in complex scenarios such as rain, snow, and fog. Attached Figure Description

[0052] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0053] Figure 1 A flowchart illustrating the training method for an autonomous driving trajectory prediction model based on a multimodal large model provided in this application embodiment. Figure 1 ;

[0054] Figure 2 A flowchart illustrating the autonomous driving trajectory prediction method based on a multimodal large model provided in this application embodiment;

[0055] Figure 3 A flowchart illustrating the training method for an autonomous driving trajectory prediction model based on a multimodal large model provided in this application embodiment. Figure 2 ;

[0056] Figure 4 A schematic diagram of the structure of a training device for an autonomous driving trajectory prediction model based on a multimodal large model provided in an embodiment of this application;

[0057] Figure 5 A schematic diagram of the structure of an autonomous driving trajectory prediction device based on a multimodal large model provided in an embodiment of this application;

[0058] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0059] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0061] Before introducing the embodiments of this application, the application background of the embodiments of this application will be explained first:

[0062] Autonomous driving technology faces the dual challenges of multimodal perception and decision-making in complex and dynamic traffic environments. In real-world road scenarios, vehicles need to collect environmental information in real time using multiple sensors such as cameras, LiDAR, and millimeter-wave radar, and combine this information with high-precision maps, traffic rules, and other multimodal data to perceive surrounding dynamic targets, predict trajectories, and plan safe paths. For example, in extreme weather conditions such as rain, snow, and fog, camera images may be distorted due to reduced visibility, LiDAR point cloud data may be noisy due to raindrop interference, and millimeter-wave radar may misjudge due to reflections from metallic objects. In such cases, autonomous driving systems need to deeply fuse multimodal data and dynamically adjust perception and decision-making logic to ensure system robustness even when sensor data fails or becomes abnormal.

[0063] Furthermore, for trajectory prediction of complex dynamic targets such as pedestrians and non-motorized vehicles on urban roads, it is necessary to combine visual semantic information, language commands, and raw sensor data to achieve joint modeling of multimodal information.

[0064] In related technologies, although multimodal large models can improve perception capabilities through training with massive amounts of data, they lack the ability to model three-dimensional spatial information and lack adaptive optimization mechanisms in the case of damaged or abnormal modal data, which makes trajectory prediction results susceptible to interference and difficult to meet the high reliability and real-time requirements of autonomous driving systems.

[0065] Existing autonomous driving systems mainly rely on two types of technological approaches:

[0066] 1. Traditional model-driven solutions: These solutions separate perception, prediction, and planning processes through a modular architecture. For example, vision-based end-to-end methods directly generate driving actions through temporal modeling, but their decision-making logic lacks interpretability and struggles to handle abnormal events in complex scenarios. Furthermore, while joint modeling solutions attempt to integrate detection, tracking, and planning tasks, their multimodal data fusion strategies remain limited to two-dimensional visual information, failing to adequately utilize three-dimensional sensors such as LiDAR and millimeter-wave radar.

[0067] 2. Large Language Model-Driven Solution: This approach utilizes multimodal large models to achieve semantic understanding and decision generation for driving scenarios. For example, it uses a question-answering mechanism to parse driving tasks and a unified framework to achieve direct reasoning from multimodal inputs to actions. However, such models do not fully consider modal corruption (such as data loss and sensor noise) and modeling defects in 3D spatial information in real-world scenarios during training, leading to significant performance degradation in extreme weather or sensor failure scenarios.

[0068] Common problems with existing technologies include:

[0069] (1) The ability to model three-dimensional sensor data (such as lidar point clouds) is limited, making it difficult to extract its geometric structure and semantic features.

[0070] (2) Lacking a dynamic intermodal coordination mechanism, the system cannot maintain robustness through compensation by other modalities when modal data is abnormal.

[0071] (3) The trajectory prediction process is sensitive to noise and is easily affected by abnormal inputs, resulting in unstable output results.

[0072] In summary, improving the accuracy of vehicle trajectory prediction has become an urgent technical problem to be solved.

[0073] To address the technical problems existing in related technologies, the inventors of this application propose the following approach: For the issue of low accuracy in vehicle trajectory prediction, this application first acquires real-time sample images and sample radar images of the vehicle's historical driving. These two types of images are then concatenated with a preset semantic guided code to obtain fused features. These features are input into a multimodal large model, which simultaneously outputs the image, radar, and three fused predicted trajectories along with their corresponding confidence scores. Subsequently, the average trajectory of the three trajectories is calculated. By comparing the deviation values ​​of each trajectory from the average trajectory, the deviation trajectory corresponding to the most deviated damaged mode (i.e., image mode, radar mode, or fused mode) is determined. Simultaneously, the trajectory with the highest confidence score is selected as the reference trajectory. Using the reference trajectory as the ground truth, the features corresponding to the deviation trajectory are input into the multimodal large model. The model parameters are updated through backpropagation using a loss function, completing robust training. The resulting trajectory prediction model effectively improves the accuracy and robustness of trajectory prediction in complex scenarios.

[0074] The parts not described in detail are disclosed in the following embodiments.

[0075] The technical solution of this application will now be described in detail through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0076] It is worth noting that the application fields of the methods, apparatus, electronic devices and storage media in this application are not limited.

[0077] The subject of this application is an electronic device, which may specifically be a server, terminal device, etc.

[0078] Figure 1 A flowchart illustrating the training method for an autonomous driving trajectory prediction model based on a multimodal large model provided in this application embodiment. Figure 1 ,like Figure 1 As shown, the method may include the following steps:

[0079] Step 11: Obtain training samples.

[0080] The training samples include real-time images and radar images from the vehicle's historical driving process.

[0081] In this step, real-time sample images are acquired by the vehicle's onboard camera, covering six environmental perspectives—front, rear, left, right, upper left, and upper right—during vehicle operation. This comprehensively captures environmental information such as pedestrians, vehicles, traffic signs, and lane markings around the vehicle. Sample radar images include LiDAR bird's-eye view images and millimeter-wave radar bird's-eye view images. These are obtained from 3D point cloud data acquired by the onboard LiDAR and millimeter-wave radar through visualization processing, accurately representing spatial information such as the distance and orientation of obstacles around the vehicle. Training samples must cover various weather scenarios, including sunny, rainy, snowy, and foggy days, as well as various road conditions, such as urban roads, highways, and rural roads, ensuring the diversity and representativeness of the training samples and thus improving the generalization ability of the subsequently trained model.

[0082] Among them, the real-time sample image corresponds to the image mode, and the sample radar image corresponds to the radar mode.

[0083] Step 12: Determine the stitching features of the real-time sample image and the sample radar image together with the preset semantic guidance code.

[0084] In this step, the preset semantic guidance code is obtained by encoding semantic guidance words based on the information from the vehicle camera and radar sensor.

[0085] For example, semantic guides include the location of the vehicle's camera viewpoint within the vehicle. They also explain what information the different types of sampling points in a radar sensor represent.

[0086] Specifically, step 12 can include the following implementation methods:

[0087] Step 1: Encode the real-time image and radar image of the sample respectively to obtain the image code and radar image code.

[0088] In this implementation, a visual network model is used to extract features from the real-time sample images, outputting an image encoding with a dimension of D1. The same visual backbone network is used to extract features from the sample radar images, outputting radar image encoding with a dimension of D1, ensuring that the dimensions of the two encodings are consistent, which facilitates subsequent stitching operations.

[0089] Step 2: Combine the image encoding and radar image encoding with the preset semantic guidance encoding to obtain the spliced ​​features of the sample real-time image and the sample radar image.

[0090] In this implementation, the preset semantic guided coding includes image semantic guided coding and radar semantic guided coding. The image coding and radar image coding are concatenated with the preset semantic guided coding to obtain the fused multimodal features (i.e., the concatenated features). The specific calculation formula is as follows:

[0091]

[0092] In the formula, For splicing features, and These are radar image coding and radar semantic guidance coding, respectively. and These are image coding and image semantic guided coding, respectively.

[0093] Step 13: Input the spliced ​​features into the multimodal large model for processing to obtain the image predicted trajectory and image trajectory confidence, radar predicted trajectory and radar trajectory confidence, and fusion predicted trajectory and fusion trajectory confidence.

[0094] In this step, the multimodal large model includes a feature processing layer, a trajectory prediction layer, and a confidence output layer. The feature processing layer performs deep feature fusion on the input concatenated features. The trajectory prediction layer outputs the predicted image trajectory for the image modality, the predicted radar trajectory for the radar modality, and the fused predicted trajectory for the image and radar fusion modality based on the fused features. The confidence output layer outputs the corresponding image trajectory confidence, radar trajectory confidence, and fused trajectory confidence based on the stability and consistency of the predicted trajectories for each modality.

[0095] The confidence level ranges from 0 to 1, with a value closer to 1 indicating higher reliability of the predicted trajectory.

[0096] In one example, the image predicts the trajectory. It can be expressed as follows:

[0097]

[0098] In the formula, the Vision-Language Model (VLM) is the model processing procedure.

[0099] Radar predicted trajectory corresponding to radar mode The specific formula is as follows:

[0100]

[0101] Fusion prediction trajectory corresponding to image and radar fusion modes The specific formula is as follows:

[0102]

[0103] It is worth noting that the radar mode mentioned above is the lidar mode. In fact, the radar mode can also be the millimeter-wave radar mode. The radar mode is not specifically limited here and can be adjusted according to the actual application scenario.

[0104] Step 14: Determine the deviation trajectory based on the image predicted trajectory, radar predicted trajectory, and fused predicted trajectory, and determine the reference trajectory based on the confidence scores of the image trajectory, radar trajectory, and fused trajectory.

[0105] In this step, the deviation trajectory is used to characterize the predicted trajectory corresponding to the modality that may have data corruption in the current scenario, and the reference trajectory is used as the ground truth benchmark for model fine-tuning. By comparing the deviation trajectory with the reference trajectory during training, the robustness of the model to modality data corruption scenarios can be improved.

[0106] Specifically, the step 14, determining the deviation trajectory based on the image-predicted trajectory, radar-predicted trajectory, and fused predicted trajectory, can be implemented in the following ways:

[0107] Step 1: Determine the average trajectory based on the image-predicted trajectory, radar-predicted trajectory, and fused predicted trajectory.

[0108] In this implementation, the position coordinates of the image predicted trajectory, radar predicted trajectory, and fused predicted trajectory at the same timestamp are arithmetically averaged to obtain the average trajectory.

[0109] Step 2: Calculate the first deviation between the fused predicted trajectory and the average trajectory, the second deviation between the image predicted trajectory and the average trajectory, and the third deviation between the radar predicted trajectory and the average trajectory.

[0110] Step 3: Determine the predicted trajectory corresponding to the maximum value among the first deviation value, the second deviation value, and the third deviation value as the deviation trajectory.

[0111] Under the above implementation, the trajectory that deviates furthest from the average trajectory from the fused predicted trajectory, image predicted trajectory, and radar predicted trajectory is identified as the off-track trajectory. The specific formula is as follows:

[0112]

[0113] In the formula, To deviate from the trajectory, for , and Any one of them, The average trajectory.

[0114] At the same time, the mode corresponding to the deviation from the trajectory is regarded as the damaged mode in the current scenario, indicating that the mode has data corruption in the current scenario.

[0115] For example, if the predicted trajectory of the image modality is off-track, it indicates that the sensor data of the image modality may be corrupted in the current scene.

[0116] Additionally, determining the reference trajectory in step 14 based on the image trajectory confidence, radar trajectory confidence, and fused trajectory confidence can be implemented in the following ways:

[0117] Step 1: Calculate the maximum value among the image trajectory confidence, radar trajectory confidence, and fused trajectory confidence.

[0118] Step 2: Determine the target trajectory corresponding to the maximum value as the reference trajectory.

[0119] The target trajectory includes any one of the following: image prediction trajectory, radar prediction trajectory, and fusion prediction trajectory.

[0120] In this implementation, the target trajectory corresponding to the maximum confidence level is determined as the reference trajectory, where the maximum confidence level can be expressed by the following formula:

[0121]

[0122] In the formula, This represents the maximum confidence level. Let be the confidence level of any trajectory point in any predicted trajectory i.

[0123] Step 15: Based on the deviation trajectory and the reference trajectory, train the multimodal large model to obtain the trained trajectory prediction model.

[0124] Specifically, step 15 may include the following implementation methods:

[0125] Step 1: Take the predicted trajectory corresponding to the maximum value among the image trajectory confidence, radar trajectory confidence, and fused trajectory confidence as the true trajectory value.

[0126] In this implementation, the true trajectory value serves as the benchmark for model fine-tuning. Selecting the predicted trajectory with the highest confidence level as the true trajectory ensures the reliability of the benchmark trajectory.

[0127] Step 2: Input the features corresponding to the deviation trajectory into the multimodal large model to obtain the trajectory to be optimized.

[0128] In this implementation, the features corresponding to the deviation from the trajectory are input into the multimodal large model for processing. The multimodal large model processes the features based on the current parameters and outputs the trajectory to be optimized, which is driven only by the features corresponding to the deviation from the trajectory.

[0129] Among them, the current predictive ability of the trajectory representation model to be optimized under this modal data input.

[0130] For example, if the deviation trajectory is the image prediction trajectory, then the image encoding is input into the multimodal large model, and the corresponding trajectory to be optimized is output.

[0131] Step 3: Calculate the loss value of the trajectory to be optimized and the true value of the trajectory.

[0132] In this implementation, a loss function is used to calculate the loss value between the trajectory to be optimized and the true value of the trajectory.

[0133] For example, the loss function could be a smooth L1 loss function.

[0134] Step 4: Backpropagate the loss value to the multimodal large model and update the parameters of the multimodal large model until the loss function corresponding to the loss value converges, thus obtaining the trained trajectory prediction model.

[0135] In this implementation, the loss value is backpropagated to the multimodal large model, and the model parameters are updated using a stochastic gradient descent optimizer. The learning rate is set to 0.001, and the number of iterations is 100. When the loss value no longer decreases for 10 consecutive iterations, the loss function is considered to have converged, training is stopped, and the trained trajectory prediction model is obtained.

[0136] This application provides a training method for an autonomous driving trajectory prediction model based on a multimodal large model. The method first acquires training samples. Based on real-time images and radar images, and using a preset semantic guidance code, the stitched features of the real-time and radar images are determined. These stitched features are then input into the multimodal large model for processing, allowing the model to simultaneously learn visual texture and radar spatial features, thus improving the accuracy of the multimodal large model. This yields the image predicted trajectory and its confidence score, the radar predicted trajectory and its confidence score, and the fused predicted trajectory and its confidence score. Next, based on the image predicted trajectory, radar predicted trajectory, and fused predicted trajectory, the deviation trajectory is determined, and based on the image trajectory confidence score, radar trajectory confidence score, and fused trajectory confidence score, a reference trajectory is determined. Finally, based on the deviation trajectory and the reference trajectory, the multimodal large model is trained to obtain a trained trajectory prediction model. This accurately corrects prediction biases in single-modal failure scenarios and enhances the model's prediction accuracy and generalization ability in complex scenarios such as rain, snow, and fog.

[0137] Figure 2 A flowchart illustrating the autonomous driving trajectory prediction method based on a multimodal large model provided in this application is shown below. Figure 2 As shown, the method may include the following steps:

[0138] Step 21: Acquire real-time images, lidar images, and millimeter-wave radar images during vehicle operation.

[0139] In this step, image data, LiDAR data, and millimeter-wave radar data are all collected synchronously by multiple sensors on the autonomous vehicle. The collection frequency is set uniformly to ensure that the data are aligned in the time dimension and to avoid feature fusion deviations caused by asynchronous collection.

[0140] Specifically, prior to step 21, the autonomous driving trajectory prediction method based on a multimodal large model also includes the following implementation methods:

[0141] Step 1: Acquire real-time lidar data and real-time millimeter-wave radar data during vehicle operation.

[0142] In this implementation, real-time LiDAR data is acquired by the vehicle-mounted LiDAR sensor, and the output format is 3D point cloud data. Where K represents the number of point clouds in a single frame, and the four dimensions correspond to the x, y, z axes and reflection intensity of the point cloud, respectively. Reflection intensity can be used to distinguish the material of obstacles (such as metal vehicles and non-metallic obstacles). Real-time millimeter-wave radar data is acquired by an onboard millimeter-wave radar sensor, and the output format is also [format missing]. It has the same data dimensions as LiDAR and is capable of withstanding severe weather such as rain, snow and fog. Even in low visibility scenarios, it can still stably capture the speed and distance information of dynamic targets, making up for the performance degradation problem of LiDAR in severe weather.

[0143] Among them, the two types of radar data are transmitted to the data processing unit in real time through the vehicle bus, with the transmission delay controlled within 10ms, to ensure that the real-time performance of the data meets the requirements of autonomous driving trajectory prediction.

[0144] Step 2: Visualize the real-time lidar data and real-time millimeter-wave radar data to obtain lidar images and millimeter-wave radar images.

[0145] In this implementation, three-dimensional radar point cloud data is mapped into two-dimensional images using methods such as bird's-eye view rasterization visualization, front view projection visualization, or spherical projection visualization. This facilitates unified input into the visual encoding network for processing, aligning the data with real-time images. Through visualization, the radar's three-dimensional spatial information can be transformed into two-dimensional image features, while preserving the location, material, and distance-related information of obstacles, achieving modal unification with real-time images.

[0146] For example, the rasterized visualization processing of the bird's-eye view is as follows: First, the detection range of the bird's-eye view is set, with the vehicle's center of gravity as the origin, covering a spatial area of ​​100m in front, 50m behind, 50m to the left, and 50m to the right. This area is divided into square grids with a resolution of 0.1m × 0.1m, forming a grid matrix with a size of 1500 × 1000 pixels. Then, the radar point cloud data is preprocessed to remove ground points and noise points (achieved through height threshold and density filtering), retaining the effective obstacle point cloud. For each grid, the average reflection intensity of all point clouds falling within that grid is calculated and used as the pixel grayscale value of the grid. The higher the reflection intensity, the larger the pixel grayscale value. This process generates both LiDAR bird's-eye view images and millimeter-wave radar bird's-eye view images.

[0147] Step 22: Encode the real-time image, lidar image, and millimeter-wave radar image respectively to obtain image code, lidar image code, and millimeter-wave radar image code.

[0148] In this step, a visual network model is used to encode real-time images, LiDAR images, and millimeter-wave radar images respectively. The visual network model divides the image into 16×16 image blocks, converts them into image block embedding vectors through an embedding layer, and then extracts deep features through a multi-head attention mechanism and a feedforward neural network. Finally, the output feature vector with a dimension of 512 is used as the encoding result, namely image encoding, LiDAR image encoding, and millimeter-wave radar image encoding.

[0149] During the encoding process, the three encoding dimensions remain consistent, which not only preserves the texture features of real-time images, the accurate spatial features of LiDAR images, and the anti-interference features of millimeter-wave radar images, but also provides dimensional compatibility for subsequent feature stitching, ensuring that multimodal features can be efficiently fused.

[0150] Step 23: Combine the image encoding, lidar image encoding, and millimeter-wave radar image encoding with the preset semantic guidance encoding to obtain the stitched features of the real-time image, lidar, and millimeter-wave radar.

[0151] In this step, semantically guided encoding is used to correlate the logical relationships between features from image encoding, LiDAR image encoding, and millimeter-wave radar image encoding, providing more targeted fusion features for trajectory prediction. After concatenation, a batch normalization operation is performed to standardize the feature distribution, accelerating the subsequent model inference process and improving feature stability.

[0152] For example, by binding spatial location features from LiDAR images with lane boundary information from semantically guided coding, the model can accurately determine the vehicle's position within the lane.

[0153] Step 24: Input the spliced ​​features into the trajectory prediction model for prediction to obtain the vehicle's predicted trajectory.

[0154] Among them, the trajectory prediction model is based on Figure 1 The method corresponding to the embodiment was trained.

[0155] In this step, the spliced ​​features are input into the trajectory prediction model for fusion processing. The weights of key features are strengthened through residual networks and attention mechanisms to suppress the influence of noisy features. Based on the fused features, the vehicle's driving trajectory in the next 5 seconds is predicted. The trajectory time interval is set to 0.5 seconds, and a total of 10 consecutive trajectory points are output. Each trajectory point contains x and y coordinates and heading angle information, which fully represents the vehicle's driving path and attitude changes.

[0156] Because the vehicle trajectory prediction model is robustly trained through steps 11-15, it can effectively cope with scenarios where single-modal data is corrupted (such as blurred images in rainy weather or sparse radar point clouds). Even if some modal features are biased, it can still output a reliable trajectory through knowledge learned from multimodal fusion and modal distillation. The output vehicle prediction trajectory is corrected by the calibration layer and then transmitted to the autonomous driving decision control unit to provide accurate basis for the vehicle's acceleration, deceleration, and steering control.

[0157] The autonomous driving trajectory prediction method based on a multimodal large model provided in this application first acquires real-time images, LiDAR images, and millimeter-wave radar images during vehicle operation. Then, the real-time images, LiDAR images, and millimeter-wave radar images are encoded separately to obtain image codes, LiDAR image codes, and millimeter-wave radar image codes. These image codes, LiDAR image codes, and millimeter-wave radar image codes are then concatenated with a preset semantic guidance code to obtain a concatenated feature of the real-time image, LiDAR, and millimeter-wave radar, improving the environmental adaptability of the trajectory prediction. Finally, the concatenated feature is input into the trajectory prediction model for prediction to obtain the predicted vehicle trajectory. This reduces the impact of single-modal failure on the predicted trajectory results, improves trajectory reliability in complex scenarios, and enhances the accuracy of trajectory prediction.

[0158] In one embodiment, Figure 3 A flowchart illustrating the training method for an autonomous driving trajectory prediction model based on a multimodal large model provided in this application embodiment. Figure 2 ,like Figure 3 As shown, this method includes three steps, as detailed below:

[0159] 1. Multimodal data processing: First, the 3D point clouds of LiDAR and millimeter-wave radar are converted into bird's-eye view images, which are then input into the image feature encoder along with images from six different perspectives to obtain the features of the corresponding modes. At the same time, semantic guided coding is introduced to provide high-level logical information for subsequent fusion.

[0160] 2. Feature Fusion and Trajectory Prediction: Features from the camera, LiDAR, and millimeter-wave radar are concatenated with semantic guided coding and input into the visual language action model (i.e., a multimodal large model). Three prediction paths are output: "image-independent", "LiDAR-independent", and "fusion of all modalities". The path with the highest confidence is selected.

[0161] 3. Modal evaluation and model fine-tuning: Calculate the prediction bias between the three paths, take the mode corresponding to the path with the largest bias as input, and take the path with the highest confidence as the true value to fine-tune the model and improve its robustness in modal corruption scenarios.

[0162] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0163] Figure 4 This is a schematic diagram of the structure of a training device for an autonomous driving trajectory prediction model based on a multimodal large model, provided in an embodiment of this application. Figure 4 As shown, the device includes:

[0164] The acquisition module 41 is used to acquire training samples, which include real-time images and radar images of the vehicle during its historical driving process.

[0165] The determination module 42 is used to determine the splicing features of the sample real-time image and the sample radar image together with the preset semantic guidance code.

[0166] Processing module 43 is used to input the spliced ​​features into a multimodal large model for processing to obtain image predicted trajectory and image trajectory confidence, radar predicted trajectory and radar trajectory confidence, and fusion predicted trajectory and fusion trajectory confidence.

[0167] The determination module 42 is also used to determine the deviation trajectory based on the image predicted trajectory, the radar predicted trajectory, and the fused predicted trajectory, and to determine the reference trajectory based on the image trajectory confidence, the radar trajectory confidence, and the fused trajectory confidence.

[0168] Training module 44 is used to train a multimodal large model based on the deviation trajectory and the reference trajectory to obtain a trained trajectory prediction model.

[0169] In one possible implementation, the determining module 42 is specifically used for:

[0170] The average trajectory is determined based on the image-predicted trajectory, radar-predicted trajectory, and fused predicted trajectory.

[0171] Calculate the first deviation between the fused predicted trajectory and the average trajectory, the second deviation between the image predicted trajectory and the average trajectory, and the third deviation between the radar predicted trajectory and the average trajectory, respectively.

[0172] The predicted trajectory corresponding to the maximum value among the first, second, and third deviation values ​​is determined as the deviation trajectory.

[0173] In one possible implementation, the determining module 42 is specifically used for:

[0174] Calculate the maximum value among the image trajectory confidence score, radar trajectory confidence score, and fused trajectory confidence score;

[0175] The target trajectory corresponding to the maximum value is determined as the reference trajectory; the target trajectory includes any one of the image prediction trajectory, radar prediction trajectory, and fusion prediction trajectory.

[0176] In one possible implementation, training module 44 is specifically used for:

[0177] The predicted trajectory corresponding to the maximum value among the image trajectory confidence, radar trajectory confidence, and fused trajectory confidence is taken as the true trajectory value.

[0178] The features corresponding to the deviation from the trajectory are input into a multimodal large model to obtain the trajectory to be optimized;

[0179] Calculate the loss value between the trajectory to be optimized and the true value of the trajectory;

[0180] The loss value is backpropagated to the multimodal large model, and the parameters of the multimodal large model are updated until the loss function corresponding to the loss value converges, thus obtaining the trained trajectory prediction model.

[0181] In one possible implementation, based on the sample real-time image and the sample radar image together with a preset semantic guidance code, the stitching features of the sample real-time image and the sample radar image are determined. The determining module 42 is specifically used for:

[0182] The real-time image and radar image of the sample are encoded separately to obtain image code and radar image code;

[0183] The image encoding and radar image encoding are concatenated together with the preset semantic guided encoding to obtain the concatenated features of the sample real-time image and the sample radar image.

[0184] Figure 5 This is a schematic diagram of the structure of an autonomous driving trajectory prediction device based on a multimodal large model provided in an embodiment of this application. Figure 5 As shown, the device includes:

[0185] The acquisition module 51 is used to acquire real-time images, lidar images and millimeter-wave radar images during the vehicle's driving process;

[0186] The encoding module 52 is used to encode the real-time image, the lidar image, and the millimeter-wave radar image respectively, to obtain the image code, the lidar image code, and the millimeter-wave radar image code.

[0187] The stitching module 53 is used to stitch together the image encoding, lidar image encoding and millimeter-wave radar image encoding with the preset semantic guidance encoding to obtain the stitched features of real-time image, lidar and millimeter-wave radar.

[0188] Prediction module 54 is used to input the stitched features into the trajectory prediction model for prediction, thereby obtaining the predicted vehicle trajectory; the trajectory prediction model is based on... Figure 1 The method used in the corresponding embodiment was trained.

[0189] In one possible implementation, before acquiring real-time images, lidar images, and millimeter-wave radar images during vehicle operation, the device further includes:

[0190] The acquisition module 51 is also used to acquire real-time lidar data and real-time millimeter-wave radar data during vehicle operation.

[0191] The processing module 55 is used to perform visualization processing on real-time lidar data and real-time millimeter-wave radar data to obtain lidar images and millimeter-wave radar images.

[0192] The apparatus provided in this application embodiment can be used to execute the determination method in any of the above embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0193] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented in software via processing element calls, while others are implemented in hardware. Additionally, these modules can be fully or partially integrated together, or implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through the integrated logic circuits in the hardware of the processor element or through software instructions.

[0194] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 6 As shown, the electronic device may include: a processor 61, a memory 62, and computer program instructions stored in the memory 62 and executable on the processor 61. When the processor 61 executes the computer program instructions, it implements the method provided in any of the foregoing embodiments.

[0195] Optionally, the various components of the electronic device can be connected via a system bus.

[0196] The memory 62 can be a separate memory unit or a memory unit integrated into the processor 61. The number of processors 61 can be one or more.

[0197] It should be understood that processor 61 can be a Central Processing Unit (CPU), or other general-purpose processor 61, digital signal processor 61 (DSP), application-specific integrated circuit (ASIC), etc. The general-purpose processor 61 can be a microprocessor 61, or any conventional processor 61. The steps of the method disclosed in this application can be directly manifested as being executed by the hardware processor 61, or being executed by a combination of hardware and software modules within the processor 61.

[0198] The system bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. Memory 62 may include random access memory (RAM) 62, and may also include non-volatile memory (NVM) 62, such as at least one disk storage device 62.

[0199] All or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory 62. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned memory 62 (storage medium) includes: read-only memory 62 (ROM), RAM, flash memory 62, hard disk, solid-state hard disk, magnetic tape, floppy disk, optical disk, and any combination thereof.

[0200] The electronic device provided in this application embodiment can be used to execute the method provided in any of the above method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0201] This application provides a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the above-described method.

[0202] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0203] Optionally, a readable storage medium can be coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Alternatively, the readable storage medium can be an integral part of the processor. Both the processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components within the device.

[0204] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium, and the at least one processor can implement the above-described method when executing the computer program.

[0205] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A training method for an autonomous driving trajectory prediction model based on a multimodal large model, characterized in that, include: Obtain training samples; The training samples include real-time images and radar images from the vehicle's historical driving process. Based on the sample real-time image and the sample radar image together with the preset semantic guidance code, the splicing features of the sample real-time image and the sample radar image are determined; The stitched features are input into a multimodal large model for processing to obtain the image predicted trajectory and image trajectory confidence, radar predicted trajectory and radar trajectory confidence, and fusion predicted trajectory and fusion trajectory confidence. Based on the image predicted trajectory, the radar predicted trajectory, and the fused predicted trajectory, a deviation trajectory is determined, and based on the confidence scores of the image trajectory, the radar trajectory, and the fused trajectory, a reference trajectory is determined. Based on the deviation trajectory and the reference trajectory, the multimodal large model is trained to obtain the trained trajectory prediction model.

2. The method according to claim 1, characterized in that, The step of determining the deviation trajectory based on the image-predicted trajectory, the radar-predicted trajectory, and the fused predicted trajectory includes: The average trajectory is determined based on the image predicted trajectory, the radar predicted trajectory, and the fused predicted trajectory; Calculate the first deviation value between the fused predicted trajectory and the average trajectory, the second deviation value between the image predicted trajectory and the average trajectory, and the third deviation value between the radar predicted trajectory and the average trajectory, respectively; The predicted trajectory corresponding to the maximum value among the first deviation value, the second deviation value, and the third deviation value is determined as the deviation trajectory.

3. The method according to claim 2, characterized in that, Determining the reference trajectory based on the image trajectory confidence level, the radar trajectory confidence level, and the fused trajectory confidence level includes: Calculate the maximum value among the image trajectory confidence score, the radar trajectory confidence score, and the fused trajectory confidence score; The target trajectory corresponding to the maximum value is determined as the reference trajectory; the target trajectory includes any one of the image prediction trajectory, the radar prediction trajectory, and the fusion prediction trajectory.

4. The method according to claim 3, characterized in that, The step of training the multimodal large model based on the deviation trajectory and the reference trajectory to obtain the trained trajectory prediction model includes: The predicted trajectory corresponding to the maximum value among the image trajectory confidence, the radar trajectory confidence, and the fused trajectory confidence is taken as the true trajectory value; The features corresponding to the deviation trajectory are input into the multimodal large model to obtain the trajectory to be optimized; Calculate the loss value between the trajectory to be optimized and the true value of the trajectory; The loss value is backpropagated to the multimodal large model, and the parameters of the multimodal large model are updated until the loss function corresponding to the loss value converges, thus obtaining the trained trajectory prediction model.

5. The method according to claim 4, characterized in that, The step of determining the stitching features of the real-time sample image and the sample radar image together with a preset semantic guidance code includes: The real-time image and the radar image of the sample are encoded respectively to obtain image code and radar image code; The image encoding and the radar image encoding are concatenated together with the preset semantic guidance encoding to obtain the concatenated features of the sample real-time image and the sample radar image.

6. An autonomous driving trajectory prediction method based on a multimodal large model, characterized in that, include: Acquire real-time images, lidar images, and millimeter-wave radar images during vehicle operation; The real-time image, the lidar image, and the millimeter-wave radar image are encoded respectively to obtain image encoding, lidar image encoding, and millimeter-wave radar image encoding. The image encoding, the lidar image encoding, and the millimeter-wave radar image encoding are concatenated together with a preset semantic guidance encoding to obtain the concatenated features of the real-time image, lidar, and millimeter-wave radar. The stitched features are input into the trajectory prediction model for prediction to obtain the vehicle's predicted trajectory; the trajectory prediction model is trained by the method according to any one of claims 1-5.

7. The method according to claim 6, characterized in that, Before acquiring real-time images, lidar images, and millimeter-wave radar images during vehicle operation, the method further includes: Acquire real-time lidar data and real-time millimeter-wave radar data during the vehicle's driving process; The real-time lidar data and the real-time millimeter-wave radar data are visualized to obtain the lidar image and the millimeter-wave radar image.

8. A training device for an autonomous driving trajectory prediction model based on a multimodal large model, characterized in that, include: The acquisition module is used to acquire training samples, which include real-time images and radar images of the vehicle during its historical driving process. The determination module is used to determine the splicing features of the sample real-time image and the sample radar image together with a preset semantic guidance code. The processing module is used to input the spliced ​​features into a multimodal large model for processing to obtain the image predicted trajectory and image trajectory confidence, radar predicted trajectory and radar trajectory confidence, and fusion predicted trajectory and fusion trajectory confidence. The determining module is further configured to determine a deviation trajectory based on the image predicted trajectory, the radar predicted trajectory, and the fused predicted trajectory, and to determine a reference trajectory based on the image trajectory confidence level, the radar trajectory confidence level, and the fused trajectory confidence level; The training module is used to train the multimodal large model based on the deviation trajectory and the reference trajectory to obtain the trained trajectory prediction model.

9. An autonomous driving trajectory prediction device based on a multimodal large model, characterized in that, include: The acquisition module is used to acquire real-time images, lidar images, and millimeter-wave radar images of the vehicle during its driving process; The encoding module is used to encode the real-time image, the lidar image, and the millimeter-wave radar image respectively to obtain image encoding, lidar image encoding, and millimeter-wave radar image encoding. The stitching module is used to stitch together the image encoding, the lidar image encoding, and the millimeter-wave radar image encoding with a preset semantic guidance encoding to obtain the stitching features of the real-time image, lidar, and millimeter-wave radar. The prediction module is used to input the spliced ​​features into the trajectory prediction model for prediction to obtain the vehicle prediction trajectory; the trajectory prediction model is trained by the method according to any one of claims 1-5.

10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 5.

Images