Automatic driving track prediction model training method and device and electronic equipment

Abstract

The invention relates to the technical field of automatic driving, in particular to an automatic driving track prediction model training method and device and electronic equipment. According to the method, the explicit risk map is constructed and is used as guidance, so that accurate and efficient fusion of two kinds of heterogeneous sensor data of the laser radar point cloud and the camera image in the aerial view space is realized. Wherein the risk map is utilized to inject prior safety knowledge such as terrains and obstacles, so that the understanding ability of the perception system to a complex physical environment is enhanced; and the expert hybrid network is adopted for track prediction, so that the model can adaptively call the most suitable expert network according to different driving scenes (such as transportation, loading and emergency), and a professional and smooth track is generated. Finally, the end-to-end training scheme significantly improves the decision safety, trajectory smoothness and overall reliability of the automatic driving system in complex scenes such as mines.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and specifically to an autonomous driving trajectory prediction model training method, apparatus, and electronic device. Background Technology

[0002] Unstructured environments such as mines and quarries are important application scenarios for autonomous driving technology. In these unstructured environments, automated vehicle operation can significantly improve production efficiency, reduce operating costs, and ensure personnel safety. However, compared with structured environments such as urban roads, these unstructured environments present unique challenges, such as: roads without clear markings, varying road surface materials (dirt roads, gravel roads), steep slopes and uneven surfaces, and potential geometric hazards such as falling rocks and deep ditches.

[0003] Currently, most autonomous driving systems in mines adopt a modular architecture of "perception-planning-control". While this architecture is logically clear and easy to debug, it has inherent drawbacks. First, the independent design and optimization of each module can easily lead to error accumulation; that is, small errors in the perception module may be amplified in the planning and control modules, affecting the global optimality of the final decision. Second, the interfaces between modules are manually designed, which may cause information loss.

[0004] In recent years, end-to-end autonomous driving methods have demonstrated great potential in urban driving scenarios. This approach uses a unified deep learning model to directly map raw sensor data to vehicle control commands or driving trajectories, avoiding the shortcomings of modular architectures. However, applying end-to-end methods directly to mining scenarios presents new challenges: the model needs a stronger understanding of the physical risks of the terrain, rather than simply learning driving behavior. How to enable the end-to-end model to explicitly understand and avoid terrain risks while ensuring the consistency of its temporal decisions in dynamically changing environments is a pressing problem that needs to be solved in current technology.

[0005] Therefore, it is necessary to propose a new end-to-end autonomous driving method specifically designed for unstructured environments such as mines, capable of explicitly modeling terrain risks and possessing temporal prediction capabilities. Summary of the Invention

[0006] This invention provides a method, apparatus, and electronic device for training an autonomous driving trajectory prediction model, in order to address the problem that existing technologies lack end-to-end autonomous driving methods capable of displaying and modeling terrain risks in unstructured environments.

[0007] In a first aspect, the present invention provides a training method for an autonomous driving trajectory prediction model. The method includes: acquiring image data, radar point cloud data of a preset area around the vehicle, and vehicle status at multiple time points during the vehicle's journey along a preset trajectory; constructing a risk map containing a multi-channel bird's-eye view tensor based on the columnarized radar point cloud data at the current time; using the risk map as a guide, extracting features from the columnarized data and image data at the current time using a preset network, and performing cross-modal fusion using a deformable cross-attention mechanism to obtain the fused features at the current time; using an expert hybrid network including multiple expert networks, and taking the fused features and vehicle status at the current time as input, performing fusion prediction of the trajectory at the next time point; constructing a first loss function based on the predicted trajectory at the next time point and the actual trajectory at the next time point, adjusting the parameters of the preset network and the expert hybrid network, and repeating the process of constructing the risk map, feature extraction, cross-modal fusion, and trajectory prediction until a preset condition is met to obtain an autonomous driving trajectory prediction model.

[0008] This invention achieves accurate and efficient fusion of heterogeneous sensor data—LiDAR point cloud and camera images—in a bird's-eye view space by constructing an explicit risk map and using it as guidance. Specifically, the risk map incorporates prior safety knowledge such as terrain and obstacles, enhancing the perception system's understanding of complex physical environments. Furthermore, the use of an expert hybrid network for trajectory prediction allows the model to adaptively call upon the most suitable expert network based on different driving scenarios (such as transportation, loading, and emergency response), generating professional and smooth trajectories. Ultimately, this end-to-end training scheme significantly improves the decision-making safety, trajectory smoothness, and overall reliability of autonomous driving systems in complex scenarios such as mines.

[0009] In an optional implementation, the method further includes: predicting the fusion features of the next moment using a world model based on the fusion features of the current moment and the trajectory of the next moment; constructing a second loss function based on the predicted fusion features of the next moment and the fusion features of the next moment obtained through feature extraction and cross-modal fusion; adjusting the parameters of the preset network, the expert hybrid network and the world model based on the first loss function and the second loss function, and repeating the process of constructing the risk map, feature extraction and cross-modal fusion, feature prediction and trajectory prediction until the preset conditions are met, thereby obtaining a time-constrained autonomous driving trajectory prediction model.

[0010] In this invention, a self-supervised prediction task based on a world model is introduced, adding a temporal consistency constraint to the entire autonomous driving trajectory prediction model. This model not only learns to match the real trajectory at the current moment but also combines the predicted trajectory to predict the environmental state at the next moment. This mechanism optimizes the features extracted by the perception network, forcing the backbone features to possess temporal consistency and foresight.

[0011] In one optional implementation, a risk map containing a multi-channel bird's-eye view tensor is constructed based on the columnarized radar point cloud data at the current moment. This includes: calculating an obstacle channel tensor based on the average height and number of point clouds in the columnarized data; calculating a drivable channel tensor and a challenge channel tensor based on the height standard deviation of the point clouds in the columnarized data; calculating a dangerous slope channel tensor based on the height gradient of adjacent grid cells in the columnarized data; and calculating an effective observation channel tensor based on whether each grid cell in the columnarized data has a point cloud. The obstacle channel tensor, drivable channel tensor, challenge channel tensor, dangerous slope channel tensor, and effective observation channel tensor constitute a risk map containing a multi-channel bird's-eye view tensor.

[0012] In this invention, raw point cloud data is transformed into a structured risk map containing multi-dimensional information such as obstacles, drivable areas, challenging terrain, dangerous slopes, and effective observation areas. This transforms the complex environmental perception problem into an analysis of explicit, computable physical semantic channels. This rule-based risk quantification method significantly enhances the system's understanding of terrain safety and the transparency of its decisions. It provides a stable and reliable perception foundation for vehicle path planning and risk avoidance in complex, unstructured environments such as mines, thereby significantly improving the safety and robustness of autonomous driving systems.

[0013] In one optional implementation, guided by the risk map, a preset network is used to extract features from the current-time columnar data and image data, and a deformable cross-attention mechanism is used for cross-modal fusion to obtain the current-time fused features. This includes: generating spatial gating based on the risk map using a first preset network; performing position-by-position gating processing on the BEV feature map using the spatial gating to obtain gating results, where the BEV feature map is obtained by extracting features from the columnar data using a second preset network; processing the risk map using a third preset network and combining the gating results to obtain point cloud features; extracting features from the image data using a fourth preset network and inverse perspective transformation projection to obtain image features; performing gating processing on the image features using the spatial gating to obtain processed image features; and using a deformable cross-attention mechanism to perform cross-modal fusion of the point cloud features and the processed image features to obtain the current-time fused features.

[0014] In this invention, by introducing an explicit, physically-based risk map as prior knowledge and utilizing a spatial gating mechanism to dynamically modulate the feature extraction process of point clouds and images, precise guidance and deep fusion of cross-modal perception are achieved. Simultaneously, a deformable cross-attention mechanism is employed during fusion, enabling the precise geometric information of the point cloud to actively and sparsely query the most relevant visual texture features, effectively overcoming sensor calibration errors and spatial misalignment problems.

[0015] In one optional implementation, a deformable cross-attention mechanism is used to fuse radar features and processed image features across modalities to obtain the fused features at the current time. This includes: performing multiple convolution operations on the radar features and processed image features to obtain two multi-scale feature pyramids; using a deformable cross-attention fusion network to fuse the vectors obtained by flattening and splicing the two multi-scale feature pyramids; and flattening, upsampling, and convolving the fused features to obtain the fused features at the current time.

[0016] This invention achieves deep and accurate fusion of radar and image features across multiple scales through a deformable cross-attention fusion network. This mechanism dynamically focuses on the most relevant local regions across modal data, thereby generating richer and more robust environmental feature representations. This fused feature provides a more reliable foundation for subsequent perception and decision-making tasks, significantly improving the overall performance of autonomous driving systems in complex scenarios.

[0017] In one optional implementation, an expert hybrid network comprising multiple expert networks is employed to perform fusion prediction of the trajectory at the next time step, using the fusion features and vehicle state at the current time step as input. This includes: determining the weights of each expert network using a gating network based on the fusion features and vehicle state at the current time step; encoding the fusion features and vehicle state separately for each expert network and adding position information to obtain processed fusion features and vehicle state; interactively decoding the processed fusion features and vehicle state using a Transformer decoder to obtain the predicted trajectory of each expert network; and fusing and predicting the trajectory at the next time step based on the predicted trajectory of each expert network and the weights of each expert network.

[0018] This invention employs an expert hybrid network architecture for trajectory prediction. A gating network dynamically evaluates the weights of each sub-expert (such as transportation, loading, and emergency experts) in the current scenario, achieving adaptive and precise matching for different working conditions. This mechanism integrates the decision-making advantages of experts from various fields and, based on real-time vehicle status and environmental characteristics, performs deep interactive reasoning through a Transformer decoder to ultimately generate an optimal trajectory that is both professional and realistic. This method significantly improves the accuracy, smoothness, and robustness of trajectory prediction under complex working conditions, effectively addressing the diversity and uncertainty of mining scenarios.

[0019] In one optional implementation, based on the fusion features at the current moment and the trajectory at the next moment, a world model is used to predict the fusion features at the next moment, including: encoding the fusion features at the current moment and the trajectory at the next moment respectively, and adding position information to obtain processed fusion features and trajectory; using a multi-layer Transformer network to perform interactive attention calculation on the processed fusion features and trajectory to obtain calculation results; and normalizing, projecting, and reshaping the calculation results to predict the fusion features at the next moment.

[0020] Secondly, the present invention provides an autonomous driving trajectory prediction model training device, the device comprising: a data acquisition module, used to acquire image data at multiple times, radar point cloud data of a preset area around the vehicle, and vehicle status during the process of the vehicle driving according to a preset trajectory; a risk map construction module, used to construct a risk map containing a multi-channel bird's-eye view tensor based on the columnar processing data of the radar point cloud data at the current time; a feature extraction and fusion module, used to extract features from the columnar processing data and image data at the current time using a preset network guided by the risk map, and to perform cross-modal fusion using a deformable cross-attention mechanism to obtain the fused features at the current time; a prediction module, used to perform fusion prediction of the trajectory at the next time using an expert hybrid network including multiple expert networks, with the fused features at the current time and the vehicle status as input; and a training module, used to construct a first loss function based on the predicted trajectory at the next time and the actual trajectory at the next time, adjust the parameters of the preset network and the expert hybrid network, and repeat the process of constructing the risk map, feature extraction, cross-modal fusion, and trajectory prediction until a preset condition is met to obtain an autonomous driving trajectory prediction model.

[0021] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the autonomous driving trajectory prediction model training method of the first aspect or any corresponding embodiment described above.

[0022] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the autonomous driving trajectory prediction model training method of the first aspect or any corresponding embodiment described above.

[0023] Fifthly, the present invention provides a computer program product, including computer instructions, which are used to cause a computer to execute the autonomous driving trajectory prediction model training method of the first aspect or any corresponding embodiment described above. Attached Figure Description

[0024] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of the first process of the autonomous driving trajectory prediction model training method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the second process of the autonomous driving trajectory prediction model training method according to an embodiment of the present invention; Figure 3 This is a structural block diagram of an autonomous driving trajectory prediction model training device according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0028] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0029] According to an embodiment of the present invention, an embodiment of a training method for an autonomous driving trajectory prediction model is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0030] This embodiment provides a method for training an autonomous driving trajectory prediction model. Figure 1 This is a flowchart of an autonomous driving trajectory prediction model training method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps: Step S101 involves acquiring image data at multiple time points, radar point cloud data of a preset area surrounding the vehicle, and vehicle status data during the vehicle's journey along a preset trajectory. The process of the vehicle traveling along the preset trajectory refers to the operation of a test vehicle equipped with sensors such as LiDAR, cameras, and high-precision GPS, driven by a professional driver in an unstructured environment such as a mine or quarry, following a specific route. This route covers all typical operational scenarios, such as routes for transportation on main roads, routes for operations in loading and unloading areas, and routes for some emergency situations.

[0031] During the journey along these routes, the vehicle's sensors simultaneously record radar point cloud data, image data, and vehicle status data, resulting in image data and radar point cloud data at multiple points in time. The radar point cloud data can be obtained by LiDAR scanning the area around the vehicle, such as a 64m radius in front of and behind it. Vehicle status data includes speed, acceleration, heading / turning angle, current gear, and load.

[0032] Step S102: Construct a risk map containing a multi-channel bird's-eye view tensor based on the columnarized radar point cloud data at the current moment. Specifically, the radar point cloud data from LiDAR scanning is a collection of tens of thousands of points in three-dimensional space. This point cloud data is disordered, sparse, and unevenly dense, making it unfavorable for network model processing. Therefore, the radar point cloud data is first columnarized. During columnarization, the area around the vehicle is first divided into grids on a horizontal plane, with each grid extending infinitely in the vertical direction. Each grid in the vertical direction is a pillar. Then, each point cloud is assigned to its corresponding pillar, obtaining the point cloud feature data in each non-empty pillar. These point cloud feature data are then scattered onto the BEV (Bird's Eye View) pseudo-image, thus obtaining the columnarized data (point cloud pillars). ,in, Indicates batch size. This is the maximum number of Pillars allowed. It is the maximum number of points allowed in each Pillar. This refers to the feature dimension of the LiDAR point, while Pillar's coordinates within the BEV grid are represented as... .

[0033] The risk map represents the different terrain risks around the vehicle, such as the presence of obstacles, steep slopes, and drivability. In other words, the risk map contains multi-dimensional tensors representing different terrain risks. Each dimension's tensor can be calculated using point cloud feature data from each pillar; that is, a multi-dimensional tensor needs to be calculated separately for each pillar. Thus, the multi-dimensional tensors from multiple pillars constitute the risk map.

[0034] Step S103: Guided by the risk map, a preset network is used to extract features from the current time-stacked columnar data and image data, and a deformable cross-attention mechanism is used for cross-modal fusion to obtain the fused features at the current time. Specifically, during feature extraction, different networks are used to extract features from the columnar data of the point cloud data and the image data respectively. During feature extraction of the point cloud data and image data, a spatial gating is generated based on the risk map. This spatial gating guides the network to focus on or ignore certain areas by adjusting the weights of different locations (e.g., suppressing high-risk areas and increasing the weights of reliable areas).

[0035] Subsequently, a deformable cross-attention mechanism is employed to fuse the extracted features from different modalities (point cloud geometric features and image features) across modalities. This mechanism uses one modality (e.g., point cloud) as the primary modality, dynamically predicting the coordinates of the most relevant reference points in another modality (e.g., image) through learnable offset parameters. Attention computation no longer involves global matching across the entire feature map, but rather weighted feature aggregation only within local regions of these predicted, sparsely populated reference points. This mechanism allows the model to adaptively focus on the most informative local regions in the cross-modal data, significantly reducing computational complexity.

[0036] Step S104: An expert hybrid network, comprising multiple expert networks, is used to perform fusion prediction of the trajectory at the next moment, taking the fusion features and vehicle status at the current moment as input. Specifically, in the expert hybrid network, corresponding expert networks are pre-constructed for different operation scenarios, such as transportation, loading, unloading, and emergency operations. Based on the fusion features and vehicle status, the expert networks can be dynamically activated, and the expert networks can predict the trajectory at the next moment. The trajectory at the next moment is obtained by fusing the prediction results of the expert networks in the expert hybrid network.

[0037] Step S105: Construct a first loss function based on the predicted trajectory at the next moment and the actual trajectory at the next moment, adjust the parameters of the preset network and the expert hybrid network, and repeat the process of constructing the risk map, feature extraction, cross-modal fusion and trajectory prediction until the preset conditions are met to obtain the autonomous driving trajectory prediction model.

[0038] Specifically, the actual trajectory at the next moment can be determined by a preset trajectory during the vehicle's movement. The first loss function can be either the L1 loss between the predicted trajectory and the actual trajectory or a smooth L1 loss. The parameters of all network models used in the trajectory prediction process are adjusted using this loss function, and steps S102 to S104 are repeated. The first loss function is recalculated using the new predicted trajectory and the actual trajectory, and the parameters are adjusted again. This process is repeated continuously until the required number of iterations or other preset conditions are met. At this point, the network model used in the above steps can be solidified as an autonomous driving trajectory prediction model.

[0039] Based on this autonomous driving trajectory prediction model, for the autonomous driving process of vehicles in unstructured environments, the point cloud data, vehicle status, and image data used by the autonomous vehicle can be processed according to the processing procedure of the autonomous driving trajectory prediction model, namely, constructing a risk map at the current moment, performing feature extraction and cross-modal fusion, and predicting the trajectory at the next moment. Thus, the autonomous vehicle can drive along the predicted trajectory.

[0040] This embodiment provides a method for training an autonomous driving trajectory prediction model, which includes the following steps: Step S201 involves acquiring image data at multiple time points, radar point cloud data of a preset area surrounding the vehicle, and vehicle status data while the vehicle is traveling along a preset trajectory. For details, please refer to [link to relevant documentation]. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0041] Step S202: Construct a risk map containing multi-channel bird's-eye view tensors based on the columnarized data of the radar point cloud data at the current moment.

[0042] Specifically, step S202 includes: Step S2021: Calculate the obstacle channel tensor based on the average height and number of point clouds in the columnarized data. Specifically, for the obstacle channel tensor of each pillar, this embodiment uses the average height of the point clouds (i.e., the average z-coordinate of all point clouds in the pillar) and the number of point clouds (i.e., the number of point clouds belonging to the pillar) to determine the obstacle channel tensor. Wherein, when the average height is greater than or equal to a height threshold... And the number of point clouds is greater than or equal to the point cloud threshold. If the value is positive, it indicates that there may be obstacles in the Pillar. In this case, the obstacle channel tensor of the Pillar can be marked as 1; otherwise, it can be marked as 0. In other embodiments, the specific obstacle channel tensor value can also be calculated using weighted formulas based on different point cloud heights and point cloud quantities.

[0043] Step S2022: Calculate the drivable path tensor and the challenge path tensor based on the height standard deviation of the point cloud in the columnarized data. Specifically, the height standard deviation represents the standard deviation of the z-coordinates of all point clouds in each pillar. In this embodiment, the drivable path tensor is determined by a first threshold: when the height standard deviation is less than or equal to the first threshold, the drivable path tensor is 1; otherwise, it is 0. Simultaneously, the challenge path tensor is determined by a first threshold and a second threshold: when the height standard deviation is greater than the first threshold and less than or equal to the second threshold, the challenge path tensor is 1; otherwise, it is 0. The drivable path tensor can be used to characterize flat, drivable areas, while the challenge path tensor is used to characterize bumpy but still passable areas.

[0044] Step S2023: Calculate the dangerous slope passage tensor based on the height gradient of adjacent grid cells in the columnarized data. Specifically, the height gradient of adjacent grid cells represents the height change rate of adjacent pillars. If a pillar exists in all four directions (up, down, left, and right), the height gradient needs to be determined by the height change rate in each of these four directions. The dangerous slope passage tensor is 1 if the height gradient is greater than or equal to a third threshold, and 0 otherwise.

[0045] Step S2024: Based on whether each grid cell in the columnarized data has a valid observation channel tensor and an obstacle channel tensor, perform point cloud computing. Driveable channel tensor Challenge Channel Tensor Dangerous slope passage tensor and effective observation channel tensor A risk map is constructed by including multi-channel bird's-eye view tensors. Specifically, the effective observation channel tensor is 1 when point clouds exist in each pillar, and 0 otherwise. That is, the effective observation channel tensor represents whether the lidar has scanned the area; if the tensor is 0, it means the area has not been scanned, and the pillar can be considered a blind spot. Using the tensors calculated above, the risk map is represented as follows: ,in, This is the number of feature channels in the risk map (in this embodiment, the number of channels is 5). These are the height and width of the BEV grid, respectively.

[0046] By constructing a risk map, its multi-channel physical semantic tensor As explicit risk priors, terrain geometry and traversal difficulty are made explicit, providing interpretable safety constraints and spatial selectivity for subsequent perception and decision-making. This also enhances the network's interpretability and controllability, and significantly improves the reliability of perception of drivable boundaries and dangerous slopes.

[0047] Step S203: Guided by the risk map, a preset network is used to extract features from the current time-based columnar data and image data, and a deformable cross-attention mechanism is used to perform cross-modal fusion to obtain the fused features at the current time.

[0048] Specifically, step S203 includes: Step S2031: Based on the risk map, a spatial gating system is generated using a first preset network; specifically, the spatial gating system is generated using the first preset network corresponding to the following formula:

[0049] In the formula, This represents the convolution operation. express Activation function express This represents the activation function.

[0050] Step S2032: The spatial gating is used to perform position-by-position gating processing on the BEV feature map to obtain the gating processing result. The BEV feature map is obtained by extracting features from the columnarized data using a second preset network. Specifically, the BEV feature map is generated using the second preset network corresponding to the following formula:

[0051] In the formula, P represents the data processed by the column element. Indicates adoption Process P. Represents the scattering operation, where, It is a feature extractor for point clouds, but it is not a fixed standard network like ResNet-50. It can be understood as an architectural paradigm that utilizes 2D CNNs to efficiently process Pillarized point clouds. Therefore, This represents the cylindrical features obtained by feature extraction from P. For the extracted cylindrical features, a scattering process is performed based on their BEV grid coordinate set C, that is, the cylindrical features are scattered onto the BEV grid to form the initial BEV feature map of the geometric flow. Here, C contains the BEV grid index set corresponding to each non-empty cylindrical element, and empty cylindrical elements are placed with 0.

[0052] The gating process is represented by the following formula:

[0053] In the formula, This indicates element-wise multiplication.

[0054] Step S2033: The risk map is processed using a third preset network, and point cloud features are obtained by combining the gating processing results. Specifically, the process of processing the risk map using the third preset network is expressed by the following formula:

[0055] The process of processing risk maps can be understood as the feature extraction process of risk maps, and the above... This can be understood as a point cloud The feature extraction process, namely the third preset network and This constitutes a two-stream feature extraction network. For features extracted from the risk map... And the above point cloud The result after gating the features First, the points are concatenated, and then convolution is performed to obtain the final extracted point cloud features. These point cloud features are determined using the following formula:

[0056]

[0057] In the formula, This indicates splicing along the channel dimension; This represents the features of a point cloud.

[0058] Step S2034: The image data is processed using a fourth preset network and inverse perspective transformation projection to extract image features. Specifically, the image feature extraction process is represented by the following formula:

[0059]

[0060] In the formula, Indicates adoption For image data Process it; Indicates the use of a sampling grid right Perform inverse perspective transformation projection processing. These represent the height and width of the image, respectively. Wherein, the height and width are used... The image feature map obtained after processing image data I is in a perspective view (camera coordinate system). Therefore, inverse perspective transformation projection is used to convert it into BEV features aligned with the point cloud features, i.e. This represents the feature representation of image data in BEV space, and... They exist in the same coordinate system and spatial dimensions. For the sampling grid... It is a continuous sampling coordinate grid of the image feature plane, determined by both camera intrinsics and the sensor coordinate system. This sampling grid defines each point in the BEV space and should be used to access the image feature map. Which location on the screen should the features be collected from?

[0061] Step S2035: The image features are gated using the spatial gating to obtain the processed image features; specifically, the gating process is represented by the following formula:

[0062] This gating process is used to modulate the spatial location of image features, suppressing high-risk areas and increasing the weight of reliable areas.

[0063] Step S2036 involves performing multiple convolution operations on the point cloud features and the processed image features to obtain two multi-scale feature pyramids. Specifically, before fusion, a multi-scale feature pyramid is constructed based on the features of different modalities, i.e., features of different depths are obtained and combined to obtain a set of features for multi-scale information fusion. The multi-scale feature pyramid is determined using the following formula:

[0064]

[0065]

[0066]

[0067]

[0068]

[0069]

[0070]

[0071] Step S2037 involves fusing the vectors obtained by flattening and stitching the two multi-scale feature pyramids using a deformable cross-attention fusion network. Specifically, this fusion process is represented by the following formula:

[0072]

[0073]

[0074]

[0075] In the formula, This represents multi-scale deformable attention. Representation layer normalization, This represents a feedforward network.

[0076] Among them, when using multi-scale deformable attention processing, firstly... Flatten and splice together, soon Each layer Flatten each vector into a single vector, then concatenate them together to obtain... (i.e., the Query of the attention mechanism); Flatten and splice together, soon Each layer Flatten each vector into a single vector, then concatenate them together to obtain... and (i.e., the Key and Value of the attention mechanism). Specifically, for and Can be respectively by The flattened and stitched vectors are obtained by projecting them through different learnable linear layers. Indicates positional encoding, because... Once flattened, its inherent spatial information is lost, and Then including The encoding vector corresponding to the coordinates of each token in the original BEV space.

[0077] This represents a reference point. In standard attention mechanisms, it is necessary to calculate... Each token in The correlation between all tokens, but in deformable attention, the entirety is no longer considered. Instead, first Each token's BEV coordinates serve as an initial reference point. An offset is then predicted, and the reference point is obtained by passing the offset from the initial reference point. At this point, the attention mechanism only needs to... The model samples and calculates values ​​near these reference points. The predicted offsets are learned during model training. Therefore, by using a deformable attention mechanism, the computational cost is significantly reduced, and the network can dynamically and adaptively determine which locations to focus on.

[0078] Step S2038 involves flattening, upsampling, and convolutionally processing the fused features to obtain the fused features at the current time step. Specifically, the fused features at the current time step are determined using the following formula:

[0079]

[0080]

[0081]

[0082] In the formula, F represents the fusion feature. This indicates upsampling. Wherein, Indicates will The result is obtained by restoring the original dimensions to the original dimensions after partitioning the multi-scale feature pyramid by token count. This way, it will be derived from... The information that was integrated there was restored to its original state. Multiscale features that perfectly correspond to scale structure Then, FPN-style upsampling fusion can be performed to obtain a final feature map.

[0083] Step S204: Using a hybrid expert network that includes multiple expert networks, the trajectory of the next moment is fused and predicted by taking the fusion features and vehicle status at the current moment as input.

[0084] Specifically, step S204 includes: Step S2041: Based on the fused features and vehicle state at the current moment, a gating network is used to determine the weights of each expert network. Specifically, the expert networks set in the expert hybrid network can be determined based on the action scenario covered by the autonomous driving system. For example, the expert networks constructed in this embodiment include a transportation expert network, a loading area expert network, an unloading area expert network, and an emergency expert network, etc. When generating weights, the fused features can be globally pooled, and then the vector formed by the fused features and the vehicle state can be input into a gating network (this gating network can be a small multilayer perceptron network or a Transformer gating network) to obtain the weights of each expert network.

[0085] Step S2042: For each expert network, the fused features and vehicle state are encoded separately, and location information is added to obtain the processed fused features and vehicle state; specifically, the processing procedure of this step is represented by the following formula:

[0086]

[0087]

[0088]

[0089] In the formula, This indicates that the fused features are encoded at location. This indicates that the encoded features will be flattened. Indicates vehicle status Encode, Feature dimensions representing vehicle status.

[0090] in, This represents a two-dimensional location encoding function, which is calculated based on the feature map itself and is used to characterize the feature location. Adding this function to the fused features is equivalent to adding a clear location label to each feature point. This represents the encoder (such as a multilayer perceptron or other coding network) corresponding to the i-th expert. Different expert networks use different encoders, meaning that different experts process the same vehicle state in different ways. This represents a learnable parameter; in this embodiment, it is a learnable query, the length of which is equal to the length of the desired output trajectory.

[0091] Step S2043: The processed fused features and vehicle state are interactively decoded using a Transformer decoder to obtain the predicted trajectory of each expert network; specifically, the predicted trajectory is determined using the following formula:

[0092]

[0093] In the formula, Indicates the method used by the i-th expert network. Decoder module. This represents the prediction head. The decoder module uses... Model pair and Interactive decoding is performed, and the decoding result is input into the prediction head to obtain the predicted trajectory of the i-th expert network. This prediction head can be a linear layer or a small multilayer perceptron, used to process abstract trajectory features. Mapped to specific BEV coordinates.

[0094] Step S2044: Based on the predicted trajectory of each expert network and the weights of each expert network, the predicted trajectory for the next time step is obtained by fusing the predictions. Specifically, the predicted trajectories obtained by each expert network in the expert hybrid network are weighted and summed to obtain the trajectory for the next time step. ,Right now , This represents the weight of the i-th expert network.

[0095] Step S205: Based on the predicted trajectory at the next moment and the actual trajectory at the next moment, a first loss function is constructed. The parameters of the preset network and the expert hybrid network are adjusted, and the processes of risk map construction, feature extraction, cross-modal fusion, and trajectory prediction are repeated until the preset conditions are met, resulting in an autonomous driving trajectory prediction model. For details, please refer to [link to details]. Figure 1 Step S105 of the illustrated embodiment will not be described again here.

[0096] This embodiment provides a method for training an autonomous driving trajectory prediction model, which includes the following steps: Step S301: Acquire image data at multiple time points, radar point cloud data of a preset area around the vehicle, and vehicle status while the vehicle is traveling along a preset trajectory; for details, please refer to [link to relevant documentation]. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0097] Step S302: Construct a risk map containing a multi-channel bird's-eye view tensor based on the columnarized radar point cloud data at the current moment; for details, please refer to [link to relevant documentation]. Figure 1 Step S102 of the illustrated embodiment will not be described again here.

[0098] Step S303: Guided by the risk map, a preset network is used to extract features from the current time-based columnar data and image data, and a deformable cross-attention mechanism is used for cross-modal fusion to obtain the fused features at the current time; for details, please refer to [link to relevant documentation]. Figure 1 Step S103 of the illustrated embodiment will not be described again here.

[0099] Step S304: An expert hybrid network, comprising multiple expert networks, is used to perform fusion prediction of the trajectory for the next time step, taking the fused features and vehicle state at the current moment as input; for details, please refer to [link to relevant documentation]. Figure 1 Step S104 of the illustrated embodiment will not be described again here.

[0100] Step S305: Based on the predicted trajectory at the next moment and the actual trajectory at the next moment, a first loss function is constructed. The parameters of the preset network and the expert hybrid network are adjusted, and the processes of risk map construction, feature extraction, cross-modal fusion, and trajectory prediction are repeated until the preset conditions are met, thus obtaining the autonomous driving trajectory prediction model. For details, please refer to [link to details]. Figure 1 Step S105 of the illustrated embodiment will not be described again here.

[0101] Step S306: Based on the fusion features at the current moment and the trajectory at the next moment, a world model is used to predict the fusion features at the next moment; specifically, this world model adopts a time-series prediction model, such as one based on... The encoder-decoder structure predicts the features for the next time step.

[0102] In an optional implementation, step S306 includes: Step S3061: Encode the fused features at the current time and the trajectory at the next time, and add position information to obtain the processed fused features and trajectory; specifically, the processing of this step is represented by the following formula:

[0103]

[0104] In the formula, This represents the fusion characteristics at time t. This represents the BEV positional encoding, which assigns two-dimensional coordinate information in the BEV space to each flattened feature point. Specifically, this positional encoding is a learnable or fixed parameter matrix with a size of [missing information]. , The dimension representing the feature. This represents the predicted trajectory for the next moment. The trajectory projection function, which could be a linear layer, projects the coordinates of each trajectory point onto a plane. Consistent feature space, This represents the trajectory location code, used to assign the location information of each point in the trajectory within the time series.

[0105] Step S3062 involves using a multi-layer Transformer network to calculate interactive attention on the processed fused features and trajectory, yielding the calculation result. Specifically, this process is represented by the following formula:

[0106] In the formula, This represents the i-th decoding layer of the world model, i.e., the i-th layer of the Transformer network, which is governed by multi-head attention. Feedforward network Composed of, etc. This represents the fused feature after processing the input of the i-th layer. Wherein, The following formula is used to determine it:

[0107]

[0108]

[0109] In the formula, This indicates random deactivation, meaning that a portion of neurons are randomly disabled during training to prevent overfitting.

[0110] Step S3063 involves normalizing, projecting, and reshaping the calculation results to predict the fusion features at the next time step. Specifically, this process is represented by the following formula:

[0111]

[0112]

[0113] In the formula, This represents the output projection function, which projects the feature dimensions. The number of channels projected back to the original BEV features. This indicates dimensionality reshaping, which involves transforming the sequence back into the BEV feature map format. This indicates the predicted fusion features.

[0114] Step S307: Based on the predicted fusion features of the next time step and the fusion features of the next time step obtained through feature extraction and cross-modal fusion, a second loss function is constructed; specifically, the second loss function is based on the mean square error of the predicted fusion features and the true fusion features. The second loss function is determined using the following formula:

[0115] In the formula, Indicates predicted fusion features, This indicates true fusion characteristics.

[0116] Step S308: Based on the first loss function and the second loss function, adjust the parameters of the preset network, the expert hybrid network and the world model, and repeat the process of constructing the risk map, feature extraction and cross-modal fusion, feature prediction and trajectory prediction until the preset conditions are met, and obtain the time-constrained autonomous driving trajectory prediction model.

[0117] Specifically, in addition to training the autonomous driving trajectory prediction model using the first loss function, after adding a world model, a new total loss can be constructed using the world model's loss (i.e., the second loss function) and the first loss function to retrain the autonomous driving trajectory prediction model. Because the world model is included in this model, and the self-supervised loss of the world model can constrain the difference between the predicted fusion features and the true fusion features, the representation of the true fusion features can be optimized, i.e., the true fusion features are forced to have temporal consistency and foresight. The model trained in this way is also called the temporally constrained autonomous driving trajectory prediction model. The total loss constructed from the first loss function (also called the main loss) and the second loss function (also called the auxiliary loss) is expressed by the following formula:

[0118] In the formula, Let λ represent the first loss function, and let λ represent the hyperparameter.

[0119] As one or more specific application embodiments of the present invention, such as Figure 2 As shown, the training method for this autonomous driving trajectory prediction model specifically includes the following steps: 1. Calculation of LiDAR risk map based on physical information.

[0120] The space surrounding the vehicle (e.g., 64 meters in front x 32 meters to the left and right) is divided into a fine two-dimensional grid (e.g., 0.25 meters resolution). For each grid cell, the following channels are calculated based on the points falling within it, forming a 5-channel BEV input tensor: Obstacle map ( (): Average height of point cloud within the grid > threshold And the number of point clouds > the threshold .

[0121] Driveable area map ( Point cloud within the grid Standard deviation of coordinates <threshold .

[0122] Challenging terrain maps ): ≤ < .

[0123] Danger map ( ): Grid height gradient | z|>threshold .

[0124] Effective observation map ( ): Marks whether a LiDAR point exists within the grid.

[0125] 2. Risk-guided multi-scale cross-modal fusion.

[0126] For LiDAR point clouds, a two-stream design is employed: one stream processes the original point cloud (e.g., using PillarNet), and the other stream processes the aforementioned 5-channel risk map using a convolutional neural network (CNN). Subsequently, the processed point cloud is gated and modulated using the risk map, and the two feature streams are then fused to obtain the LiDAR BEV features. .

[0127] For image data captured by the camera, features are extracted using a standard image backbone network (such as ResNet), and then projected onto the BEV space through inverse perspective transformation (IPM) to obtain the camera BEV features. The risk map is then used to generate spatial gates via a convolutional neural network, which are then applied to... ,get .

[0128] When performing multi-scale fusion, first analyze the following: and Downsampling is performed using convolutional layers to construct a multi-scale feature pyramid. Then, at each scale, a deformable cross-attention operation is performed using LiDAR features as the query and camera features as the key and value. Each feature point in the LiDAR is predicted with a sampling offset and fused to the corresponding position in the camera feature map. Finally, a neck network similar to FPN is used to upsample and merge the fused multi-scale features, outputting the final high-resolution BEV feature map. .

[0129] 3. Self-supervised training of the world model.

[0130] The world model adopts the Transformer architecture.

[0131] The input to the world model is the fused features at the current moment. (As a query) and the planned trajectory (Encoded and used as Key and Value).

[0132] The output of the world model is the predicted BEV feature for the next time step. .

[0133] During the training of the world model, the loss is calculated. ,in The BEV features are calculated from the real data of the next frame. This loss is backpropagated to the perception architecture.

[0134] 4. Trajectory decoder based on expert hybrid (MoE).

[0135] The trajectory decoder includes: Gated networks are small neural networks whose input is global pooling. The encoder outputs the vehicle state vector, and the output consists of the weights of each expert. .

[0136] The network of operation experts includes: a transportation expert focused on generating smooth, efficient trajectories that adhere to road boundaries; a loading area operation expert focused on generating short-distance, low-speed, high-precision alignment trajectories; an unloading area operation expert focused on generating precise reversing trajectories and possessing a high degree of avoidance of hazards such as unloading area edges; and an emergency condition expert focused on generating trajectories prioritizing safety, such as emergency braking and parking. Each expert network predicts trajectories based on vehicle status and fused features.

[0137] Based on the weights of each expert and the trajectory predicted by each expert, the final trajectory is obtained. That is, all expert output trajectories Weighted sum: .

[0138] 5. End-to-end training.

[0139] The model's total loss function consists of two parts: First, trajectory planning loss That is, predicting the trajectory The actual trajectory marked by experts The L1 or SmoothL1 loss between them.

[0140] Second, world model loss This refers to the mean squared error loss between the predicted future features and the actual future features.

[0141] Total loss ,in These are the hyperparameters that balance the two tasks. The entire model is optimized end-to-end using gradient descent.

[0142] The beneficial effects of this invention are as follows: 1. Enhanced safety. Explicit risk maps and the "gating first, fusion later" strategy can reduce the probability of vehicles mistakenly entering dangerous slopes and rough terrain areas, thereby improving operational safety in complex terrain.

[0143] 2. Planning accuracy and smoothness. Asymmetric fusion of geometric vision and collaboration with MoE scene experts help reduce average displacement error and final displacement error and suppress trajectory jitter.

[0144] 3. Temporal consistency and robustness. Introducing self-supervised constraints from the world model can improve the temporal continuity of features and decisions, reduce the amplitude of short-term trajectory jitter and the frequency of replanning, and maintain higher performance fidelity in degraded scenarios such as low light, dust, rain, and snow.

[0145] 4. Explainability and Maintainability. The visualization of risk channels, gating weights, and expert activation results facilitates troubleshooting, participation adjustment, and regression analysis. Compared to purely black-box end-to-end solutions, it is more conducive to security assessment and third-party acceptance, and helps to shorten the offline regression and problem localization cycle.

[0146] 5. Adaptable to multiple work areas and domain migrations. MoE structures the prior knowledge of operations such as transportation / loading / unloading. When adding new mining areas or new working conditions, only experts or gating strategies need to be added / adjusted to adapt, without the need to reconstruct the overall model, thus reducing the cost and debugging workload of cross-mining area migration.

[0147] 6. Data and computing efficiency. The world model utilizes unlabeled data to provide self-supervised signals, reducing labeling overhead; risk priors and asymmetric fusion suppress redundant branches, making inference computing power controllable and facilitating deployment at vehicle / edge.

[0148] This embodiment also provides an autonomous driving trajectory prediction model training device, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0149] This embodiment provides an autonomous driving trajectory prediction model training device, such as... Figure 3 As shown, it includes: The data acquisition module 31 is used to acquire image data at multiple times, radar point cloud data of a preset area around the vehicle, and vehicle status during the process of the vehicle traveling along a preset trajectory. Risk map construction module 32 is used to construct a risk map containing a multi-channel bird's-eye view tensor based on the columnar processing data of the radar point cloud data at the current moment. The feature extraction and fusion module 33 is used to extract features from the current time columnar data and image data using a preset network guided by the risk map, and to perform cross-modal fusion using a deformable cross-attention mechanism to obtain the fused features at the current time. The prediction module 34 is used to perform fusion prediction of the trajectory at the next moment by using a hybrid expert network that includes multiple expert networks, taking the fusion features and vehicle state at the current moment as input. Training module 35 is used to construct a first loss function based on the predicted trajectory at the next moment and the actual trajectory at the next moment, and to adjust the parameters of the preset network and the expert hybrid network. It repeats the process of constructing risk map, feature extraction and cross-modal fusion and trajectory prediction until the preset conditions are met, and obtains the autonomous driving trajectory prediction model.

[0150] The autonomous driving trajectory prediction model training device provided in this embodiment of the invention can execute the autonomous driving trajectory prediction model training method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method. Further functional descriptions of the various modules and units are the same as in the corresponding embodiments described above, and will not be repeated here.

[0151] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0152] The following is a detailed reference. Figure 4 This diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 11, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 12 or a program loaded from memory 18 into random access memory (RAM) 13. The RAM 13 also stores various programs and data required for the operation of the electronic device. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0153] Typically, the following devices can be connected to I / O interface 15: input devices 16 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 17 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 18 including, for example, magnetic tapes, hard disks, etc.; and communication devices 19. Communication device 19 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0154] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 19, or installed from a memory 18, or installed from a ROM 12. When the computer program is executed by the processor 11, it performs the functions defined in the autonomous driving trajectory prediction model training method of the embodiments of the present invention.

[0155] Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0156] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the autonomous driving trajectory prediction model training method shown in the above embodiments is implemented.

[0157] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0158] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for training an automatic driving trajectory prediction model, characterized in that, The method comprises: acquiring image data, radar point cloud data of a preset area around the vehicle, and vehicle state at multiple time points during driving of the vehicle according to a preset trajectory; constructing a risk map containing a multi-channel bird's eye view (BEV) tensor based on columnar processing data of the radar point cloud data at the current time point; using a preset network to extract features from the columnar processing data and the image data at the current time point under the guidance of the risk map, and using a deformable cross-attention mechanism to perform cross-modal fusion to obtain fusion features at the current time point; using an expert hybrid network comprising multiple expert networks to perform fusion prediction on the trajectory at the next time point by taking the fusion features at the current time point and the vehicle state as inputs; constructing a first loss function based on the predicted trajectory at the next time point and the real trajectory at the next time point, adjusting parameters of the preset network and the expert hybrid network, and repeating the processes of constructing the risk map, feature extraction, cross-modal fusion, and trajectory prediction until a preset condition is met to obtain an automatic driving trajectory prediction model.

2. The method of claim 1, wherein, The method further comprises: predicting the fusion features at the next time point using a world model based on the fusion features at the current time point and the trajectory at the next time point; constructing a second loss function based on the predicted fusion features at the next time point and the fusion features at the next time point obtained through feature extraction and cross-modal fusion; adjusting parameters of the preset network, the expert hybrid network, and the world model based on the first loss function and the second loss function, and repeating the processes of constructing the risk map, feature extraction, cross-modal fusion, feature prediction, and trajectory prediction until a preset condition is met to obtain a time-constrained automatic driving trajectory prediction model.

3. The method of claim 1, wherein, Constructing a risk map containing a multi-channel bird's eye view (BEV) tensor based on columnar processing data of the radar point cloud data at the current time point comprises: calculating an obstacle channel tensor based on the average height and the number of point clouds in the columnar processing data; calculating a drivable channel tensor and a challenge channel tensor based on the height standard deviation of the point clouds in the columnar processing data; calculating a dangerous slope channel tensor based on the height gradient of adjacent grid cells in the columnar processing data; calculating an effective observation channel tensor based on whether each grid cell in the columnar processing data has a point cloud, and the obstacle channel tensor, the drivable channel tensor, the challenge channel tensor, the dangerous slope channel tensor, and the effective observation channel tensor constitute the risk map containing the multi-channel BEV tensor.

4. The method of claim 1, wherein, Using a preset network to extract features from the columnar processing data and the image data at the current time point under the guidance of the risk map, and using a deformable cross-attention mechanism to perform cross-modal fusion to obtain fusion features at the current time point comprises: generating a spatial gate using a first preset network based on the risk map; performing position-by-position gating processing on a BEV feature map using the spatial gate to obtain a gating processing result, the BEV feature map being obtained by extracting features from the columnar processing data using a second preset network; processing the risk map using a third preset network and combining the gating processing result to obtain a point cloud feature; extracting features from the image data using a fourth preset network and inverse perspective transformation projection to obtain an image feature; The image features are gated using the spatial gating to obtain processed image features; The point cloud features and the processed image features are fused in a cross-modal manner using a deformable cross-attention mechanism to obtain fusion features at the current time.

5. The method of claim 4, wherein, The radar features and the processed image features are fused in a cross-modal manner using a deformable cross-attention mechanism to obtain fusion features at the current time, including: The radar features and the processed image features are respectively subjected to multiple convolution operations to obtain two multi-scale feature pyramids; The vector obtained by flattening and splicing the two multi-scale feature pyramids is fused using a deformable cross-attention fusion network; The fused features are flattened, up-sampled, and convolved to obtain fusion features at the current time.

6. The method of claim 1, wherein, An expert hybrid network including multiple expert networks is used to fuse predict a trajectory at the next time based on the fusion features at the current time and the vehicle state, including: Based on the fusion features at the current time and the vehicle state, a gating network is used to determine the weights of the expert networks; For each expert network, the fusion features and the vehicle state are respectively encoded and position information is added to obtain processed fusion features and vehicle state; A Transformer decoder is used to interactively decode the processed fusion features and the vehicle state to obtain a predicted trajectory of each expert network; Based on the predicted trajectory of each expert network and the weights of the expert networks, a trajectory at the next time is fused predicted.

7. The method of claim 2, wherein, Based on the fusion features at the current time and the trajectory at the next time, a world model is used to predict fusion features at the next time, including: The fusion features at the current time and the trajectory at the next time are respectively encoded and position information is added to obtain processed fusion features and trajectory; A multi-layer Transformer network is used to perform interactive attention calculation on the processed fusion features and the trajectory to obtain a calculation result; The calculation result is normalized, projected, and reshaped to predict fusion features at the next time.

8. An automatic driving trajectory prediction model training apparatus, characterized by, The device includes: A data acquisition module for acquiring image data, radar point cloud data of a predetermined area around the vehicle, and vehicle state at multiple times during driving of the vehicle according to a predetermined trajectory; A risk map construction module for constructing a risk map including a multi-channel bird's eye view tensor based on columnar processing data of the radar point cloud data at the current time; A feature extraction and fusion module for using a predetermined network to extract features from the columnar processing data and the image data at the current time using the risk map as a guide, and using a deformable cross-attention mechanism to fuse in a cross-modal manner to obtain fusion features at the current time; A prediction module for using an expert hybrid network including multiple expert networks to fuse predict a trajectory at the next time based on the fusion features at the current time and the vehicle state. The training module is configured to construct a first loss function based on the predicted next-time trajectory and the real next-time trajectory, adjust parameters of the preset network and the expert hybrid network, and repeat the process of constructing the risk map, feature extraction, cross-modal fusion, and trajectory prediction until a preset condition is reached, thereby obtaining an automatic driving trajectory prediction model.

9. An electronic device, comprising: The method comprises the following steps: A memory and a processor are communicatively connected, and computer instructions are stored in the memory; the processor executes the computer instructions to perform the automatic driving trajectory prediction model training method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer instructions are stored on the computer readable storage medium, and the computer instructions are used to make the computer execute the automatic driving trajectory prediction model training method according to any one of claims 1 to 7.

Images