Track prediction method and device, electronic equipment, storage medium and vehicle

Abstract

The invention relates to the technical field of vehicle track prediction, in particular to a track prediction method and device, electronic equipment, a storage medium and a vehicle, and the method comprises the steps: obtaining the driving data of the vehicle from different collection sources; inputting the driving data into a pre-constructed trajectory prediction model to extract spatial features and semantic features in the driving data by using a feature extraction layer, and determining interactive features of the vehicle by using an interactive perception layer based on the spatial features and the semantic features, the track decoding layer is used for predicting at least one track of the vehicle and the confidence coefficient corresponding to the at least one track; a final trajectory of the vehicle is determined based on the at least one trajectory and the confidence. Therefore, the problems of large prediction error, insufficient safety boundary, high time delay, difficulty in meeting real-time control, poor model interpretability, remarkable challenge to reliable deployment of function safety auditing and high-order automatic driving and the like in complex scenes such as congestion and multi-vehicle interaction in the related technology are solved.

Description

Technical Field

[0001] This application relates to the field of vehicle trajectory prediction technology, and in particular to a trajectory prediction method, device, electronic device, storage medium, and vehicle. Background Technology

[0002] In related technologies, the motion law of the target can be described by a preset kinematic model, and a state transition equation can be established using physical parameters (such as velocity, acceleration, and turning angle). Then, the future trajectory position can be recursively inferred by combining sensor data (such as radar point clouds and camera detection results). Alternatively, features (such as target position, velocity, historical trajectory, and environmental context) can be extracted manually, and the mapping relationship between features and trajectory can be learned through statistical learning models (such as hidden Markov models for modeling state transitions and support vector machines for classifying trajectory patterns) to achieve prediction. Furthermore, based on the raw data (such as image sequences or point cloud sequences) from a single sensor (such as a camera or millimeter-wave radar), the spatiotemporal features in the data can be automatically learned through deep neural networks to directly output the future trajectory sequence.

[0003] However, in related technologies, the prediction error is large in scenarios such as congestion, multi-vehicle interaction, and ramp merging, making it difficult to meet the safety boundary requirements of L3 and above autonomous driving for trajectory planning. The latency is high, which cannot meet the real-time control requirements. The interpretability is poor, which is not conducive to functional safety auditing, and it urgently needs to be improved. Summary of the Invention

[0004] This application provides a trajectory prediction method, device, electronic device, storage medium, and vehicle to address the problems in related technologies, such as large prediction errors, insufficient safety boundaries, high latency making it difficult to meet real-time control requirements in complex scenarios like congestion and multi-vehicle interaction, and poor model interpretability, which pose significant challenges to functional safety audits and the reliable deployment of advanced autonomous driving.

[0005] The first aspect of this application provides a method for predicting vehicle trajectory, comprising the following steps: acquiring driving data of a vehicle from different collection sources; inputting the driving data into a pre-constructed trajectory prediction model, using a feature extraction layer to extract spatial and semantic features from the driving data, and using an interaction perception layer to determine the interaction features of the vehicle based on the spatial and semantic features, and using a trajectory decoding layer to predict at least one trajectory of the vehicle and the confidence level corresponding to the at least one trajectory; and determining the final trajectory of the vehicle based on the at least one trajectory and the confidence level.

[0006] Optionally, in one embodiment of this application, before inputting the driving data into the pre-built trajectory prediction model, the method further includes: acquiring target driving data of the target vehicle from different acquisition sources; extracting target spatial features and target semantic features from the target driving data respectively; generating corresponding target fusion features based on the target spatial features and the target semantic features; and constructing a feature extraction layer in the trajectory prediction model based on the target fusion features.

[0007] Optionally, in one embodiment of this application, before inputting the driving data into the pre-built trajectory prediction model, the method further includes: determining neighboring vehicles of the target vehicle based on the target driving data; determining neighboring driving data of the neighboring vehicles based on the neighboring vehicles, the target vehicle, and the target driving data; determining graph node features of the interaction perception layer in the trajectory prediction model based on the neighboring vehicles and the target vehicle; determining graph edge features of the interaction perception layer based on the target driving data and the neighboring driving data; and determining target interaction features of the target vehicle based on the graph node features and the graph edge features, thereby constructing the interaction perception layer.

[0008] Optionally, in one embodiment of this application, before inputting the driving data into the pre-built trajectory prediction model, the method further includes: predicting the location information of the target vehicle within a preset time period based on the target interaction features; generating at least one target trajectory of the target vehicle and a target confidence level corresponding to the at least one target trajectory based on the location information; and constructing a trajectory decoding layer in the trajectory prediction model based on the at least one target trajectory and the target confidence level.

[0009] Optionally, in one embodiment of this application, before inputting the driving data into the pre-built trajectory prediction model, the method further includes: obtaining training data of the trajectory prediction model; and training the trajectory prediction model based on the training data until a preset training stop condition is met to obtain the trained trajectory prediction model.

[0010] Optionally, in one embodiment of this application, determining the final trajectory of the vehicle based on the at least one trajectory and the confidence level includes: calculating the collision time of the vehicle under the corresponding trajectory based on the at least one trajectory and the confidence level; determining the response level of the corresponding trajectory based on the confidence level and the collision time; and determining the final trajectory based on the response level.

[0011] A second aspect of this application provides a vehicle trajectory prediction device, comprising: a first acquisition module for acquiring driving data of a vehicle from different acquisition sources; a first prediction module for inputting the driving data into a pre-constructed trajectory prediction model, thereby extracting spatial and semantic features from the driving data using a feature extraction layer, and determining the interaction features of the vehicle using an interaction perception layer based on the spatial and semantic features, and predicting at least one trajectory of the vehicle and the confidence level corresponding to the at least one trajectory using a trajectory decoding layer; and a first determination module for determining the final trajectory of the vehicle based on the at least one trajectory and the confidence level.

[0012] Optionally, in one embodiment of this application, it further includes: a second acquisition module, configured to acquire target driving data of the target vehicle from different acquisition sources before inputting the driving data into the pre-built trajectory prediction model; an extraction module, configured to extract target spatial features and target semantic features from the target driving data respectively; a first generation module, configured to generate corresponding target fusion features based on the target spatial features and the target semantic features; and a first construction module, configured to construct a feature extraction layer in the trajectory prediction model based on the target fusion features.

[0013] Optionally, in one embodiment of this application, it further includes: a second determining module, configured to determine neighboring vehicles of the target vehicle based on the target driving data before inputting the driving data into the pre-built trajectory prediction model; a third determining module, configured to determine neighboring driving data of the neighboring vehicles based on the neighboring vehicles, the target vehicle, and the target driving data; a fourth determining module, configured to determine graph node features of the interaction perception layer in the trajectory prediction model based on the neighboring vehicles and the target vehicle; a fifth determining module, configured to determine graph edge features of the interaction perception layer based on the target driving data and the neighboring driving data; and a second constructing module, configured to determine the target interaction features of the target vehicle based on the graph node features and the graph edge features, so as to construct the interaction perception layer.

[0014] Optionally, in one embodiment of this application, it further includes: a second prediction module, configured to predict the location information of the target vehicle within a preset time period based on the target interaction features before inputting the driving data into the pre-built trajectory prediction model; a second generation module, configured to generate at least one target trajectory of the target vehicle and a target confidence level corresponding to the at least one target trajectory based on the location information; and a third construction module, configured to construct a trajectory decoding layer in the trajectory prediction model based on the at least one target trajectory and the target confidence level.

[0015] Optionally, in one embodiment of this application, it further includes: a third acquisition module, configured to acquire training data of the trajectory prediction model before inputting the driving data into the pre-built trajectory prediction model; and a training module, configured to train the trajectory prediction model based on the training data until a preset training stop condition is met, so as to obtain the trained trajectory prediction model.

[0016] Optionally, in one embodiment of this application, the first determining module includes: a calculation unit, configured to calculate the collision time of the vehicle under the corresponding trajectory based on the at least one trajectory and the confidence level; a first determining unit, configured to determine the response level of the corresponding trajectory based on the confidence level and the collision time; and a second determining unit, configured to determine the final trajectory based on the response level.

[0017] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the vehicle trajectory prediction method as described in the above embodiments.

[0018] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the vehicle trajectory prediction method described above.

[0019] A fifth aspect of this application provides a vehicle that includes the electronic equipment described above.

[0020] A sixth aspect of this application provides a computer program product, including a computer program that, when executed, implements the vehicle trajectory prediction method described above.

[0021] This application's embodiments can input driving data from different data sources into a pre-built trajectory prediction model to predict the vehicle's trajectory and corresponding confidence level, thereby determining the vehicle's final trajectory. Multi-source data fusion significantly improves the robustness of environmental perception, hierarchical feature extraction enhances the model's expressive power, and multi-trajectory prediction and confidence assessment ensure safety boundaries. It adapts to the latency requirements of autonomous driving systems, can handle complex scenarios, and provides key technical support for high-level autonomous driving systems. Simultaneously, it reduces the difficulty of functional safety audits and accelerates the technology's deployment. Therefore, it solves the problems in related technologies, such as large prediction errors, insufficient safety boundaries, high latency making real-time control difficult in complex scenarios like traffic congestion and multi-vehicle interactions, and poor model interpretability, which pose significant challenges to functional safety audits and the reliable deployment of high-level autonomous driving.

[0022] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0023] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of a vehicle trajectory prediction method provided according to an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a lightweight spatiotemporal feature network (Lite-STFNet) provided according to an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a Vehicle Interaction Graph Neural Network (VIG-NN) according to an embodiment of this application; Figure 4 A heatmap corresponding to the trajectory and confidence level provided according to one embodiment of this application; Figure 5 This is a schematic diagram of a secure arbitration state machine according to an embodiment of this application; Figure 6 A flowchart illustrating the working principle of a vehicle trajectory prediction method according to an embodiment of this application; Figure 7 This is a block diagram of a vehicle trajectory prediction device provided according to an embodiment of this application; Figure 8 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0024] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0025] The following description, with reference to the accompanying drawings, outlines a trajectory prediction method, apparatus, electronic device, storage medium, and vehicle according to embodiments of this application. Addressing the issues mentioned in the background art, such as large prediction errors, insufficient safety boundaries, and high latency making real-time control difficult in complex scenarios like congestion and multi-vehicle interactions, coupled with poor model interpretability, which significantly challenge functional safety audits and the reliable deployment of advanced autonomous driving, this application provides a vehicle trajectory prediction method. In this method, driving data from different data sources is input into a pre-built trajectory prediction model to predict the vehicle's trajectory and corresponding confidence level, thereby determining the vehicle's final trajectory. Multi-source data fusion significantly improves the robustness of environmental perception, hierarchical feature extraction enhances model expressive power, multi-trajectory prediction and confidence assessment ensure safety boundaries, adapts to the latency requirements of autonomous driving systems, and can adapt to complex scenarios. This provides key technical support for advanced autonomous driving systems, while reducing the difficulty of functional safety audits and accelerating the technology's implementation. This solves the problems in related technologies, such as large prediction errors, insufficient safety boundaries, high latency making it difficult to meet real-time control in complex scenarios like traffic congestion and multi-vehicle interaction, and poor model interpretability, which pose significant challenges to functional safety audits and the reliable deployment of advanced autonomous driving.

[0026] Specifically, Figure 1 This is a flowchart of a vehicle trajectory prediction method provided according to an embodiment of this application.

[0027] like Figure 1 As shown, the method for predicting vehicle trajectories includes the following steps: In step S101, driving data of the vehicle from different data collection sources is acquired.

[0028] It is understood that, in the embodiments of this application, the acquisition source may include, but is not limited to, a forward-facing camera with a resolution of 8MP and a frame rate of 30Hz, which is not specifically limited in this application; a millimeter-wave radar with an operating frequency of 77GHz, a data refresh rate of 20Hz, and a pitch angle of ±15°, which is not specifically limited in this application; and an automotive-grade inertial measurement unit with a reference frequency of 100Hz, which is not specifically limited in this application.

[0029] In some embodiments, the present application can acquire driving data of a vehicle from different data collection sources.

[0030] For example, in this embodiment, a forward-facing camera can be installed on the inside of the windshield at a downward tilt angle of 4° and a horizontal field of view of 120°. A millimeter-wave radar can be installed in the center of the front bumper at a horizontal field of view of 150° and a distance accuracy of ±0.1m. An inertial measurement unit (IMU) can be installed on the domain controller PCB at a position where the gyroscope has a zero bias of ±0.5° / s. This allows for the acquisition of forward-facing image data from the forward-facing camera, radar point cloud data from the millimeter-wave radar, and motion compensation data from the IMU. The camera thread operates at 30Hz, the radar thread at 20Hz, and the main inference thread and arbitration thread communicate via a lock-free circular queue.

[0031] Furthermore, in this embodiment, the radar point cloud data is rasterized using BEV (Bird's Eye View) and spatiotemporally synchronized with the forward image data using bicubic interpolation. The error can be less than 5ms, and this application does not impose any specific limitations.

[0032] Furthermore, in this embodiment of the application, the radar point cloud data is converted to the current vehicle coordinate system based on the motion compensation data of the inertial measurement unit, thereby achieving the alignment of driving data from different acquisition sources.

[0033] Optionally, in one embodiment of this application, before inputting the driving data into the pre-built trajectory prediction model, the method further includes: acquiring target driving data of the target vehicle from different acquisition sources; extracting target spatial features and target semantic features from the target driving data respectively; generating corresponding target fusion features based on the target spatial features and target semantic features; and constructing a feature extraction layer in the trajectory prediction model based on the target fusion features.

[0034] In some embodiments, the present application embodiments may first acquire target driving data of the target vehicle from different acquisition sources, and extract target spatial features and target semantic features from the target driving data respectively, thereby obtaining corresponding fusion features and constructing a feature extraction layer.

[0035] It should be noted that, in this embodiment of the application, the Lite-STFNet network can be used as the feature extraction layer of the trajectory prediction model. The Lite-STFNet network has only 1.8M parameters and can complete inference in 15ms on an NVIDIA Olin system-on-a-chip.

[0036] For example, a schematic diagram of the Lite-STFNet network structure provided in this application embodiment is shown below. Figure 2 As shown, it includes a data input layer, a data processing layer, and a data fusion layer.

[0037] In this embodiment, forward image data acquired by a forward-facing camera and radar point cloud data acquired by millimeter-wave radar can be used as input data for the Lite-STFNet network. 3D sparse convolution is used to extract features from the radar point cloud data to obtain target radar spatial features with a dimension of 64*256*256. EfficientNet-B3 is used to extract features from the forward image data to obtain target image semantic features with a dimension of 1280*16*16. Furthermore, this embodiment obtains 256-dimensional target fusion features through dual-tower structure shared weights and cross-attention fusion.

[0038] Optionally, in one embodiment of this application, before inputting driving data into a pre-built trajectory prediction model, the method further includes: determining neighboring vehicles of a neighboring target vehicle based on the target driving data; determining neighboring driving data of neighboring vehicles based on neighboring vehicles, the target vehicle, and the target driving data; determining graph node features of the interaction perception layer in the trajectory prediction model based on neighboring vehicles and the target vehicle; determining graph edge features of the interaction perception layer based on the target driving data and neighboring driving data; and determining target interaction features of the target vehicle based on graph node features and graph edge features, so as to construct the interaction perception layer.

[0039] In some embodiments, this application can determine the neighboring vehicles of the target vehicle based on the target driving data, and then obtain the neighboring driving data of the neighboring vehicles. The neighboring vehicles and the target vehicle are used as graph node features of the interaction perception layer, and the target driving data and the neighboring driving data are used as graph edge features of the interaction perception layer. The target interaction features of the target vehicle are then determined, thereby constructing the corresponding interaction perception layer.

[0040] In this embodiment of the application, the VIG-NN network can be used as the interactive perception layer of the trajectory prediction model.

[0041] For example, a schematic diagram of the VIG-NN network structure provided in the embodiments of this application is shown below. Figure 3 As shown, by Figure 3 As can be seen, in this embodiment, the vehicle and neighboring vehicles, such as the vehicle in front and five adjacent vehicles, are used as graph nodes. The relative distance, relative speed, difference in vehicle heading, and lane line relationship (encoded as an 8-dimensional vector) are used as graph edges to construct a dynamic directed graph. The graph nodes are updated using a gated graph neural network, which iterates three times to output 128-dimensional target interaction features containing interactive effects.

[0042] Optionally, in one embodiment of this application, before inputting driving data into a pre-built trajectory prediction model, the method further includes: predicting the location information of the target vehicle within a preset time period based on target interaction features; generating at least one target trajectory of the target vehicle and the target confidence level corresponding to the at least one target trajectory based on the location information; and constructing a trajectory decoding layer in the trajectory prediction model based on the at least one target trajectory and the target confidence level.

[0043] In some embodiments, this application can predict the location information of a target vehicle within a certain time period based on the target interaction characteristics, thereby generating the target trajectory of the target vehicle and the corresponding target confidence level, thus constructing the trajectory decoding layer in the trajectory prediction model. The "certain time period" can be set by those skilled in the art according to actual circumstances, and this application does not impose specific limitations.

[0044] In this embodiment of the application, a Multiple Hypothesis Trajectory Decoder (MHD-Decoder) can be used as the trajectory decoding layer of the trajectory prediction model.

[0045] For example, the MHD-Decoder provided in this application embodiment uses a hybrid density network and outputs K=5 Gaussian trajectories, each Gaussian trajectory including two-dimensional position information within the next 5 seconds. The uncertainty is described using the covariance matrix, and the trajectory confidence is... , The result is as follows Figure 4 As shown, the spatial distribution of each Gaussian trajectory can be explained using a heatmap and interpreted in natural language, such as "the vehicle in front has an 82% probability of changing lanes to the left because the left lane has a 120m empty length." The specific settings can be configured by those skilled in the art according to the actual situation, and this application does not impose any specific restrictions.

[0046] Optionally, in one embodiment of this application, before inputting driving data into a pre-built trajectory prediction model, the method further includes: obtaining training data for the trajectory prediction model; and training the trajectory prediction model based on the training data until a preset training stop condition is met, so as to obtain a trained trajectory prediction model.

[0047] In some embodiments, the present application embodiments can use offline SLAM (Simultaneous Localization and Mapping), multi-target tracking, manual quality inspection, and other methods to annotate the collected 1200 hours of highway, urban expressway, and ramp scene data, thereby obtaining training data with a trajectory true value error of <0.1m.

[0048] It should be noted that the embodiments of this application can perform data augmentation on the training data, such as random frame dropping (simulating sensor failure); radar noise injection ( This application does not impose specific restrictions on camera image lighting disturbances, etc.

[0049] Furthermore, in this embodiment, the trajectory prediction model is trained using training data until a certain training stopping condition is met, thereby obtaining the trained trajectory prediction model. The certain training stopping condition can be set by those skilled in the art according to actual circumstances, and this application does not impose specific limitations.

[0050] For example, embodiments of this application can construct the model's loss function using negative log-likelihood and KL divergence regularization. Training is then stopped when the minimum distance between the predicted trajectory and the actual trajectories of adjacent vehicles exceeds a safety threshold, thus determining that a certain training stop condition has been met, thereby obtaining the trained trajectory prediction model. The safety threshold can be set by those skilled in the art according to actual circumstances, and this application does not impose specific limitations.

[0051] In step S102, driving data is input into a pre-built trajectory prediction model to extract spatial and semantic features from the driving data using a feature extraction layer. Based on the spatial and semantic features, the interaction features of the vehicle are determined using an interaction perception layer. The trajectory decoding layer is then used to predict at least one trajectory of the vehicle and the confidence level corresponding to at least one trajectory.

[0052] In some embodiments, driving data can be input into a pre-built trajectory prediction model, and then spatial and semantic features in the driving data can be extracted using a feature extraction layer. Based on the spatial and semantic features, the interaction features of the vehicle can be determined using an interaction perception layer, thereby using a trajectory decoding layer to predict the vehicle's trajectory and corresponding confidence level.

[0053] It should be noted that, in this embodiment, the software operating system can be QNX 7.1, the middleware can be AdaptiveAUTOSAR (Adaptive AUTomotive Open System ARchitecture), and the deep learning framework can be to first export the model from the PyTorch 2.0 framework to the ONNX format (Open Neural Network Exchange), and then use TensorRT 8.6 to optimize and accelerate the model, thereby achieving efficient deployment of the model.

[0054] In step S103, the final trajectory of the vehicle is determined based on at least one trajectory and a confidence level.

[0055] In some embodiments, the present application can determine the final trajectory of a vehicle based on the trajectory and the corresponding confidence level.

[0056] It should be noted that, on the publicly available nuScenes dataset, the embodiments of this application can reduce the 3-second trajectory prediction error from 0.97m to 0.5m; achieve an end-to-end latency of 27ms in a vehicle speed of 80km / h scenario; and reduce the proportion of "unknown unsafe scenarios" from 18% to 5% in functional safety audits through interpretable branches.

[0057] For example, in this embodiment of the application, forward image data from the camera can be acquired at t0, and the nearest radar point cloud data can be waited for simultaneously. After spatiotemporal alignment, feature extraction by the Lite-STFNet network, and interactive updates by the VIG-NN network, five trajectories and the confidence scores corresponding to different trajectories are output using the MHD-Decoder at t0+15ms. Thus, the final trajectory of the vehicle is generated at t0+20ms, and acceleration and curvature requests are sent to the chassis via the CAN (Controller Area Network) bus at t0+27ms.

[0058] Optionally, in one embodiment of this application, determining the final trajectory of a vehicle based on at least one trajectory and a confidence level includes: calculating the collision time of the vehicle under the corresponding trajectory based on at least one trajectory and a confidence level; determining the response level of the corresponding trajectory based on the confidence level and the collision time; and determining the final trajectory based on the response level.

[0059] It is understood that in the embodiments of this application, the Time To Collision (TTC) can be understood as the remaining time expected before the vehicle, continuing to travel at its current speed and trajectory, collides with the obstacle ahead. For example, TTC = 5s indicates that the vehicle is far from the obstacle and the risk is low; TTC < 2.7s indicates that the vehicle is approaching the obstacle, the risk of collision is significantly increased, and immediate intervention is required.

[0060] In some embodiments, the present application can first calculate the collision time of the vehicle under the corresponding trajectory based on the trajectory and the corresponding confidence level, thereby determining the response level of the corresponding trajectory, and then determining the final trajectory.

[0061] In this embodiment, the response level can be divided into three levels. Level 1 can be represented by a green zone, indicating that the vehicle can track normally; Level 2 can be represented by a yellow zone, indicating that the vehicle needs to reduce its own acceleration limit by 30%; Level 3 can be represented by a red zone, indicating that the vehicle triggers a braking request and switches to a conservative prediction based on a physics model.

[0062] For example, embodiments of this application use confidence levels. A graded response is triggered based on a TTC threshold (<2.7s). For example, in this embodiment, a graded response can be triggered when TTC = 2.5s. =0.8 seconds, triggering Level 1 response, alerting the driver through flashing dashboard icons and audible alerts, but without mandatory intervention; when TTC=1.8 seconds, =0.6, triggering a level 2 response, which involves seat vibration, repeated alarms, and automatic slight braking or steering wheel adjustment to assist the driver in avoiding hazards; when TTC=s, =0.4, triggering a level three response, and avoiding collisions to the greatest extent possible through automatic emergency braking and full-force avoidance. The specific settings can be configured by those skilled in the art according to actual conditions; this application does not impose specific limitations. A schematic diagram of the safety arbitration state machine is shown below. Figure 5 As shown.

[0063] The method for predicting vehicle trajectories proposed in this application will be described below with reference to a specific embodiment.

[0064] in, Figure 6 This is a flowchart illustrating the working principle of a vehicle trajectory prediction method provided according to an embodiment of this application.

[0065] Step S601: Install the data acquisition source.

[0066] Among them, the camera is located on the inside of the windshield with a downward tilt angle of 4° and a horizontal field of view of 120°; the millimeter-wave radar is located in the center of the front bumper with a horizontal field of view of 150° and a distance accuracy of ±0.1m; and the inertial measurement unit is integrated on the domain controller PCB with a gyroscope zero bias of ±0.5° / s.

[0067] Step S602: Software deployment.

[0068] The system includes: Operating system: QNX 7.1; Middleware: Adaptive AUTOSAR; Deep learning framework: PyTorch 2.0 → ONNX → TensorRT 8.6; Task scheduling: Camera thread 30Hz, radar thread 20Hz, and the main inference thread and arbitration thread communicate via a lock-free circular queue.

[0069] Step S603: Data Acquisition Layer.

[0070] In this embodiment, forward image data can be acquired at time t0, while simultaneously waiting for the latest radar point cloud data and motion compensation data from the inertial measurement unit.

[0071] Step S604: Data alignment layer.

[0072] In this embodiment, the forward image data, radar point cloud data, and motion compensation data at time t0 can be spatiotemporally synchronized, motion compensated, and BEV rasterized to obtain aligned data.

[0073] Step S605: Feature extraction layer.

[0074] In this application embodiment, the following is utilized: Figure 2 The Lite-STFNet network shown is used for feature extraction, resulting in 256-dimensional fused features.

[0075] Step S606: Interaction Perception Layer.

[0076] In this application embodiment, the following is utilized: Figure 3 The VIG-NN network shown is used for interactive perception, thereby obtaining 128-dimensional interactive features.

[0077] Step S607: Trajectory Decoding Layer.

[0078] In this embodiment, the MHD-Decoder can output 5 trajectories and their corresponding confidence levels at t0+15ms, as shown in the schematic diagram below. Figure 4 As shown.

[0079] Step S608: Security Arbitration Layer.

[0080] In this embodiment, the final trajectory of the vehicle can be generated at t0+20ms, and at t0+27ms, acceleration and curvature requests are sent to the chassis via the CAN bus. The safety arbitration process is as follows: Figure 5 As shown.

[0081] The vehicle trajectory prediction method proposed in this application can input driving data from different data sources into a pre-built trajectory prediction model to predict the vehicle's trajectory and corresponding confidence level, thereby determining the vehicle's final trajectory. Multi-source data fusion significantly improves the robustness of environmental perception, hierarchical feature extraction enhances the model's expressive power, multi-trajectory prediction and confidence assessment ensure safety boundaries, adapt to the latency requirements of autonomous driving systems, and can adapt to complex scenarios. This provides key technical support for high-level autonomous driving systems while reducing the difficulty of functional safety audits and accelerating the technology's deployment. Therefore, it solves the problems in related technologies, such as large prediction errors, insufficient safety boundaries, high latency making real-time control difficult in complex scenarios like congestion and multi-vehicle interaction, and poor model interpretability, which pose significant challenges to functional safety audits and the reliable deployment of high-level autonomous driving.

[0082] Next, the vehicle trajectory prediction device according to an embodiment of this application is described with reference to the accompanying drawings.

[0083] Figure 7 This is a block diagram of a vehicle trajectory prediction device provided according to an embodiment of this application.

[0084] like Figure 7 As shown, the vehicle trajectory prediction device 10 includes: a first acquisition module 100, a first prediction module 200, and a first determination module 300.

[0085] The first acquisition module 100 is used to acquire driving data of the vehicle from different collection sources.

[0086] The first prediction module 200 is used to input driving data into a pre-built trajectory prediction model, so as to extract spatial and semantic features from the driving data using a feature extraction layer, and determine the vehicle's interaction features using an interaction perception layer based on the spatial and semantic features, so as to predict at least one trajectory of the vehicle and the confidence level corresponding to at least one trajectory using a trajectory decoding layer.

[0087] The first determining module 300 is used to determine the final trajectory of the vehicle based on at least one trajectory and a confidence level.

[0088] Optionally, in one embodiment of this application, it further includes: a second acquisition module, an extraction module, a first generation module, and a first construction module.

[0089] The second acquisition module is used to acquire target driving data of the target vehicle from different acquisition sources before inputting the driving data into the pre-built trajectory prediction model.

[0090] The extraction module is used to extract target spatial features and target semantic features from the target driving data, respectively.

[0091] The first generation module is used to generate corresponding target fusion features based on target spatial features and target semantic features.

[0092] The first building module is used to construct the feature extraction layer in the trajectory prediction model based on the target fusion features.

[0093] Optionally, in one embodiment of this application, it further includes: a second determining module, a third determining module, a fourth determining module, a fifth determining module, and a second constructing module.

[0094] The second determining module is used to determine the neighboring vehicles of the target vehicle based on the target driving data before inputting the driving data into the pre-built trajectory prediction model.

[0095] The third determination module is used to determine the neighboring driving data of neighboring vehicles based on the neighboring vehicles, the target vehicle, and the target driving data.

[0096] The fourth determination module is used to determine the graph node features of the interactive perception layer in the trajectory prediction model based on nearby vehicles and the target vehicle.

[0097] The fifth determination module is used to determine the graph edge features of the interactive perception layer based on the target driving data and the adjacent driving data.

[0098] The second construction module is used to determine the target interaction features of the target vehicle based on graph node features and graph edge features, so as to construct the interaction perception layer.

[0099] Optionally, in one embodiment of this application, it further includes: a second prediction module, a second generation module, and a third construction module.

[0100] The second prediction module is used to predict the location information of the target vehicle within a preset time period based on the target interaction characteristics before inputting the driving data into the pre-built trajectory prediction model.

[0101] The second generation module is used to generate at least one target trajectory of the target vehicle and the target confidence level corresponding to at least one target trajectory based on the location information.

[0102] The third building module is used to construct the trajectory decoding layer in the trajectory prediction model based on at least one target trajectory and target confidence.

[0103] Optionally, in one embodiment of this application, it further includes: a third acquisition module and a training module.

[0104] The third acquisition module is used to acquire training data for the trajectory prediction model before inputting driving data into the pre-built trajectory prediction model.

[0105] The training module is used to train the trajectory prediction model based on the training data until the preset training stopping condition is met, so as to obtain the trained trajectory prediction model.

[0106] Optionally, in one embodiment of this application, the first determining module 300 includes: a calculation unit, a first determining unit, and a second determining unit.

[0107] The calculation unit is used to calculate the collision time of a vehicle on a corresponding trajectory based on at least one trajectory and a confidence level.

[0108] The first determining unit is used to determine the response level of the corresponding trajectory based on the confidence level and the collision time.

[0109] The second determining unit is used to determine the final trajectory based on the response level.

[0110] It should be noted that the explanation of the aforementioned vehicle trajectory prediction method embodiment also applies to the vehicle trajectory prediction device of this embodiment, and will not be repeated here.

[0111] The vehicle trajectory prediction device proposed in this application can input driving data from different data sources into a pre-built trajectory prediction model to predict the vehicle's trajectory and corresponding confidence level, thereby determining the vehicle's final trajectory. Multi-source data fusion significantly improves the robustness of environmental perception, hierarchical feature extraction enhances the model's expressive power, multi-trajectory prediction and confidence assessment ensure safety boundaries, adapt to the latency requirements of autonomous driving systems, and can adapt to complex scenarios. It provides key technical support for high-level autonomous driving systems while reducing the difficulty of functional safety audits and accelerating the technology's deployment. Therefore, it solves the problems in related technologies, such as large prediction errors, insufficient safety boundaries, high latency making real-time control difficult in complex scenarios like congestion and multi-vehicle interaction, and poor model interpretability, which pose significant challenges to functional safety audits and the reliable deployment of high-level autonomous driving.

[0112] Figure 8 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. The electronic device may include: The memory 801, the processor 802, and the computer program stored on the memory 801 and capable of running on the processor 802.

[0113] When the processor 802 executes the program, it implements the vehicle trajectory prediction method provided in the above embodiments.

[0114] Furthermore, electronic devices also include: Communication interface 803 is used for communication between memory 801 and processor 802.

[0115] The memory 801 is used to store computer programs that can run on the processor 802.

[0116] The memory 801 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0117] If the memory 801, processor 802, and communication interface 803 are implemented independently, then the communication interface 803, memory 801, and processor 802 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized into address buses, data buses, control buses, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0118] Optionally, in a specific implementation, if the memory 801, processor 802, and communication interface 803 are integrated on a single chip, then the memory 801, processor 802, and communication interface 803 can communicate with each other through an internal interface.

[0119] The processor 802 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0120] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the vehicle trajectory prediction method described above.

[0121] This application also provides a vehicle that includes the electronic devices described above.

[0122] This application also provides a computer program product, including a computer program that, when executed, implements the vehicle trajectory prediction method described above.

[0123] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0124] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0125] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0126] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0127] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or more of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0128] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0129] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0130] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for predicting vehicle trajectories, characterized in that, Includes the following steps: Acquire driving data from different data sources for the vehicle; The driving data is input into a pre-built trajectory prediction model, and the spatial and semantic features in the driving data are extracted using a feature extraction layer. Based on the spatial and semantic features, the interaction features of the vehicle are determined using an interaction perception layer, and the trajectory decoding layer is used to predict at least one trajectory of the vehicle and the confidence level corresponding to the at least one trajectory. The final trajectory of the vehicle is determined based on the at least one trajectory and the confidence level.

2. The method according to claim 1, characterized in that, Before inputting the driving data into the pre-built trajectory prediction model, the following steps are also included: Acquire target driving data of the target vehicle from different data sources; Extract the target spatial features and target semantic features from the target driving data respectively; Based on the target spatial features and the target semantic features, generate corresponding target fusion features; Based on the target fusion features, a feature extraction layer is constructed in the trajectory prediction model.

3. The method according to claim 2, characterized in that, Before inputting the driving data into the pre-built trajectory prediction model, the following steps are also included: Based on the target driving data, determine the neighboring vehicles that are adjacent to the target vehicle; Based on the nearby vehicles, the target vehicle, and the target driving data, determine the nearby driving data of the nearby vehicles; Based on the nearby vehicles and the target vehicle, determine the graph node features of the interactive perception layer in the trajectory prediction model; Based on the target driving data and the adjacent driving data, the graph edge features of the interactive perception layer are determined; Based on the graph node features and the graph edge features, the target interaction features of the target vehicle are determined to construct the interaction perception layer.

4. The method according to claim 3, characterized in that, Before inputting the driving data into the pre-built trajectory prediction model, the following steps are also included: Based on the target interaction features, predict the location information of the target vehicle within a preset time period; Based on the location information, at least one target trajectory of the target vehicle and the target confidence level corresponding to the at least one target trajectory are generated; Based on the at least one target trajectory and the target confidence level, a trajectory decoding layer is constructed in the trajectory prediction model.

5. The method according to claim 1, characterized in that, Before inputting the driving data into the pre-built trajectory prediction model, the following steps are also included: Obtain the training data of the trajectory prediction model; Based on the training data, the trajectory prediction model is trained until a preset training stop condition is met to obtain the trained trajectory prediction model.

6. The method according to claim 1, characterized in that, Determining the final trajectory of the vehicle based on the at least one trajectory and the confidence level includes: Based on the at least one trajectory and the confidence level, the collision time of the vehicle under the corresponding trajectory is calculated; Based on the confidence level and the collision time, the response level of the corresponding trajectory is determined; The final trajectory is determined based on the response level.

7. A vehicle trajectory prediction device, characterized in that, include: The first acquisition module is used to acquire driving data of the vehicle from different collection sources; The first prediction module is used to input the driving data into a pre-built trajectory prediction model, so as to extract spatial and semantic features from the driving data using a feature extraction layer, and determine the interaction features of the vehicle using an interaction perception layer based on the spatial and semantic features, so as to predict at least one trajectory of the vehicle and the confidence level corresponding to the at least one trajectory using a trajectory decoding layer. The first determining module is used to determine the final trajectory of the vehicle based on the at least one trajectory and the confidence level.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the vehicle trajectory prediction method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the vehicle trajectory prediction method as described in any one of claims 1-6.

10. A vehicle, characterized in that, The vehicle includes the electronic equipment as described in claim 8.

Images