Dynamic obstacle trajectory prediction method and device, equipment and storage medium

Abstract

The invention discloses a dynamic obstacle trajectory prediction method, device and equipment and a storage medium, and relates to the technical field of intelligent driving, and the dynamic obstacle trajectory prediction method comprises the steps: receiving multi-modal data from a sensor; extracting a semantic feature of the camera image data, a geometric feature of the laser radar point cloud data and a motion feature of the millimeter wave radar data, and fusing the semantic feature, the geometric feature and the motion feature through a cross-modal cross attention module to generate a fused feature vector; according to the fusion feature vector, generating a space interaction feature in a space dimension through deformable convolution, and according to the fusion feature vector, analyzing a high-order motion parameter of a historical track in a time dimension to generate a time feature; and processing the space interaction features and the time features through a hybrid sequential network to obtain a multi-modal trajectory prediction result. According to the invention, the accuracy of obstacle trajectory prediction can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent driving, and particularly relates to a dynamic obstacle trajectory prediction method and device, equipment and a storage medium. BACKGROUND

[0002] With the development of automatic driving, real-time motion intention reasoning and trajectory prediction of dynamic obstacles such as vehicles and pedestrians in complex traffic scenes have become a key technology for automatic driving. At present, the means for obstacle prediction is to use traditional Kalman filtering for prediction, but due to the significant nonlinearity and non-Gaussian characteristics of the motion of dynamic obstacles in actual traffic scenes. For example: sudden lateral acceleration when a vehicle changes lanes in an emergency, random gait adjustment when a pedestrian dodges, which will cause the inaccuracy of the prediction results of traditional Kalman filtering. Therefore, how to improve the accuracy of obstacle trajectory prediction is a problem that still needs to be solved.

[0003] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0004] The main purpose of the present application is to provide a dynamic obstacle trajectory prediction method, device, equipment and storage medium, aiming at solving the technical problem of how to improve the accuracy of obstacle trajectory prediction.

[0005] To achieve the above purpose, the present application provides a dynamic obstacle trajectory prediction method, which comprises:

[0006] Receiving multi-modal data from a sensor, the multi-modal data comprising camera image data, laser radar point cloud data and millimeter wave radar data;

[0007] Extracting semantic features of the camera image data, geometric features of the laser radar point cloud data and motion features of the millimeter wave radar data, and fusing the semantic features, the geometric features and the motion features through a cross-modal cross-attention module to generate a fusion feature vector;

[0008] Generating a spatial interaction feature through deformable convolution in the spatial dimension according to the fusion feature vector, and generating a time feature by analyzing high-order motion parameters of a historical trajectory in the time dimension according to the fusion feature vector;

[0009] Processing the spatial interaction feature and the time feature through a hybrid temporal network to obtain a multi-modal trajectory prediction result.

[0010] In one embodiment, the step of extracting semantic features of the camera image data, geometric features of the lidar point cloud data, and motion features of the millimeter wave radar data, and fusing the semantic features, the geometric features, and the motion features through a cross-modal cross-attention module to generate a fused feature vector includes:

[0011] Extracting semantic features of the camera image data through a lightweight encoder;

[0012] Extracting geometric features of the lidar point cloud data through a sparse three-dimensional convolutional network;

[0013] extracting motion features of the millimeter-wave radar data through a gated recurrent unit temporal encoder;

[0014] The semantic features, the geometric features, and the motion features are input into the cross-modal cross attention module for fusion to generate the fused feature vector.

[0015] In one embodiment, the step of fusing the semantic features, the geometric features, and the motion features through a cross-modal attention module to generate a fused feature vector includes:

[0016] Obtaining a query vector according to a fusion vector of the geometric features and the motion features, and obtaining a key vector and a value vector according to the semantic features;

[0017] Mapping the query vector, the key vector, and the value vector to a unified latent space through a linear projection layer to obtain a target query vector, a target key vector, and a target value vector;

[0018] Calculate an attention weight matrix based on the target query vector, the target key vector, and the target value vector;

[0019] The target value vector is weightedly fused according to the attention weight matrix to generate the fused feature vector.

[0020] In one embodiment, the step of generating a spatial interaction feature by deformable convolution in the spatial dimension according to the fused feature vector includes:

[0021] Based on the fused feature vector, the interactive hotspot areas between obstacles are dynamically focused through deformable convolution to obtain an interactive area feature map;

[0022] generating a spatial interaction energy field according to the interaction region feature map;

[0023] A spatial interaction feature is generated according to the spatial interaction energy field.

[0024] In an embodiment, the step of generating time features according to the high-order motion parameters of the historical trajectories in the time dimension based on the fusion feature vector comprises:

[0025] extracting historical trajectory data based on the fusion feature vector;

[0026] analyzing the historical trajectory data by using a sliding window multi-head attention mechanism to obtain a trajectory analysis result;

[0027] extracting high-order motion parameters from the trajectory analysis result;

[0028] obtaining a behavior pattern according to the high-order motion parameters to generate time features.

[0029] In an embodiment, the step of processing the spatial interaction features and the time features by a mixed temporal network to obtain a multi-modal trajectory prediction result comprises:

[0030] processing the spatial interaction features and the time features by a temporal convolution network to obtain local acceleration data;

[0031] processing the spatial interaction features and the time features by a transformer encoder to obtain a periodic behavior model;

[0032] obtaining a multi-modal trajectory prediction result according to the local acceleration data and the periodic behavior model.

[0033] In an embodiment, the step of obtaining a multi-modal trajectory prediction result according to the local acceleration data and the periodic behavior model comprises:

[0034] obtaining a set of obstacle candidate trajectories according to the local acceleration data and the periodic behavior model, and scoring the set of obstacle candidate trajectories to obtain a trajectory set score result;

[0035] selecting a multi-modal trajectory prediction result from the set of obstacle candidate trajectories according to the trajectory set score result.

[0036] In addition, to achieve the above-mentioned purposes, the present application further provides a dynamic obstacle trajectory prediction device, which comprises:

[0037] a receiving module configured to receive multi-modal data from a sensor, wherein the multi-modal data comprises camera image data, laser radar point cloud data and millimeter wave radar data;

[0038] a fusion module configured to extract semantic features of the camera image data, geometric features of the LiDAR point cloud data, and motion features of the mmWave radar data, and to generate a fusion feature vector by fusing the semantic features, the geometric features, and the motion features through a cross-modal cross-attention module;

[0039] a generation module configured to generate a spatial interaction feature by deformable convolution in a spatial dimension according to the fusion feature vector, and to generate a temporal feature by analyzing high-order motion parameters of a historical trajectory in a temporal dimension according to the fusion feature vector;

[0040] a prediction module configured to obtain a multi-modal trajectory prediction result by processing the spatial interaction feature and the temporal feature through a hybrid temporal network.

[0041] In addition, to achieve the above object, the present application further provides a dynamic obstacle trajectory prediction device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the dynamic obstacle trajectory prediction method as described above.

[0042] In addition, to achieve the above object, the present application further provides a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the dynamic obstacle trajectory prediction method as described above.

[0043] In addition, to achieve the above object, the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the dynamic obstacle trajectory prediction method as described above.

[0044] The present application provides a dynamic obstacle trajectory prediction method, which receives multi-modal data from sensors, the multi-modal data comprising camera image data, LiDAR point cloud data, and mmWave radar data; extracts semantic features of the camera image data, geometric features of the LiDAR point cloud data, and motion features of the mmWave radar data, and generates a fusion feature vector by fusing the semantic features, the geometric features, and the motion features through a cross-modal cross-attention module; generates a spatial interaction feature by deformable convolution in a spatial dimension according to the fusion feature vector, and generates a temporal feature by analyzing high-order motion parameters of a historical trajectory in a temporal dimension according to the fusion feature vector; and obtains a multi-modal trajectory prediction result by processing the spatial interaction feature and the temporal feature through a hybrid temporal network.

[0045] In conclusion, the application processes multiple sensor data through spatial and time decoupling attention mechanism, and uses hybrid temporal network for trajectory prediction, thereby improving the accuracy of obstacle trajectory prediction. BRIEF DESCRIPTION OF DRAWINGS

[0046] The drawings incorporated in and forming a part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application.

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, those drawings can also provide other drawings for those skilled in the art without creative labor.

[0048] Figure 1 A flowchart is provided for the dynamic obstacle trajectory prediction method embodiment one of the present application;

[0049] Figure 2 A trajectory prediction structure diagram is provided for the dynamic obstacle trajectory prediction method embodiment one of the present application;

[0050] Figure 3 A flowchart is provided for the dynamic obstacle trajectory prediction method embodiment two of the present application;

[0051] Figure 4 A brief flowchart of the dynamic obstacle trajectory prediction method is provided for the embodiment one of the present application;

[0052] Figure 5 A module structure diagram of the dynamic obstacle trajectory prediction device is provided for the embodiment of the present application;

[0053] Figure 6 A device structure diagram of the hardware running environment involved in the dynamic obstacle trajectory prediction method in the embodiment of the present application.

[0054] The purpose of the present application, functional characteristics and advantages will be further described with reference to the drawings combined with the embodiments. DETAILED DESCRIPTION

[0055] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the present application.

[0056] In order to better understand the technical solutions of the present application, the following will be described in detail combined with the drawings in the specification and specific embodiments.

[0057] The main solution of the present application is to receive multi-modal data from sensors, the multi-modal data including camera image data, laser radar point cloud data and millimeter wave radar data; extract semantic features of the camera image data, geometric features of the laser radar point cloud data and motion features of the millimeter wave radar data, and fuse the semantic features, the geometric features and the motion features through a cross-modal cross-attention module to generate a fusion feature vector; generate a spatial interaction feature through deformable convolution in the spatial dimension according to the fusion feature vector, and generate a time feature by analyzing high-order motion parameters of a historical trajectory in the time dimension according to the fusion feature vector; and obtain a multi-modal trajectory prediction result by processing the spatial interaction feature and the time feature through a hybrid temporal network.

[0058] At present, the means for obstacle prediction is to use traditional Kalman filtering for prediction, but due to the significant nonlinearity and non-Gaussian characteristics of dynamic obstacle motion in actual traffic scenes. For example: sudden lateral acceleration when a vehicle changes lanes in an emergency, random gait adjustment when pedestrians avoid, which will cause the inaccuracy of the prediction result of the traditional Kalman filtering, therefore, how to improve the accuracy of obstacle trajectory prediction is still a problem to be solved.

[0059] The present application processes various sensor data through a spatial and temporal decoupling attention mechanism, and uses a hybrid temporal network for trajectory prediction, thereby improving the accuracy of obstacle trajectory prediction.

[0060] Based on this, the embodiment of the present application provides a dynamic obstacle trajectory prediction method, referring to Figure 1 , Figure 1 The flowchart of the first embodiment of the dynamic obstacle trajectory prediction method of the present application is shown.

[0061] In this embodiment, the dynamic obstacle trajectory prediction method includes steps S10-S40:

[0062] Step S10: receiving multi-modal data from sensors, the multi-modal data including camera image data, laser radar point cloud data and millimeter wave radar data;

[0063] It should be noted that the execution subject of the present embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, a dynamic obstacle trajectory prediction device, etc. capable of realizing the above functions. In the following, the dynamic obstacle trajectory prediction device is taken as an example to explain the present embodiment and the following embodiments.

[0064] It can be understood that the embodiment receives multiple data, and by combining different sensor data, different levels of data can be extracted, such as image semantic features, point cloud geometric features, and millimeter wave radar motion features.

[0065] Step S20: extracting semantic features of the camera image data, geometric features of the lidar point cloud data, and motion features of the millimeter wave radar data, and fusing the semantic features, the geometric features, and the motion features through a cross-modal cross-attention module to generate a fusion feature vector;

[0066] It can be understood that the semantic features include vehicle categories, heading angles, etc.; the geometric features include obstacle surface curvature, space occupancy state, etc.; and the motion features include radial velocity, angular velocity change trend, etc.

[0067] In one possible implementation, the step of extracting semantic features of the camera image data, geometric features of the lidar point cloud data, and motion features of the millimeter wave radar data, and fusing the semantic features, the geometric features, and the motion features through a cross-modal cross-attention module to generate a fusion feature vector includes: extracting the semantic features of the camera image data through a lightweight encoder; extracting the geometric features of the lidar point cloud data through a sparse three-dimensional convolutional network; extracting the motion features of the millimeter wave radar data through a gated recurrent unit temporal encoder; and inputting the semantic features, the geometric features, and the motion features into the cross-modal cross-attention module for fusion to generate the fusion feature vector.

[0068] It should be noted that the embodiment uses an improved real-time detection transformer (RT-DETR) framework to synchronously process camera images, lidar point clouds, and millimeter wave radar data. The lightweight encoder is a real-time processing optimized convolutional neural network architecture, including an image feature extractor with a depth separable convolutional layer and a parameter compression module with a channel pruning technique, which can process camera raw image data and output a semantic feature vector containing obstacle categories, heading angles, and bounding boxes. The sparse three-dimensional convolutional network is a three-dimensional voxel feature extractor for processing lidar point clouds, including a convolution kernel activation device based on non-empty voxel index and an octree space partition accelerator, which can process a set of three-dimensional coordinates of lidar point clouds and output a geometric feature tensor representing surface curvature and space occupancy state. The gated recurrent unit temporal encoder is a recurrent neural network processor for analyzing millimeter wave radar sequences, including a loop calculation unit with reset gate / update gate and a time step sliding window buffer, which can process radial velocity and angular velocity time series data in the millimeter wave radar to obtain motion features.

[0069] In an implementable manner, the step of generating a fusion feature vector by fusing the semantic feature, the geometric feature and the motion feature through the cross-modal cross-attention module comprises: obtaining a query vector according to the fusion vector of the geometric feature and the motion feature, and obtaining a key vector and a value vector according to the semantic feature; mapping the query vector, the key vector and the value vector to a unified hidden space through a linear projection layer to obtain a target query vector, a target key vector and a target value vector; calculating an attention weight matrix according to the target query vector, the target key vector and the target value vector; and performing weighted fusion on the target value vector according to the attention weight matrix to generate the fusion feature vector.

[0070] It can be understood that, in order to realize deep fusion of multi-modal data, the cross-modal cross-attention module is used in the embodiment: taking the fusion vector of the laser radar point cloud geometric feature and the millimeter wave radar motion feature as Query, and taking the camera image semantic feature as Key / Value. First, the heterogeneous features are mapped to a unified hidden space through a linear projection layer, and then an attention weight matrix is calculated to realize adaptive alignment of the features. At the same time, a sensor confidence gating mechanism is embedded to monitor the quality of each modal data (such as camera overexposure or millimeter wave radar noise-to-signal ratio anomaly) in real time, dynamically adjust the feature fusion weight, and finally output the fusion feature vector aligned across modalities, and also output a real-time confidence score. The effective modal weight can be automatically increased when the sensor is partially failed, ensuring the prediction robustness. Further, a vehicle dynamics constraint term (such as minimum turning radius and maximum longitudinal acceleration) is introduced, and the kinematic equation is embedded into the loss function to force the predicted trajectory to comply with the physical law and avoid generating infeasible paths that invade the safety boundaries of other obstacles.

[0071] Step S30: generating spatial interaction features by deformable convolution in the spatial dimension according to the fusion feature vector, and generating time features by analyzing high-order motion parameters of the historical trajectory in the time dimension according to the fusion feature vector;

[0072] It should be noted that the fusion feature vector is processed by the spatio-temporal decoupling attention mechanism. Spatially, the deformable convolution is a convolution kernel with dynamic deformation capability, which can dynamically adjust the sampling position of the convolution kernel and adaptively focus on the key area according to the input feature.

[0073] In an implementable manner, the step of generating spatial interaction features by deformable convolution in the spatial dimension according to the fusion feature vector comprises: dynamically focusing on the interaction hotspot area between obstacles by deformable convolution based on the fusion feature vector to obtain an interaction area feature map; generating a spatial interaction energy field according to the interaction area feature map; and generating spatial interaction features according to the spatial interaction energy field.

[0074] It should be noted that the interaction hotspot area is the spatial area where the probability of motion conflict between obstacles exceeds the threshold, specifically including the lane speed mutation area caused by vehicle cutting in and the path deviation area caused by pedestrian avoidance. The spatial interaction energy field is a three-dimensional vector field that characterizes the intensity of motion conflict between obstacles.

[0075] In a feasible manner, the step of generating time features by analyzing the high-order motion parameters of the historical trajectory in the time dimension based on the fused feature vector includes: extracting historical trajectory data based on the fused feature vector; analyzing the historical trajectory data using a sliding window multi-head attention mechanism to obtain a trajectory analysis result; extracting high-order motion parameters from the trajectory analysis result; and obtaining a behavior pattern based on the high-order motion parameters to generate time features.

[0076] It's important to note that the sliding window multi-head attention mechanism is a fixed-length data slice processor used for time series analysis. Key parameters include the window length, number of heads, and sliding step size. High-order motion parameters are a set of physical quantities that characterize the rate of change of motion states. Behavioral patterns include sudden braking and zigzagging around obstacles.

[0077] Step S40: Processing the spatial interaction features and the temporal features through a hybrid temporal network to obtain a multimodal trajectory prediction result.

[0078] It should be noted that the hybrid temporal network includes short-term memory units (TCN) and long-term memory units (Transformer). The short-term memory unit captures local acceleration trends, and the long-term memory unit models periodic behaviors (such as fixed lane change points for commuter vehicles), generating a set of candidate trajectories with intentions such as conservative following and aggressive overtaking. Each trajectory is accompanied by an uncertainty score, providing explainable prediction results for the downstream decision module. The final trajectory prediction result can be selected based on the score. The overall process can be referred to Figure 2 , Figure 2 Schematic diagram of trajectory prediction structure. Figure 2 The system is divided into an input layer, a fusion layer, a core modeling layer, and an output layer. The input layer extracts semantic features from camera images, performs collective feature extraction on lidar point clouds, and extracts motion features from millimeter-wave radar. The results of the input layer are fed into the fusion layer for fusion through cross-module cross-attention. The fused features are then fed into the core modeling layer. The core modeling layer includes spatial attention focusing on interaction hotspots and temporal attention analyzing historical behaviors. The results are fed into the output layer, where a hybrid temporal network is used to generate physically constrained trajectories and obtain a set of candidate trajectories.

[0079] The embodiment receives multi-modal data from a sensor, the multi-modal data including camera image data, laser radar point cloud data, and millimeter wave radar data; extracts semantic features of the camera image data, geometric features of the laser radar point cloud data, and motion features of the millimeter wave radar data, and fuses the semantic features, the geometric features, and the motion features through a cross-modal cross-attention module to generate a fusion feature vector; generates a spatial interaction feature through deformable convolution in a spatial dimension according to the fusion feature vector, and generates a time feature according to analysis of high-order motion parameters of a historical trajectory in a time dimension; and processes the spatial interaction feature and the time feature through a hybrid temporal network to obtain a multi-modal trajectory prediction result.

[0080] In summary, the embodiment processes various sensor data through a spatial and time decoupling attention mechanism, and uses a hybrid temporal network for trajectory prediction, thereby improving the accuracy of obstacle trajectory prediction.

[0081] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above-mentioned first embodiment can be referred to the above introduction, and the subsequent will not be described in detail. On this basis, please refer to Figure 3 , step S40 further includes steps S401-S403:

[0082] Step S401: processing the spatial interaction feature and the time feature through a time convolution network to obtain local acceleration data;

[0083] It should be noted that the time convolution network is a causal convolution processor for local time sequence feature extraction, which uses an expanded causal convolution structure as the core, gradually expands the time receptive field through a hierarchical expansion mechanism, and is dedicated to capturing short-term motion trend features, including acceleration change direction and intensity, emergency turning signal, and dynamic behavior patterns in a short time.

[0084] Step S402: processing the spatial interaction feature and the time feature through a transformer encoder to obtain a periodic behavior model;

[0085] It should be noted that the transformer encoder is an attention processor for long-term behavior pattern mining, which analyzes the full sequence correlation of historical trajectories, identifies periodic behavior rules (such as fixed lane changing points of commuter vehicles) and intention transmission features (such as the spatio-temporal correlation between turn signal and trajectory deviation), and establishes a long-term behavior memory model.

[0086] Step S403: obtaining a multi-modal trajectory prediction result according to the local acceleration data and the periodic behavior model.

[0087] In an implementable manner, the step of obtaining the multimodal trajectory prediction result according to the local acceleration data and the periodic behavior model comprises: obtaining a set of obstacle candidate trajectories according to the local acceleration data and the periodic behavior model, and scoring the set of obstacle candidate trajectories to obtain a trajectory set score result; and selecting the multimodal trajectory prediction result from the set of obstacle candidate trajectories according to the trajectory set score result.

[0088] It should be noted that when scoring the set of obstacle candidate trajectories, the pixel saturation variance is detected for camera overexposure or underexposure, the Doppler spectrum entropy value is detected for abnormal millimeter wave radar signal-to-noise ratio, the effective echo rate is detected for abnormal laser radar point cloud, and the real-time confidence coefficient in the 0-1 interval of each modality is output. Moreover, dynamic weight adjustment can be performed when scoring, and the online calculation module of confidence-based optimization feature fusion is specifically based. When the confidence coefficient of a certain modality is lower than a threshold (for example, the camera is less than 0.3), the weight thereof is automatically reduced to 20% of the baseline value; at the same time, the weight of an effective modality (for example, the millimeter wave radar is greater than 0.8) is increased to 180% of the baseline value, so as to ensure the robustness of the fusion vector. The multimodal trajectory prediction result can be selected from the set of obstacle candidate trajectories through the trajectory set score result.

[0089] In this embodiment, the local acceleration data is obtained by processing the spatial interaction feature and the time feature through the time convolution network; the periodic behavior model is obtained by processing the spatial interaction feature and the time feature through the transformer encoder; and the multimodal trajectory prediction result is obtained according to the local acceleration data and the periodic behavior model.

[0090] In this embodiment, the local time sequence feature is extracted and the periodic behavior in the historical trajectory is captured through the hybrid time sequence network, so as to obtain the set of obstacle candidate trajectories, and the final trajectory prediction result is obtained according to the score result of the set of obstacle candidate trajectories, thereby improving the accuracy of the obstacle trajectory prediction.

[0091] By way of example, in order to facilitate understanding of the implementation process of the dynamic obstacle trajectory prediction method obtained after the above-mentioned embodiment one, please refer to Figure 4 , Figure 4A brief flowchart of a dynamic obstacle trajectory prediction method is provided, specifically: using a lightweight editor to process camera images, using a sparse 3D convolution network to process laser radar point clouds, using a GRU time sequence editor to process millimeter wave radars, and inputting the processing results into a cross-modal cross-attention module, the cross-modal cross-attention module inputs the processed data into a spatial attention layer and a time attention layer for processing, wherein the spatial attention layer includes deformable convolution, and the time attention layer includes a sliding window multi-head attention, and then the processing results are input into a hybrid time sequence network, the hybrid time sequence network includes a TCM short-term memory unit and a Transformer long-term memory unit, so as to obtain a physically constrained trajectory, and then realize multi-modal trajectory output.

[0092] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the dynamic obstacle trajectory prediction method of the present application. More forms of simple transformation based on this technical concept are within the protection scope of the present application.

[0093] The present application also provides a dynamic obstacle trajectory prediction device, please refer to Figure 5 , the dynamic obstacle trajectory prediction device comprises:

[0094] The receiving module 10 is used for receiving multi-modal data from sensors, and the multi-modal data includes camera image data, laser radar point cloud data and millimeter wave radar data;

[0095] The fusion module 20 is used for extracting semantic features of the camera image data, geometric features of the laser radar point cloud data and motion features of the millimeter wave radar data, and fusing the semantic features, the geometric features and the motion features through a cross-modal cross-attention module to generate a fusion feature vector;

[0096] The generation module 30 is used for generating spatial interaction features in the spatial dimension through deformable convolution according to the fusion feature vector, and generating time features by analyzing high-order motion parameters of historical trajectories in the time dimension according to the fusion feature vector;

[0097] The prediction module 40 is used for processing the spatial interaction features and the time features through a hybrid time sequence network to obtain a multi-modal trajectory prediction result.

[0098] The embodiment receives multi-modal data from sensors, the multi-modal data including camera image data, laser radar point cloud data and millimeter wave radar data; semantic features of the camera image data, geometric features of the laser radar point cloud data and motion features of the millimeter wave radar data are extracted, and the semantic features, the geometric features and the motion features are fused through a cross-modal cross-attention module to generate a fusion feature vector; spatial interaction features are generated through deformable convolution in the spatial dimension according to the fusion feature vector, and time features are generated by analyzing high-order motion parameters of historical trajectories in the time dimension according to the fusion feature vector; the multi-modal trajectory prediction result is obtained by processing the spatial interaction features and the time features through a hybrid temporal network.

[0099] In summary, the embodiment processes various sensor data through a spatial and temporal decoupling attention mechanism, and uses a hybrid temporal network for trajectory prediction, thereby improving the accuracy of obstacle trajectory prediction.

[0100] In an embodiment, the fusion module 20 is further configured to extract semantic features of the camera image data through a lightweight encoder; extract geometric features of the laser radar point cloud data through a sparse three-dimensional convolution network; extract motion features of the millimeter wave radar data through a gated recurrent unit temporal encoder; and input the semantic features, the geometric features and the motion features into the cross-modal cross-attention module for fusion to generate the fusion feature vector.

[0101] In an embodiment, the fusion module 20 is further configured to obtain a query vector according to a fusion vector of the geometric features and the motion features, and obtain a key vector and a value vector according to the semantic features; map the query vector, the key vector and the value vector to a unified hidden space through a linear projection layer to obtain a target query vector, a target key vector and a target value vector; calculate an attention weight matrix according to the target query vector, the target key vector and the target value vector; and perform weighted fusion on the target value vector according to the attention weight matrix to generate the fusion feature vector.

[0102] In an embodiment, the generation module 30 is further configured to dynamically focus on hot spot regions of interaction between obstacles based on the fusion feature vector through deformable convolution to obtain an interaction region feature map; generate a spatial interaction energy field according to the interaction region feature map; and generate spatial interaction features according to the spatial interaction energy field.

[0103] In an embodiment, the generation module 30 is further configured to extract historical trajectory data based on the fused feature vector; analyze the historical trajectory data by using a sliding window multi-head attention mechanism to obtain trajectory analysis results; extract high-order motion parameters from the trajectory analysis results; obtain a behavior pattern according to the high-order motion parameters to generate time features.

[0104] In an embodiment, the prediction module 40 is further configured to obtain local acceleration data by processing the spatial interaction features and the time features through a temporal convolution network; obtain a periodic behavior model by processing the spatial interaction features and the time features through a transformer encoder; and obtain a multi-modal trajectory prediction result according to the local acceleration data and the periodic behavior model.

[0105] In an embodiment, the prediction module 40 is further configured to obtain a set of obstacle candidate trajectories according to the local acceleration data and the periodic behavior model, score the set of obstacle candidate trajectories to obtain a trajectory set score result, and select a multi-modal trajectory prediction result from the set of obstacle candidate trajectories according to the trajectory set score result.

[0106] The dynamic obstacle trajectory prediction device provided in the present application adopts the dynamic obstacle trajectory prediction method in the above embodiments, and can solve the technical problem of how to improve the accuracy of obstacle trajectory prediction. Compared with the prior art, the dynamic obstacle trajectory prediction device provided in the present application has the same beneficial effects as the dynamic obstacle trajectory prediction method provided in the above embodiments, and other technical features in the dynamic obstacle trajectory prediction device are the same as the features disclosed in the above embodiments, which will not be repeated here.

[0107] The present application provides a dynamic obstacle trajectory prediction device, which comprises at least one processor and a memory connected in communication with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the dynamic obstacle trajectory prediction method in the above embodiment one.

[0108] The following will be described with reference to the drawings Figure 6, which shows a schematic diagram of the structure of a dynamic obstacle trajectory prediction device suitable for implementing the embodiments of the present application. The dynamic obstacle trajectory prediction device in the embodiments of the present application can include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The dynamic obstacle trajectory prediction device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0109] like Figure 6 As shown, the dynamic obstacle trajectory prediction device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in ROM (Read Only Memory) 1002 or the program loaded from the storage device 1003 to RAM (Random Access Memory) 1004. Various programs and data required for the operation of the dynamic obstacle trajectory prediction device are also stored in RAM 1004. The processing device 1001, ROM 1002, and RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the dynamic obstacle trajectory prediction device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a dynamic obstacle trajectory prediction device with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented or have instead.

[0110] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiments disclosed in the present application are executed.

[0111] The dynamic obstacle trajectory prediction device provided by the present application adopts the dynamic obstacle trajectory prediction method in the above-mentioned embodiments, and can solve the technical problem of how to improve the accuracy of obstacle trajectory prediction. Compared with the prior art, the dynamic obstacle trajectory prediction device provided by the present application has the same beneficial effects as the dynamic obstacle trajectory prediction method provided by the above-mentioned embodiments, and other technical features in the dynamic obstacle trajectory prediction device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0112] It should be understood that parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above-mentioned embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0113] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0114] The present application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e. computer program) for executing the dynamic obstacle trajectory prediction method in the above-mentioned embodiments.

[0115] The computer readable storage medium provided in the application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiment, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency), etc., or any suitable combination of the above.

[0116] The above computer readable storage medium can be included in the dynamic obstacle trajectory prediction device, or can exist separately without being assembled into the dynamic obstacle trajectory prediction device.

[0117] The above computer readable storage medium carries one or more programs, which, when executed by the dynamic obstacle trajectory prediction device, cause the dynamic obstacle trajectory prediction device to: receive multi-modal data from a sensor, the multi-modal data including camera image data, laser radar point cloud data, and millimeter wave radar data; extract semantic features of the camera image data, geometric features of the laser radar point cloud data, and motion features of the millimeter wave radar data, and fuse the semantic features, the geometric features, and the motion features through a cross-modal cross-attention module to generate a fusion feature vector; generate a spatial interaction feature in a spatial dimension according to the fusion feature vector through a deformable convolution, and generate a time feature by analyzing high-order motion parameters of a historical trajectory in a time dimension according to the fusion feature vector; and obtain a multi-modal trajectory prediction result by processing the spatial interaction feature and the time feature through a hybrid temporal network.

[0118] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0119] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0120] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0121] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer program) for executing the dynamic obstacle trajectory prediction method described above, and can solve the technical problem of how to improve the accuracy of obstacle trajectory prediction. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the dynamic obstacle trajectory prediction method provided by the above-mentioned embodiments, which will not be described here.

[0122] The application further provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the dynamic obstacle trajectory prediction method as described above.

[0123] The computer program product provided by the application can solve the technical problem of how to improve the accuracy of obstacle trajectory prediction. Compared with the prior art, the beneficial effects of the computer program product provided by the application are the same as those of the dynamic obstacle trajectory prediction method provided by the above-mentioned embodiments, and are not described here.

[0124] The above only describes some embodiments of the application, and does not limit the patent scope of the application. Any equivalent structural transformation, direct / indirect application in other related technical fields, or direct / indirect application in other related technical fields based on the technical concept of the application and the content of the specification and drawings are included in the patent protection scope of the application.

Claims

1. A dynamic obstacle trajectory prediction method, characterized in that: The method includes: Receiving multimodal data from a sensor, the multimodal data including camera image data, lidar point cloud data, and millimeter-wave radar data; extracting semantic features of the camera image data, geometric features of the lidar point cloud data, and motion features of the millimeter-wave radar data, and fusing the semantic features, the geometric features, and the motion features through a cross-modal cross-attention module to generate a fused feature vector; Generate spatial interaction features through deformable convolution in the spatial dimension based on the fused feature vector, and generate temporal features by analyzing high-order motion parameters of the historical trajectory in the temporal dimension based on the fused feature vector; The spatial interaction features and the temporal features are processed by a hybrid temporal network to obtain a multimodal trajectory prediction result.

2. The method according to claim 1, wherein The step of extracting the semantic features of the camera image data, the geometric features of the lidar point cloud data, and the motion features of the millimeter wave radar data, and fusing the semantic features, the geometric features, and the motion features through a cross-modal cross-attention module to generate a fused feature vector includes: Extracting semantic features of the camera image data through a lightweight encoder; Extracting geometric features of the lidar point cloud data through a sparse three-dimensional convolutional network; extracting motion features of the millimeter-wave radar data through a gated recurrent unit temporal encoder; The semantic features, the geometric features, and the motion features are input into the cross-modal cross attention module for fusion to generate the fused feature vector.

3. The method according to claim 1, wherein The step of fusing the semantic features, the geometric features, and the motion features through a cross-modal attention module to generate a fused feature vector includes: Obtaining a query vector according to a fusion vector of the geometric features and the motion features, and obtaining a key vector and a value vector according to the semantic features; Mapping the query vector, the key vector, and the value vector to a unified latent space through a linear projection layer to obtain a target query vector, a target key vector, and a target value vector; Calculate an attention weight matrix based on the target query vector, the target key vector, and the target value vector; The target value vector is weightedly fused according to the attention weight matrix to generate the fused feature vector.

4. The method according to claim 1, wherein The step of generating spatial interaction features by deformable convolution in the spatial dimension according to the fused feature vector includes: Based on the fused feature vector, the interactive hotspot areas between obstacles are dynamically focused through deformable convolution to obtain an interactive area feature map; generating a spatial interaction energy field according to the interaction region feature map; A spatial interaction feature is generated according to the spatial interaction energy field.

5. The method according to claim 1, wherein The step of analyzing the high-order motion parameters of the historical trajectory in the time dimension according to the fused feature vector to generate a time feature comprises: Extracting historical trajectory data based on the fused feature vector; A sliding window multi-head attention mechanism is used to analyze the historical trajectory data to obtain a trajectory analysis result; extracting high-order motion parameters from the trajectory analysis results; A behavior pattern is obtained according to the high-order motion parameters, and a time feature is generated.

6. The method according to claim 1, wherein The step of processing the spatial interaction features and the temporal features through a hybrid temporal network to obtain a multimodal trajectory prediction result includes: Processing the spatial interaction features and the temporal features through a temporal convolutional network to obtain local acceleration data; Processing the spatial interaction features and the temporal features through a transformer encoder to obtain a periodic behavior model; A multimodal trajectory prediction result is obtained according to the local acceleration data and the periodic behavior model.

7. The method according to claim 6, wherein The step of obtaining a multimodal trajectory prediction result according to the local acceleration data and the periodic behavior model includes: Obtaining a candidate obstacle trajectory set according to the local acceleration data and the periodic behavior model, and scoring the candidate obstacle trajectory set to obtain a trajectory set scoring result; A multimodal trajectory prediction result is selected from the obstacle candidate trajectory set according to the trajectory set scoring result.

8. A dynamic obstacle trajectory prediction device, characterized in that: The device comprises: A receiving module, configured to receive multimodal data from a sensor, wherein the multimodal data includes camera image data, lidar point cloud data, and millimeter-wave radar data; a fusion module, configured to extract semantic features of the camera image data, geometric features of the lidar point cloud data, and motion features of the millimeter-wave radar data, and fuse the semantic features, the geometric features, and the motion features through a cross-modal cross-attention module to generate a fused feature vector; a generation module, configured to generate spatial interaction features by deformable convolution in the spatial dimension based on the fused feature vector, and to generate temporal features by analyzing high-order motion parameters of historical trajectories in the temporal dimension based on the fused feature vector; The prediction module is used to process the spatial interaction features and the temporal features through a hybrid temporal network to obtain a multimodal trajectory prediction result.

9. A dynamic obstacle trajectory prediction device, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the dynamic obstacle trajectory prediction method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the dynamic obstacle trajectory prediction method according to any one of claims 1 to 7 is implemented.

Images