Vehicle track prediction method and device and vehicle

Abstract

The invention provides a vehicle track prediction method and device and a vehicle. According to the vehicle trajectory prediction method provided by the embodiment of the invention, the point cloud feature and the image feature are fused through the local-global mixed attention mechanism to obtain the multi-modal fusion feature, and the causal time sequence modeling is performed based on the multi-modal fusion feature and the historical trajectory data of the vehicle by using the causal mask and the time convolution network to obtain the time sequence fusion feature. And the time sequence fusion features are used to predict future trajectory data of the vehicle. According to the invention, on-line real-time sequential reasoning of vehicle trajectory prediction can be realized, and the high-real-time requirements of scenes such as automatic driving and the like are met.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving, and in particular to a vehicle trajectory prediction method, device, and vehicle. Background Art

[0002] Vehicle trajectory prediction is a core technology of autonomous driving and intelligent transportation systems, which aims to predict the future movement path of a vehicle.

[0003] Currently, the implementation of vehicle trajectory prediction widely relies on multimodal data fusion. The real-time inference latency and computational efficiency of multimodal data fusion directly affect the real-time performance of vehicle trajectory prediction. Multimodal data fusion based on non-causal time series modeling uses a bidirectional long short-term memory network (LSTM) or future frame information to fuse multiple frames of data to obtain multimodal fusion features. Because this technology relies on future frames, it needs to cache the next 3-5 frames of data, resulting in a real-time inference latency of more than 100ms. Therefore, vehicle trajectory prediction solutions using this technology cannot support online decision-making and cannot meet the real-time requirements of scenarios such as autonomous driving. At the same time, due to the high computational complexity and low computational efficiency of multimodal data fusion, vehicle trajectory prediction relying on multimodal data fusion cannot meet the real-time requirements of scenarios such as autonomous driving. Summary of the Invention

[0004] In view of this, the present disclosure provides a vehicle trajectory prediction method, device, and vehicle.

[0005] According to a first aspect of the present disclosure, a vehicle trajectory prediction method is provided. The method is applied to a vehicle equipped with a multi-view camera array and a laser radar. The method comprises:

[0006] Acquire multi-view images captured by the multi-view camera array and point clouds captured by the laser radar;

[0007] Acquiring image features using the multi-view images;

[0008] obtaining point cloud features using the point cloud;

[0009] Performing spatiotemporal fusion on the image features and the point cloud features to obtain a temporal fusion feature, wherein the spatiotemporal fusion includes: fusing the point cloud features and the image features through a local-global hybrid attention mechanism to obtain a multimodal fusion feature, and performing causal temporal modeling based on the multimodal fusion feature and the historical trajectory data of the vehicle using a causal mask and a temporal convolutional network (TCN) to obtain a temporal fusion feature, wherein each time step in the temporal fusion feature corresponds to an output feature vector, and the output feature vector of each time step represents a feature extraction result of the fusion feature of the time step and the previous time step;

[0010] The future trajectory data of the vehicle is predicted based on the time series fusion feature.

[0011] In some implementations of the first aspect of the present disclosure, fusing the point cloud features and the image features through a local-global hybrid attention mechanism to obtain a multimodal fusion feature includes:

[0012] Obtaining locally enhanced image features and locally enhanced point cloud features based on the point cloud features and the image features through a local attention module, wherein the local attention module adopts a sliding window local attention mechanism;

[0013] Obtaining globally enhanced image features and globally enhanced point cloud features based on the point cloud features, the image features, the locally enhanced point cloud features, and the locally enhanced image features through a global attention module, wherein the global attention module adopts a global hash sparse attention mechanism based on a local sensitive hash (LSH) function;

[0014] The multimodal fusion feature is obtained based on the locally enhanced point cloud feature, the globally enhanced point cloud feature, the locally enhanced image feature, the globally enhanced image feature, the point cloud feature and the image feature by adopting a gated dynamic fusion and weighted summation method.

[0015] In some embodiments of the first aspect of the present disclosure, the method further includes: obtaining IMU data from the IMU mounted on the vehicle; and before the causal time series modeling, using the IMU data to predict the feature offset of the historical trajectory data of the vehicle to compensate for the feature misalignment caused by vehicle motion.

[0016] In some implementations of the first aspect of the present disclosure, predicting the feature offset of the historical trajectory data using the IMU data to compensate for feature misalignment caused by vehicle motion includes:

[0017] A first feature offset is predicted based on the IMU data using a bidirectional long short-term memory network (LSTM);

[0018] Inputting the first characteristic offset into a kinematic model to calculate a second characteristic offset based on a kinematic equation of the vehicle;

[0019] The historical trajectory data of the vehicle is compensated according to the second offset to eliminate feature misalignment in the historical trajectory data caused by vehicle motion.

[0020] In some implementations of the first aspect of the present disclosure, the method further includes: dynamically discarding redundant features according to channel confidence in the causal time series modeling.

[0021] In some embodiments of the first aspect of the present disclosure, redundant features are dynamically discarded according to channel confidence in the causal temporal modeling, including: performing the following processing for each output channel in each layer or specific layer of the TCN: calculating a confidence score of the output channel through a gating network, the confidence score representing the feature importance of the output channel; and processing the confidence score using Gumbel-Softmax to generate a discrete gating decision value, and dynamically retaining or discarding the features corresponding to the output channel according to the gating decision value.

[0022] In some embodiments of the first aspect of the present disclosure, the method further includes: evaluating whether the cross-modal consistency of the image features and the point cloud features meets the requirements; when the cross-modal consistency of the image features and the point cloud features does not meet the requirements, using a generative adversarial network to generate a pseudo multi-view image based on the point cloud, obtaining features of the pseudo multi-view image, and merging the features of the pseudo multi-view image into the image features, and the loss function of the generative adversarial network includes: adversarial loss, cycle consistency loss and cross-modal alignment loss.

[0023] In some implementations of the first aspect of the present disclosure, the evaluating whether the cross-modal consistency of the image features and the point cloud features meets the requirements includes one or more of the following:

[0024] Calculating the similarity between the point cloud feature and the image feature at the same spatial position through BEV spatial contrast learning, and determining that the cross-modal consistency of the point cloud feature and the image feature does not meet the requirement when the similarity is less than a first predetermined similarity threshold;

[0025] Calculate the cross-modal feature similarity between the forward view image and the point cloud in the multi-view image, and when the cross-modal feature similarity is less than a second predetermined similarity threshold, determine that the cross-modal consistency of the point cloud and the forward view image does not meet the requirements.

[0026] According to a second aspect of the present disclosure, a vehicle trajectory prediction device is provided, comprising:

[0027] A data acquisition unit, configured to acquire the multi-view images acquired by the multi-view camera array and the point cloud acquired by the laser radar;

[0028] An image feature extraction unit, configured to obtain image features using the multi-view images;

[0029] a point cloud feature extraction unit, configured to obtain point cloud features using the point cloud;

[0030] a spatiotemporal fusion unit, configured to perform spatiotemporal fusion on the image features and the point cloud features to obtain a temporal fusion feature, wherein the spatiotemporal fusion comprises: fusing the point cloud features and the image features through a local-global hybrid attention mechanism to obtain a multimodal fusion feature, and performing causal temporal modeling based on the multimodal fusion feature and the historical trajectory data of the vehicle using a causal mask and a temporal convolutional network (TCN) to obtain a temporal fusion feature, wherein each time step in the temporal fusion feature corresponds to an output feature vector, and the output feature vector of each time step represents a feature extraction result of the fusion feature of the time step and the previous time step;

[0031] A trajectory determination unit is used to predict and obtain future trajectory data of the vehicle based on the time series fusion feature.

[0032] According to a third aspect of the present disclosure, a vehicle is provided, wherein the vehicle is equipped with a multi-view camera sequence and a laser radar, and the vehicle includes the aforementioned vehicle trajectory prediction device.

[0033] The disclosed embodiments utilize causal masks and TCN-based causal temporal modeling to achieve spatiotemporal fusion of point cloud and image features. This utilizes only the vehicle's historical trajectory data, preventing leakage of future frames. This enables online, real-time temporal reasoning for vehicle trajectory prediction, meeting the high real-time requirements of scenarios such as autonomous driving. Furthermore, the local-global hybrid attention mechanism effectively reduces computational complexity while capturing both local detail features and long-range dependencies, resulting in high computational efficiency. This further reduces the time required for real-time inference of vehicle trajectory prediction, better meeting the high real-time requirements of scenarios such as autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0035] Figure 1 A schematic diagram of the system architecture to which the embodiments of the present disclosure are applicable;

[0036] Figure 2 A schematic diagram of a process flow of a vehicle trajectory prediction method provided by an embodiment of the present disclosure;

[0037] Figure 3 A schematic diagram of a process for obtaining image features using multi-view images according to an embodiment of the present disclosure;

[0038] Figure 4 A schematic diagram of the process of space-time fusion involved in an embodiment of the present disclosure;

[0039] Figure 5 Another schematic diagram of a process flow of a vehicle trajectory prediction method provided by an embodiment of the present disclosure;

[0040] Figure 6 A schematic diagram of the structure of a vehicle trajectory prediction device provided in an embodiment of the present disclosure;

[0041] Figure 7 A schematic structural block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present disclosure in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0043] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure. The singular forms "a," "an," "the," and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0044] As used herein, the words “if,” “if,” and the like may be interpreted as “at the time of,” “when,” “in response to determining,” or “in response to detecting,” depending on the context. Similarly, the phrases “if it is determined” or “if (stated condition or event) is detected” may be interpreted as “when it is determined” or “in response to the determination” or “when detecting (stated condition or event)” or “in response to detecting (stated condition or event),” depending on the context.

[0045] To facilitate understanding, the system structure applicable to the embodiments of the present disclosure is briefly described below.

[0046] Figure 1 Schematic diagram of the structure of the system to which the embodiment of the present disclosure is applicable. Figure 1 The system to which the embodiments of the present disclosure are applicable may include: an electronic device and a peripheral sensor component connected to the electronic device, the peripheral sensor component including but not limited to a multi-view camera array and a lidar.

[0047] A multi-view camera array can be mounted on a vehicle to capture multi-view images of the vehicle's surroundings. In some examples, the multi-view camera array can be implemented as, but not limited to, a six-view camera group, including a front-view camera, a rear-view camera, a left-view camera, a right-view camera, an upward-view camera, and a downward-view camera. In specific applications, the cameras can be evenly distributed around the vehicle to ensure full coverage, with each camera capturing a portion of the surrounding scene.

[0048] LiDAR is mounted on a vehicle and can be used to collect point clouds in real time. In specific applications, the LiDAR's installation position can be flexibly adjusted to suit the specific application, ensuring that the point cloud collected by the LiDAR covers the vehicle's forward, rearward, or surrounding environment.

[0049] Each point in the point cloud collected by the laser radar can contain three-dimensional coordinates, reflection intensity and timestamp. Among them, the three-dimensional coordinates can be calculated by the laser flight time and determined in combination with the radar rotation angle (azimuth, pitch angle). The three-dimensional coordinates of each point in the point cloud can be the three-dimensional coordinates in the laser radar coordinate system, which can be converted to the vehicle body coordinate system, world coordinate system, etc. through coordinate conversion. The reflection intensity (also called reflectivity) depends on the surface material of the object and can be used to distinguish the material type (such as road marking recognition). The timestamp can be used to record the time difference between laser emission and reception, which can be used for analysis of dynamic scenes such as moving object trajectory tracking.

[0050] Furthermore, the peripheral sensor assembly may also include: an inertial measurement unit (IMU). The IMU is installed on the vehicle and connected to the electronic device, and can be used to measure IMU data and provide the IMU data to the electronic device. IMU data may include but is not limited to the vehicle's three-axis acceleration, three-axis angular velocity, etc. The IMU data can be used to determine the vehicle's instantaneous motion state, assist in positioning, attitude estimation, etc. For example, the IMU data can be used to determine the vehicle's roll angle, pitch angle, yaw angle and other attitude information.

[0051] Furthermore, the above system may also include an on-board communication module (for example, a 4G / 5G, Wi-Fi, Internet of Vehicles, and other communication modules) and / or an on-board environmental sensor. Among them, the on-board communication module can be used to connect to the cloud to obtain meteorological data. For example, it is possible to receive meteorological data on the driving route pushed in real time by Amap, Google Maps, etc. For another example, it is possible to receive meteorological data provided by devices around the vehicle (such as other vehicles, roadside units, etc.). The on-board environmental sensor can be used to detect the weather conditions of the vehicle's surrounding environment in real time and generate corresponding meteorological data. The on-board environmental sensor may include but is not limited to an ambient light sensor, a humidity sensor, a temperature sensor, and the like.

[0052] The system to which the embodiments of the present disclosure are applicable may be, but is not limited to, any system requiring multimodal data fusion. For example, the system may be, but is not limited to, an autonomous driving system, an intelligent assisted driving system, a traffic congestion assistance system, etc. Figure 1 The system provided by the embodiments of the present disclosure can be installed in a vehicle and used as, but not limited to, an intelligent assisted driving system, a traffic congestion assistance system, an automatic driving system, etc. of the vehicle.

[0053] It should be noted that the “vehicle” described in the embodiments of the present disclosure may be implemented as, but not limited to, multiple wheeled mobile robots, wheeled mobile robots, mobile robots, general-purpose vehicles, aircraft, ships, Autonomous Rail Rapid Transit (ART) systems, industrial automation equipment, etc. General-purpose vehicles may include, but are not limited to, passenger cars, commercial vehicles (e.g., trucks, buses, vans, etc.), special-purpose vehicles (e.g., ambulances, fire trucks, engineering vehicles, rescue vehicles, etc.), agricultural and industrial vehicles (e.g., harvesters, forklifts, etc.), transportation and logistics vehicles (e.g., container trucks, refrigerated trucks, etc.), new energy vehicles (e.g., electric vehicles, hybrid electric vehicles), and special vehicles (e.g., garbage trucks, water trucks, etc.).

[0054] The embodiments of the present disclosure can be applied to various scenarios such as urban transportation, highways, ports, mines, farms, closed parks, and industrial production. They can be applied to many aspects such as travel by car, public transportation, logistics distribution, unmanned transportation, terminal delivery, automated agricultural operations, and automated sanitation. Of course, the embodiments of the present disclosure can also be applied to any other intelligent control scenarios involving, for example, multimodal data fusion. The present disclosure does not limit the application scenarios and applicable fields of the embodiments of the present disclosure.

[0055] The specific implementation of the embodiment of the present disclosure is described in detail below.

[0056] Figure 2The flowchart of the vehicle trajectory prediction method provided by the embodiment of the present disclosure is shown. The vehicle trajectory prediction method can be executed by the electronic device below, which can be implemented as but not limited to a domain controller or other similar devices. Figure 2 The vehicle trajectory prediction method of the embodiment of the present disclosure includes the following steps:

[0057] Step 201: Acquire multi-view images captured by a multi-view camera array and point clouds captured by a laser radar;

[0058] Step 202: obtaining image features using multi-view images;

[0059] Step 203, using the point cloud to obtain point cloud features;

[0060] Step 204: Perform spatiotemporal fusion on the image features and the point cloud features to obtain a temporal fusion feature. The spatiotemporal fusion includes: fusing the point cloud features and the image features through a local-global hybrid attention mechanism to obtain a multimodal fusion feature, and using a causal mask and a temporal convolutional network (TCN) to perform causal temporal modeling based on the multimodal fusion feature and the vehicle's historical trajectory data to obtain a temporal fusion feature. Each time step in the temporal fusion feature corresponds to an output feature vector, and the output feature vector of each time step represents the feature extraction result of the fusion feature of the time step and the previous time step.

[0061] Step 205 : obtaining the future trajectory data of the vehicle based on the time series fusion feature prediction.

[0062] The method of the disclosed embodiment achieves spatiotemporal fusion of point cloud and image features based on causal masking and TCN causal temporal modeling. It only uses historical vehicle trajectory data and prohibits leakage of future frames. This enables online, real-time temporal reasoning for vehicle trajectory prediction, meeting the high real-time requirements of scenarios such as autonomous driving. Furthermore, the local-global hybrid attention mechanism effectively reduces complexity while capturing local detail features and long-range dependencies, resulting in high computational efficiency. This further reduces the time required for real-time inference of vehicle trajectory prediction, better meeting the high real-time requirements of scenarios such as autonomous driving.

[0063] In step 201, the multi-view images and point cloud are synchronized. Specifically, the synchronization of the multi-view images and point cloud can be achieved through mechanisms such as synchronous triggering and time alignment of the multi-view camera array and the lidar.

[0064] Figure 3 FIG2 shows a flow chart of obtaining image features using multi-view images in step 202. Figure 3 , the process may include steps 301 to 303:

[0065] Step 301: Determine the illumination level of each view image in the multi-view image using a lightweight convolutional neural network (CNN) classifier.

[0066] A frame of image can be captured from the camera corresponding to each viewpoint image, and processed using a lightweight CNN classifier to obtain the illumination level of the corresponding viewpoint image. Specifically, for each viewpoint image, a lightweight CNN classifier can be used to extract global illumination features to obtain the illumination level. Lightweight CNN classifiers can be used, but are not limited to, MobileNetV3. Estimating illumination levels using a lightweight CNN classifier can more accurately capture features under complex lighting conditions, such as shadows and mixed light sources, and thus accurately assess the illumination level of each viewpoint image.

[0067] In specific applications, other methods can be used to determine the illumination level of each viewing angle image as needed. The embodiment of the present disclosure does not limit the specific method for determining the illumination level of each viewing angle image.

[0068] In step 302, the following processing is performed on each view image in the multi-view image to extract its features: an image feature extraction module including conditional convolution is used to dynamically adjust the RGB channel weights of the basic convolution kernel according to the illumination level of the view image to generate a dynamic convolution kernel that adapts to the illumination conditions of the view image and use the dynamic convolution kernel to extract the features of the view image.

[0069] Specifically, the process of extracting features of a certain perspective image using an image feature extraction module including conditional convolution may include: generating an RGB channel weight scaling factor based on the illumination level of the current perspective image, adjusting the RGB channel weights of the basic convolution kernel using the RGB channel weight scaling factor to generate a dynamic convolution kernel, and using the dynamic convolution kernel to perform feature extraction on the current perspective image to obtain the features of the current perspective image. As a result, in low-light areas, the dynamic convolution kernel can enhance the ability to extract high-frequency details to restore dark information, and in overexposed areas, the dynamic convolution kernel can suppress noise and balance brightness, thereby adaptively optimizing its feature extraction process based on the illumination conditions of each perspective image, enhancing color invariance under low light conditions, and suppressing noise in overexposed areas.

[0070] The features of a single-view image can include one or more of the following: color, texture, shape, spatial relationships, and semantics. Spatial relationships include object position, which describes the absolute or relative position of segmented objects in the image. Semantic information includes object category information, which can be used to describe the object category and its confidence at the pixel level.

[0071] Step 303: align and fuse the features of the images from each perspective to obtain multi-scale features.

[0072] Specifically, multi-scale features can be obtained by geometric alignment and attention weighting. The embodiments of this disclosure do not limit the specific implementation method of obtaining multi-scale features.

[0073] Here, multi-scale features are image features of multi-view images. Multi-scale features can include one or more of the following information: color, texture, shape, semantics, and spatial constraints. Semantic information includes object type information.

[0074] Furthermore, in step 302, a lightweight semantic segmentation network can be used to generate a semantic segmentation mask based on the view image, and the semantic segmentation mask can be used as an attention weight in the process of extracting features from the view image using a dynamic convolution kernel. In this way, the lightweight semantic segmentation network can dynamically guide the image feature extraction model to focus on key areas such as roads and obstacles, while suppressing irrelevant background interference.

[0075] For example, the lightweight semantic segmentation network may be, but is not limited to, DeepLabv3+Mobile. The semantic segmentation mask generated based on the view image may be a probability map, a binary mask, or other applicable forms.

[0076] use Figure 3 The feature extraction method shown can achieve efficient feature extraction from multi-view images by dynamically adjusting the convolution kernel to adapt to lighting changes in multi-view scenarios while accurately focusing on key areas. This improves the efficiency of multi-view image feature extraction and further reduces computational time, thereby reducing latency and improving real-time performance without reducing the accuracy of vehicle predicted trajectory.

[0077] Furthermore, before step 203, the vehicle trajectory prediction method of the disclosed embodiment may also include: preprocessing the point cloud. This preprocessing may include, but is not limited to, downsampling the point cloud based on vehicle speed, and performing sparse region interpolation and dense region downsampling on the downsampled point cloud. This allows for "global lightweighting and local refinement" in a dynamic environment, while responding to both the dynamic environment (i.e., speed changes) and static characteristics (i.e., local density distribution), thereby further reducing computational complexity and time without compromising accuracy.

[0078] When adaptively downsampling the point cloud based on vehicle speed, the resolution of the point cloud can be dynamically adjusted based on changes in vehicle speed, resulting in a sparse point cloud in high-speed scenarios and a higher resolution in low-speed scenarios. Specifically, a piecewise linear interpolation method based on vehicle speed can be used to dynamically adjust the downsampling ratio of the point cloud to achieve adaptive downsampling of the point cloud. The vehicle speed can be provided in real time by an IMU installed on the vehicle. Alternatively, the vehicle speed can be determined based on an extended Kalman filter (EKF) using IMU data provided by the IMU installed on the vehicle.

[0079] Interpolation and completion of sparse regions of the point cloud and further downsampling of dense regions can prevent information loss in sparse regions while avoiding computational redundancy in dense regions. In some examples, sparse and dense regions can be dynamically divided based on the local density of the point cloud. For example, the local density of the point cloud can be estimated based on K-Nearest Neighbors (KNN) and sparse and dense regions can be dynamically distinguished based on this local density.

[0080] Furthermore, in step 203, point cloud features can be obtained by adaptively voxelizing the point cloud to obtain voxel data by adjusting the voxel grid size based on the local point cloud density, and processing the voxel data using separable 3D convolution to obtain point cloud features. Separable 3D convolution decomposes the three-dimensional convolution into two independent low-dimensional operations, which can reduce redundant calculations and reduce the waste of computation on empty voxels. This can further improve computational efficiency and reduce computational time without sacrificing accuracy.

[0081] In some examples, point clouds can be voxelized using dynamic meshing (density-based voxelization) to obtain voxel data. This can preserve details, avoid information loss in sparse areas, and suppress computational redundancy in dense areas, further improving computational efficiency, reducing computational complexity, and shortening computational time.

[0082] Voxel data can include the following information: spatial position, voxel size, voxel attribute information, etc. Voxel coordinates represent the position of each voxel in three-dimensional space and can be represented by grid coordinates. Voxel size represents the length, width and height of each voxel. The attribute information of the voxel may include but is not limited to one or more of the following: point density, intensity, color information, normal vector, height feature, etc. Among them, point density represents the number of original point cloud points contained in the voxel, reflecting the object density or occlusion of the area; intensity is the signal strength returned by the lidar, which can be used to distinguish materials; normal vector represents the surface direction of the point in the voxel, which can be used to reconstruct the geometric shape of the object. The height feature represents the height of the voxel relative to the ground, which can be used to distinguish between ground and non-ground objects.

[0083] Figure 4 FIG2 shows an exemplary process of fusing point cloud features and image features through a local-global hybrid attention mechanism in step 204. Figure 4 , the process may include the following steps:

[0084] Step 401: obtaining locally enhanced image features and locally enhanced point cloud features based on point cloud features and image features through a local attention module;

[0085] The local attention module can adopt a sliding window local attention mechanism. The sliding window local attention limits the calculation to a small window, greatly reducing the amount of calculation. In the sliding window local attention, each pixel only needs to calculate the attention relationship with the pixels in the window (taking a 7×7 neighborhood as an example, 48 (7×7-1) pixels), and the computational complexity is significantly reduced. At the same time, the sliding window local attention can effectively capture the correlation between local pixels. In point cloud features and image features, many features (such as edges, textures, etc.) are local. By calculating self-attention within the window, these local features can be better extracted, and local detail features such as local textures and edges (such as vehicle contours, lane lines, etc.) can be captured.

[0086] The local attention module processes both image and point cloud data. In image processing, self-attention can be calculated within a 7×7 window of image features to enhance texture and edge features. In point cloud processing, the point cloud can be projected onto the image to guide local regions of the point cloud using the image's 7×7 window, or for each point cloud, points projected into the same image window can be selected as the local neighborhood.

[0087] Specifically, image-side processing can include: projecting the point cloud features onto the image plane to obtain the image area corresponding to each point, taking the 7×7 neighborhood (49 pixels in total) around each image pixel in the image feature as a local window, and finding the set P of all point cloud points projected to this area by the point cloud feature within the window. If the set P of point cloud points is not empty, the point cloud features are aggregated; if the set P of point cloud points is empty, a zero vector can be filled in. Using the image window feature as the query (Query) and the aggregated point cloud feature as the key (Key) and value (Value), a cross-modal attention calculation is performed to obtain a locally enhanced image feature, which incorporates the geometric information of the point cloud. In this way, edge features such as vehicle contours and lane lines can be enhanced.

[0088] Specifically, point cloud processing can include: using the camera's intrinsic parameter matrix K and extrinsic parameter matrix to project the points in the point cloud feature onto the image plane to obtain their projected positions, corresponding to the pixel positions in the image; for each point cloud point, taking a 7×7 neighborhood in the image based on its projected position, extracting image features from the 7×7 neighborhood and performing average pooling on the image features to obtain a local image feature vector; using the point cloud feature as the query (Query) and the local image feature vector as the key (Key) and value (Value) to perform cross-modal attention calculation to obtain locally enhanced point cloud features. The locally enhanced point cloud features incorporate the semantic information of the image. This can enhance edge features such as vehicle outlines and lane lines.

[0089] In image side processing and point cloud side processing, a sliding window mechanism can also be used for cross-window information interaction. If the local attention module includes multiple local attention layers and these local attention layers are continuous, a new non-aligned window can be formed by cyclically shifting 3 pixels (i.e., half of the window size 7) in the lower right direction in the continuous layer, and local attention calculations are performed in the new non-aligned window. In this way, the window boundary restrictions can be broken, the associated information of adjacent windows can be introduced, the receptive field can be expanded to the global level, and the accuracy can be further improved. If necessary, the attention connection of illegal areas can be shielded by the mask matrix in the cross-window information interaction.

[0090] Although the above description uses a 7×7 neighborhood as an example to illustrate an exemplary implementation process of calculating self-attention using sliding window local attention, it should be noted that in specific applications, the neighborhood can be adjusted to a 5×5 neighborhood or other window size as needed, and the embodiments of the present disclosure do not limit this.

[0091] Step 402: obtaining globally enhanced image features and globally enhanced point cloud features based on the point cloud features, image features, locally enhanced point cloud features, and locally enhanced image features through a global attention module;

[0092] The global attention module can use a global hash sparse attention mechanism. For example, a global hash sparse attention mechanism based on the Locality-Sensitive Hashing (LSH) function can be used. Global hash sparse attention uses a hash function to quickly filter key areas and only calculates attention weights between important locations. This mechanism can reduce the computational complexity of global attention while retaining the ability to model long-range dependencies such as distant vehicles and cross-frame objects.

[0093] In some examples, the global attention module can employ, but is not limited to, the Reformer or other similar model architectures based on LSH attention. Specifically, the Reformer's encoder and decoder both consist of multiple stacked reversible residual layers, each of which contains an LSH attention unit and a block-wise feedforward network. The LSH attention unit uses a hash index to select key regions. When calculating attention, it aggregates elements that may have long-range dependencies into the same hash bucket. This allows computation to span longer distances within the sequence, even though elements with similar semantic features may be far apart in the sequence. This effectively captures long-range dependencies. Furthermore, the Reformer uses LSH bucketing to only calculate attention within the bucket, skipping non-critical background. This significantly reduces complexity and computation by 50%, enabling real-time performance of 30 FPS. The LSH attention unit uses a hash table to select key regions for attention computation, skipping non-critical background, reducing computation by 50%, ensuring 30 FPS real-time performance.

[0094] The processing of the global attention module may include image-side processing and point cloud-side processing. Image-side processing may include: mapping the locally enhanced image features and point cloud features to the same hash space through the LSH function, performing cross-modal attention calculations with the locally enhanced image features as queries and the point cloud features in the same bucket as keys and values ​​to obtain globally enhanced image features. Point cloud-side processing may include: mapping the locally enhanced point cloud features and image features to the same hash space through the LSH function, performing cross-modal attention calculations with the locally enhanced point cloud features as queries and the image features in the same bucket as keys and values ​​to obtain globally enhanced point cloud features.

[0095] If necessary, before performing image-side processing and point cloud-side processing, each feature (i.e., point cloud features, image features, locally enhanced point cloud features, and locally enhanced image features) can be projected into the same latent space through a multi-layer perceptron (MLP) to obtain corresponding features with consistent dimensions, distribution, and semantic expressions.

[0096] In step 403 , a gated dynamic fusion and weighted summation method is used to obtain multimodal fusion features based on the locally enhanced point cloud features, the globally enhanced point cloud features, the locally enhanced image features, the globally enhanced image features, the point cloud features, and the image features.

[0097] A first gating matrix is ​​generated based on the locally enhanced point cloud features and the globally enhanced point cloud features. The first gating matrix is ​​used to perform weighted summation of the locally enhanced point cloud features and the globally enhanced point cloud features to obtain point cloud fusion features. The point cloud fusion features and the original point cloud features are spliced ​​together through residual connection to obtain the final point cloud features.

[0098] A second gating matrix is ​​generated based on the locally enhanced image features and the globally enhanced image features. The second gating matrix is ​​used to perform weighted summation of the locally enhanced image features and the globally enhanced image features to obtain image fusion features. The image fusion features and the original image are spliced ​​together through residual connection to obtain the final image features.

[0099] The final point cloud features and the final image features are weighted and summed to obtain multimodal fusion features. Specifically, the final point cloud features and the final image features can be weighted and summed using a pre-set fixed weight, or the final point cloud features and the final image features can be weighted and summed using a dynamic weight generated based on an attention mechanism. The specific method of weighted summation is not limited in the embodiments of the present disclosure. Obtaining multimodal fusion features by weighted summation can further reduce computational complexity and improve computational efficiency while maintaining the stability of feature dimensions.

[0100] The multimodal fusion features obtained through the local-global hybrid attention mechanism can simultaneously contain local geometric details and global semantic context, effectively balancing details and context, and can further reduce computational complexity and improve computational efficiency while improving the accuracy of multimodal fusion features.

[0101] In step 204, the historical trajectory data of the vehicle can be observed in real time. The historical trajectory data of the vehicle can include trajectory data of a predetermined number of frames in the past, for example, trajectory data of the past 5 frames.

[0102] In step 204, the causal mask can be a two-dimensional matrix, where all elements below the diagonal are 1, indicating that information before the current position is allowed to flow, and all elements above the diagonal are 0, indicating that information after the current position is blocked. In TCN, the causal mask can be used to block the attention weights of future positions to ensure that the attention calculation at time step t only accesses the positions at time step t and before, that is, only historical frames are used, and future frames are prohibited from leaking.

[0103] In step 204, the size of the time series fusion feature is the same as the size of the multimodal fusion feature. For example, if the multimodal fusion feature is a three-dimensional tensor, the time series fusion feature is also a three-dimensional tensor. The three-dimensional tensor can be expressed as (B, S, C), where B represents the number of data samples processed in one forward propagation. A sample in the B dimension is a two-dimensional matrix with a shape of (S, C), and each row of the two-dimensional matrix represents the output feature vector of a time step; S represents the number of time steps in each sequence, and each time step corresponds to an output feature vector. The output feature vector of each time step represents the feature extraction result of the fusion feature of the time step and the previous time step. The output feature vector can also be called the fused feature point, that is, the feature point after the fusion of the point cloud feature and the image feature; C represents the number of channels of the output feature, which is determined by the number of output channels of the last convolutional layer of the TCN.

[0104] Under the constraints of the causal mask, the output feature vector of each time step depends only on the fusion features of that time step and the previous time step. The output feature vector contains the result of TCN's feature extraction and processing of multimodal fusion features in the time series. For each time step t, the calculation process of the output feature vector can be expressed as follows (1):

[0105] h t =TCN causal (x t , x t-1 , ..., x1) (1)

[0106] Among them, x t represents the fused feature vector at time step t, h t represents the output feature vector at time step t.

[0107] Furthermore, in step 204, a learnable attention mask can be introduced into the convolution kernel of the TCN to dynamically adjust the weight of each time step.

[0108] In step 204, by combining the causal mask with TCN, it is possible to efficiently capture local details and global dependencies in the time series data while ensuring strict causality, thereby further improving computational efficiency.

[0109] Figure 5 Another flow chart of the vehicle trajectory prediction method provided by the embodiment of the present disclosure is shown. Figure 5 Before step 204, the method provided by the embodiment of the present disclosure may further include: step 206, obtaining IMU data collected by the IMU mounted on the vehicle, and using the IMU data to predict the feature offset of the vehicle's historical trajectory data to compensate for the feature misalignment caused by the vehicle movement and improve the timing consistency.

[0110] In some examples, feature misalignment caused by vehicle motion can be compensated by bidirectional LSTM and kinematic models. Specifically, step 206 may include: using bidirectional LSTM to predict a first feature offset based on IMU data, inputting the first feature offset into the kinematic model to calculate a second feature offset in combination with the vehicle's kinematic equation, and compensating the vehicle's historical trajectory data according to the second offset to eliminate feature misalignment caused by vehicle motion in the historical trajectory data. Performing LSTM motion compensation before causal time series modeling can correct feature misalignment of historical trajectory data, improve the quality of historical trajectory data input to TCN, reduce the difficulty of causal time series modeling, and further improve the computational efficiency of TCN and the accuracy of time series fusion features.

[0111] It has been verified that motion compensation of the vehicle's historical trajectory data through bidirectional LSTM and kinematic models before causal time series modeling can reduce the vehicle trajectory prediction error by approximately 12% and reduce the end-to-end delay to less than 35 milliseconds.

[0112] Furthermore, in step 204, redundant features can be dynamically discarded based on channel confidence in causal temporal modeling. For example, a learnable gating mechanism can be used to dynamically determine whether to retain or discard features of each feature channel by learning the importance of that channel. Dynamically discarding redundant features based on channel confidence can further improve the efficiency and performance of TCNs using causal masks.

[0113] Specifically, the following processing is performed for each output channel in each layer or specific layer of the TCN: the confidence score of the output channel is calculated through the gating network, and the confidence score represents the feature importance of the output channel; and the confidence score is processed using Gumbel-Softmax to generate a discrete gating decision value, and the corresponding features of the output channel are dynamically retained or discarded according to the gating decision value. Among them, a gating decision value of 0 indicates discarding, and a gating decision value of 1 indicates retaining. Here, in the TCN using causal masks, in order to ensure that the gating mechanism does not violate the constraints of the causal mask, only the information of the current time step and before can be used when calculating the channel confidence.

[0114] As mentioned above, by introducing gating networks and Gumbel-Softmax at each layer or specific layers, TCN can learn which feature channels are important and which can be discarded, thereby better supporting the processing of high-dimensional features and tasks with high real-time requirements.

[0115] See also Figure 5 Furthermore, before step 204, the method of the embodiment of the present disclosure may further include:

[0116] Step 207: evaluating whether the cross-modal consistency of the image features and the point cloud features meets the requirements;

[0117] In step 208, when the cross-modal consistency of the image features and the point cloud features does not meet the requirements, a generative adversarial network (GAN) is used to generate a pseudo multi-view image based on the point cloud, obtain features of the pseudo multi-view image, and merge the features of the pseudo multi-view image into the image features.

[0118] In some embodiments, in step 208, whether the cross-modal consistency of image features and point cloud features meets the requirements can be evaluated in one or both of the following ways: 1) calculating the similarity between point cloud features and image features at the same spatial position through BEV spatial contrast learning, and when the similarity is less than a first predetermined similarity threshold, determining that the cross-modal consistency of point cloud features and image features does not meet the requirements; 2) calculating the cross-modal feature similarity between the forward view image and the point cloud in the multi-view image, and when the cross-modal feature similarity is less than a second predetermined similarity threshold, determining that the cross-modal consistency of the point cloud and the forward view image does not meet the requirements.

[0119] Specifically, the average cosine similarity between the point cloud features and image features at the same spatial location can be calculated. If the average cosine similarity is greater than or equal to a first predetermined similarity threshold, the evaluation is considered passed, and the image features obtained in step 202 and the point cloud features obtained in step 203 can be directly used to perform spatiotemporal fusion in step 204. If the average cosine similarity is less than the first predetermined similarity threshold, the image features of the pseudo-multi-view image generated by the GAN can be mixed with the image features obtained in step 203 to obtain new image features. This new image feature can then be used together with the point cloud features obtained in step 203 to perform spatiotemporal fusion in step 204. Thus, BEV spatial contrast learning can be used to dynamically evaluate the consistency of feature distributions between the lidar and camera in BEV space, addressing projection deviation issues and further reducing fusion deviations caused by the external environment in extreme scenarios (e.g., rainy days, nighttime, etc.). This improves the reliability and robustness of multimodal data fusion, thereby further enhancing the real-time performance of multimodal data fusion without compromising accuracy.

[0120] When the image features of the pseudo multi-view image generated by the GAN are mixed with the image features obtained in step 203 to obtain new image features, this can be achieved through adaptive mixing weights. Specifically, a mixing ratio can be dynamically assigned based on the difference between the average cosine similarity and the first predetermined similarity threshold (for example, the larger the difference, the higher the GAN feature weight), and the image features of the pseudo multi-view image generated by the GAN are mixed with the image features obtained in step 203 according to the mixing ratio.

[0121] The first predetermined similarity threshold can be a dynamic threshold that can be adaptively adjusted based on scene complexity. Scene complexity involves multiple factors such as weather and lighting. For example, the first predetermined similarity threshold can be set to 0.6 on rainy days and 0.65 at night.

[0122] Furthermore, during the model training phase, the parameters of the image feature-related model and the point cloud feature-related model can be further optimized by adding the contrast loss (InfoNCE Loss) function in BEV spatial contrast learning to the loss function, forcing cross-modal features to be aligned in the BEV space, thereby solving the projection deviation problem and further reducing the fusion deviation caused by the external environment in extreme scenarios (such as thunderstorms, nighttime, etc.), thereby improving the reliability and robustness of multimodal data fusion.

[0123] Specifically, the average cosine similarity between the forward view image features (the forward view image features can be obtained in step 202) and the point cloud features at the same spatial position can be calculated, and the average cosine similarity can be used as the cross-modal feature similarity between the forward view image and the point cloud in the multi-view image. When the average cosine similarity is greater than or equal to the second predetermined similarity threshold, the evaluation is passed, and the image features obtained in step 202 and the point cloud features obtained in step 203 can be directly used to perform the spatiotemporal fusion of step 204. When the average cosine similarity is less than the second predetermined similarity threshold, the image features of the pseudo multi-view image generated by GAN can be mixed with the image features obtained in step 202 to obtain new image features, and the new image features and the point cloud features obtained in step 203 can be used to perform the spatiotemporal fusion of step 204. Therefore, the consistency of feature distribution of lidar and camera can be evaluated by forward graph feature matching, the projection deviation problem can be solved, and the fusion deviation caused by the external environment in extreme scenarios (such as thunderstorms, nighttime, etc.) can be further reduced, thereby improving the reliability and robustness of multimodal data fusion, thereby further improving the real-time performance of multimodal data fusion without reducing accuracy.

[0124] Furthermore, during the model training phase, by adding the mutual information (MI) related to the forward graph feature matching into the loss function, the parameters of the image feature-related model and the point cloud feature-related model can be further optimized, thereby enhancing the representation consistency of the same object in different modalities, solving the projection deviation problem, and further reducing the fusion deviation caused by the external environment (such as occlusion, lighting changes, etc.) in extreme scenarios (such as thunderstorms, nighttime, etc.), thereby improving the reliability and robustness of multimodal data fusion.

[0125] In step 208, the GAN can be, but is not limited to, a pixel-level generative adversarial network. Specifically, the GAN can be, but is not limited to, a cycle-consistent generative adversarial network (CycleGAN). The GAN includes a generator and a discriminator. The generator is responsible for generating pseudo data of one modality based on data of another modality, while the discriminator is used to distinguish true from false, thereby driving the generator to produce more realistic pseudo data.

[0126] In step 208, the loss function of GAN includes: adversarial loss, cycle consistency loss (Cycle Consistency Loss) and cross-modal alignment loss (Cross-modal Alignment Loss). Among them, the adversarial loss (GAN Loss) is used to train the generator and the discriminator to conduct adversarial training, so that the features generated by the generator are as realistic as possible. The cycle consistency loss ensures that the generated features are consistent with the original features after being converted back to the original modality, and the cross-modal alignment loss can be used to measure the degree of alignment between the generated features and the target modality features. In this way, GAN can maintain cross-modal alignment, and the generated features should be semantically and structurally consistent with the target modality features, thereby improving the robustness of feature alignment of images and point clouds in extreme scenarios such as camera failure.

[0127] In step 205, the vehicle's future trajectory data can be obtained using a time-series fusion feature, such as a trajectory prediction head. The vehicle's future trajectory data may include, but is not limited to, a sequence of trajectory points and motion parameters (e.g., speed, acceleration, etc.) for a predetermined future duration (e.g., the next 5 seconds). Assuming the predetermined duration is 5 seconds, the vehicle's future trajectory data may include the position coordinates for each second in the next 5 seconds and the vehicle's speed, acceleration, and other information for the next 5 seconds.

[0128] See also Figure 5 After step 205, the method may further include: step 209, using a combined algorithm of Kalman filtering and polynomial fitting to perform smooth interpolation on multiple frames of vehicle future trajectory data to obtain optimized vehicle future trajectory data, thereby eliminating the impact of single-frame detection jitter on vehicle trajectory prediction and improving the accuracy of vehicle trajectory prediction.

[0129] Figure 6 FIG2 shows a schematic diagram of the structure of the vehicle trajectory prediction device provided by the embodiment of the present disclosure. Figure 6 , the vehicle trajectory prediction device 600 of the embodiment of the present disclosure may include:

[0130] A data acquisition unit 601 is configured to acquire multi-view images acquired by a multi-view camera array and point clouds acquired by a laser radar;

[0131] An image feature extraction unit 602 is configured to obtain image features using multi-view images;

[0132] The point cloud feature extraction unit 603 is used to obtain point cloud features using the point cloud;

[0133] The spatiotemporal fusion unit 604 is configured to perform spatiotemporal fusion on the image features and the point cloud features to obtain a temporal fusion feature. The spatiotemporal fusion includes: fusing the point cloud features and the image features through a local-global hybrid attention mechanism to obtain a multimodal fusion feature; performing causal temporal modeling based on the multimodal fusion feature and the vehicle's historical trajectory data using a causal mask and TCN to obtain a temporal fusion feature. Each time step in the temporal fusion feature corresponds to an output feature vector, and the output feature vector of each time step represents the feature extraction result of the fusion feature of the time step and the previous time step.

[0134] The trajectory determination unit 605 is configured to predict the future trajectory data of the vehicle based on the time series fusion feature.

[0135] Furthermore, the spatiotemporal fusion unit 604 can be specifically used to obtain multimodal fusion features in the following manner: locally enhanced image features and locally enhanced point cloud features are obtained based on point cloud features and image features through the local attention module, and the local attention module adopts a sliding window local attention mechanism; globally enhanced image features and globally enhanced point cloud features are obtained based on point cloud features, image features, locally enhanced point cloud features and locally enhanced image features through the global attention module, and the global attention module adopts a global hash sparse attention mechanism based on the LSH function; multimodal fusion features are obtained based on locally enhanced point cloud features, globally enhanced point cloud features, locally enhanced image features, globally enhanced image features, point cloud features and image features by adopting gated dynamic fusion and weighted summation.

[0136] Furthermore, the vehicle trajectory prediction device 600 may also include a motion compensation unit 606 that uses IMU data to predict feature offsets of the vehicle's historical trajectory data before causal time series modeling to compensate for feature misalignment caused by vehicle motion. The data acquisition unit may also be configured to acquire IMU data from an inertial measurement unit (IMU) mounted on the vehicle.

[0137] Furthermore, the motion compensation unit 606 can be specifically used to: use a bidirectional long short-term memory network LSTM to predict a first feature offset based on IMU data; input the first feature offset into a kinematic model to calculate a second feature offset based on the kinematic equation of the vehicle; and compensate the vehicle's historical trajectory data according to the second offset to eliminate feature dislocation in the historical trajectory data caused by vehicle motion.

[0138] Furthermore, the vehicle trajectory prediction device 600 may further include: a dynamic pruning unit 607 for dynamically discarding redundant features according to channel confidence in the causal time series modeling performed by the spatiotemporal fusion unit 604 .

[0139] Furthermore, the dynamic pruning unit 607 can be specifically used to: perform the following processing for each output channel in each layer or specific layer of the TCN: calculate the confidence score of the output channel through the gating network, where the confidence score represents the feature importance of the output channel; and use Gumbel-Softmax to process the confidence score to generate a discrete gating decision value, and dynamically retain or discard the features corresponding to the output channel according to the gating decision value.

[0140] Furthermore, the vehicle trajectory prediction device 600 may further include:

[0141] A consistency evaluation unit 608 is used to evaluate whether the cross-modal consistency of image features and point cloud features meets the requirements;

[0142] The forging unit 609 is used to use a generative adversarial network to generate a pseudo multi-view image based on the point cloud when the cross-modal consistency of the image features and the point cloud features does not meet the requirements, obtain the features of the pseudo multi-view image, and merge the features of the pseudo multi-view image into the image features. The loss function of the generative adversarial network includes: adversarial loss, cycle consistency loss and cross-modal alignment loss.

[0143] Furthermore, the consistency evaluation unit 608 can be specifically used to evaluate whether the cross-modal consistency of image features and point cloud features meets the requirements in one or more of the following ways: 1) calculating the similarity between point cloud features and image features at the same spatial position through BEV spatial contrast learning, and when the similarity is less than a first predetermined similarity threshold, determining that the cross-modal consistency of point cloud features and image features does not meet the requirements; 2) calculating the cross-modal feature similarity between the forward perspective image and the point cloud in the multi-perspective image, and when the cross-modal feature similarity is less than a second predetermined similarity threshold, determining that the cross-modal consistency of the point cloud and the forward perspective image does not meet the requirements.

[0144] Furthermore, the vehicle trajectory prediction device 600 may further include: a smoothing interpolation unit 610, configured to perform smoothing interpolation on multiple frames of vehicle future trajectory data using a combined algorithm of Kalman filtering and polynomial fitting to obtain optimized vehicle future trajectory data.

[0145] In specific applications, the vehicle trajectory prediction device 600 can be implemented by software, hardware, or a combination of both. For example, the vehicle trajectory prediction device 600 can be implemented as the electronic device 700 described below or as software in the electronic device 700 described below.

[0146] In addition, an embodiment of the present disclosure further provides a computer-readable storage medium on which a computer program is stored. The program includes instructions, and when the instructions are executed by one or more processors of a computing device, the steps of the aforementioned vehicle trajectory prediction method are executed.

[0147] Figure 7 Schematic diagram of the structure of the electronic device provided by the embodiment of the present disclosure is shown. Figure 7 The electronic device 700 may include: one or more processors 701, and a memory 702 storing one or more programs, which are executed by the one or more processors 701 to implement the method flow shown in the above embodiments of the present disclosure and / or the program units corresponding to each unit in the device.

[0148] The various components are interconnected using different buses and can be mounted on a common motherboard or in other ways as needed. The processor 701 can process instructions executed within the electronic device, including instructions stored in or on the memory to display graphical information of the user interface on an external input / output device (such as a display device coupled to the interface). In other embodiments, if desired, multiple processors and / or multiple buses can be used with multiple memories and multiple memories.

[0149] The processor 701 may include one or more single-core processors or multi-core processors. The processor 401 may include any combination of general-purpose processors or specialized processors (such as image processors, application processors, baseband processors, etc.).

[0150] The memory 702 is a computer-readable storage medium provided by the present disclosure, which can be used to store non-transient software programs, non-transient computer executable programs and units, such as the embodiment of the present disclosure. Figure 2 The program instructions / units corresponding to the vehicle trajectory prediction method shown. The processor 401 executes the non-transient software programs, instructions and units stored in the memory 702, thereby executing the above method embodiments. Figure 2 The programs, instructions, and units corresponding to the vehicle trajectory prediction method shown.

[0151] The electronic device 700 may further include: an input device 703, an output device 704, a communication component 705, etc. The processor 701, the memory 702, the input device 703, the output device 704 and the communication component 705 may be connected via a bus or other means. Figure 7 The bus connection is taken as an example.

[0152] The programs (also referred to as software, software applications, or code) described above include machine instructions for a programmable processor, and these computer programs may be implemented using a vehicle-specific programming language, assembly, or machine language.

[0153] As time goes by and technology develops, the meaning of medium becomes more and more extensive, and the propagation path of computer programs is no longer limited to tangible media, but can also be downloaded directly from the Internet, etc. Any combination of one or more computer-readable storage media can be used. Computer-readable storage media can adopt but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or components, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium that contains or stores a program that can be used by or in combination with an instruction execution system, device or device.

[0154] In specific applications, the electronic device 700 can be implemented as, but not limited to, a domain controller or other similar devices.

[0155] The embodiment of the present disclosure further provides a vehicle equipped with a multi-perspective image sequence and a laser radar. The vehicle may include the aforementioned vehicle trajectory prediction device 600 and / or electronic device 700.

[0156] The technical solutions provided by the present disclosure are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present disclosure. The descriptions of the above embodiments are intended only to help understand the methods and core concepts of the present disclosure. Furthermore, those skilled in the art will appreciate that variations in the specific implementation methods and scope of application may occur based on the concepts of the present disclosure. In summary, the contents of this specification should not be construed as limiting the present disclosure.

[0157] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A vehicle trajectory prediction method, characterized in that: The method is applied to a vehicle equipped with a multi-view camera array and a laser radar; the method comprises: Acquire multi-view images captured by the multi-view camera array and point clouds captured by the laser radar; Acquiring image features using the multi-view images; obtaining point cloud features using the point cloud; Performing spatiotemporal fusion on the image features and the point cloud features to obtain a temporal fusion feature, wherein the spatiotemporal fusion includes: fusing the point cloud features and the image features through a local-global hybrid attention mechanism to obtain a multimodal fusion feature, and performing causal temporal modeling based on the multimodal fusion feature and the historical trajectory data of the vehicle using a causal mask and a temporal convolutional network (TCN) to obtain a temporal fusion feature, wherein each time step in the temporal fusion feature corresponds to an output feature vector, and the output feature vector of each time step represents a feature extraction result of the fusion feature of the time step and the previous time step; The future trajectory data of the vehicle is predicted based on the time series fusion feature.

2. The method according to claim 1, characterized in that The fusing of the point cloud features and the image features through a local-global hybrid attention mechanism to obtain a multimodal fusion feature includes: Obtaining locally enhanced image features and locally enhanced point cloud features based on the point cloud features and the image features through a local attention module, wherein the local attention module adopts a sliding window local attention mechanism; Obtaining globally enhanced image features and globally enhanced point cloud features based on the point cloud features, the image features, the locally enhanced point cloud features, and the locally enhanced image features through a global attention module, wherein the global attention module adopts a global hash sparse attention mechanism based on a local sensitive hash (LSH) function; The multimodal fusion feature is obtained based on the locally enhanced point cloud feature, the globally enhanced point cloud feature, the locally enhanced image feature, the globally enhanced image feature, the point cloud feature and the image feature by adopting a gated dynamic fusion and weighted summation method.

3. The method according to claim 1, characterized in that The method further comprises: Acquire IMU data from an inertial measurement unit (IMU) mounted on the vehicle; Before the causal time series modeling is performed, the IMU data is used to predict the feature offset of the historical trajectory data of the vehicle to compensate for the feature misalignment caused by vehicle motion.

4. The method according to claim 3, characterized in that The method of using the IMU data to predict the feature offset of the historical trajectory data to compensate for feature misalignment caused by vehicle motion includes: A first feature offset is predicted based on the IMU data using a bidirectional long short-term memory network (LSTM); Inputting the first characteristic offset into a kinematic model to calculate a second characteristic offset based on a kinematic equation of the vehicle; The historical trajectory data of the vehicle is compensated according to the second offset to eliminate feature misalignment in the historical trajectory data caused by vehicle motion.

5. The method according to claim 1, wherein The method further includes dynamically discarding redundant features according to channel confidence in the causal temporal modeling.

6. The method according to claim 5, characterized in that In the causal temporal modeling, redundant features are dynamically discarded according to channel confidence, including: The following processing is performed for each output channel in each layer or a specific layer of the TCN: a confidence score of the output channel is calculated through a gating network, where the confidence score represents the feature importance of the output channel; and the confidence score is processed using Gumbel-Softmax to generate a discrete gating decision value, and the feature corresponding to the output channel is dynamically retained or discarded according to the gating decision value.

7. The method according to claim 1, characterized in that The method further comprises: evaluating whether cross-modal consistency of the image features and the point cloud features meets requirements; When the cross-modal consistency of the image features and the point cloud features does not meet the requirements, a generative adversarial network is used to generate a pseudo multi-view image based on the point cloud, the features of the pseudo multi-view image are obtained, and the features of the pseudo multi-view image are merged into the image features. The loss function of the generative adversarial network includes: adversarial loss, cycle consistency loss and cross-modal alignment loss.

8. The method according to claim 7, characterized in that The evaluating whether the cross-modal consistency of the image features and the point cloud features meets the requirements includes one or more of the following: Calculating the similarity between the point cloud feature and the image feature at the same spatial position through BEV spatial contrast learning, and determining that the cross-modal consistency of the point cloud feature and the image feature does not meet the requirement when the similarity is less than a first predetermined similarity threshold; Calculate the cross-modal feature similarity between the forward view image and the point cloud in the multi-view image, and when the cross-modal feature similarity is less than a second predetermined similarity threshold, determine that the cross-modal consistency of the point cloud and the forward view image does not meet the requirements.

9. A vehicle trajectory prediction device, characterized in that: include: A data acquisition unit, configured to acquire the multi-view images acquired by the multi-view camera array and the point cloud acquired by the laser radar; An image feature extraction unit, configured to obtain image features using the multi-view images; a point cloud feature extraction unit, configured to obtain point cloud features using the point cloud; a spatiotemporal fusion unit, configured to perform spatiotemporal fusion on the image features and the point cloud features to obtain a temporal fusion feature, wherein the spatiotemporal fusion comprises: fusing the point cloud features and the image features through a local-global hybrid attention mechanism to obtain a multimodal fusion feature, and performing causal temporal modeling based on the multimodal fusion feature and the historical trajectory data of the vehicle using a causal mask and a temporal convolutional network (TCN) to obtain a temporal fusion feature, wherein each time step in the temporal fusion feature corresponds to an output feature vector, and the output feature vector of each time step represents a feature extraction result of the fusion feature of the time step and the previous time step; A trajectory determination unit is used to predict and obtain future trajectory data of the vehicle based on the time series fusion feature.

10. A vehicle equipped with a multi-view camera sequence and a laser radar, characterized in that: The vehicle comprises the device of claim 9.

Images