Trajectory prediction method, device and medium for surrounding vehicles in autonomous driving scenario

Abstract

The embodiment of the application provides a kind of trajectory prediction method of surrounding vehicle in automatic driving scene, equipment and medium, belong to automatic driving technical field;Method includes obtaining the multiple continuous time stamp corresponding multi-view image data and sensor point cloud data of target prediction vehicle;Respectively to each multi-view image data is carried out BEV feature extraction, obtains multiple image BEV features;Respectively to each sensor point cloud data is carried out BEV feature extraction, obtains and image BEV feature one-to-one corresponding point cloud BEV feature;Each image BEV feature and corresponding point cloud BEV feature are fused, obtain multiple first fusion features;According to each first fusion feature, spatiotemporal fusion is obtained, and spatiotemporal information is obtained;According to spatiotemporal information, trajectory prediction is carried out to target prediction vehicle, and multiple continuous prediction time stamp corresponding prediction instance is obtained.The precision of the vehicle driving trajectory prediction of the embodiment of the application is higher and the prediction efficiency is higher.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to a trajectory prediction method for surrounding vehicles in an automatic driving scene, a device and a medium. BACKGROUND

[0002] The trajectory prediction for surrounding vehicles in an automatic driving scene is mainly instance prediction based on BEV perception; in the related art, the BEV perception mainly adopts pure visual BEV perception. The core challenge of the pure visual BEV perception lies in how to convert two-dimensional image features to a three-dimensional BEV space. Early methods such as LSS achieve the conversion through depth discretization and depth distribution prediction, laying a foundation for subsequent 3D target detection. LSS can be based on methods such as BEVFormer and BEVSegFormer, which further improve efficiency and performance by using attention mechanisms and camera calibration information. However, the pure visual BEV perception method lacks robustness, resulting in low accuracy of vehicle driving trajectory prediction. SUMMARY

[0003] The main purpose of the embodiments of the present application is to propose a trajectory prediction method for surrounding vehicles in an automatic driving scene, a device and a medium, which can improve the accuracy of vehicle driving trajectory prediction.

[0004] To achieve the above purpose, the first aspect of the embodiments of the present application proposes a trajectory prediction method for surrounding vehicles in an automatic driving scene, which comprises:

[0005] obtaining acquisition data corresponding to multiple continuous timestamps of a target prediction vehicle, each of the acquisition data comprising multi-view image data and sensor point cloud data corresponding to the multi-view image data;

[0006] respectively performing bird's eye view BEV feature extraction on each of the multi-view image data to obtain multiple image BEV features;

[0007] respectively performing BEV feature extraction on each of the sensor point cloud data to obtain point cloud BEV features corresponding one-to-one to the image BEV features;

[0008] fusing each of the image BEV features and the corresponding point cloud BEV features to obtain multiple first fusion features;

[0009] performing spatio-temporal fusion according to each of the first fusion features to obtain spatio-temporal information;

[0010] performing trajectory prediction on the target prediction vehicle according to the spatio-temporal information to obtain prediction instances corresponding to multiple continuous prediction timestamps.

[0011] To achieve the above object, a second aspect of the embodiments of the present application provides an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the trajectory prediction method of the surrounding vehicle in the autonomous driving scene according to any one of the first aspect when executing the computer program.

[0012] To achieve the above object, a third aspect of the embodiments of the present application provides a computer readable storage medium, the storage medium stores a computer program, and the computer program implements the trajectory prediction method of the surrounding vehicle in the autonomous driving scene according to any one of the first aspect when executed by a processor.

[0013] The trajectory prediction method of the surrounding vehicle in the autonomous driving scene, the device and the medium provided by the present application can make the data source for instance prediction contain more feature information, so that the accuracy of instance prediction is higher. Compared with the pure visual BEV method, the embodiments of the present application can make up for the shortcomings of the pure visual method in complex environments such as light change and bad weather, and improve the stability and reliability of BEV perception. Moreover, the sensor information is used for BEV instance prediction, which can realize more accurate scene understanding and target perception. Therefore, the embodiments of the present application can improve the accuracy of vehicle driving trajectory prediction. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a flowchart of the trajectory prediction method of the surrounding vehicle in the autonomous driving scene provided by the present application;

[0015] Figure 2 is a network architecture diagram of the prediction model applied in one embodiment of the trajectory prediction method of the surrounding vehicle in the autonomous driving scene provided by the present application;

[0016] Figure 3 is a flowchart of the first feature fusion in one embodiment of the trajectory prediction method of the surrounding vehicle in the autonomous driving scene provided by the present application;

[0017] Figure 4 is a schematic diagram of the image BEV feature generation process in one embodiment of the trajectory prediction method of the surrounding vehicle in the autonomous driving scene provided by the present application;

[0018] Figure 5 is a structural schematic diagram of the full convolution mask autoencoder in one embodiment of the trajectory prediction method of the surrounding vehicle in the autonomous driving scene provided by the present application;

[0019] Figure 6is a schematic diagram of dynamic fusion in an embodiment of the trajectory prediction method of surrounding vehicles in an autonomous driving scene provided by the present application;

[0020] Figure 7 is a structural schematic diagram of the hardware structure corresponding to the trajectory prediction method of surrounding vehicles in an autonomous driving scene provided by the present application. DETAILED DESCRIPTION

[0021] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0022] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0024] First, several terms involved in the present application are analyzed:

[0025] BEV (Bird's Eye View): bird's eye view, which represents a perspective from directly above the vehicle, providing global road environment information.

[0026] Instance segmentation: each target in the image is segmented at the pixel level to form different target instances.

[0027] Instance prediction: predicting the position and state of each target instance in the future period of time, such as position, speed, direction, etc.

[0028] Global Response Normalization, abbreviated as GRN.

[0029] The trajectory prediction of surrounding vehicles in an autonomous driving scene is mainly based on BEV perception for instance prediction. In the prior art, BEV perception mainly uses pure vision BEV perception. The core challenge of pure vision BEV perception is how to convert two-dimensional image features to a three-dimensional BEV space. Early methods such as LSS achieve conversion through depth discretization and depth distribution prediction, laying the foundation for subsequent 3D target detection. Among them, LSS can be based on Transformer-based methods such as BEVFormer and BEVSegFormer, and further improve efficiency and performance by using attention mechanisms and camera calibration information. However, pure vision BEV perception methods lack robustness, such as strong dependence on the accuracy of camera internal and external parameter calibration. Therefore, the calibration error of the camera internal and external parameters will directly affect the construction of the BEV space, resulting in deviations in the pure vision BEV perception results. In addition, image quality has a great influence on the final prediction results under pure vision BEV. For example, in complex environments such as strong light, shadows, night, and bad weather such as rain, snow, and fog, due to the decline in image quality, feature extraction is difficult, which in turn affects the generation of BEV features and the performance of subsequent perception tasks. In addition, pure vision BEV perception methods also involve inferring 3D scene information from 2D images, but during the process of inferring 3D scene information from 2D images, it is easily affected by factors such as occlusion and texture loss, resulting in inaccurate depth estimation, which in turn affects the construction of the BEV space and the accuracy of target positioning. Similarly, the training data of pure vision BEV perception methods is usually limited to specific scenes and weather conditions, resulting in limited generalization ability of the prediction model based on pure vision BEV perception methods. When encountering new scenes or extreme weather conditions, the performance of the prediction model will decrease significantly. Therefore, the precision of vehicle driving trajectory prediction using pure vision BEV perception is low. Although there are also laser radar-vision fusion BEV perception methods in the prior art to construct more robust BEV representations. However, this method is mainly used for 3D target detection and semantic segmentation tasks, and cannot achieve instance prediction. Compared with the instance prediction method, the prediction accuracy of 3D target detection and semantic segmentation tasks is lower, based on which the embodiment of the present application provides a trajectory prediction method, device and medium for surrounding vehicles in an autonomous driving scene, which can improve the precision of vehicle driving trajectory prediction.

[0030] The trajectory prediction method, device and medium for surrounding vehicles in an autonomous driving scene provided by the embodiments of the present application are specifically described as follows. First, the trajectory prediction method for surrounding vehicles in an autonomous driving scene in the embodiments of the present application is described.

[0031] The trajectory prediction method of surrounding vehicles in an autonomous driving scene of the present application can be used in many vehicle-mounted systems. For example: car-mounted systems, truck-mounted systems, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform particular tasks or implement particular abstract data types. The present application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are connected through a communication network. In a distributed computing environment, program modules can be located in both local and remote computer storage media, including storage devices.

[0032] It can be understood that, referring to Figure 1 The trajectory prediction method of surrounding vehicles in an autonomous driving scene provided by the embodiments of the present application is shown in the method, which comprises:

[0033] Step S100, acquiring acquisition data corresponding to multiple continuous timestamps of a target prediction vehicle, each acquisition data comprising multi-view image data and sensor point cloud data corresponding to the multi-view image data;

[0034] Step S200, performing bird's eye view (BEV) feature extraction on each multi-view image data to obtain multiple image BEV features;

[0035] Step S300, performing BEV feature extraction on each sensor point cloud data to obtain point cloud BEV features corresponding to the image BEV features one by one;

[0036] Step S400, fusing each image BEV feature and the corresponding point cloud BEV feature to obtain multiple first fusion features;

[0037] Step S500, performing spatio-temporal fusion according to each first fusion feature to obtain spatio-temporal information;

[0038] Step S600, performing trajectory prediction on the target prediction vehicle according to the spatio-temporal information to obtain prediction instances corresponding to multiple continuous prediction timestamps.

[0039] Therefore, by performing BEV feature extraction on the multi-view image data and the sensor point cloud data respectively, and fusing the image BEV features and the point cloud BEV features extracted respectively, the application embodiment makes the data source for instance prediction contain more feature information, so that the accuracy of instance prediction is higher. Compared with the pure visual BEV method, the application embodiment can make up for the shortcomings of the pure visual method in complex environments such as light changes and bad weather, and improve the stability and reliability of BEV perception. Moreover, using sensor information for BEV instance prediction can realize more accurate scene understanding and target perception. Therefore, the application embodiment can improve the accuracy of vehicle driving trajectory prediction.

[0040] The application embodiment does not limit the duration of a single timestamp, and a person skilled in the art can selectively set it according to the real-time requirement of autonomous driving. The prediction timestamp is a timestamp after the current time. For example, if the corresponding timestamp of the current time is T1, the collection data corresponding to T1, T1-1 and T1-2 can be used as the collection data corresponding to multiple consecutive timestamps of the target prediction vehicle, where T1-1 is a timestamp before T1, and T1-2 is a timestamp before T1-1. At this time, the prediction timestamp is a timestamp after T1.

[0041] The multi-view image data represents a set of RGB images of different perspectives acquired by multiple cameras at a timestamp. The sensor point cloud data represents a set of point cloud data collected by at least one sensor at a timestamp, wherein the sensor is set as a laser radar. When multiple sensors are provided, the remaining sensors can be one or more of a traditional radar (Radar), an ultrasonic sensor (Ultrasonic Sensors) and an inertial measurement unit (IMU). A person skilled in the art can selectively deploy different types of sensors according to actual conditions. For example, a traditional radar sensor can provide high-precision speed information and stable performance in bad weather conditions, so when a laser radar and a camera are used as the main data source of the first fusion feature, the data collected by the traditional radar sensor can be added as a supplement. For example, an ultrasonic sensor can be used for close-range object detection and can provide additional environmental information when the vehicle is driving at low speed or is parked. At this time, the laser radar sensor and the ultrasonic sensor can be combined to collect point cloud data, so that point cloud data with more feature information can be obtained. For example, an IMU can provide acceleration and rotation information of the vehicle, which helps to improve the motion prediction accuracy in dynamic scenes. Therefore, a person skilled in the art can combine different types of sensor point cloud data according to different stages of autonomous driving, so as to optimize the point cloud BEV feature representation and improve the accuracy and robustness of environment perception and motion prediction.

[0042] This application does not limit the method of obtaining predicted instances based on spatiotemporal information. Those skilled in the art can obtain instances based on existing instance acquisition methods.

[0043] By extracting and fusing BEV features from multi-view image data and sensor point cloud data separately, performance can be improved while controlling computational complexity, making it suitable for application scenarios with high real-time requirements.

[0044] Steps S2200 to S600 in this embodiment are obtained based on a pre-trained prediction model. For example, using LiDAR point cloud data as an example, refer to... Figure 2 As shown, the network architecture of the prediction model consists of a perception module, a prediction module, and a post-processing module. The current timestamp and the data collected at the previous two timestamps are input into the prediction model. The perception module implements steps S200 to S500; the prediction module and the post-processing module implement step S600 to obtain the prediction instance. In some embodiments, the perception module can extract image BEV features from multi-view image data based on the LSS (Lift, Splat, Shoot) method. For example, efficient convolutional neural networks (CNNs) such as ConvNeXtV2Atto, MobileNetV3, ResNet series, or EfficientNet V2 can be used, allowing the backbone network of the perception module to maintain high performance while reducing computational resource consumption, thereby meeting the real-time requirements of autonomous driving systems. In some embodiments, image BEV features can be provided by first extracting perspective features corresponding to the multi-view image data. Then, based on the estimated depth distribution, the perspective features corresponding to the multi-view image data are upscaled to 3D space. By utilizing the intrinsic and extrinsic parameters of the cameras, the 3D features of each camera are transformed into a vehicle-centric coordinate system. The image BEV features and point cloud BEV features are then projected onto a unified BEV plane, achieving the fusion of image BEV features and point cloud BEV features to obtain the first fused features for each time stamp. Finally, the first fused features from different time stamps are fused and stacked to form a global dynamic state independent of the vehicle's own position, i.e., spatiotemporal information. In some embodiments, reference... Figure 2As shown, the prediction module of the prediction model is responsible for predicting future segmentation maps and centripetal reverse optical flow; the post-processing module mainly generates the final future multi-frame instance prediction according to the predicted segmentation map and the centripetal reverse optical flow. In some embodiments, the prediction module contains two parallel branches (segmentation prediction branch network and optical flow prediction branch network, respectively), which are used for segmentation and optical flow prediction, respectively. The two branches share the same network structure, but the parameters are independent. Each branch adopts a multi-scale U-Net architecture, including an encoder, a predictor and a decoder. The encoder is composed of multiple down-sampling blocks, each block containing a 2D convolution layer, a normalization layer and a LeakyReLU activation function, which gradually reduces the spatial resolution of the feature map and increases the number of feature channels through the convolution step length, so as to extract multi-scale features. The predictor is located between the encoder and the decoder and is composed of multiple convolution layers, which map the multi-scale features extracted by the encoder to the future time step, realizing the conversion of the number of channels from C i ×T in to C i ×T out , where T in is the input time step (i.e., the number of time stamps in step S100), and T out is the predicted future time step (i.e., the number of predicted time stamps in step S600). The decoder is symmetrical to the encoder structure and is composed of multiple up-sampling blocks, each block containing a transpose convolution layer, a normalization layer and a LeakyReLU activation function, which gradually increases the spatial resolution of the feature map through up-sampling, and finally the segmentation prediction branch network outputs the predicted segmentation map; the optical flow prediction branch network outputs the optical flow data. Each up-sampling block of the decoder also combines features from the corresponding scale of the encoder to fuse multi-scale information through a skip connection, helping to restore fine details. In some embodiments, the post-processing module first determines the instance center by extracting the local maximum value from the segmentation map. Then, using the optical flow data for pixel-level association, each pixel point is associated to the corresponding instance ID in the previous frame, thereby generating the instance segmentation result. For the first frame, since there is no information of the previous frame, the instance ID is assigned by grouping the pixels to the past instance center. Finally, through the reverse optical flow distortion operation, the instance ID of the previous frame is propagated to the current frame, realizing the cross-time instance association, and obtaining the predicted instances corresponding to multiple consecutive predicted time stamps.

[0045] It can be understood that the bird's eye view BEV feature extraction is performed on each multi-view image data respectively, and multiple image BEV features are obtained, including:

[0046] The object feature and the scene feature are extracted from each multi-view image data respectively, and the image vector features corresponding to the multi-view image data are obtained;

[0047] The image vector features are distributed to a BEV grid according to preset camera internal and external parameters, to obtain view frustum point cloud data corresponding to the multi-view image data.

[0048] The view frustum point cloud data corresponding to each multi-view image data is encoded to obtain image BEV features corresponding to the multi-view image data.

[0049] In some embodiments, the object feature and scene feature extraction can be performed by a convolutional neural network arranged in the camera encoder to extract semantic features, to obtain image vector features corresponding to the multi-view image data.

[0050] Each image vector feature represents a feature vector formed by the combination of object features and scene features in the image data of one view in the corresponding multi-view image data. If the multi-view image data includes n views of image data, the image vector features also have n.

[0051] The view frustum point cloud data is three-dimensional data obtained by converting the image vector features from two dimensions to a BEV perspective.

[0052] For example, as shown in Figure 3 The camera encoder can identify object features and scene features in the images of different views in the multi-view image data, and convert these information into compact feature vectors to obtain image vector features, which can facilitate subsequent view transformation and task processing. The object features refer to the attributes of specific objects in the image, such as the type (sedan, truck, bus), color, shape, size, etc. of the vehicle, as well as the attributes of other road users such as pedestrians, traffic lights, road signs, etc. The scene features refer to the overall environmental information in the image, such as the layout of the road, the distribution of buildings, the density of vegetation, the lighting conditions, etc.

[0053] Since each camera has an independent coordinate system, all 3D points need to be converted to a unified world coordinate system for subsequent fusion; as shown in Figure 4 In some embodiments, the depth of each camera can be estimated separately, and the view frustum can be merged using the internal and external parameter matrices of the camera, and the image vector features can be distributed to a BEV grid, so that the image vector features of each view can be lifted from 2D pixel coordinates to 3D camera coordinates for BEV perspective conversion by a multi-view geometry method; and the image BEV features can be obtained by encoding processing through a pooling operation.

[0054] As shown in Figure 4 The object features and scene features of each multi-view image data are extracted to obtain image vector features corresponding to the multi-view image data, including:

[0055] The object and scene information of each multi-view image data is extracted based on the ConvNeXtV2 Atto to obtain two-dimensional image features corresponding to the multi-view image data.

[0056] The two-dimensional image features are up-sampled, and two-dimensional convolution calculation is performed on the up-sampled two-dimensional image features to obtain image vector features corresponding to the multi-view image data.

[0057] It can be understood that multi-sensor fusion usually needs to process a large amount of data, and the calculation complexity is high, which poses a challenge to application scenarios with high real-time requirements (such as autonomous driving). Especially for instance prediction tasks, more delicate processing is required, and the calculation burden will be further increased. Referring to Figure 4 It is shown that the embodiments of the present application can significantly improve performance on various recognition benchmarks by using ConvNeXtV2 Atto as the backbone network for visual information extraction, while maintaining efficient use of computing resources. Thus, the requirements for hardware can be reduced while meeting the requirements for computational complexity.

[0058] ConvNeXtV2 Atto has a deep convolutional structure and optimized feature extraction capability, and can generate more rich and representative two-dimensional feature maps. ConvNeXtV2 Atto has the characteristics of high efficiency and light weight, which can greatly reduce the model parameter amount and calculation demand while maintaining high performance, and meets the design goal of light weight and high efficiency pursued by the end-to-end autonomous driving system. ConvNeXtV2 Atto also has better adaptability, and can achieve superior performance in multi-task learning, support the collaborative work of segmentation and flow field prediction in the algorithm framework, and simplify the complexity of multi-task learning. ConvNeXtV2 Atto can optimize computing resources, and can ensure that the model can realize real-time instance and motion prediction with lower delay and calculation cost in resource-constrained application scenarios.

[0059] Among them, ConvNeXtV2 Atto also combines the full convolutional mask autoencoder (FCMAE) pre-training framework and the global response normalization (GRN) layer, so that ConvNeXtV2 Atto can extract more discriminative and robust features, improve the accuracy and stability of instance prediction, and achieve the above effects with only about 3.7M parameters.

[0060] Among them, the full convolutional mask autoencoder (FCMAE) is used for pre-training, and the core idea is to randomly mask part of the input image (masking rate is 60%), and train the model to reconstruct the masked part. Referring to Figure 5As shown, FCMAE consists of a hierarchical encoder and a lightweight decoder, and the overall architecture is asymmetric. Among them, the hierarchical encoder in FCMAE only processes visible pixels. The input of ConvNeXtV2 Atto can be an image with random masking, and the hierarchical encoder of FCMAE can effectively extract features by using sparse convolution, ignoring the masked part in some embodiments. The lightweight decoder in FCMAE receives the output of the hierarchical encoder and the position information (used to indicate the position of the masked block), and tries to reconstruct the original image. And the lightweight decoder in FCMAE is lighter than the hierarchical encoder in FCMAE, usually only contains a ConvNeXt block, and a 1x1 convolution layer is set after the ConvNeXt block, so as to adjust the output channel number to be consistent with the channel number of the input image, and in some embodiments, FCMAE uses mean square error (MSE) as a loss function to measure the difference between the reconstructed image and the target image; The loss function of FCMAE only calculates on the masked block, because the information of the visible block can be directly obtained from the input. At this time, let x be the original image, x masked be the masked image, f(x masked ) be the output of the model (reconstructed image), and M be the masking mask, then the loss function can be represented as shown in the following formula (1):

[0061]

[0062] Therefore, since the hierarchical encoder in FCMAE only processes visible pixels, the amount of calculation is small; while the lightweight decoder in FCMAE needs to reconstruct the entire image, but the structure is simpler, and such an asymmetric design of encoder and decoder can balance the calculation efficiency and reconstruction quality. In some embodiments, in order to avoid the shortcut of the model based on ConvNeXtV2 Atto to learn to copy and paste information from the visible area, the encoder part in FCMAE can use sparse convolution. By using the feature that sparse convolution only performs convolution operation on non-masked pixels, the calculation efficiency is improved. In practical application, a special sparse convolution library can be used or by setting the value of the masked pixel to zero and using standard convolution. After pre-training is completed, the sparse convolution layer is converted into a standard dense convolution layer, so as to fine-tune on downstream tasks.

[0063] GRN is a new global response normalization method that makes FCMAE pre-training more effective when combined with the ConvNeXt architecture. During self-supervised learning using convolutional neural networks, especially when pre-training based on Masked Autoencoders (MAE), the phenomenon of "feature collapse" often occurs, where different feature channels become highly redundant and the amount of information decreases, limiting the model's expressive power. Therefore, after feature extraction based on FCMAE, normalization processing using GRN can enhance inter-channel feature competition and prevent feature collapse, especially in convolutional neural networks using masked autoencoders. For example, for a given input feature, the processing process of GRN includes three steps: 1) global feature aggregation; 2) feature normalization; 3) feature calibration.

[0064] In step 1), global feature aggregation can extract global context information from the input feature The result of global feature aggregation is a global response vector representing the overall feature performance of each channel. Such global aggregation allows capturing global information while preserving local details, which is particularly important for understanding the overall layout of complex scenes and objects. In this way, better feature competition and selection can be achieved, promoting comprehensive information expression. In addition, global feature aggregation provides an important foundation for subsequent steps, enabling feature normalization and calibration to be performed based on a full understanding of global information.

[0065] In step 2), feature normalization is used to ensure that all channel features are compared on a unified scale. By standardizing the global response vector, scale differences between different channels can be eliminated, allowing the relative importance of each feature to be evaluated on the same basis. The normalization process usually involves calculating the mean and standard deviation of each channel, then adjusting the feature values to conform to the standard normal distribution. This standardization not only enhances inter-channel competition, but also reduces redundant information, allowing each feature channel to provide unique and valuable information. Through normalization, the learning process of the prediction model is more stable and has better robustness when dealing with different data distributions.

[0066] In step 3), the feature calibration can be optimized for each channel based on the normalized features. Just like the prediction model can redistribute feature weights according to the needs of specific tasks, highlighting important features while suppressing irrelevant or redundant information. The feature calibration process may involve linear transformation or nonlinear activation functions to enhance the expressive power of features, ensuring that each channel effectively contributes its information in the final output, thereby improving the overall performance of the model. Therefore, by adding feature calibration, the prediction model can better adapt to different input data and task requirements, enhancing its generalization ability. In addition, feature calibration can further reduce the risk of feature collapse, ensuring that the prediction model maintains efficient performance in diverse scenarios.

[0067] Therefore, the image vector features obtained by ConvNeXtV2 Atto are lifted to a three-dimensional space and projected onto the bird's eye view (BEV) plane, providing a basic feature grid for subsequent instance prediction and motion prediction. Compared with EfficientNet, ConvNeXtV2 Atto has more flexibility and adaptability in feature expression, better capturing details and contextual information in complex scenes.

[0068] It can be understood that the BEV features of each sensor point cloud data are extracted respectively to obtain point cloud BEV features corresponding to the image BEV features, including:

[0069] The point cloud data of each sensor is voxelized respectively to obtain a plurality of voxel interval data corresponding to the sensor point cloud data;

[0070] Sparse convolution calculation is performed on the plurality of voxel interval data corresponding to each sensor point cloud data to obtain point cloud BEV features corresponding to the image BEV features.

[0071] It can be understood that the image BEV features and the corresponding point cloud BEV features are fused to obtain a plurality of first fusion features, including:

[0072] The image BEV features and the corresponding point cloud BEV features are processed by static fusion to obtain second fusion features corresponding to the multi-view image data;

[0073] Each second fusion feature is processed by dynamic fusion mechanism to obtain first fusion features corresponding to the multi-view image data.

[0074] The point cloud data has the characteristics of sparsity and high dimension, and direct processing of the original sensor point cloud data is often computationally intensive and inefficient. By voxelization and sparse 3D convolution, sensor point cloud data can be processed more efficiently. Among them, point cloud voxelization can convert the original sensor point cloud data into voxel (small cube) form, thereby reducing the complexity of the data in the vertical direction. After converting the point cloud data into voxel form, the sparse convolution algorithm can use a sparse 3D convolution network to efficiently extract features from these voxelized data.

[0075] Among them, point cloud voxelization is the process of scattering point cloud data in three-dimensional space into a uniform grid. The key to voxelization is to convert continuous three-dimensional coordinates into discrete voxel grids. Each voxel is a small three-dimensional cube that contains all point information at that location. Through voxelization, complex point cloud data can be converted into data with regular grid structure, facilitating subsequent convolution operations. For example, if the position of each point in the point cloud data is represented by three-dimensional coordinates (x, y, z), and the edge length of the voxel is Δx, Δy, Δz, then the coordinate interval of each voxel in space can be represented as shown in equation (2):

[0076]

[0077] Among them, for each point p i = (x i , y i , z i ), its falling voxel coordinates (v x , v y , v z ) can be calculated, which is determined by dividing (x i , y i , z i ) by the integer part of the voxel edge length: the original continuous point cloud data {p1, p2, …, p N]mapped to a discrete voxel grid {(v1), (v2),...}, each voxel can contain multiple points, and the value of each voxel can be aggregated in some way, such as using the average position or maximum value of the points within the voxel to represent the characteristics of the voxel. At this time, the voxelized data can be represented by a sparse three-dimensional tensor, in which only the voxels containing point cloud data store feature information. In order to facilitate subsequent calculations, the voxels are usually flattened and mapped to a two-dimensional plane (such as the BEV plane), which simplifies subsequent processing. After voxelization, the data is not only compressed and converted into a regular grid form, but also effectively reduces the computational complexity. In a 3D convolutional network, voxelized point cloud data can be used as input for feature extraction and target detection tasks. For example, voxelized point cloud data can be used for further sparse convolution operations, effectively improving the network's ability to process sparse point clouds.

[0078] wherein the sparse convolution can be used for convolution operation designed for sparse data, which can avoid calculating the spatial positions without active values (i.e., zero values), thus improving the computational efficiency. The core idea of sparse convolution is to perform convolution operation in the local area with non-zero data only. Compared with traditional convolution operation (suitable for dense data), sparse convolution operation is more efficient in memory and calculation. For example, the input is a three-dimensional sparse tensor X with size (L, H, W), where L is the depth, and H and W are the height and width, respectively. Traditional convolution operation needs to perform sliding operation of convolution kernel K in the whole space, and calculates each position as follows:

[0079]

[0080] wherein K L ,K H ,K W are the sizes of the convolution kernel in the depth, height and width dimensions, respectively. However, in sparse convolution, only the non-zero voxels (valid data) participate in the convolution operation. Assuming that the set of valid points in the input tensor is {(l1, h1, w1), (l2, h2, w2),...}, the convolution calculation corresponding to each valid point can be represented as follows:

[0081] Y(l, h, w) = ∑ （l',h',w'）∈S（l,h,w) X(l', h', w') · K(l'-l, h'-h, w'-w) formula (4);

[0082] wherein S(l, h, w) represents the set of valid points around voxel (l, h, w), and K(l'-l, h'-h, w'-w) is the corresponding weight of the convolution kernel. By calculating only the positions of non-zero voxels, sparse convolution effectively reduces the amount of calculation.

[0083] Therefore, sparse convolution reduces most of the zero-value operations by introducing sparse matrix representation, and computation and storage are only concentrated on non-zero voxels. Therefore, sparse convolution can significantly improve computational efficiency and save memory when processing such sparse data as sensors.

[0084] It can be understood that, as shown in Figure 6 , each second fusion feature is dynamically fused by an attention mechanism to obtain a first fusion feature corresponding to the multi-view image data, including:

[0085] Each second fusion feature is subjected to a global average pooling operation to obtain a global feature corresponding to the second fusion feature;

[0086] The global feature is subjected to channel feature learning by two fully connected layers to obtain a plurality of feature channel weights corresponding to the second fusion feature;

[0087] The plurality of feature channel weights and the corresponding second fusion feature are subjected to weighted calculation to obtain a first fusion feature corresponding to the multi-view image data.

[0088] The feature map of a sensor such as a laser radar is a sparse three-dimensional point cloud, and the feature map of an image is a dense two-dimensional pixel array. There are significant differences in features and dimensions between the two kinds of data, and direct fusion may cause difficulty in feature alignment and loss of information. By performing global average pooling on the fused features based on the SE (Squeeze-and-Excitation) attention mechanism, and then applying a nonlinear transformation and a sigmoid function, the importance of each feature channel is dynamically adjusted.

[0089] Wherein SE dynamically adjusts the feature channel weight, intelligently enhances the key features and suppresses the irrelevant features. This can effectively improve the expression ability of the fused features and ensure the integrity and relevance of the information. Ultimately, this method enhances the performance and robustness of the model, making the processing of complex data more efficient and accurate.

[0090] For example, as shown in Figure 6 , SE (Squeeze-and-Excitation) adjusts the weight of the feature channel by the following steps to enhance the feature representation ability:

[0091] First, the global average pooling is used to convert the input feature map into a global feature vector at the channel level, which realizes the squeezing operation. Specifically, the global feature vector z is realized by calculating the average value on each channel, which is specifically represented as follows:

[0092]

[0093] where c denotes the channel index.

[0094] Next, the correlation between channels is learned through a two-layer fully connected network, generating a weight for each channel and implementing the excitation operation. Specifically, the dimension is reduced by a scaling factor r, and then the dimension is restored, where the feature channel weight is represented as follows:

[0095] s = σ(W2δ(W1z));

[0096] where where W1 and W2 are trainable parameters in SE, W1 is the weight matrix for dimension reduction to map the channel descriptors to a smaller space. W2 is the weight matrix for dimension restoration to restore the transformed channel descriptors to the same dimension as the original input channels. δ is the ReLU activation function, and σ is the Sigmoid activation function. Finally, the learned weight s is applied to each channel of the input feature map through feature reweighting:

[0097]

[0098] In this way, the reweighted feature map can highlight key features and suppress irrelevant features, thereby improving the performance and robustness of the model. Through these steps, the SE module effectively enhances the expressive ability of the features.

[0099] It can be understood that the image BEV features and the corresponding point cloud BEV features are processed by static fusion to obtain second fusion features corresponding to the multi-view image data, including:

[0100] The image BEV features and the corresponding point cloud BEV features are mapped to the corresponding preset BEV space to obtain BEV space data.

[0101] The BEV space data is convolved by a preset 3x3 convolution to obtain second fusion features corresponding to the multi-view image data.

[0102] The design of the static fusion module aims to preliminarily integrate the features extracted by the camera and the laser radar, to reduce the differences in feature distribution and dimension between the point cloud data of the two sensors, and to ensure the effectiveness of subsequent processing.

[0103] For example, referring to Figure 3As shown, taking the sensor point cloud data as an example, the laser radar point cloud data is set, firstly, the features of the camera and the laser radar point cloud data are mapped to a unified BEV space, and a 3x3 convolution layer is adopted for convolution calculation to realize BEV feature splicing. The convolution layer can not only extract local spatial features, but also realize the deep integration of the two features in the channel dimension through cross-channel calculation operation, thereby enhancing the correlation of the multi-modal features. In addition, the weight sharing mechanism is combined in the fusion process to improve the calculation efficiency, and the ReLU activation function is introduced to further enrich the feature expression. Such a static fusion method not only simplifies the model complexity, but also effectively balances the differences between the dense image features and the sparse point cloud features, providing better input features for the subsequent dynamic fusion module.

[0104] It can be understood that the target prediction vehicle is trajectory predicted according to the space-time information, and a plurality of prediction instances corresponding to a plurality of continuous prediction timestamps are obtained, including:

[0105] The space-time information is input into a preset first parallel network branch and a second parallel network branch, respectively; wherein the first parallel network branch and the second parallel network branch have the same network structure;

[0106] The first parallel network branch is used for feature classification of the space-time information, and at least one segmentation map is obtained;

[0107] The second parallel network branch is used for optical flow information extraction of the space-time information, and optical flow data is obtained;

[0108] The instance center is determined by the segmentation map;

[0109] According to the instance center and the optical flow data, a plurality of prediction instances corresponding to a plurality of continuous prediction timestamps are obtained.

[0110] In some embodiments, both feature classification and optical flow extraction are based on a Transformer architecture; processing sequence data using a Transformer architecture can improve the modeling ability of time series information and enhance the accuracy of motion prediction. And the Transformer has an advantage in capturing long-distance dependency and global feature integration, which can further improve the performance of instance prediction.

[0111] For the Transformer architecture, the encoder is the core part of the Transformer architecture, which is responsible for the layer-by-layer down-sampling of the input tensor F in Firstly, the encoder performs spatial down-sampling through multiple convolution layers, thereby extracting features of different scales. For example, referring to Figure 1As shown, the first parallel network branch is a segmentation prediction branch network, and the second parallel network branch is an optical flow prediction branch network. The encoders in the segmentation prediction branch network and the optical flow prediction branch network generate a series of multi-scale BEV features where E ∈ {1, …, 5} represents different scale layers. At each scale layer, the encoders in the segmentation prediction branch network and the optical flow prediction branch network extract spatial features through convolution operations and combine temporal information with spatial information to gradually obtain higher-level semantic information. Through this multi-scale processing, the encoders in the segmentation prediction branch network and the optical flow prediction branch network can capture changes in the environment and dynamic information at different spatial resolutions.

[0112] For the Transformer architecture, refer to Figure 2 As shown, the multi-scale features f enc extracted by the encoder are mapped to a higher-level feature space to generate predictions of future segmentation maps and centripetal backward flow fields, respectively, through corresponding predictors. Specifically, the input feature tensor F in is mapped to a new feature space F out to generate the final prediction results where T out is the output time step, and F out represents the prediction results of the future. These outputs include future segmentation maps and centripetal backward flow field information to describe the future positions and motion states of objects in the scene. The predictors are responsible for performing time series prediction and generating segmentation maps and flow fields of the future scene based on the spatio-temporal features extracted by the encoder.

[0113] For the Transformer architecture, refer to Figure 2 As shown, the decoder of the Transformer architecture is another key part of the codec architecture, which functions to upsample the features processed by the encoder to reconstruct the future BEV features at the original scale. The decoder has a similar structure to the encoder but in the opposite direction, which gradually recovers the low-resolution feature maps generated by the encoder to the original spatial resolution to generate the final output feature map i.e. the future BEV feature map. In this process, the decoder passes the feature information in the encoder to the corresponding layer of the decoder through skip connections, ensuring the recovery of spatial details. Each decoder layer uses deconvolution or upsampling operations to gradually restore the resolution of the feature map, thereby generating a high-resolution output of the future scene. At this time, the segmentation prediction result and the reverse motion flow prediction result are obtained respectively, the segmentation prediction result is the position where the traffic participants (vehicles) around the autonomous vehicle may be in the future, and the reverse motion flow prediction result is the possible motion mode and motion trajectory of the traffic participants, which is taken as the input of the post-processing module. After fusion by the post-processing module, the motion trajectory and position prediction of the traffic participants in the future are obtained.

[0114] Therefore, based on the encoding-decoding architecture (i.e. the Transformer architecture), the low-level spatial features and the high-level semantic information can be effectively combined on the basis of inputting a large number of good BEV spatio-temporal features, the details are retained and the global spatio-temporal changes can be captured, and the position trajectory and future motion trend direction of the traffic participants around the autonomous vehicle can be well predicted. At the same time, the encoder-decoder architecture of the prediction module designed by us is a pure CNN architecture, which has a faster inference speed and can better adapt to the task requirements of the autonomous vehicle for real-time perception prediction and decision-making. The good prediction effect is due to the radar data added in the perception feature fusion module and the good feature extraction capability of the designed feature fusion module.

[0115] Reference Figure 2As shown, after obtaining the segmentation map and optical flow data output by the prediction module, the segmentation map and optical flow data are taken as inputs of the post-processing module to realize correct association in time series through prediction of future instances. Specifically, the traditional method projects the instance center to the next frame by using forward optical flow, and matches the instance centers by using the Hungarian matching algorithm. These methods usually associate in units of instances, only use the displacement vector of the object center position, and ignore the rotation information of the object. In addition, this method is susceptible to errors, especially in long-time prediction, which can cause multiple instances to overlap and cause problems of incorrect ID assignment. The embodiments of the present application realize pixel-level association by using backward centripetal flow, effectively overcoming the shortcomings of traditional algorithms. For example, each foreground grid cell propagates the corresponding instance ID from the previous frame to the current frame by using backward optical flow. This way makes the ID assignment at the pixel level more fine-grained and more robust than traditional methods. Even if there is a local error in optical flow prediction, since the grid cells near the instance center usually share the same ID, the algorithm can effectively alleviate the impact of errors. In addition, with the help of the distortion characteristics of the backward flow, multiple future positions can be associated with the same pixel of the previous frame, enhancing the performance of the model in the multi-modal future prediction task. Among them, for the first frame (t = 0), since there is no instance information of the previous frame (t = -1), our algorithm assigns instance IDs by grouping pixels to the past instance centers. Unlike algorithms that rely on additional centerness head, the present application can directly extract instance centers from semantic segmentation results, simplifying the generation process of the first frame instance segmentation. This strategy not only ensures the accuracy of the initial segmentation, but also lays a solid foundation for subsequent pixel-level association. Therefore, by fusing the segmentation result (spatial result) obtained in the prediction module and the backward motion flow result (temporal result) through the post-processing module, the future action trajectory of the traffic participants perceived by the autonomous vehicle is finally formed. Whether it is the prediction module or the post-processing module, whether the prediction is accurate depends on whether the spatio-temporal feature information extracted by the perception module is comprehensive and accurate. Therefore, the sensor point cloud data fused in the perception module and the backbone network based on ConvNeXtV2Atto can extract better and more accurate spatio-temporal feature information, thereby having better prediction effect.

[0116] The embodiment of the present application can realize the behavior prediction of surrounding traffic participants (vehicles or pedestrians, etc.) of an autonomous vehicle in the field of autonomous driving. Through correct prediction of the behavior of surrounding vehicles, the autonomous vehicle decision system can make correct driving behavior decisions. Compared with the existing instance prediction method based on multi-camera data, the present application fuses lidar data, so that the original input data is more abundant, thereby providing more valuable features for extraction, and the behavior of the surrounding traffic participants of the autonomous vehicle can be more reasonably predicted, so that the autonomous vehicle can make safer and more reasonable decisions. Moreover, the embodiment of the present application can combine a multi-sensor fusion pre-processing module and a PowerBEV instance prediction post-processing module to construct a new BEV instance prediction algorithm based on lidar and multi-camera fusion. The three-dimensional reconstruction of the BEV perspective is realized to complete downstream tasks such as target detection and panoramic segmentation. Compared with the pure visual scheme, the embodiment of the present application has better robustness and can stably and reliably obtain the information of the surrounding environment under complex lighting environments and high-speed vehicle running conditions, and predict the future running state in the BEV space. The future instance prediction algorithm represented by PowerBEV only relies on visual information for BEV three-dimensional reconstruction, and uses the feature information extracted by vision to predict the future motion state of each instance in the BEV space. The instance prediction has higher robustness and efficiency. Moreover, by fusing lidar data in the data embedding module, the expression ability of the BEV feature is enhanced, especially the robustness of scene reconstruction; the spatial features of the surrounding traffic participants of the autonomous vehicle are fully extracted, which is conducive to better positioning and future behavior prediction of these traffic participants. At the same time, it effectively solves the limitations of the pure visual method in complex scenes such as light changes and occlusions. Moreover, the backbone network of PowerBEV is improved in the embodiment of the present application, for example, ConvNeXtV2 Atto is used, which significantly reduces the complexity of the backbone network while significantly improving the visual feature extraction capability, speeds up the model operation speed, and realizes more efficient processing. The ConvNeXtV2 Atto backbone feature extraction network has better feature extraction capability, can better extract the features of the input data, and supply the subsequent prediction module to better predict the future instance segmentation result of the traffic participant, and then better predict the future moving track of the traffic participant. Therefore, the vehicle driving track prediction of the embodiment of the present application has higher precision and higher prediction efficiency.

[0117] The embodiment of the present application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the above-mentioned trajectory prediction method of surrounding vehicles in an autonomous driving scene when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0118] Please refer to Figure 7 , Figure 7 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device comprises:

[0119] The processor 701 can be implemented in a manner of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the embodiments of the present application;

[0120] The memory 702 can be a NAND flash, and related program codes are saved in the memory 702 and called by the processor 701 to execute the trajectory prediction method of surrounding vehicles in an automatic driving scene according to the embodiments of the present application;

[0121] The input / output interface 703 is used to realize information input and output;

[0122] The communication interface 704 is used to realize the communication interaction between the device and other devices, and the communication can be realized in a wired manner (for example, USB, network cable, etc.) or in a wireless manner (for example, mobile network, WIFI, Bluetooth, etc.);

[0123] The bus 705 is used to transmit information between various components (for example, the processor 701, the memory 702, the input / output interface 703, and the communication interface 704) of the device;

[0124] The processor 701, the memory 702, the input / output interface 703, and the communication interface 704 are connected to each other in the device through the bus 705.

[0125] The embodiments of the present application also provide a computer readable storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the trajectory prediction method of surrounding vehicles in an automatic driving scene.

[0126] The memory is a non-transient computer readable storage medium, which can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transient memory, for example, at least one magnetic disk storage device, a flash memory device, or other non-transient solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0127] The embodiments described in the present application are for more clearly illustrating the technical solutions of the present application, and do not constitute a limitation on the technical solutions provided by the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the present application are also applicable to similar technical problems.

[0128] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the present application, and can include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0129] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment.

[0130] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the function modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0131] The terms "first", "second", "third", "fourth" and the like (if any) in the specification of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0132] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the front and rear associated objects. "At least one of the following" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b and c can be single or multiple.

[0133] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed mutual ones can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0134] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0135] In addition, each functional unit in each embodiment of the application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0136] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0137] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method for predicting the trajectories of surrounding vehicles in an autonomous driving scenario, characterized in that, The method includes: Acquire collection data corresponding to multiple consecutive timestamps of the target predicted vehicle, each of the collection data includes multi-view image data and sensor point cloud data corresponding to the multi-view image data; Bird’s-eye view BEV features are extracted from each of the multi-view image data to obtain multiple image BEV features; BEV features are extracted from the point cloud data of each sensor to obtain point cloud BEV features that correspond one-to-one with the BEV features of the image. The BEV features of each image and the corresponding BEV features of the point cloud are fused to obtain multiple first fused features; Spatiotemporal fusion is performed based on each of the first fusion features to obtain spatiotemporal information; Based on the spatiotemporal information, the trajectory of the target vehicle is predicted to be obtained, resulting in multiple prediction instances corresponding to consecutive prediction timestamps. Specifically, the step of extracting bird's-eye view BEV features from each of the multi-view image data yields multiple image BEV features, including: Object features and scene features are extracted from each of the multi-view image data to obtain image vector features corresponding to the multi-view image data; The image vector features are assigned to a BEV grid with preset camera intrinsic and extrinsic parameters to obtain view frustum cloud data corresponding to the multi-view image data. Encode the view frustum cloud data corresponding to each of the multi-view image data to obtain image BEV features that correspond one-to-one with the multi-view image data. The step of predicting the trajectory of the target vehicle based on the spatiotemporal information to obtain multiple prediction instances corresponding to consecutive prediction timestamps includes: The spatiotemporal information is input into a preset first parallel network branch and a second parallel network branch, respectively; wherein the first parallel network branch and the second parallel network branch have the same network structure; The first parallel network branch is used to perform feature classification on the spatiotemporal information to obtain at least one segmentation map; The second parallel network branch is used to extract optical flow information from the spatiotemporal information to obtain optical flow data. The instance center is determined using the segmentation map; Based on the instance center and the optical flow data, multiple prediction instances corresponding to consecutive prediction timestamps are obtained.

2. The method for predicting the trajectory of surrounding vehicles in an autonomous driving scenario according to claim 1, characterized in that, The step of extracting object features and scene features from each of the multi-view image data to obtain image vector features corresponding to the multi-view image data includes: Based on ConvNeXtV2 Atto, object and scene information are extracted from each of the multi-view image data to obtain two-dimensional image features corresponding to the multi-view image data; The two-dimensional image features are upsampled, and two-dimensional convolution is performed on the upsampled two-dimensional image features to obtain image vector features corresponding to the multi-view image data.

3. The method for predicting the trajectories of surrounding vehicles in an autonomous driving scenario according to claim 1, characterized in that, The step of extracting BEV features from the point cloud data of each of the aforementioned sensors to obtain point cloud BEV features that correspond one-to-one with the image BEV features includes: The point cloud data of each sensor is voxelized to obtain multiple voxel interval data corresponding to the point cloud data of the sensor. Sparse convolution calculations are performed on multiple voxel intervals corresponding to each sensor point cloud data to obtain point cloud BEV features corresponding to the image BEV features one-to-one.

4. The method for predicting the trajectory of surrounding vehicles in an autonomous driving scenario according to claim 1, characterized in that, The step involves fusing the BEV features of each image with the corresponding BEV features of the point cloud to obtain multiple first fused features, including: The image BEV features and the corresponding point cloud BEV features are statically fused to obtain a second fused feature that corresponds one-to-one with the multi-view image data. Each of the second fusion features is dynamically fused using an attention mechanism to obtain a first fusion feature that corresponds one-to-one with the multi-view image data.

5. The trajectory prediction method for surrounding vehicles in an autonomous driving scenario according to claim 4, characterized in that, The step of dynamically fusing each of the second fusion features through an attention mechanism to obtain a first fusion feature corresponding one-to-one with the multi-view image data includes: Perform global average pooling on each of the second fusion features to obtain the global features corresponding to the second fusion features; The global features are respectively passed through two fully connected layers to learn channel features, thereby obtaining multiple feature channel weights corresponding to the second fused feature; The weights of the multiple feature channels and the corresponding second fusion features are weighted and calculated to obtain the first fusion features that correspond one-to-one with the multi-view image data.

6. The method for predicting the trajectory of surrounding vehicles in an autonomous driving scenario according to claim 4, characterized in that, The step of statically fusing the BEV features of each image and the corresponding BEV features of the point cloud to obtain a second fused feature that corresponds one-to-one with the multi-view image data includes: The BEV features of each image and the corresponding BEV features of the point cloud are mapped to the corresponding preset BEV space to obtain BEV space data. The BEV spatial data is convolved through a preset 3x3 convolutional layer to obtain a second fusion feature that corresponds one-to-one with the multi-view image data.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the trajectory prediction method for surrounding vehicles in an autonomous driving scenario as described in any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the trajectory prediction method for surrounding vehicles in the autonomous driving scenario as described in any one of claims 1 to 6.

Images