Sparse BEV feature-based driving track prediction method, apparatus and device, and medium

Abstract

The invention discloses a driving track prediction method and device based on sparse BEV features, equipment and a medium, and relates to the technical field of vehicle simulation, and the method comprises the steps: obtaining driving scene information, carrying out the mapping according to the driving scene information, obtaining sparse vehicle features, carrying out the interaction prediction according to the driving scene information, and obtaining motion prediction features, according to the action prediction features and the sparse automobile features, enhanced automobile features are obtained, and according to the enhanced automobile features, a self-automobile prediction trajectory is obtained. Through obtaining the scene information, generating the sparse features through mapping and obtaining the predicted action features, the two features are fused to obtain the enhanced automobile features, the self-vehicle trajectory prediction is realized according to the enhanced automobile features, the driving trajectory prediction efficiency and accuracy are improved, and the collision risk is reduced.

Description

Technical Field

[0001] The present invention relates to the field of vehicle simulation technology, and in particular to a driving trajectory prediction method, device, equipment and medium for sparse BEV features. Background Art

[0002] With the development of intelligent transportation systems and autonomous driving technology, environmental perception has become one of the key technologies for achieving safe driving. Autonomous vehicles need to accurately understand their surroundings, including other vehicles, pedestrians, road signs and signals, in order to make correct driving decisions. The development of deep learning technology has provided powerful tools for environmental perception, especially methods based on deep neural networks, which have made significant progress in image recognition, object detection, and trajectory prediction.

[0003] The current vehicle trajectory prediction mainly makes trajectory decisions by fusing RGB image features and point cloud data using large visual language models and natural language models.

[0004] Although the existing vehicle trajectory prediction technology has made some progress, it still has obvious shortcomings. The performance of the machine vision system depends largely on the quality of the input image. In bad weather conditions or insufficient lighting, the image quality may degrade, thus affecting the accuracy of the feature extraction network. In addition, the use of large visual language models and natural language models may introduce black box problems, that is, the decision-making process of the model is not transparent enough and difficult to explain and understand. Therefore, there is an urgent need for an intelligent driving method that can efficiently process environmental information and make fast and accurate decisions to predict driving trajectories. Summary of the invention

[0005] The main purpose of this application is to provide a driving trajectory prediction method, device, equipment and medium for sparse BEV features, aiming to solve the technical problem of how to improve the efficiency and accuracy of driving trajectory prediction.

[0006] To achieve the above objectives, the present application proposes a driving trajectory prediction method based on sparse BEV features, the method comprising:

[0007] Obtain driving scene information;

[0008] Obtaining a perception detection result and a map perception result according to the driving scene information;

[0009] Generate sparse vehicle features by sparse mapping the perception detection result and the map perception result;

[0010] Perform interactive prediction based on the driving scene information to obtain action prediction features;

[0011] Obtaining enhanced vehicle features according to the action prediction features and the sparse vehicle features;

[0012] According to the enhanced vehicle features, a predicted trajectory of the vehicle is obtained.

[0013] In one embodiment, the step of obtaining enhanced vehicle features according to the action prediction features and the sparse vehicle features comprises:

[0014] Decoding the action prediction features by inputting them into a multi-layer perception model to obtain specific data of the predicted action;

[0015] Obtaining automobile feature data according to the sparse automobile features;

[0016] Obtaining a first weight and a second weight, wherein the first weight corresponds to a ratio of specific data of the predicted action, and the second weight corresponds to a ratio of the vehicle feature data;

[0017] Enhanced vehicle features are obtained based on the first weight, the second weight, the specific data of the predicted action and the vehicle feature data.

[0018] In one embodiment, the step of obtaining enhanced vehicle features according to the first weight, the second weight, the specific data of the predicted action and the sparse vehicle features includes:

[0019] Based on the first weight and the second weight, the specific data of the predicted action and the sparse vehicle feature are fused to obtain a fused feature;

[0020] Calculation is performed based on the fused features to obtain enhanced vehicle features.

[0021] In one embodiment, the step of mapping the driving scene information through sparse query to obtain sparse vehicle features includes:

[0022] Preprocessing the driving scene information to obtain processed information;

[0023] Inputting the processed information into a preset deep learning model to obtain an object detection result;

[0024] Performing semantic segmentation on the processed information to obtain a semantic result;

[0025] Identify and locate the object according to the object detection result to obtain a perception detection result;

[0026] Identification and classification are performed according to the semantic results to obtain a map perception result.

[0027] In one embodiment, the perception detection result is 3D detection result data, and the 3D detection result data includes center position, size, heading angle and speed in three-dimensional direction. The map perception result is multiple point coordinates of the map, and a scene vector map is composed according to the multiple point coordinates.

[0028] In one embodiment, the step of performing interactive prediction according to the driving scene information to obtain action prediction features includes:

[0029] Get interactive features;

[0030] Performing normalization processing according to the driving scene information to obtain undecoded features;

[0031] The action prediction feature is obtained according to the undecoded feature and the interaction feature through the functional relationship between the action prediction feature, the undecoded feature and the interaction feature.

[0032] In one embodiment, the step of obtaining the predicted trajectory of the vehicle according to the enhanced vehicle features includes:

[0033] Constructing a multi-layer perception model, the multi-layer perception model at least comprising an input layer, a hidden layer and an output layer, the input layer comprising a first preset number of nodes, and the output layer comprising a second preset number of nodes;

[0034] The enhanced vehicle features are input into the multi-layer perception model for regression prediction to obtain a predicted trajectory of the vehicle.

[0035] In addition, to achieve the above purpose, the present application also proposes a driving trajectory prediction device based on sparse BEV features, and the driving trajectory prediction device based on sparse BEV features includes:

[0036] An acquisition module, used to acquire driving scene information;

[0037] A computing module, used to obtain a perception detection result and a map perception result according to the driving scene information; and also used to generate a sparse vehicle feature by sparse mapping the perception detection result and the map perception result;

[0038] An interaction module, used for performing interaction prediction according to the driving scene information to obtain action prediction features;

[0039] A fusion module, used for obtaining enhanced vehicle features according to the action prediction features and the sparse vehicle features;

[0040] The result module is used to obtain the predicted trajectory of the vehicle according to the enhanced vehicle features.

[0041] In addition, to achieve the above-mentioned purpose, the present application also proposes a medium, which is a computer-readable medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the driving trajectory prediction method of sparse BEV features as described above are implemented.

[0042] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the driving trajectory prediction method of sparse BEV features as described above.

[0043] The present application obtains driving scene information, obtains perception detection results and map perception results based on the driving scene information, generates sparse car features through sparse mapping of the perception detection results and the map perception results, performs interactive prediction based on the driving scene information, obtains action prediction features, obtains enhanced car features based on the action prediction features and the sparse car features, and obtains the predicted trajectory of the vehicle based on the enhanced car features. By obtaining scene information, mapping to generate sparse features and obtaining predicted action features, the enhanced car features are fused to obtain the enhanced car features, and the vehicle trajectory prediction is realized based on the enhanced car features, thereby improving the efficiency and accuracy of driving trajectory prediction and reducing the risk of collision. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0045] Figure 1 This is a flow chart of the first embodiment of the driving trajectory prediction method based on sparse BEV features of the present application;

[0046] Figure 2 This is a flow chart of a second embodiment of the driving trajectory prediction method based on sparse BEV features of the present application;

[0047] Figure 3 A schematic flow chart of a third embodiment of the driving trajectory prediction method based on sparse BEV features of the present application;

[0048] Figure 4 This is a schematic diagram of the module structure of the driving trajectory prediction device with sparse BEV features according to an embodiment of the present application;

[0049] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the driving trajectory prediction method of sparse BEV features in the embodiment of the present application.

[0050] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0052] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0053] The main solution of the embodiment of the present application is: obtaining driving scene information, mapping according to the driving scene information, obtaining sparse vehicle features, performing interactive prediction according to the driving scene information, obtaining action prediction features, obtaining enhanced vehicle features based on the action prediction features and the sparse vehicle features, and obtaining the predicted trajectory of the vehicle based on the enhanced vehicle features.

[0054] Based on this, the embodiment of the present application provides a driving trajectory prediction method with sparse BEV features, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the driving trajectory prediction method based on sparse BEV features of the present application.

[0055] In this embodiment, the driving trajectory prediction method of the sparse BEV features includes steps S10 to S60:

[0056] Step S10, obtaining driving scene information.

[0057] It should be noted that the driving scene information is obtained through the precise perception of the vehicle's surrounding environment by sensors. For example, information is collected through on-board sensors such as LiDAR, cameras, millimeter-wave radar, etc. These sensors can provide rich environmental information, such as road structure, obstacle locations, and behavioral patterns of other traffic participants. By fusing data from different types of sensors, a more complete and accurate driving scene model can be constructed. In addition, CAN bus data is also used to obtain the vehicle's own status information, such as speed, acceleration, etc. Secondly, computer vision technology and machine learning algorithms play an important role in processing the collected data. Traffic signs, pedestrians and other vehicles are identified from the images captured by the front-view camera; at the same time, the real driving process can be restored and key scene information can be extracted from the analysis of the dashcam video data. Map data is also indispensable. They provide geographical background knowledge to help better understand and predict future driving paths. It is worth noting that in-car voice commands can also be used as an additional source of information to infer the driver's intentions or emotional state, all of which belong to driving scene information.

[0058] Step S20, obtaining perception detection results and map perception results based on the driving scene information.

[0059] It should be noted that the process of obtaining perception detection results and map perception results based on driving scene information is intended to ensure that the vehicle can accurately understand the surrounding environment and make reasonable decisions accordingly.

[0060] Further, the driving scene information is preprocessed to obtain processed information, and the processed information is input into a preset deep learning model to obtain an object detection result, and the object detection result is identified and located to obtain a perception detection result. Specifically, the acquisition of the perception detection result depends on a variety of sensors (such as cameras, laser radars, etc.) to monitor the surrounding environment in real time. These sensors provide information about road conditions, other traffic participants (such as pedestrians, vehicles), traffic signals, and static obstacles. For example, different objects can be identified and classified from images or point cloud data using a deep learning algorithm. For dynamic objects, their motion trajectories also need to be tracked in order to predict their future movements. The above-mentioned perception detection results are 3D detection result data, and the above-mentioned 3D detection result data include center position, size, heading angle, and speed in three-dimensional directions. Specifically, the 3D detection result data provides key information about objects in the surrounding environment. These data usually contain parameters such as the center position, size (length, width, and height), heading angle, and speed in three-dimensional directions of each detected target. Specifically, the center position is represented by (x, y, z) coordinates, which define the exact position of the target in the world coordinate system; the size is described by (l, w, h), where l represents the length, w is the width, and h is the height; the heading angle θ reflects the direction of the target relative to the reference axis. For ground vehicles, it mainly refers to the angle between its forward direction and the reference direction; and the speed in the three-dimensional direction (v x , v y , v z ) gives the velocity components of the target along three orthogonal axes, which is crucial for predicting the target's future motion trajectory. 3D detection result data not only helps the vehicle understand the current environmental state, but also supports more complex tasks such as path planning and obstacle avoidance decisions. For example, when encountering pedestrians crossing the road or the vehicle in front suddenly slows down, accurate 3D detection results can enable autonomous vehicles to take appropriate measures in time to ensure safety. In addition, when combined with other sensor data (such as radar reflection intensity), 3D detection results can also enhance the recognition accuracy of target types and distinguish different types of objects, such as motor vehicles, non-motor vehicles or pedestrians.

[0061] Furthermore, the driving scene information is preprocessed to obtain processed information, the processed information is semantically segmented to obtain semantic results, and recognition and classification are performed based on the semantic results to obtain map perception results. Specifically, the perception module is not limited to target detection and tracking, but also includes semantic segmentation tasks, that is, assigning a category label to each pixel to more carefully depict the various components in the environment. The map perception result refers to the combination of high-precision maps (HD Maps), which contain detailed geographic information such as lane lines, intersections, crosswalks, etc. High-precision maps help vehicles achieve precise positioning and provide a framework so that the data captured by sensors can be interpreted in a unified spatial coordinate system. When the vehicle is driving, it can match data from different sensors with pre-built maps to enhance the understanding of the current location and support activities such as path planning. The above-mentioned map perception results are multiple point coordinates of the map, and the scene vector map is composed according to the above-mentioned multiple point coordinates. Specifically, in the process of constructing the scene vector map, the multiple point coordinates provided by the map perception results play a core role. These point coordinates not only represent specific locations in geographic space, but also carry information about key elements such as roads, buildings, and traffic signs. By analyzing and processing these point coordinates, accurate vectorized map elements can be generated, such as road boundaries represented by line segments, building outlines depicted by polygons, and other geographic features. First, based on the collected point coordinate data, a series of spatial analysis operations are performed, such as cluster analysis to identify the entity objects that dense point groups may correspond to, or to connect discrete points into continuous curves or surface structures through fitting algorithms. For road networks, this step can effectively restore the complete road form and provide a basic framework for subsequent map editing. Then, on this basis, the topological relationship between each vector element is further defined to ensure that adjacent or connected objects maintain the correct logical connection, such as the correct association between lanes in different directions at intersections. In order to ensure the quality and practicality of the vectorized map, the original point coordinates need to be cleaned and optimized. Remove noise points to avoid misjudgment; adjust outliers to make the shape smoother and more natural; finally, apply simplification algorithms to reduce unnecessary details, thereby improving rendering efficiency without losing important information. In addition, considering the needs of actual application scenarios, additional attributes can be added to the vector map, such as road grade, speed limit, number of lanes, etc., so that high-level autonomous driving vehicles can better understand and respond to complex traffic conditions. It is worth noting that with the development of technology, more and more research is devoted to automatically extracting high-quality vector maps from data obtained by sensors. For example, DuMapNet proposed by Baidu uses the Transformer architecture to achieve end-to-end city-scale lane-level vector map generation, greatly reducing the cost of manual annotation and improving production efficiency.Similar technological advances are driving intelligent transportation systems to a higher level, making vector-based map services more accurate and reliable, while also paving the way for future smart city construction and autonomous driving technology.

[0062] Step S30, generating sparse vehicle features by sparse mapping the perception detection results and the map perception results.

[0063] It can be explained that the perception detection results and the map perception results are sparsely mapped to generate sparse car features. Before sparse mapping, the perception detection results and the map perception results are first normalized, as shown below:

[0064]

[0065] Where max(B pred ) is the maximum value mapped to the BEV space, min(B pred ) is the minimum value mapped to the BEV space, x represents the initial data, and then the final sparse Query BEV features mapped to the BEV space are obtained.

[0066] Step S40: performing interactive prediction based on the driving scene information to obtain action prediction features.

[0067] It is important to understand that interaction prediction relies on a comprehensive understanding of the driving scenario, which includes but is not limited to dynamic information such as the vehicle's position, speed, acceleration, direction, and static or quasi-static environmental factors such as road layout, traffic light status, and pedestrian activity. Through the comprehensive processing of these data, a detailed scenario model can be constructed, which serves as the basis for subsequent predictions. For example, using a multi-layer perception model (MLP) or a Transformer-based architecture can effectively capture the complex relationships between traffic participants and simulate their potential interactions.

[0068] Next, a variety of methods have been developed to extract useful action prediction features from scene models. A common approach is to use an attention mechanism, which allows the model to focus on the most relevant parts, such as vehicles that are about to intersect or pedestrians that suddenly appear. This mechanism can help the model identify which objects' behavior will have the greatest impact on the future, and then concentrate resources to make more detailed predictions for them. In addition, game theory concepts have been introduced, such as the GameFormer model, which iteratively refines predictions through a hierarchical decoding structure and models the interactions between agents based on the spirit of k-level game theory, making the prediction results closer to the behavior patterns in the real world.

[0069] Step S50, obtaining enhanced vehicle features according to the action prediction features and the sparse vehicle features.

[0070] It should be noted that when combining action prediction features with sparse car features, richer and more representative enhanced car features can be obtained. This step usually involves the design of a feature fusion mechanism, with the goal of creating a synergistic effect between different features. For example, the Sparse4D series of work proposed a deformable attention-based method for extracting features from multi-view images, and enhanced feature representation capabilities through a multi-scale feature aggregation (LMFA) module and a global feature aggregation (GFA) module. In addition, in order to better capture local details and global contextual information, graph neural networks (GNNs) or Transformer-based architectures can also be used. They are good at processing data with non-Euclidean structures, enabling the model to fully understand complex interactions while maintaining sparsity.

[0071] Step S60, obtaining the predicted trajectory of the vehicle according to the enhanced vehicle features.

[0072] It should be noted that in order to obtain the predicted trajectory of the vehicle, it is necessary to infer the most likely future path based on the enhanced vehicle features. This step can be achieved in many ways. In this embodiment, a multi-layer perception model is used for prediction.

[0073] Further, a multi-layer perception model is first constructed. Constructing a multi-layer perception model (MLP) for the task of predicting the trajectory of the self-driving car in autonomous driving involves key steps such as carefully designing the network architecture, selecting appropriate activation functions, and determining hyperparameters. As a feedforward neural network, the basic structure of MLP at least includes an input layer, one or more hidden layers, and an output layer. For this embodiment, we will use the enhanced car features as input data and perform regression prediction through MLP to obtain the future trajectory of the self-car. First, define the input layer of MLP. According to the dimension of the enhanced car features, the input layer will be set to contain a first preset number of nodes, which correspond one to one to each element in the feature vector. For example, if the enhanced car features consist of vehicle position, speed, acceleration, direction angle and its rate of change, the number of nodes in the input layer should be equal to the sum of the number of all these feature values. This is done to ensure that each influencing factor can be effectively passed to the subsequent processing layer. Next is the design of the hidden layer. In order to capture nonlinear relationships and improve the expressiveness of the model, MLP usually adds one or more hidden layers. The number of nodes in each hidden layer is an important hyperparameter that determines the learning ability and complexity of the model. In theory, increasing the number of hidden layers can improve the representation ability, but it also brings the risk of overfitting and increased computational costs. Therefore, in practice, it is often necessary to find the best configuration through experiments. A common practice is to start with a small number of nodes and gradually increase them until the generalization error no longer decreases significantly. In addition, regularization techniques such as Dropout can also be used to alleviate the overfitting problem. Regarding the choice of activation function, ReLU (Rectified Linear Unit) has become one of the default options due to its simplicity and effectiveness. It can introduce nonlinearity while maintaining gradient propagation, which helps to break the limitations of linear combinations. Of course, other types of activation functions, such as Sigmoid or Tanh, can also be considered depending on the specific situation, especially in the output layer, which may be more suitable for normalization. Finally, determine the structure of the output layer. Since our goal is to predict the future trajectory of the vehicle, which is essentially a regression problem, the output layer should contain a second preset number of nodes to represent the coordinate position or other related parameters at different time points. For example, if the position is predicted for several moments in the future, the output layer may have a corresponding number of nodes corresponding to the x, y coordinate values ​​at each moment. It is worth noting that the activation function of the output layer usually chooses a linear function or does not use any activation function in order to directly output continuous numerical results. After completing the above preparations, the enhanced car features can be input into the constructed MLP model for training. During the training process, we will use the back propagation algorithm to adjust the weight matrix to minimize the prediction error.Specifically, the loss function can be selected in the form of mean square error (MSE), mean absolute error (MAE) and other forms according to task requirements, and combined with optimizers such as Adam to achieve efficient parameter updates. As the number of iterations increases, the model gradually learns to extract useful information from the input features and accurately map them to the desired output.

[0074] Furthermore, the enhanced vehicle features are input into the multi-layer perception model for regression prediction to obtain the predicted trajectory of the ego vehicle. The final predicted trajectory of the ego vehicle not only helps to plan a safe and reliable driving path, but also provides an important basis for subsequent decision-making. For example, when approaching an intersection, you can slow down or avoid the action in advance based on the predicted movement trend of other vehicles; when driving on the highway, you can adjust your own following distance according to the behavior of the vehicle in front. In short, by effectively utilizing the enhanced vehicle features, the MLP model can achieve accurate ego vehicle trajectory prediction in complex traffic environments, thereby improving the safety and reliability of autonomous driving.

[0075] This embodiment provides a driving trajectory prediction method of sparse BEV features, by acquiring driving scene information, obtaining perception detection results and map perception results according to the driving scene information, generating sparse vehicle features through sparse mapping of the perception detection results and the map perception results, performing interactive prediction according to the driving scene information, obtaining action prediction features, obtaining enhanced vehicle features according to the action prediction features and the sparse vehicle features, and obtaining the predicted trajectory of the vehicle according to the enhanced vehicle features. By acquiring scene information, mapping to generate sparse features and obtaining predicted action features, the enhanced vehicle features are fused to obtain the enhanced vehicle features, and the vehicle trajectory prediction is realized according to the enhanced vehicle features, thereby improving the efficiency and accuracy of driving trajectory prediction and reducing the risk of collision.

[0076] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can refer to the above introduction, and will not be repeated later. Figure 2 The driving trajectory prediction method of the sparse BEV feature step S40 further includes steps S401 to S403:

[0077] Step S401, obtaining interaction features.

[0078] Step S402: performing normalization processing according to the driving scene information to obtain undecoded features.

[0079] It should be noted that data from multiple sensors (such as cameras, radars, lidars, etc.) may have different dimensions and numerical ranges. In order for these multi-source heterogeneous data to work together, they must be preprocessed first. This step includes not only conventional operations such as removing outliers and filling missing values, but also an important normalization step. Through min-max normalization or other appropriate normalization methods, all input features can be guaranteed to be of similar order of magnitude, thereby avoiding some features from dominating the learning process due to excessive values. The undecoded features obtained after normalization, although they have been preliminarily processed, still retain their original semantics and are now more suitable for further feature engineering or directly as input to machine learning models. Next, these undecoded features can be further processed by the encoding part of the encoder-decoder architecture and converted into a more abstract and expressive form. For example, in deep learning-based methods, convolutional neural networks (CNNs) can be used to capture local patterns in images; recurrent neural networks (RNNs) and their variants are good at processing time series data, such as vehicle trajectories.

[0080] Step S403, obtaining the action prediction feature according to the undecoded feature and the interaction feature through the functional relationship between the action prediction feature and the undecoded feature and the interaction feature.

[0081] It should be noted that the action prediction feature is obtained by learning the undecoded features and interaction features, which reflects the probability distribution of a series of actions that are most likely to occur in the current state. For example, in an autonomous driving system, action prediction may include the probability of operations such as acceleration, deceleration, and steering. Once the undecoded features and interaction features are obtained, the next step is to establish a functional relationship between them and the action prediction features. The specific functional relationship is as follows:

[0082] Q=MHCH(Q motion ,B pred )

[0083] Where Q motion It represents the action features of n objects in the predicted scene in the future T time. MHCH represents multi-head cross attention, which is calculated using Softmax.

[0084] This embodiment obtains the interaction features, performs normalization processing according to the driving scene information, obtains the undecoded features, and obtains the action prediction features according to the undecoded features and the interaction features through the functional relationship between the action prediction features and the undecoded features and the interaction features. The undecoded features are obtained by normalizing the driving scene data, combining the interaction features, and using the functional relationship to obtain the action prediction features, thereby improving the safety and efficiency of autonomous driving.

[0085] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description, and will not be described in detail later. Figure 3 The driving trajectory prediction method of the sparse BEV feature step S50 further includes steps S501 to S504:

[0086] Step S501, decoding the action prediction features by inputting into a multi-layer perception model to obtain specific data of the predicted action.

[0087] It should be noted that in this process, the first thing to prepare is the action prediction features extracted from the video frames or sensor data, which include spatial features, temporal features, and frequency domain features. Once the feature vectors are prepared, they will be fed into the pre-trained MLP model to obtain the predicted action data. The specific data of the predicted action can not only reflect the current action category or state, but also contain information about the action trend that may occur in the future.

[0088] Step S502, obtaining automobile feature data according to the sparse automobile features.

[0089] It should be noted that after obtaining a series of images or point cloud data containing vehicles, it is first necessary to remove the influence of background noise and other irrelevant factors. For image data, background and foreground objects can be separated by methods such as GMM to ensure that only the parts related to the vehicle are retained. After that, sparse coding algorithms such as dictionary learning are applied to map each vehicle into a set of sparse coefficients to obtain vehicle feature data. In order to further enrich the feature expression, multiple types of input sources can also be combined, such as fusing information from cameras with different perspectives or integrating the three-dimensional spatial coordinates provided by lidar. This step often involves cross-modal feature extraction technology to ensure that all available information can be fully utilized. Finally, the sparse features obtained after the above processing can be used as input for subsequent classifiers (such as SVM support vector machines) or other machine learning models to complete tasks such as brand type recognition, vehicle type classification, and even speed and flow estimation.

[0090] Step S503, obtaining a first weight and a second weight, wherein the first weight corresponds to a ratio of specific data of the predicted action, and the second weight corresponds to a ratio of characteristic data of the vehicle.

[0091] Step S504, obtaining enhanced vehicle features according to the first weight, the second weight, the specific data of the predicted action and the vehicle feature data.

[0092] It should be noted that the first weight and the second weight are usually calculated from historical energy consumption data or similar time series data through a neural network model, reflecting the degree of influence of each device (such as engine, headlights, air conditioner, etc.) on the overall system performance at different time points. Combined with specific data of the predicted action. This type of data often comes from a trained MLP or other type of regression prediction model, which infers upcoming actions or change trends based on the current state of the vehicle and the surrounding environment, allowing the algorithm to prepare countermeasures in advance, thereby improving response speed and safety. Finally, all the above information is combined with the existing vehicle feature data to form enhanced vehicle features. The vehicle feature data here covers static attributes (such as brand, model, color, etc.) and dynamic parameters (such as speed, acceleration, direction angle, etc.). By introducing an attention mechanism or a graph neural network (GNN), the model's ability to capture interactions between vehicles can be further enhanced, making the enhanced features of the final output closer to the actual situation.

[0093] Further, step S504 also includes: fusing the specific data of the predicted action and the sparse vehicle features based on the first weight and the second weight to obtain a fused feature; and calculating according to the fused feature to obtain an enhanced vehicle feature. Specifically, the weights are used to adjust the specific data of the predicted action and the sparse vehicle features. The specific data of the predicted action usually comes from MLP, which provides information about possible actions or change trends in the future. The sparse vehicle features are obtained by applying a sparse coding algorithm to the original sensor data, aiming to reduce redundancy and highlight the most important features. When we add these two types of information according to their corresponding weights, a preliminary fused feature is formed. In order to further optimize the quality of the fused feature, an attention mechanism or GNN can also be introduced. These advanced technologies can help the model better understand the relationship between the various components, especially when it comes to interactions between multiple vehicles. For example, in a typical autonomous driving scenario, it is necessary to consider not only the behavior of a single vehicle, but also the actions of other surrounding vehicles and their potential impact. In this way, the fused feature can not only capture the unique attributes of individual vehicles, but also reflect the state of the entire traffic flow. Calculation is performed according to the fused feature to obtain the final enhanced vehicle feature. This step may involve a variety of mathematical operations and technical means, such as principal component analysis (PCA) and entropy weight method (ENT). Its goal is to add additional semantic information while retaining the original information, so as to make the features richer and more discriminative. For example, the enhanced car features can not only be used to improve existing detection and classification tasks, but also support higher-level functions such as path planning and collision warning.

[0094] In this embodiment, the action prediction feature is decoded by inputting a multi-layer perception model to obtain specific data of the predicted action, and the vehicle feature data is obtained according to the sparse vehicle features, and the first weight and the second weight are obtained. The enhanced vehicle features are obtained according to the first weight, the second weight, the specific data of the predicted action and the vehicle feature data. The action prediction feature is decoded by a multi-layer perceptron, combined with the sparse vehicle features, and the predicted action data and vehicle features are fused using the first and second weights to generate enhanced vehicle features, thereby improving the richness and accuracy of the features, optimizing the decision-making ability of the autonomous driving system, and enhancing safety and efficiency.

[0095] This application also provides a driving trajectory prediction device for sparse BEV features, please refer to Figure 4 , the device comprises:

[0096] The acquisition module 10 is used to acquire driving scene information.

[0097] The calculation module 20 is used to obtain the perception detection result and the map perception result according to the driving scene information; and is also used to generate sparse vehicle features by sparse mapping the perception detection result and the map perception result.

[0098] The interaction module 30 is used to perform interaction prediction according to the driving scene information to obtain action prediction features.

[0099] The fusion module 40 is used to obtain enhanced vehicle features according to the action prediction features and the sparse vehicle features.

[0100] The result module 50 is used to obtain the predicted trajectory of the vehicle according to the enhanced vehicle characteristics.

[0101] The driving trajectory prediction device with sparse BEV features provided in the present application adopts the driving trajectory prediction method with sparse BEV features in the above-mentioned embodiment, which can solve the technical problem of how to improve the efficiency and accuracy of driving trajectory prediction. Compared with the prior art, the beneficial effects of the driving trajectory prediction device with sparse BEV features provided in the present application are the same as the beneficial effects of the driving trajectory prediction method with sparse BEV features provided in the above-mentioned embodiment, and the other technical features in the driving trajectory prediction device with sparse BEV features are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0102] In one embodiment, the fusion module 40 is also used to decode the action prediction features by inputting a multi-layer perception model to obtain specific data of the predicted action; obtain vehicle feature data based on the sparse vehicle features; obtain a first weight and a second weight, the first weight corresponding to the ratio of the action prediction features, and the second weight corresponding to the ratio of the sparse vehicle features; obtain enhanced vehicle features based on the first weight, the second weight, the specific data of the predicted action and the vehicle feature data.

[0103] In one embodiment, the fusion module 40 is further used to fuse the specific data of the predicted action with the sparse vehicle features based on the first weight and the second weight to obtain fused features; and perform calculations based on the fused features to obtain enhanced vehicle features.

[0104] In one embodiment, the computing module 20 is also used to pre-process the driving scene information to obtain processed information; input the processed information into a preset deep learning model to obtain object detection results; perform semantic segmentation on the processed information to obtain semantic results; perform identification and positioning based on the object detection results to obtain perception detection results; perform identification and classification based on the semantic results to obtain map perception results.

[0105] In one embodiment, the interaction module 30 is also used to obtain interaction features; perform normalization processing based on the driving scene information to obtain undecoded features; and obtain action prediction features based on the undecoded features and the interaction features through the functional relationship between the action prediction features and the undecoded features and the interaction features.

[0106] In one embodiment, the result module 50 is also used to construct a multi-layer perception model; the enhanced vehicle features are input into the multi-layer perception model for regression prediction to obtain the vehicle prediction trajectory.

[0107] The present application provides a driving trajectory prediction device for sparse BEV features, and the driving trajectory prediction device for sparse BEV features includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the driving trajectory prediction method for sparse BEV features in the above-mentioned embodiment one.

[0108] Reference below Figure 5 , which shows a schematic diagram of the structure of a driving trajectory prediction device suitable for implementing the sparse BEV features of the embodiment of the present application. The driving trajectory prediction device of the sparse BEV features in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The driving trajectory prediction device of the sparse BEV features shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0109] like Figure 5 As shown, the driving trajectory prediction device of sparse BEV features may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: Read Only Memory) 1002 or the program loaded from the storage device 1003 to the random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the driving trajectory prediction device of sparse BEV features are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other via a bus 1005. The input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the driving trajectory prediction device with sparse BEV features to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a driving trajectory prediction device with sparse BEV features having various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0110] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0111] The driving trajectory prediction device with sparse BEV features provided by the present application adopts the driving trajectory prediction method with sparse BEV features in the above-mentioned embodiment, which can solve the technical problem of how to improve the efficiency and accuracy of driving trajectory prediction. Compared with the prior art, the beneficial effects of the driving trajectory prediction device with sparse BEV features provided by the present application are the same as the beneficial effects of the driving trajectory prediction method with sparse BEV features provided by the above-mentioned embodiment, and the other technical features in the driving trajectory prediction device with sparse BEV features are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0112] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0113] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0114] The present application provides a computer-readable medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the driving trajectory prediction method of sparse BEV features in the above-mentioned embodiment.

[0115] The computer-readable medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0116] The computer-readable medium may be included in the driving trajectory prediction device for sparse BEV features; or may exist independently without being assembled into the driving trajectory prediction device for sparse BEV features.

[0117] The computer-readable medium carries one or more programs. When the one or more programs are executed by the driving trajectory prediction device of sparse BEV features, the driving trajectory prediction device of sparse BEV features can be written in one or more programming languages ​​or a combination thereof to write computer program codes for performing the operations of the present application. The programming languages ​​include object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0118] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0119] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0120] The readable medium provided in the present application is a computer-readable medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned sparse BEV feature driving trajectory prediction method, which can solve the technical problem of how to improve the efficiency and accuracy of driving trajectory prediction. Compared with the prior art, the beneficial effects of the computer-readable medium provided in the present application are the same as the beneficial effects of the sparse BEV feature driving trajectory prediction method provided in the above-mentioned embodiment, and will not be repeated here.

[0121] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the driving trajectory prediction method based on sparse BEV features as described above.

[0122] The computer program product provided in this application can solve the technical problem of how to improve the efficiency and accuracy of driving trajectory prediction. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the driving trajectory prediction method of sparse BEV features provided in the above embodiment, which will not be repeated here.

[0123] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A driving trajectory prediction method based on sparse BEV features, characterized in that: The method comprises: Obtain driving scene information; Obtaining a perception detection result and a map perception result according to the driving scene information; Generate sparse vehicle features by sparse mapping the perception detection result and the map perception result; Perform interactive prediction based on the driving scene information to obtain action prediction features; Obtaining enhanced vehicle features according to the action prediction features and the sparse vehicle features; According to the enhanced vehicle features, a predicted trajectory of the vehicle is obtained.

2. The method according to claim 1, characterized in that The step of obtaining enhanced vehicle features according to the action prediction features and the sparse vehicle features comprises: Decoding the action prediction features by inputting them into a multi-layer perception model to obtain specific data of the predicted action; Obtaining automobile feature data according to the sparse automobile features; Obtaining a first weight and a second weight, wherein the first weight corresponds to a ratio of specific data of the predicted action, and the second weight corresponds to a ratio of the vehicle feature data; Enhanced vehicle features are obtained based on the first weight, the second weight, the specific data of the predicted action and the vehicle feature data.

3. The method according to claim 2, characterized in that The step of obtaining enhanced vehicle features according to the first weight, the second weight, the specific data of the predicted action and the sparse vehicle features comprises: Based on the first weight and the second weight, the specific data of the predicted action and the sparse vehicle feature are fused to obtain a fused feature; Calculation is performed based on the fused features to obtain enhanced vehicle features.

4. The method according to claim 1, characterized in that The step of obtaining a perception detection result and a map perception result according to the driving scene information includes: Preprocessing the driving scene information to obtain processed information; Inputting the processed information into a preset deep learning model to obtain an object detection result; Performing semantic segmentation on the processed information to obtain a semantic result; Identify and locate the object according to the object detection result to obtain a perception detection result; Identification and classification are performed according to the semantic results to obtain a map perception result.

5. The method according to claim 4, characterized in that The perception detection result is 3D detection result data, which includes center position, size, heading angle and speed in three-dimensional directions. The map perception result is multiple point coordinates of the map, and a scene vector map is composed according to the multiple point coordinates.

6. The method according to claim 1, characterized in that The step of performing interactive prediction according to the driving scene information to obtain action prediction features includes: Get interactive features; Performing normalization processing according to the driving scene information to obtain undecoded features; The action prediction feature is obtained according to the undecoded feature and the interaction feature through the functional relationship between the action prediction feature, the undecoded feature and the interaction feature.

7. The method according to claim 1, characterized in that The step of obtaining the predicted trajectory of the vehicle according to the enhanced vehicle features comprises: Constructing a multi-layer perception model, the multi-layer perception model at least comprising an input layer, a hidden layer and an output layer, the input layer comprising a first preset number of nodes, and the output layer comprising a second preset number of nodes; The enhanced vehicle features are input into the multi-layer perception model for regression prediction to obtain a predicted trajectory of the vehicle.

8. A driving trajectory prediction device based on sparse BEV features, characterized in that: The device comprises: An acquisition module, used to acquire driving scene information; A computing module, used to obtain a perception detection result and a map perception result according to the driving scene information; and also used to generate a sparse vehicle feature by sparse mapping the perception detection result and the map perception result; An interaction module, used for performing interaction prediction according to the driving scene information to obtain action prediction features; A fusion module, used for obtaining enhanced vehicle features according to the action prediction features and the sparse vehicle features; The result module is used to obtain the predicted trajectory of the vehicle according to the enhanced vehicle features.

9. A driving trajectory prediction device based on sparse BEV features, characterized in that: The device includes: a memory, a processor, and a driving trajectory prediction program based on sparse BEV features stored in the memory and running on the processor, wherein the driving trajectory prediction program based on sparse BEV features is configured to implement the steps of the driving trajectory prediction method based on sparse BEV features as described in any one of claims 1-7.

10. A storage medium, characterized in that: The storage medium stores a driving trajectory prediction program based on sparse BEV features, and when the driving trajectory prediction program based on sparse BEV features is executed by the processor, the steps of the driving trajectory prediction method based on sparse BEV features as described in any one of claims 1-7 are implemented.

Images