Intelligent driving track prediction method and device, equipment and storage medium

Abstract

The invention discloses an intelligent driving track prediction method, device and equipment and a storage medium, relates to the technical field of intelligent driving, and discloses an intelligent driving track prediction method which comprises the following steps: acquiring environment laser radar data, environment radar data and environment image data; converting the environment laser radar data, the environment radar data and the environment image data into aerial view features; performing feature alignment on the aerial view features based on a sensor field angle identifier to obtain multi-modal sensing features; determining a candidate trajectory according to the multi-modal sensing features; and determining a target trajectory from the candidate trajectories according to a target cost function to complete intelligent driving trajectory prediction. According to the scheme, the complex driving environment can be accurately understood in the intelligent driving process so as to predict the optimal driving track.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, and in particular to intelligent driving trajectory prediction methods, devices, equipment and storage media. Background Art

[0002] With the development of intelligent driving technology, higher requirements are placed on the accuracy and real-time performance of trajectory prediction. A trajectory prediction method with good generalization ability that can accurately reflect vehicle dynamics, environmental changes, and interactions between vehicles is needed to adapt to different driving scenarios and conditions.

[0003] At present, some solutions use long short-term memory (LSTM) networks for trajectory prediction and behavior intention classification, and then input this information into a multimodal LSTM network to generate the final predicted trajectory, but the real-time changes in the surrounding environment are not fully considered during the prediction process. Therefore, in the process of intelligent driving, how to accurately understand the complex driving environment to predict the optimal driving trajectory has become a problem to be solved.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide an intelligent driving trajectory prediction method, device, equipment and storage medium, aiming to solve the technical problem of how to accurately understand the complex driving environment in the process of intelligent driving to predict the optimal driving trajectory.

[0006] To achieve the above objectives, the present application proposes an intelligent driving trajectory prediction method, the method comprising:

[0007] Acquire environmental lidar data, environmental radar data, and environmental image data;

[0008] Converting the environmental laser radar data, the environmental radar data, and the environmental image data into bird's-eye view features;

[0009] Performing feature alignment on the bird's-eye view features based on the sensor field of view angle identifier to obtain a multimodal perception feature;

[0010] Determining candidate trajectories according to the multimodal perception features;

[0011] A target trajectory is determined from the candidate trajectories according to a target cost function to complete intelligent driving trajectory prediction.

[0012] In one embodiment, the step of determining candidate trajectories according to the multimodal perception features includes:

[0013] Determining path planning language information according to the multimodal perception features and the environmental image data;

[0014] Obtain the vehicle's driving status information, road environment information, obstacle status information, and traffic rules information;

[0015] A candidate trajectory is generated according to the path planning language information, the environmental image data, the vehicle driving state information, the road environment information, the obstacle dynamic information and the traffic rule information.

[0016] In one embodiment, the step of determining the path planning language information according to the multimodal perception feature and the environmental image data includes:

[0017] Decoding the multimodal perception features to obtain a bird's-eye view visual marker;

[0018] Encoding the environmental image data to obtain a text label;

[0019] Path planning language information is determined based on the bird's-eye view visual mark and the text mark.

[0020] In one embodiment, the step of generating a candidate trajectory according to the path planning language information, the environmental image data, the vehicle driving state information, the road environment information, the obstacle dynamic information and the traffic rule information includes:

[0021] Encoding the trajectory planning language information to obtain a trajectory planning vector;

[0022] A candidate trajectory is generated according to the trajectory planning vector, the environmental image data, the vehicle driving state information, the road environment information, the obstacle dynamic information and the traffic rule information.

[0023] In one embodiment, the sensor field of view angle identifier includes a laser radar field of view angle identifier, a radar field of view angle identifier, and an image field of view angle identifier;

[0024] The step of aligning the features of the bird's-eye view based on the sensor field of view angle identifier to obtain multimodal perception features includes:

[0025] Acquire the sensor field of view angle identifiers corresponding to the environmental laser radar data, the environmental radar data, and the environmental image data, and obtain the laser radar field of view angle identifier, the radar field of view angle identifier, and the image field of view angle identifier;

[0026] The bird's-eye view features are aligned based on the laser radar field of view angle identifier, the radar field of view angle identifier and the image field of view angle identifier to obtain multimodal perception features.

[0027] In one embodiment, the step of aligning the bird's-eye view features based on the laser radar field of view angle identifier, the radar field of view angle identifier, and the image field of view angle identifier to obtain a multimodal perception feature includes:

[0028] Assigning a modal weight to the bird's-eye view feature based on the laser radar field of view angle identifier, the radar field of view angle identifier, and the image field of view angle identifier;

[0029] The bird's-eye view features are aligned according to the modal weights to obtain multimodal perception features.

[0030] In one embodiment, the step of determining the target trajectory from the candidate trajectories according to the target cost function to complete the intelligent driving trajectory prediction includes:

[0031] Constructing the target cost function based on safety cost, comfort cost, compliance cost, prediction uncertainty cost and path continuity cost;

[0032] A target trajectory is determined from the candidate trajectories according to the target cost function to complete intelligent driving trajectory prediction.

[0033] In addition, to achieve the above-mentioned purpose, the present application also proposes an intelligent driving trajectory prediction device, which includes:

[0034] A data acquisition module, used to acquire environmental laser radar data, environmental radar data and environmental image data;

[0035] A feature extraction module, used for converting the environmental laser radar data, the environmental radar data and the environmental image data into bird's-eye view features;

[0036] A feature alignment module, used to align the features of the bird's-eye view based on the sensor field of view angle identifier to obtain a multimodal perception feature;

[0037] A trajectory generation module, used to determine candidate trajectories according to the multimodal perception features;

[0038] The trajectory prediction module is used to determine the target trajectory from the candidate trajectories according to the target cost function to complete the intelligent driving trajectory prediction.

[0039] In addition, to achieve the above-mentioned purpose, the present application also proposes an intelligent driving trajectory prediction device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the intelligent driving trajectory prediction method as described above.

[0040] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the intelligent driving trajectory prediction method described above are implemented.

[0041] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the intelligent driving trajectory prediction method as described above.

[0042] One or more technical solutions proposed in this application have at least the following technical effects:

[0043] Acquire environmental laser radar data, environmental radar data and environmental image data; convert the environmental laser radar data, environmental radar data and environmental image data into bird's-eye view features; align the bird's-eye view features based on the sensor field of view angle identifier to obtain multimodal perception features; determine candidate trajectories based on the multimodal perception features; determine the target trajectory from the candidate trajectories based on the target cost function to complete intelligent driving trajectory prediction. By integrating data from different sensors such as laser radar, radar and camera, a richer and more accurate environmental perception is provided, the data of each sensor is converted into bird's-eye view features, and the features are aligned using the field of view of the sensor to obtain multimodal perception features containing perspective information from different sensors, which can comprehensively describe the behavior of traffic flow, pedestrians, obstacles and other vehicles in the vehicle driving environment, identify and predict potential trajectories based on these multimodal perception features, that is, candidate trajectories, and optimize the selection of the optimal trajectory through the target cost function, so as to realize the intelligent driving system's accurate understanding and prediction of complex driving environments and provide richer and more accurate environmental perception. The solution of the present application enables the intelligent driving system to better understand the complex driving environment, including traffic flow, pedestrians, obstacles and the behavior of other vehicles, in order to predict the optimal driving trajectory. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0046] Figure 1A flowchart of the first embodiment of the intelligent driving trajectory prediction method of the present application is provided;

[0047] Figure 2 A flow chart of the second embodiment of the intelligent driving trajectory prediction method of the present application;

[0048] Figure 3 A brief flowchart of the intelligent driving trajectory prediction method provided in Example 2 of the present application;

[0049] Figure 4 This is a schematic diagram of the module structure of the intelligent driving trajectory prediction device according to an embodiment of the present application;

[0050] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the intelligent driving trajectory prediction method in the embodiment of the present application.

[0051] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0053] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0054] The main solutions of the embodiments of this application are:

[0055] Acquire environmental lidar data, environmental radar data, and environmental image data;

[0056] Converting the environmental laser radar data, the environmental radar data, and the environmental image data into bird's-eye view features;

[0057] Performing feature alignment on the bird's-eye view features based on the sensor field of view angle identifier to obtain a multimodal perception feature;

[0058] Determining candidate trajectories according to the multimodal perception features;

[0059] A target trajectory is determined from the candidate trajectories according to a target cost function to complete intelligent driving trajectory prediction.

[0060] In this embodiment, for ease of description, the following description is made by taking the identification of the intelligent driving system as the execution subject.

[0061] With the development of intelligent driving technology, higher requirements are placed on the accuracy and real-time performance of trajectory prediction. A trajectory prediction method with good generalization ability that can accurately reflect vehicle dynamics, environmental changes, and interactions between vehicles is needed to adapt to different driving scenarios and conditions.

[0062] At present, some solutions use long short-term memory (LSTM) networks for trajectory prediction and behavior intention classification, and then input this information into a multimodal LSTM network to generate the final predicted trajectory, but the real-time changes in the surrounding environment are not fully considered during the prediction process. Therefore, in the process of intelligent driving, how to accurately understand the complex driving environment to predict the optimal driving trajectory has become a problem to be solved.

[0063] The present application provides a solution that provides richer and more accurate environmental perception by integrating data from different sensors such as lidar, radar and camera, converting the data of each sensor into bird's-eye view features, aligning the features using the field of view of the sensor, and obtaining multimodal perception features containing perspective information from different sensors, which can comprehensively describe the traffic flow, pedestrians, obstacles and other vehicle behaviors in the vehicle driving environment, identify and predict potential trajectories based on these multimodal perception features, i.e. candidate trajectories, and optimize the selection of the optimal trajectory through the target cost function, so as to enable the intelligent driving system to accurately understand and predict the complex driving environment, and provide richer and more accurate environmental perception. The solution of the present application enables the intelligent driving system to better understand the complex driving environment, including the traffic flow, pedestrians, obstacles and other vehicle behaviors, in order to predict the optimal driving trajectory.

[0064] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, an intelligent driving system, etc. The following takes the intelligent driving system as an example to illustrate this embodiment and the following embodiments.

[0065] Based on this, the present application embodiment provides an intelligent driving trajectory prediction method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the intelligent driving trajectory prediction method of the present application.

[0066] In this embodiment, the intelligent driving trajectory prediction method includes steps S10 to S40:

[0067] Step S10, acquiring environmental laser radar data, environmental radar data and environmental image data;

[0068] It should be understood that environmental lidar data is the three-dimensional point cloud data of the surrounding environment collected by the lidar during the vehicle's driving process, reflecting the three-dimensional spatial coordinates (x, y, z) and reflection intensity information of each point. Environmental radar data is the distance, speed and angle information collected by radar sensors (Radar) such as millimeter-wave radar during the vehicle's driving process. Environmental image data is the image data of the surrounding environment collected by the vehicle's onboard camera during the vehicle's driving process.

[0069] Step S20, converting the environmental laser radar data, the environmental radar data and the environmental image data into bird's-eye view features;

[0070] It should be understood that the bird's-eye view is a two-dimensional representation of the vehicle's surroundings, which provides a perspective of looking down on the vehicle and its surrounding objects from above, and projects all target objects in the view space (such as vehicles, pedestrians, obstacles, etc.) onto the ground to form a simplified view. The Bird's Eye View (BEV) feature includes a first bird's-eye view feature, a second bird's-eye view feature, and a third bird's-eye view feature. Among them, the first bird's-eye view feature is a 3D feature extracted from the environmental lidar data through a 3D encoder (such as PointPillars); the second bird's-eye view feature is a 3D feature extracted from the environmental radar data through a 3D encoder (such as PointPillars); the third bird's-eye view feature is a visual feature extracted from the environmental image data through a visual encoder (such as ResNet) and a multi-view feature fusion module (such as a Transformer model).

[0071] Specifically, PointPillars divides the three-dimensional space into a certain grid, and multiple columnar units Pillars are obtained. Each Pillar represents a small spatial area. A fixed size (such as 0.1m×0.1m×3m) can be selected to discretize the environmental space. For the 3D features of LiDAR and Radar sensors, PointPillars will map the information of each point to the corresponding Pillar on the BEV plane, and each Pillar will contain a certain number of features. The features of the Pillars can be extracted to obtain a unified BEV feature map, that is, the first bird's-eye view feature or the second bird's-eye view feature corresponding to the environmental LiDAR data or the environmental radar data. By extracting features from the environmental image data through ResNet, multi-view visual features containing key information of the environmental image (such as road signs, traffic lights, pedestrians and vehicles, etc.) can be obtained. The multi-view visual features extracted by different cameras can be integrated through Transformer to obtain the third bird's-eye view feature with image feature representation.

[0072] Step S30, aligning the features of the bird's-eye view based on the sensor field of view angle identifier to obtain a multimodal perception feature;

[0073] It should be noted that considering the bird's-eye view features to extract information from different sources (i.e., lidar, radar, and camera) or different perspectives (such as the front, back, left, and right), the bird's-eye view features need to be aligned through the sensor fusion model. Each sensor can be assigned a specific label through the sensor field of view angle identification to help the sensor fusion model identify and distinguish sensors in different orientations and obtain multimodal perception features.

[0074] In a feasible implementation, step S30 may include steps S31 to S32:

[0075] Step S31, assigning a modal weight to the bird's-eye view feature based on the laser radar field of view angle identifier, the radar field of view angle identifier, and the image field of view angle identifier;

[0076] It should be noted that the sensor field of view angle identifier includes the laser radar field of view angle identifier, the radar field of view angle identifier and the image field of view angle identifier. Among them, the laser radar field of view angle identifier is a label assigned to laser radars with different viewing angles. For example, the label corresponding to the laser radar on the right can be<RIGHT_LIDAR> ; The radar field of view angle is the label assigned to radars with different viewing angles. For example, the label corresponding to the radar on the left can be<LEFT_RADAR> ; The image field of view angle is identified by the label assigned to the camera with different viewing angles. For example, the label corresponding to the front-view camera can be<FRONT_CAM> . By comparing sensor signals at different field of view angles, the sensor fusion model can infer the relative position of objects, the direction of vehicle movement, and the geometric structure of the road. Specifically, it can identify vehicles in the adjacent lane ahead based on the forward visual image of the environmental image data, and verify these vehicles based on the side radar information of the environmental radar data, thereby unifying data from different perspectives into a complete BEV view space so that the network of the sensor fusion model can learn efficiently without the need to retrieve data from different perspectives separately. In the process of feature alignment, a weight, namely the modal weight, is assigned to the feature set corresponding to each bird's-eye view feature. The modal weight can be learned through the training process of the preset sensor fusion model using a multimodal dataset.

[0077] Step S32, aligning the features of the bird's-eye view according to the modal weights to obtain multimodal perception features.

[0078] It should be understood that after the bird's-eye view features and the corresponding sensor field of view angle identifiers are input into the sensor fusion model, the bird's-eye view features of each sensor (i.e., lidar, radar, and camera) will be properly projected and aligned according to the sensor field of view angle identifier, and the data of different sensors will be unified into the BEV view space to obtain multimodal perception features. The weighted bird's-eye view features are added according to the modal weights to obtain the fused feature representation F fused , that is, multimodal perception features. fused The expression formula is as follows:

[0079] F fused =F1·w1+F2·w2+...+F n ·w n

[0080] Among them, F1, F2, ..., F n are bird’s-eye view features from different sources, w1, w2, ..., w n is the corresponding weight, and n depends on the number of sensors of different types and orientations. For example, if the vehicle is equipped with a laser radar with one viewing orientation (such as the front), radars with four viewing orientations (such as the front, rear, left and right), and cameras with four viewing orientations (such as the front, rear, left and right), then n is 9.

[0081] Step S40, determining candidate trajectories according to the multimodal perception features;

[0082] It should be noted that the candidate trajectories are multiple possible vehicle driving trajectories. Based on the kinematic model or trajectory planning algorithm, the candidate trajectories can be obtained by combining multimodal perception features, vehicle driving state information and other information.

[0083] In a feasible implementation, step S40 may include steps S41 to S43:

[0084] Step S41, determining path planning language information according to the multimodal perception feature and the environmental image data;

[0085] It should be noted that the path planning language information is a high-level planning decision in the form of natural language, such as "go straight", "turn left" or "avoid pedestrians". After feature processing of multimodal perception features and environmental image data, corresponding visual tokens and text tokens can be generated. The visual tokens and text tokens are input into the pre-trained large language model (LLM) to obtain the corresponding path planning language information.

[0086] In a feasible implementation, step S41 may include steps S411 to S413:

[0087] Step S411, decoding the multimodal perception features to obtain a bird's-eye view visual mark;

[0088] It should be noted that the multimodal perception features can be decoded by the BEV decoder to obtain the corresponding bird's-eye view visual markers, namely visual tokens. Visual tokens are images containing information such as road boundaries, traffic signs, obstacles, and vehicle positions. While retaining depth information, the image highlights the spatial distribution of each object or obstacle, providing the intelligent driving system with a vehicle-centric bird's-eye view.

[0089] Step S412, encoding the environment image data to obtain a text label;

[0090] It should be noted that the environmental image data can be encoded by a text encoder to obtain corresponding text tags, namely text tokens. Specifically, the environmental image data can be first processed by a convolutional neural network to obtain a high-dimensional feature vector, which contains object type information (such as lane lines, obstacles, traffic lights, etc.), object location information (such as obstacle coordinates, etc.) and object state information (such as stationary or moving) in the environmental image data. The high-dimensional feature vector is input into the text encoder, and the high-dimensional feature vector is mapped to text tokens to obtain a natural language description of the image, such as "There is a parked vehicle 50 meters ahead."

[0091] Step S413: determining the path planning language information according to the bird's-eye view visual mark and the text mark.

[0092] It should be understood that by inputting the bird's-eye view visual markers (i.e., visual tokens) and text markers (i.e., text tokens) into the pre-trained large language model (Large Language Model, LLM), the corresponding path planning language information will be obtained. The large language model includes a self-attention mechanism and a multi-layer perceptron (Multi-Layer Perceptron, MLP). Specifically, LLM can process the input bird's-eye view visual markers and text markers through the self-attention mechanism, identify the semantic association between the bird's-eye view visual markers and the text markers, and assign corresponding weights to perform feature fusion to obtain fusion features. The fusion features are successively nonlinearly transformed and feature mapped by the multi-layer perceptron, and path planning language information that is easy to understand and explain is generated at the end of the large language model. The large language model based on vision and semantics can process and interpret natural language instructions, improve the generalization ability of the intelligent driving system, and enable the intelligent driving system to work in different driving scenarios and conditions. At the same time, its interpretability enhances the transparency of the intelligent driving system, allowing developers and users to understand the decision-making process of the intelligent driving system and increase trust.

[0093] Step S42, obtaining vehicle driving status information, road environment information, obstacle status information, and traffic rules information;

[0094] It should be noted that the vehicle driving state information is the dynamic information of the vehicle during driving (such as vehicle speed, steering angle, acceleration, etc.). The road environment information is the road type information (urban roads, highways, etc.) and road surface type information (such as wet, dry, icy, etc.) of the road section where the vehicle is driving. The obstacle state information includes static obstacles (such as parked vehicles, roadblocks, etc.) and dynamic obstacles (such as pedestrians, other moving vehicles, etc.) on the current road where the vehicle is driving. The traffic rules information is the traffic signs, signal light status, speed limit regulations, etc. on the current road where the vehicle is driving.

[0095] Step S43, generating a candidate trajectory according to the path planning language information, the environmental image data, the vehicle driving state information, the road environment information, the obstacle dynamic information and the traffic rule information.

[0096] It should be noted that after data processing of path planning language information, environmental image data, vehicle driving status information, road environment information, obstacle dynamic information and traffic rule information, and inputting the processed data into a pre-trained end-to-end trajectory planning model (such as a Transformer model), several possible trajectories, namely candidate trajectories, can be obtained. The end-to-end trajectory planning model can realize the complete trajectory planning process from the initial state of the vehicle to the target state, which can be obtained by training the preset Transformer model. Each candidate trajectory is a possible driving path of the vehicle in the future, represented by a series of location information (such as longitude and latitude or coordinate points in a local coordinate system).

[0097] In a feasible implementation, step S43 may include steps S431 to S432:

[0098] Step S431, encoding the trajectory planning language information to obtain a trajectory planning vector;

[0099] It should be noted that the trajectory planning vector is a high-dimensional feature vector, and each high-dimensional feature vector represents a specific meta-action. The trajectory planning language information can be encoded into a trajectory planning vector through the meta-action encoder for subsequent calculations. Specifically, the meta-action encoder maps the output of the language model (i.e., the trajectory planning language information) to a set of predefined meta-actions, encodes the meta-actions into high-dimensional feature vectors, and realizes the conversion of meta-actions to trajectory planning feature vectors through a set of learnable embedding vectors (Embeddings). Each meta-action corresponds to an embedding vector, which can be learned in the process of training the meta-action encoder. Predefined meta-actions may include lateral meta-actions and longitudinal meta-actions. Horizontal meta-actions include at least left turn, straight and right turn, etc., and longitudinal meta-actions include at least acceleration, holding, deceleration and stop, etc.

[0100] Exemplarily, the mapping process of meta-actions is explained by taking the trajectory planning language information "go straight", "turn left" or "avoid pedestrians" as an example: "go straight" is mapped into the lateral meta-action "go straight" and the longitudinal meta-action "keep the speed constant"; "turn left" is mapped into the lateral meta-action "turn left" and the longitudinal meta-action "slow down"; "avoid pedestrians" is mapped into the sequential execution of the lateral meta-action "turn right" and the longitudinal meta-action "slow down", the lateral meta-action "turn left" and the longitudinal meta-action "keep the speed constant", the lateral meta-action "turn left" and the longitudinal meta-action "keep the speed constant", the lateral meta-action "turn right" and the longitudinal meta-action "keep the speed constant", and the lateral meta-action "go straight" and the longitudinal meta-action "accelerate".

[0101] Step S432, generating a candidate trajectory according to the trajectory planning vector, the environmental image data, the vehicle driving state information, the road environment information, the obstacle dynamic information and the traffic rule information.

[0102] It should be understood that after obtaining the trajectory planning vector corresponding to the meta-action mapped by the trajectory planning vector, the trajectory planning vector, environmental image data, vehicle driving state information, road environment information, obstacle dynamic information and traffic rule information are input into a pre-trained end-to-end trajectory planning model (such as a Transformer model), and several possible trajectories, namely candidate trajectories, can be obtained.

[0103] Step S50, determining a target trajectory from the candidate trajectories according to a target cost function to complete intelligent driving trajectory prediction.

[0104] It should be understood that the target cost function is a mathematical function designed based on multiple factors (such as safety, driving comfort, traffic rule compliance, obstacle avoidance, etc.), which can calculate the cost of each candidate trajectory and then select the optimal trajectory. After obtaining the target cost function score of each candidate trajectory, an optimization algorithm (gradient descent) can be used to find the trajectory that minimizes the cost function and select the optimal trajectory from the predicted trajectories.

[0105] This embodiment provides a method for predicting intelligent driving trajectories, which includes obtaining environmental laser radar data, environmental radar data, and environmental image data; converting the environmental laser radar data, the environmental radar data, and the environmental image data into bird's-eye view features; aligning the bird's-eye view features based on the sensor field of view angle identifier to obtain multimodal perception features; determining candidate trajectories based on the multimodal perception features; and determining the target trajectory from the candidate trajectories based on the target cost function to complete intelligent driving trajectory prediction. By integrating data from different sensors such as laser radar, radar, and camera, a richer and more accurate environmental perception is provided, the data of each sensor is converted into a bird's-eye view feature, and the features are aligned using the field of view angle of the sensor to obtain a multimodal perception feature containing perspective information from different sensors, which can comprehensively describe the behavior of traffic flow, pedestrians, obstacles, and other vehicles in the vehicle driving environment. Based on these multimodal perception features, potential trajectories, i.e., candidate trajectories, are identified and predicted, and the optimal trajectory is selected through the target cost function optimization, so as to realize the intelligent driving system's accurate understanding and prediction of complex driving environments and provide richer and more accurate environmental perception. The solution of the present application enables the intelligent driving system to better understand the complex driving environment, including traffic flow, pedestrians, obstacles and the behavior of other vehicles, in order to predict the optimal driving trajectory.

[0106] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 2 , step S50 may include steps S51 to S52:

[0107] Step S51, constructing a target cost function based on safety cost, comfort cost, compliance cost, prediction uncertainty cost and path continuity cost;

[0108] It should be understood that the safety cost is used to evaluate the potential risks and collision possibilities in the candidate trajectory, including the assessment of the risk of collision between the vehicle and obstacles, pedestrians, other vehicles, etc. in the candidate trajectory. The comfort cost is used to evaluate the comfort felt by passengers during the vehicle's driving process, including the assessment of the vehicle's sudden braking, sudden acceleration, or sharp steering in the candidate trajectory. The compliance cost is used to evaluate the degree to which the vehicle's driving behavior in the candidate trajectory complies with traffic rules (such as speed limits, traffic lights, traffic signs, lane keeping, etc.), including the assessment of the vehicle's violation of traffic signals, speeding, irregular lane changes, etc. in the candidate trajectory. The prediction uncertainty cost is used to evaluate the errors introduced by environmental factors and the behavior of other traffic participants in the trajectory prediction process, including the assessment of the prediction errors of other vehicles and pedestrians in the candidate trajectory, the uncertainty of road condition changes, etc. The path continuity cost is used to evaluate the smoothness and continuity of the vehicle's driving trajectory, including the assessment of sudden changes in the trajectory, too sharp turns, and too high path curvature in the candidate trajectory.

[0109] It should be noted that the safety cost, comfort cost, compliance cost, prediction uncertainty cost and path continuity cost can be calculated for the candidate trajectory through vehicle body information (such as vehicle speed, steering angle, acceleration, etc.). According to the safety cost, comfort cost, compliance cost, prediction uncertainty cost and path continuity cost, a cost function C, namely the target cost function, can be constructed and defined to quantify the pros and cons of each candidate trajectory. The construction formula of the target cost function C is as follows:

[0110] C=λ1·C safety +λ2·C comfort +λ3·C compliance +λ4·C uncertainty +λ5·C continuity

[0111] Among them, λ i is the weight of the i-th cost factor, reflecting the importance of this factor in the total cost. safety Used to represent security cost; C comfort Used to represent comfort cost; C complianceUsed to represent compliance costs; C uncertainty Used to represent the forecast uncertainty cost; C continuity Used to represent path continuity cost.

[0112] Step S52, determining a target trajectory from the candidate trajectories according to the target cost function to complete intelligent driving trajectory prediction.

[0113] It should be understood that after obtaining the scores of all candidate trajectories and their corresponding safety costs, comfort costs, compliance costs, prediction uncertainty costs, and path continuity costs, the optimal trajectory can be screened out from the candidate trajectories through an optimization algorithm. Specifically, the optimization algorithm can be gradient descent. After setting the initial trajectory and the target cost function C, the trajectory is updated along the direction of the cost function descent by calculating the gradient. Each iteratively updated trajectory will be adjusted in the direction of reducing the cost. Among all the generated candidate trajectories, after the above optimization process, the trajectory that minimizes the cost function is the optimal trajectory, that is, the target trajectory.

[0114] This embodiment provides an intelligent driving trajectory prediction method, which constructs a target cost function based on safety cost, comfort cost, compliance cost, prediction uncertainty cost and path continuity cost, and determines a target trajectory from the candidate trajectories according to the target cost function to complete the intelligent driving trajectory prediction. By constructing an efficient trajectory screening strategy, the intelligent system can quickly identify and select the optimal trajectory, i.e., the target trajectory, from multiple possible trajectories, i.e., candidate trajectories, so that the vehicle can be navigated more effectively during the intelligent driving process, reduce vehicle energy consumption, and improve driving smoothness, safety and comfort.

[0115] For example, in order to help understand the implementation process of the intelligent driving trajectory prediction method obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 3 , Figure 3 A brief flowchart of an intelligent driving trajectory prediction method is provided, specifically:

[0116] The three sensors of Lidar, Radar and Camera collect data about the surrounding environment of the vehicle respectively, and obtain environmental Lidar data, environmental radar data and environmental image data. The 3D encoder PointPillars can be used to extract features from environmental Lidar data and environmental radar data respectively to obtain corresponding bird's-eye view features. The visual encoder (such as ResNet) and multi-view feature fusion module (such as Transformer model) can be used to extract features from environmental image data to obtain corresponding bird's-eye view features. The bird's-eye view features are aligned to obtain multimodal perception features. The multimodal perception features that are multimodally fused can be decoded by the BEV decoder to obtain corresponding BEV features (i.e., bird's-eye view visual tags), i.e., visual tokens. The environmental image data can be converted into text tokens by the text encoder. The visual tokens and text tokens are input into the large language model LLM together to obtain high-level planning decisions (i.e., path planning language information). The high-level planning decisions are encoded through the meta-action encoder to obtain the trajectory planning vector. The trajectory planning vector can be combined with the vehicle body information (such as vehicle speed, steering angle, acceleration, etc.) to perform end trajectory planning, generate vehicle trajectory (i.e. candidate trajectory), and verify the vehicle trajectory according to the vehicle body information to achieve the optimal trajectory screening and output the final trajectory planning result, i.e. the target trajectory.

[0117] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the intelligent driving trajectory prediction method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0118] This application also provides an intelligent driving trajectory prediction device, please refer to Figure 4 , the intelligent driving trajectory prediction device comprises:

[0119] The data acquisition module 10 is used to acquire environmental laser radar data, environmental radar data and environmental image data;

[0120] A feature extraction module 20, for converting the environmental laser radar data, the environmental radar data and the environmental image data into bird's-eye view features;

[0121] A feature alignment module 30, configured to perform feature alignment on the bird's-eye view features based on the sensor field of view angle identifier to obtain a multimodal perception feature;

[0122] A trajectory generation module 40, configured to determine candidate trajectories according to the multimodal perception features;

[0123] The trajectory prediction module 50 is used to determine the target trajectory from the candidate trajectories according to the target cost function to complete the intelligent driving trajectory prediction.

[0124] In one embodiment, the trajectory generation module 40 is also used to determine path planning language information based on the multimodal perception features and the environmental image data; obtain vehicle driving status information, road environment information, obstacle status information, and traffic rule information; and generate candidate trajectories based on the path planning language information, the environmental image data, the vehicle driving status information, the road environment information, obstacle dynamic information, and the traffic rule information.

[0125] In one embodiment, the trajectory generation module 40 is further used to decode the multimodal perception features to obtain a bird's-eye view visual mark; encode the environmental image data to obtain a text mark; and determine the path planning language information based on the bird's-eye view visual mark and the text mark.

[0126] In one embodiment, the trajectory generation module 40 is also used to encode the trajectory planning language information to obtain a trajectory planning vector; and generate a candidate trajectory according to the trajectory planning vector, the environmental image data, the vehicle driving state information, the road environment information, the obstacle dynamic information and the traffic rule information.

[0127] In one embodiment, the feature alignment module 30 is also used to obtain the sensor field of view angle identifiers corresponding to the environmental lidar data, the environmental radar data, and the environmental image data, and obtain the lidar field of view angle identifier, the radar field of view angle identifier, and the image field of view angle identifier; based on the lidar field of view angle identifier, the radar field of view angle identifier, and the image field of view angle identifier, feature alignment is performed on the bird's-eye view features to obtain multimodal perception features.

[0128] In one embodiment, the feature alignment module 30 is also used to assign modal weights to the bird's-eye view features based on the laser radar field of view angle identifier, the radar field of view angle identifier, and the image field of view angle identifier; and perform feature alignment on the bird's-eye view features according to the modal weights to obtain multimodal perception features.

[0129] In one embodiment, the trajectory prediction module 50 is also used to construct a target cost function based on safety cost, comfort cost, compliance cost, prediction uncertainty cost and path continuity cost; determine the target trajectory from the candidate trajectories according to the target cost function to complete the intelligent driving trajectory prediction.

[0130] The intelligent driving trajectory prediction device provided by the present application adopts the intelligent driving trajectory prediction method in the above embodiment, which can solve the technical problem of how to accurately understand the complex driving environment in the process of intelligent driving to predict the optimal driving trajectory. Compared with the prior art, the beneficial effects of the intelligent driving trajectory prediction device provided by the present application are the same as the beneficial effects of the intelligent driving trajectory prediction method provided by the above embodiment, and the other technical features in the intelligent driving trajectory prediction device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0131] The present application provides an intelligent driving trajectory prediction device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the intelligent driving trajectory prediction method in the above-mentioned embodiment one.

[0132] Reference below Figure 5 , which shows a schematic diagram of the structure of an intelligent driving trajectory prediction device suitable for implementing an embodiment of the present application. The intelligent driving trajectory prediction device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The intelligent driving trajectory prediction device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0133] like Figure 5As shown, the intelligent driving trajectory prediction device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the intelligent driving trajectory prediction device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the intelligent driving trajectory prediction device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows an intelligent driving trajectory prediction device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0134] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0135] The intelligent driving trajectory prediction device provided by the present application adopts the intelligent driving trajectory prediction method in the above embodiment, which can solve the technical problem of how to accurately understand the complex driving environment in the process of intelligent driving to predict the optimal driving trajectory. Compared with the prior art, the beneficial effects of the intelligent driving trajectory prediction device provided by the present application are the same as the beneficial effects of the intelligent driving trajectory prediction method provided by the above embodiment, and the other technical features in the intelligent driving trajectory prediction device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0136] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0137] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0138] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the intelligent driving trajectory prediction method in the above-mentioned embodiment.

[0139] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0140] The above-mentioned computer-readable storage medium may be included in the intelligent driving trajectory prediction device; or it may exist independently without being assembled into the intelligent driving trajectory prediction device.

[0141] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the intelligent driving trajectory prediction device, the intelligent driving trajectory prediction device:

[0142] Acquire environmental lidar data, environmental radar data, and environmental image data;

[0143] Converting the environmental laser radar data, the environmental radar data, and the environmental image data into bird's-eye view features;

[0144] Performing feature alignment on the bird's-eye view features based on the sensor field of view angle identifier to obtain a multimodal perception feature;

[0145] Determining candidate trajectories according to the multimodal perception features;

[0146] A target trajectory is determined from the candidate trajectories according to a target cost function to complete intelligent driving trajectory prediction.

[0147] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0148] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0149] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0150] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned intelligent driving trajectory prediction method, and can solve the technical problem of how to accurately understand the complex driving environment in the process of intelligent driving to predict the optimal driving trajectory. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the intelligent driving trajectory prediction method provided in the above-mentioned embodiment, and will not be repeated here.

[0151] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned intelligent driving trajectory prediction method when executed by a processor.

[0152] The computer program product provided by this application can solve the technical problem of how to accurately understand the complex driving environment in the process of intelligent driving to predict the optimal driving trajectory. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the intelligent driving trajectory prediction method provided by the above embodiment, which will not be repeated here.

[0153] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. An intelligent driving trajectory prediction method, characterized in that: The intelligent driving trajectory prediction method comprises: Acquire environmental lidar data, environmental radar data, and environmental image data; Converting the environmental laser radar data, the environmental radar data, and the environmental image data into bird's-eye view features; Performing feature alignment on the bird's-eye view features based on the sensor field of view angle identifier to obtain a multimodal perception feature; Determining candidate trajectories according to the multimodal perception features; A target trajectory is determined from the candidate trajectories according to a target cost function to complete intelligent driving trajectory prediction.

2. The method according to claim 1, characterized in that The step of determining candidate trajectories according to the multimodal perception features comprises: Determining path planning language information according to the multimodal perception features and the environmental image data; Obtain the vehicle's driving status information, road environment information, obstacle status information, and traffic rules information; A candidate trajectory is generated according to the path planning language information, the environmental image data, the vehicle driving state information, the road environment information, the obstacle dynamic information and the traffic rule information.

3. The method according to claim 2, characterized in that The step of determining the path planning language information according to the multimodal perception feature and the environmental image data comprises: Decoding the multimodal perception features to obtain a bird's-eye view visual marker; Encoding the environmental image data to obtain a text label; Path planning language information is determined based on the bird's-eye view visual mark and the text mark.

4. The method according to claim 2, characterized in that The step of generating candidate trajectories according to the path planning language information, the environmental image data, the vehicle driving state information, the road environment information, the obstacle dynamic information and the traffic rule information comprises: Encoding the trajectory planning language information to obtain a trajectory planning vector; A candidate trajectory is generated according to the trajectory planning vector, the environmental image data, the vehicle driving state information, the road environment information, the obstacle dynamic information and the traffic rule information.

5. The method according to claim 1, characterized in that The sensor field of view angle identifier includes a laser radar field of view angle identifier, a radar field of view angle identifier and an image field of view angle identifier; The step of aligning the features of the bird's-eye view based on the sensor field of view angle identifier to obtain multimodal perception features includes: Acquire the sensor field of view angle identifiers corresponding to the environmental laser radar data, the environmental radar data, and the environmental image data, and obtain the laser radar field of view angle identifier, the radar field of view angle identifier, and the image field of view angle identifier; The bird's-eye view features are aligned based on the laser radar field of view angle identifier, the radar field of view angle identifier and the image field of view angle identifier to obtain multimodal perception features.

6. The method according to claim 5, characterized in that The feature alignment of the bird's-eye view features is performed based on the laser radar field of view angle identifier, the radar field of view angle identifier and the image field of view angle identifier, The steps of obtaining multimodal perception features include: Assigning a modal weight to the bird's-eye view feature based on the laser radar field of view angle identifier, the radar field of view angle identifier, and the image field of view angle identifier; The bird's-eye view features are aligned according to the modal weights to obtain multimodal perception features.

7. The method according to any one of claims 1 to 6, characterized in that The step of determining the target trajectory from the candidate trajectories according to the target cost function to complete the intelligent driving trajectory prediction includes: Constructing the target cost function based on safety cost, comfort cost, compliance cost, prediction uncertainty cost and path continuity cost; A target trajectory is determined from the candidate trajectories according to the target cost function to complete intelligent driving trajectory prediction.

8. An intelligent driving trajectory prediction device, characterized in that: The device comprises: A data acquisition module, used to acquire environmental laser radar data, environmental radar data and environmental image data; A feature extraction module, used for converting the environmental laser radar data, the environmental radar data and the environmental image data into bird's-eye view features; A feature alignment module, used to align the features of the bird's-eye view based on the sensor field of view angle identifier to obtain a multimodal perception feature; A trajectory generation module, used to determine candidate trajectories according to the multimodal perception features; The trajectory prediction module is used to determine the target trajectory from the candidate trajectories according to the target cost function to complete the intelligent driving trajectory prediction.

9. An intelligent driving trajectory prediction device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the intelligent driving trajectory prediction method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the intelligent driving trajectory prediction method according to any one of claims 1 to 7 are implemented.

Images