Automatic driving model and method based on end-to-end joint optimization

Abstract

The invention discloses an automatic driving model and method based on end-to-end joint optimization, and particularly relates to the field of visual automatic driving, and the method comprises the following steps: processing an input image and a point cloud, and constructing instance features; static and dynamic interaction scene representations based on sparse instances are constructed through query and attention mechanisms; constructing a parallel prediction-planning module based on the tree structure; extracting time sequence information of a historical track to construct a probability space; and modeling and predicting a time sequence dependency relationship of the trajectory through a gating loop unit, and screening an optimal future trajectory by using a confidence score. According to the method, the dynamic feasibility and safety of the planned trajectory can be improved, meanwhile, the calculation overhead is reduced, and an efficient and robust solution is provided for actual landing of the automatic driving technology.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and specifically to an autonomous driving model and method based on end-to-end joint optimization. Background Art

[0002] The application of autonomous driving technology in the transportation field is becoming more and more extensive. Its rapid development and application requirements stem from the continuous pursuit of travel efficiency, safety, and intelligent management. Autonomous driving can not only improve driving safety and reduce traffic accidents, but also enhance traffic efficiency and alleviate urban traffic congestion. With the progress of computer vision, deep learning, and intelligent sensor technologies, autonomous driving has gradually evolved from traditional assisted driving to higher-level autonomous driving, and is gradually moving towards the goal of full automation and full perception, posing higher requirements for real-time processing, environmental understanding, path planning, and control response of vehicles. Among them, accurately predicting future trajectories and generating reasonable planning results are the core issues to ensure safe and stable driving of vehicles.

[0003] Most existing autonomous driving methods start from BEV features, regard trajectory prediction and planning as independent tasks, and ignore the interaction between the two, resulting in predicted trajectories not conforming to feasible planning schemes and affecting the rationality of decision-making. Although some technologies attempt parallel designs to reduce error accumulation, they do not fully utilize environmental information and the interaction relationships between traffic participants, nor do they provide multiple candidate trajectory schemes. Therefore, how to generate more accurate and feasible trajectories through parallel modeling of trajectory prediction and planning has become a key research issue in the current field of autonomous driving decision-making. Summary of the Invention

[0004] Aiming at the deficiencies of the prior art, the present invention provides an autonomous driving model and method based on end-to-end joint optimization, which has the characteristics of accurate trajectory planning, small cumulative error, and low computational resource consumption. It can improve the accuracy and rationality of future trajectory prediction while optimizing computational resources, thereby achieving more reliable autonomous driving decision-making, and is particularly suitable for autonomous driving environments in complex traffic scenarios.

[0005] To achieve the above object, the present invention provides an autonomous driving model and method based on end-to-end joint optimization, including the following steps:

[0006] S1: Process the input image and point cloud to construct instance features;

[0007] S2: Construct static and dynamic interaction scene representations based on sparse instances through query and attention mechanisms;

[0008] S3: Construct a parallel prediction-planning module based on a tree structure;

[0009] S4: Extract the temporal information of historical trajectories to construct a probability space;

[0010] S5: Model the temporal dependencies of the predicted trajectory through a gated recurrent unit and use the confidence score to filter the optimal future trajectory.

[0011] Preferably, in step S1, the processing of the input image and point cloud to construct instance features includes the following steps:

[0012] S11: For the input image sequence, use a convolutional neural network to extract high-level semantic features and generate multi-view 2D image feature maps; for the input point cloud sequence, use a point cloud feature extraction network to encode the sparse point cloud data into voxel features, and then extract spatial features through a 3D convolutional neural network to generate a sparse 3D point cloud feature representation;

[0013] S12: Through a feature fusion network, align the image features with the point cloud features and use geometric transformation to uniformly map them to the BEV space to generate a BEV feature representation of the scene, denoted as B;

[0014] S13: Expand B into a sequence form X = Flatten(B), and add positional encoding X through a single-layer MLP pos = X + PE(X), where PE is the positional encoding;

[0015] S14: Define K learnable embedding vectors, which are automatically learned during training to obtain a learnable query vector Q ∈ R K×C , where K is the maximum number of target instances and C is the query feature dimension;

[0016] S15: Calculate the interaction between B and Q through a cross-attention block to obtain the global instance feature I = CA(Q, X pos , X pos ), where CA represents a cross-attention block composed of interleaved self-attention and cross-attention layers;

[0017] Preferably, in step S2, the construction of a static and dynamic interaction scene representation based on sparse instances through query and attention mechanisms includes the following steps:

[0018] S21: Use a cross-attention block to construct an interaction scene representation of the ego vehicle and the static semantic map based on instances from I, Map = CA(Map0, I, I), where CA represents a cross-attention block composed of interleaved self-attention and cross-attention layers, and Map0 is a randomly initialized learnable map query vector;

[0019] S22: Use a deformable cross-attention block to construct an instance-based representation of the interaction scenario between the ego vehicle and dynamic other agents from I, i.e., Agent = DA(Agent0, I, I), where DA represents the deformable cross-attention block and Agent0 is a learnable agent query vector initialized randomly.

[0020] Preferably, in step S3, the construction of the parallel prediction-planning module includes the following steps:

[0021] S31: The prediction module uses the scenario interaction Map and Agent to make an intermediate estimate of the future states of surrounding dynamic vehicles; meanwhile, the planning module generates a set of candidate ego vehicle driving action plans according to vehicle kinematic constraints, the current ego vehicle state, and the intermediate information from the prediction module.

[0022] S32: Design a bidirectional interaction mechanism based on tree-structured policy planning.

[0023] S33: Use the KL divergence to design the prediction-planning interaction loss L interact , which is a global joint optimization function updated through backpropagation in end-to-end training.

[0024] Preferably, the design of the bidirectional interaction mechanism based on tree-structured policy planning in step S3 includes the following steps:

[0025] S321: Perform autoregressive conditional prediction on each candidate trajectory of the ego vehicle to obtain an estimate of the future states of surrounding vehicles and the scenario under this trajectory, generating tree-like branches, and each branch will obtain a conditional prediction result, forming child nodes in the tree structure.

[0026] S322: Use autoregressive decoding to construct a tree structure for continuous prediction. For each candidate branch, perform autoregressive prediction step by step, and the prediction result of each step is used as the input for the next step to generate a continuous state sequence, and then integrate the branches of each continuous prediction into a tree, with each branch representing a future scenario.

[0027] S323: Implement bidirectional information flow interaction between the ego vehicle planning and the surrounding vehicle prediction in the tree structure, i.e., the upstream information passes through the predicted branches in the tree structure to feedback the state prediction of the surrounding vehicles to the ego vehicle planning module; the downstream information makes a preliminary decision based on the candidate trajectory generated by the ego vehicle planning module to further constrain and correct the conditional prediction.

[0028] S324: For each candidate trajectory branch, use the dynamic programming algorithm to calculate its cumulative cost according to a predefined global cost function. If a certain branch exceeds the rationality threshold in the intermediate stage, then prune this branch, and finally obtain a set of ego vehicle candidate trajectories that are reasonable under the global cost evaluation, and the number is denoted as k.

[0029] Preferably, in step S4, the extraction of the temporal information of the historical trajectory to construct a probability space includes the following steps:

[0030] S41: Extract the historical trajectory s past ={s t-n , …, s t}, input it into the linear transformation layer to convert it into high-dimensional features, where s t represents the state of the vehicle at time t, including information such as position, speed, acceleration, yaw angle, steering angle, etc., and n represents the historical time length;

[0031] S42: Use the gated recurrent unit g to gradually receive the trajectory sequence and calculate the latent state h t =g(s past ) at the current timestamp;

[0032] S43: Through the variational autoencoder map the latent state h t to a latent space to generate the probability distribution of the latent variable z. The encoder assumes that the latent space follows a Gaussian distribution: where μ(h t ) and σ(h t ) represent the mean and variance of z, which are the outputs of the encoder, indicating that the latent space is a parameterized probability space of a Gaussian distribution;

[0033] Preferably, in step S5, the modeling of the temporal dependence of the predicted trajectory through the gated recurrent unit and the screening of the optimal future trajectory using the confidence score include the following steps:

[0034] S51: In the probability space, sample the latent variable z = μ(h t ) + σ(h t )·∈, where ∈ represents the Gaussian distribution. Use the gated recurrent unit g as the future trajectory generator to simulate the time evolution of the instance features. Take the sampled latent variable z and the latent state h t at the current timestamp as inputs and convert them to obtain the state h t+1 =g(z, h t ) at the next timestamp;

[0035] S52: Use the MLP-based waypoint decoder to decode the next waypoint at the (t + 1)-th timestamp according to the state h t+1 to generate the future trajectory, and obtain the distribution of the trajectory set T = {T1, T2, …, T k} in the probability space, where k is the number of trajectories;

[0036] S53: Introduce the mean square error Simulate and predict the similarity between the predicted trajectory and the historical trajectory, and normalize the trajectory distribution in the form of a softmax function where τ is the temperature hyperparameter that controls the smoothness of the distribution;

[0037] S54: Calculate the trajectory score score(T i ) = -log P(T i |s past ) using cross-entropy. The confidence score reflects the relative credibility of the trajectory, and the lower the score, the higher the quality of the trajectory;

[0038] S55: Use a cascaded Transformer decoder to interact with the scenario information Map and Agent. According to the confidence score, select the trajectory T with the lowest score(T i ) from the candidate trajectories * as the true future trajectory of the ego vehicle, T * = arg minscore(T i ).

[0039] The present invention has the following advantages:

[0040] (1) The present invention constructs an end-to-end autonomous driving system architecture centered on sparse instances, which is more suitable for the case of parallel design of prediction and planning, and also represents the driving scenario more accurately and has higher computational efficiency;

[0041] (2) The present invention integrates prediction and planning into an interdependent module, which can improve decision consistency and adaptability to complex scenarios, enabling the model to make more accurate and reliable planning results;

[0042] (3) The present invention simultaneously predicts multiple future trajectories and evaluates them in the probability space, and selects the optimal solution through a trajectory scoring mechanism, thereby improving the accuracy and stability of trajectory planning. Description of the Drawings

[0043] Figure 1 It is a model architecture diagram of an autonomous driving model and method based on end-to-end joint optimization of the present invention. Detailed Embodiments

[0044] The following specifically introduces the present invention in combination with the drawings and specific examples. Those familiar with this technology can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of them. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0045] CombinedFigure 1 , this embodiment mentions an end-to-end jointly optimized autonomous driving model and method, including the following steps:

[0046] First, preprocess the input data to obtain instance features.

[0047] First, use a convolutional neural network to extract high-level semantic features from 2D images to obtain multi-view 2D image feature maps; use a point cloud feature extraction network to extract spatial features from 3D point clouds to obtain a sparse 3D point cloud feature representation. Then, fuse the extracted image and point cloud features through a feature fusion network and uniformly map them to the BEV space through geometric transformation to generate the preliminary BEV feature B of the scene. Next, expand the BEV feature B into a sequence form and add position encoding. Then, define a learnable embedding query vector Q, which will ultimately be mapped to specific object instances, such as vehicles, pedestrians, etc. After that, calculate the interaction between B and Q through a cross-attention block composed of interleaved self-attention and cross-attention layers to obtain the instance feature I.

[0048] Second, construct a static and dynamic interaction scene representation based on sparse instances through query and attention mechanisms.

[0049] First, construct an instance-centered interaction scene representation Map = CA(Map0, I, I) of the ego vehicle and the static semantic map from the instance feature I obtained in the previous step through a cross-attention block composed of interleaved self-attention and cross-attention layers. Then, construct an instance-centered interaction scene representation Agent = DA(Agent0, I, I) of the ego vehicle and dynamic other agents from I through a deformable cross-attention block.

[0050] Third, construct a parallel prediction-planning module.

[0051] First, use the scene intersections Map and Agent obtained in the previous step to make an intermediate estimate of the future states of surrounding dynamic vehicles and generate a set of candidate ego vehicle driving action plans. Then, design a bidirectional interaction mechanism based on tree-structured policy planning. The specific steps for designing a bidirectional interaction mechanism based on tree-structured policy planning are as follows:

[0052] (1) Perform autoregressive conditional prediction on each candidate trajectory of the ego vehicle to obtain an estimate of the future states of surrounding vehicles and the scene under this trajectory, generating tree-like branches, that is, the parallel prediction-planning module is designed as a decision tree, and each node represents a possible future scenario branch, and each branch will obtain a conditional prediction result, forming sub-nodes in the tree structure;

[0053] (2) Construct a tree structure for continuous prediction using autoregressive decoding. For each candidate branch, perform autoregressive prediction step by step. The prediction result of each step is used as the input for the next step to generate a continuous state sequence. Then, integrate the branches of each continuous prediction into a tree, where each branch represents a future scenario.

[0054] (3) Implement two-way information flow interaction between the ego-vehicle planning and the surrounding vehicle prediction in the tree structure. That is, the upstream information feeds back the state prediction of the surrounding vehicles to the ego-vehicle's planning module through the predicted branches in the tree structure. The downstream information makes a preliminary decision based on the candidate trajectories generated by the ego-vehicle planning module to further constrain and correct the conditional prediction. Both two-way interactions are carried out at each branch node, enabling each branch to be continuously updated under the interactive influence of prediction and planning information.

[0055] (4) For each candidate trajectory branch, use the dynamic programming algorithm to calculate its cumulative cost according to a predefined global cost function. If a branch exceeds the rationality threshold in the intermediate stage, prune this branch. After screening multiple candidate trajectories, finally obtain a set of ego-vehicle candidate trajectories that are reasonable under the global cost evaluation, and the number is denoted as k.

[0056] Finally, define the globally optimized KL divergence to design the prediction-planning interaction loss L interact , and update it through backpropagation during the end-to-end training process.

[0057] Fourth, extract the temporal information of the historical trajectory to construct a probability space.

[0058] First, extract the historical trajectory s past = {s t-n , …, s t}, input it into a linear transformation layer to convert it into high-dimensional features, where s t represents the state of the vehicle at time t, including information such as position, speed, acceleration, yaw angle, steering angle, etc., and n represents the length of the historical time. Then, use the gated recurrent unit g to gradually receive the trajectory sequence and calculate the latent state h t = g(s past ). Finally, map the latent state h to a latent space through a variational autoencoder t to generate the probability distribution of the latent variable z. The encoder assumes that the latent space follows a Gaussian distribution: The output of the encoder is the mean μ(h t ) and the variance σ(h t ). This means that given the input h t, the latent space z samples historical trajectories according to a Gaussian distribution. Therefore, the latent space is actually a space of probability distributions. One of the learning objectives of the model is to make the constructed latent space distribution close to the standard normal distribution, thereby discretizing the continuous historical trajectory action space into a probability space.

[0059] Fifth, model the temporal dependencies of the predicted trajectories through a gated recurrent unit and use confidence scores to screen the optimal future trajectories.

[0060] In the probability space obtained in the fourth step, first, sample the latent variable z = μ(h t ) + σ(h t )·∈ based on the Gaussian distribution, where ∈ represents the Gaussian distribution. Use the gated recurrent unit g as the future trajectory generator to simulate the temporal evolution of instance features. Take the sampled latent variable z and the latent state h t at the current timestamp as inputs, and transform to obtain the state h t+1 at the next timestamp = g(z, h t ). Then, for each candidate trajectory output by the parallel prediction-planning module, sequentially input it into the MLP-based waypoint decoder. The decoder decodes the next waypoint at the (t + 1)-th timestamp according to the state h t+1 to generate the future trajectory, and obtain the distribution of the future trajectory set T = {T1, T2,..., T k} in the probability space, where k is the number of trajectories. Next, introduce the mean squared error to simulate the similarity between the predicted trajectory and the historical trajectory, and normalize the trajectory distribution in the form of the softmax function where τ is the temperature hyperparameter that controls the smoothness of the distribution. Then, calculate the trajectory score score(T i ) = -log P(T i |s past ) using cross-entropy. The confidence score reflects the relative credibility of the trajectory, and the lower the score, the higher the quality of the candidate trajectory. Finally, use the cascaded transformer decoder to interact with the scene information Map and Agent, and select the trajectory T i with the lowest score score(T * ) from the k candidate trajectories as the true future trajectory of the ego vehicle, T * = arg min score(T i ).

[0061] The above are only the preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions falling within the concept of the present invention belong to the protection scope of the present invention. It should be noted that for those of ordinary skill in the art, several improvements and refinements made without departing from the principle of the present invention should be regarded as within the protection scope of the present invention.

Claims

1. An end-to-end jointly optimized autonomous driving model and method, characterized in that: It includes the following steps: S1: Process the input image and point cloud to construct instance features; S2: Construct static and dynamic interaction scene representations based on sparse instances through query and attention mechanisms; S3: Construct a parallel prediction - planning module based on a tree structure; S4: Extract the temporal information of the historical trajectory to construct a probability space; S5: Model the temporal dependencies of the predicted trajectory through a gated recurrent unit and use confidence scores to filter the optimal future trajectory.

2. The end-to-end jointly optimized autonomous driving model and method according to claim 1, characterized in that: In step S1, the process of processing the input image and point cloud to construct instance features includes the following steps: S11: For the input image sequence, use a convolutional neural network to extract high - level semantic features and generate multi - view 2D image feature maps; for the input point cloud sequence, use a point cloud feature extraction network to encode sparse point cloud data into voxel features, and then use a 3D convolutional neural network to extract spatial features to generate a sparse 3D point cloud feature representation; S12: Through a feature fusion network, align the image features with the point cloud features and use geometric transformation to uniformly map them to the BEV space to generate a BEV feature representation of the scene, denoted as B; S13: Expand B into a sequence form X = Flatten(B), and add positional encoding X through a single-layer MLP pos = X + PE(X), where PE is the positional encoding; S14: Define K learnable embedding vectors, which are automatically learned during the training process to obtain a learnable query vector Q ∈ R K ×C , where K is the maximum number of target instances and C is the query feature dimension; S15: Calculate the interaction between B and Q through the cross-attention block to obtain the global instance feature I = CA(Q, X pos , X pos ), where CA represents the cross-attention block composed of interleaved self-attention and cross-attention layers.

3. The end-to-end jointly optimized autonomous driving model and method according to claim 2, characterized in that: In step S2, the construction of static and dynamic interaction scene representations based on sparse instances through query and attention mechanisms includes the following steps: S21: Use a cross - attention block to construct an interaction scene representation of the ego - vehicle and the static semantic map based on instances from I, Map = CA(Map0, I, I), where CA represents a cross - attention block composed of interleaved self - attention and cross - attention layers, and Map0 is a learnable map query vector initialized randomly; S22: Use a deformable cross - attention block to construct an interaction scene representation of the ego - vehicle and dynamic other agents based on instances from I, Agent = DA(Agent0, I, I), where DA represents a deformable cross - attention block, and Agent0 is a learnable agent query vector initialized randomly.

4. The end-to-end jointly optimized autonomous driving model and method according to claim 3, characterized in that: In step S3, the construction of a parallel prediction - planning module based on a tree structure includes the following steps: S31: The prediction module uses the scene interactions Map and Agent to make an intermediate estimate of the future states of surrounding dynamic vehicles; meanwhile, the planning module generates a set of candidate ego - vehicle driving action plans according to vehicle kinematic constraints, the current ego - vehicle state, and the intermediate information from the prediction module; S32: Design a bidirectional interaction mechanism based on tree - structured policy planning; S33: Use the KL divergence as the interaction loss L of the prediction-planning module interact , which is a global joint optimization function updated through backpropagation in end-to-end training.

5. The end-to-end jointly optimized autonomous driving model and method according to claim 4, wherein: In step S32, the design of a bidirectional interaction mechanism based on tree - structured policy planning includes the following steps: S321: Perform autoregressive conditional prediction on each candidate trajectory of the ego - vehicle to obtain an estimate of the future states of surrounding vehicles and the scene under this trajectory, generating tree - like branches, and each branch will get a conditional prediction result, forming child nodes in the tree - like structure; S322: Use autoregressive decoding to construct a tree - like structure for continuous prediction. For each candidate branch, perform autoregressive prediction step by step, and the prediction result of each step is used as the input for the next step to generate a continuous state sequence. Then, integrate the branches of each continuous prediction into a tree, and each branch represents a future scenario; S323: Implement bidirectional information flow interaction between the ego vehicle planning and the surrounding vehicle prediction in the tree structure, that is, the upstream information feeds back the state prediction of the surrounding vehicles to the ego vehicle's planning module through the predicted branches in the tree structure; the downstream information makes a preliminary decision based on the candidate trajectories generated by the ego vehicle's planning module, and further constrains and corrects the conditional prediction. S324: For each candidate trajectory branch, use the dynamic programming algorithm to calculate its cumulative cost according to the predefined global cost function. If a certain branch has exceeded the rationality threshold in the intermediate stage, prune this branch. Finally, obtain a set of ego vehicle candidate trajectories that are reasonable under the global cost evaluation, and the number is denoted as k.

6. The end-to-end jointly optimized autonomous driving model and method according to claim 5, characterized in that: In step S4, the extracting the temporal information of the historical trajectories to construct the probability space includes the following steps: S41: Extract the historical trajectory s past = {s t-n , …, s t}, and input it into the linear transformation layer to be converted into high-dimensional features, where s t represents the state of the vehicle at time t, including information such as position, speed, acceleration, yaw angle, steering angle, etc., and n represents the historical time length; S42: Gradually receive the trajectory sequence using the gated recurrent unit g, and calculate the latent state h at the current timestamp t = g(s past ) S43: Through the variational autoencoder Map the latent state h t to a latent space to generate the probability distribution of the latent variable z. The encoder assumes that the latent space follows a Gaussian distribution: where μ(h t ) and σ(h t ) represent the mean and variance of z, which are the outputs of the encoder, indicating that the latent space is a parameterized probability space with a Gaussian distribution.

7. The end-to-end jointly optimized autonomous driving model and method according to claim 6, characterized in that: In step S5, the modeling of the temporal dependence relationship of the predicted trajectories by the gated recurrent unit and the screening of the optimal future trajectories using the confidence score include the following steps: S51: In the probability space, sample the latent variable z = μ(h t ) + σ(h t )·ε, where ε represents the Gaussian distribution. Use the gated recurrent unit g as the future trajectory generator to simulate the temporal evolution of the instance features. Take the sampled latent variable z and the latent state h t at the current timestamp as inputs, and transform them to obtain the state h t+1 at the next timestamp = g(z, h t ); S52: At the (t + 1)-th timestamp, use the MLP-based waypoint decoder to decode the next waypoint according to the state h t+1 to generate the future trajectory, and obtain the distribution of the trajectory set T = {T1, T2, …, T k} in the probability space, where k is the number of trajectories; S53: Introduce the mean squared error Simulate and predict the similarity between the trajectory and the historical trajectory, and normalize the trajectory distribution in the form of the softmax function where τ is the temperature hyperparameter that controls the smoothness of the distribution; S54: Calculate the trajectory score score(T i ) = -log P(T i |s past ). The confidence score reflects the relative credibility of the trajectory, and the lower the score, the higher the quality of the trajectory; S55: Interact with the scene information Map and the Agent using a cascaded Transformer decoder, and select the trajectory T with the lowest confidence score from the candidate trajectories as the true future trajectory of the ego vehicle, where T = arg min score(T). i ) * * i )​​

Images