Method and device for determining route track of automatic driving and medium

Abstract

The invention discloses an automatic driving route track determination method and device and a medium. Belongs to the technical field of automatic driving route trajectory planning. The method comprises the steps of obtaining real-time environment sensing data, vehicle state data and map basic data of a real vehicle; inputting the real vehicle real-time environment perception data, the vehicle state data and the map basic data into an automatic driving model to obtain a plurality of candidate route tracks with confidence output by the model; wherein the automatic driving model comprises a decision planning branch related to route trajectory planning, and the decision planning branch is obtained by pre-training based on an offline reinforcement learning training method and double-source training data; a target route trajectory for automated driving execution is determined based on the candidate route trajectories of the plurality of stripe confidence. The technical effect of improving the route planning adaptability and accuracy is achieved.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving route trajectory planning technology, and in particular to methods, devices and media for determining autonomous driving routes and trajectories. Background Technology

[0002] The field of autonomous driving is gradually evolving from the traditional modular architecture (perception-prediction-planning-control) to an end-to-end integrated architecture. An end-to-end autonomous driving system is an intelligent computing system that takes raw sensor data (such as camera images and LiDAR data) as input, fully utilizes the perceived information, and outputs planned trajectories or control commands through neural network models to achieve global optimization of autonomous driving. End-to-end autonomous driving systems reduce redundant information transmission and error accumulation between multiple modules, react more quickly to complex dynamic scenarios, and achieve a high degree of integration of different functions in the development process.

[0003] Currently, common end-to-end autonomous driving systems are typically modular end-to-end systems. The perception module no longer outputs results based on human understanding, but instead provides feature vectors. Correspondingly, the integrated model of the prediction, decision-making, and planning modules outputs motion planning results based on these feature vectors. In terms of training, these models must support gradient propagation across modules; the two modules cannot be trained independently and must be trained simultaneously. This results in weak generalization ability, inaccurate trajectory evaluation, and high training complexity. Summary of the Invention

[0004] This invention provides a method, device, and medium for determining the route trajectory of autonomous driving, in order to solve the technical problems of weak generalization ability, inaccurate trajectory evaluation, and high training complexity of traditional methods.

[0005] According to one aspect of the present invention, a method for determining the route trajectory of autonomous driving is provided, the method comprising:

[0006] Acquire real-time environmental perception data, vehicle status data, and basic map data for the actual vehicle;

[0007] The real-time environmental perception data of the actual vehicle, the vehicle status data, and the basic map data are input into the autonomous driving model to obtain multiple candidate route trajectories with confidence scores output by the model; wherein, the autonomous driving model includes a decision planning branch related to route trajectory planning, and the decision planning branch is pre-trained based on offline reinforcement learning training method and dual-source training data;

[0008] The target route trajectory for autonomous driving execution is determined based on multiple candidate route trajectories with confidence scores.

[0009] According to another aspect of the present invention, an autonomous driving route trajectory determination device is provided, the device comprising:

[0010] The data acquisition module is used to acquire real-time environmental perception data of the actual vehicle, vehicle status data, and basic map data;

[0011] The candidate route output module is used to input the real-time environmental perception data of the actual vehicle, the vehicle status data, and the basic map data into the autonomous driving model to obtain multiple candidate route trajectories with confidence scores output by the model; wherein, the autonomous driving model includes a decision planning branch related to route trajectory planning, and the decision planning branch is pre-trained based on offline reinforcement learning training method and dual-source training data;

[0012] The target route determination module is used to determine the target route trajectory for autonomous driving execution based on multiple candidate route trajectories with confidence scores.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] At least one processor;

[0015] and memory that is communicatively connected to at least one processor;

[0016] The memory stores a computer program that can be executed by at least one processor, which enables the at least one processor to execute the autonomous driving route trajectory determination method of any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute and implement the route trajectory determination method for autonomous driving according to any embodiment of the present invention.

[0018] The technical solution of this invention acquires real-time environmental perception data, vehicle status data, and basic map data; inputs these data into an autonomous driving model to obtain multiple candidate route trajectories with confidence scores. The autonomous driving model includes a decision-making branch related to route planning, which is pre-trained using offline reinforcement learning and dual-source training data. Based on these multiple candidate route trajectories with confidence scores, a target route trajectory for autonomous driving execution is determined. This solves the technical problems of weak generalization ability, inaccurate trajectory evaluation, and high training complexity in traditional methods, achieving improved adaptability, accuracy, and training efficiency in route planning.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart illustrating a method for determining the route trajectory of autonomous driving, provided as an embodiment of the present invention;

[0022] Figure 2a A flowchart of another method for determining the route trajectory of autonomous driving provided in an embodiment of the present invention;

[0023] Figure 2b A flowchart of the real vehicle incremental update data processing for an autonomous driving model, which is another method for determining the route trajectory of autonomous driving provided in an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of the structure of an autonomous driving route trajectory determination device provided in an embodiment of the present invention;

[0025] Figure 4 A schematic diagram of the structure of an electronic device for implementing an autonomous driving route trajectory determination method according to an embodiment of the present invention. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] Figure 1 This is a flowchart illustrating a method for determining the route trajectory of autonomous driving according to an embodiment of the present invention. This embodiment is applicable to the determination of the route trajectory of autonomous driving. The method can be executed by an autonomous driving route trajectory determination device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method specifically includes the following steps:

[0029] S110: Acquire real-time environmental perception data, vehicle status data, and basic map data of the actual vehicle.

[0030] Among these, real-time vehicle environmental perception data can be understood as the currently collected information about the surrounding environment. Vehicle status data can be understood as the vehicle's current operating status information. Basic map data can be understood as the basic map information required for vehicle operation.

[0031] Preferably, the real-time environmental perception data includes images of the vehicle's surroundings captured by the vehicle's surround-view cameras, such as road markings, traffic lights, pedestrians, non-motorized vehicles, and surrounding vehicles. Real-time dynamics of surrounding vehicles are obtained through the fusion of sensors such as LiDAR, millimeter-wave radar, and cameras, including vehicle position, speed, acceleration, direction of travel, relative distance to the vehicle, and relative orientation. Vehicle status data includes the vehicle's current speed, position information, and steering angle information. Basic map data includes lane distribution data, intersection signs, and road speed limits.

[0032] Specifically, the vehicle uses sensors and a mapping system to collect information about the real-time environment around the vehicle, its own status, and a basic map.

[0033] In some possible implementations, the offline reinforcement learning training method employs an objective function with adaptive constraints; the adaptive constraints are KL divergence terms, and the weights of the KL divergence terms are calculated using the safety loss from the collision probability of the vehicle during the training process.

[0034] The adaptive constraint term can be understood as a term in the objective function whose constraint strength is automatically adjusted according to certain conditions. The KL divergence term can be understood as a term used to measure the difference between two probability distributions. The weight of the KL divergence term can be understood as a coefficient used to adjust the degree of influence of the KL divergence term in the objective function. The safety loss due to collision probability can be understood as a loss value that measures the magnitude of the risk of a collision during vehicle training; the higher the collision probability, the greater the safety loss.

[0035] In some possible implementations, the real-time environmental perception data of the vehicle includes surround view images collected by onboard sensors and dynamic information of surrounding vehicles; the vehicle status data includes the vehicle's current speed, position, and steering angle information; and the map base data includes lane line distribution, intersection signs, and road speed limit information.

[0036] Among these, onboard sensors can be understood as various sensing devices installed on autonomous vehicles. Surrounding vehicle dynamics information can be understood as information such as the speed, direction, and position changes of other vehicles around the vehicle. Current vehicle speed can be understood as the vehicle's current speed. Current location can be understood as the vehicle's current geographical location. Steering angle information can be understood as the steering angle of the vehicle's steering wheels. Lane markings can be understood as the arrangement, number, and type of lane markings on the road. Intersection markings can be understood as traffic signs, markings, and other signage information at intersections. Road speed limit information can be understood as the maximum speed limit stipulated on the road.

[0037] S120. Input the real-time environmental perception data of the actual vehicle, the vehicle status data and the basic map data into the autonomous driving model to obtain multiple candidate route trajectories with confidence scores output by the model; wherein, the autonomous driving model includes a decision planning branch related to route trajectory planning, and the decision planning branch is pre-trained based on offline reinforcement learning training method and dual-source training data.

[0038] In this context, the autonomous driving model can be understood as a decision-making model that outputs corresponding driving decisions or routes based on input data. The decision-making and planning branch can be understood as the part of the autonomous driving model responsible for route planning. Offline reinforcement learning training methods can be understood as training methods that optimize the model through trial and error by the agent interacting with the environment in a non-real-time environment, using reward feedback. Dual-source training data can be understood as training data from two different sources. Candidate route trajectories can be understood as possible driving routes, each with its own level of confidence in being selected.

[0039] Specifically, the three types of collected data are input into the autonomous driving model. The decision-making and planning branches in the model generate multiple possible routes and the confidence level of each route based on the capabilities obtained in advance through offline reinforcement learning and dual-source training data, which facilitates the subsequent selection of the optimal route.

[0040] In some possible implementations, before inputting the real-time environmental perception data of the actual vehicle, the vehicle status data, and the map basic data into the autonomous driving model, the following steps are also included:

[0041] Obtain a pre-trained end-to-end autonomous driving model, divide the autonomous driving model into an environmental perception branch and a decision planning branch related to route trajectory planning, and freeze the parameters of the environmental perception branch.

[0042] Acquire dual-source training data, and divide the dual-source training data into a training set and a test set;

[0043] The decision planning branch is iteratively trained based on a preset number of training iterations and a training set. In each training iteration, the current policy network parameters are copied to the reference network. Each training batch is traversed. In each batch, the old policy network is updated to the current policy network parameters first, and then real vehicle data is input to generate candidate training trajectories.

[0044] The pre-trained reward model is called to calculate the reward value of the candidate training trajectory. The advantage function value is determined by a preset standardized formula. The loss is calculated based on the objective function with adaptive KL divergence constraint. The decision planning branch parameters are updated by mini-batch gradient descent to complete single-batch optimization.

[0045] After each iteration, the current policy network is validated using a test dataset. The output trajectory is scored using a reward model. If the trajectory score is not lower than a preset score threshold, the completed decision planning branch is determined as the decision planning branch for route trajectory planning in the autonomous driving model.

[0046] In this context, a pre-trained end-to-end autonomous driving model can be understood as a complete autonomous driving model that has undergone preliminary training and can directly receive environmental inputs and output driving control commands. For example, an end-to-end autonomous driving model includes the VADv2 model. The environmental perception branch can be understood as the part of the autonomous driving model responsible for perceiving the surrounding environment, processing sensor data to identify objects in the environment, etc. The training set can be understood as the dataset used for parameter adjustment during model training. The test set can be understood as the dataset used to evaluate the model's training effectiveness. The policy network can be understood as the network model used to generate route trajectories in reinforcement learning. The reference network can be understood as a benchmark network whose parameters are periodically copied from the current policy network. The old policy network can be understood as the network that records the parameters of the current policy network before the update in each training batch. The reward model can be understood as the model used to score the generated route trajectories and provide reward values. The advantage function value can be understood as a numerical value used to measure the advantage of a particular route trajectory relative to other trajectories, helping the model learn better. The adaptive KL divergence constraint term can be understood as a term in the objective function used to limit the difference between the current policy network and the old policy network. Mini-batch gradient descent can be understood as an optimization algorithm.

[0047] Specifically, after acquiring the end-to-end autonomous driving model, it is divided into two parts: environment perception and route planning. Freezing the parameters of the perception part avoids affecting the already trained perception capabilities, allowing focus on optimizing route planning capabilities. Training data from two sources is collected and divided into training and testing parts. In some special cases, the two data sources can be the same. The training set is used for model learning, and the test set is used for objectively evaluating model performance. Following a set number of training iterations, the parameters of the reference network and the current policy network are synchronized each time, followed by batch training. After each batch updates the old policy network, candidate trajectories are generated. Through multiple iterations, the model gradually improves its route planning capabilities. A reward model is used to score the candidate trajectories, and after calculating the advantage function value, the parameters are updated based on an objective function with adaptive constraints to make the model-generated trajectories better. After each round of training, the model is tested using the test set. If the trajectory score meets the requirements, the model training effect is satisfactory and can be used as the final part for route planning.

[0048] In some possible implementations, the dual-source training data includes a real-vehicle historical driving dataset and a public benchmark dataset; the real-vehicle historical driving dataset contains historical surround view images, historical vehicle states, and corresponding historical driving trajectory labels under different scenarios; the public benchmark dataset contains a closed-loop simulation framework, a multi-dimensional trajectory scoring mechanism, map template data, and interaction trajectory records between the vehicle and surrounding vehicles.

[0049] The dataset consists of several key components: Real-world vehicle history datasets (which can be understood as collections of data recorded during past driving of autonomous vehicles), publicly available benchmark datasets (which are standard datasets used to evaluate and train autonomous driving models), historical surround-view images (images of the surrounding environment captured by surround-view cameras during real-world vehicle driving), historical vehicle state (information on the vehicle's own state recorded during real-world vehicle driving), historical driving trajectory labels (marks of the routes actually taken by the vehicle during real-world vehicle driving), closed-loop simulation framework (a simulation environment within the publicly available benchmark dataset that simulates the vehicle's driving process in various scenarios and forms a closed-loop feedback loop), multi-dimensional trajectory scoring mechanism (rules and methods for evaluating driving trajectories from multiple perspectives), map template data (standard map template information within the publicly available benchmark dataset), and interaction trajectory records between the vehicle and surrounding vehicles (information on the vehicle's own trajectory during interactions with other vehicles recorded in the publicly available benchmark dataset).

[0050] In some possible implementations, before calling the pre-trained reward model to calculate the reward value of the candidate training trajectory, the following steps are also included:

[0051] The closed-loop simulation framework of the publicly available benchmark dataset is invoked to obtain the map template data and the interaction trajectory records of surrounding vehicles in the dataset, and the reference trajectory of the vehicle in different scenarios is generated based on the preset simple planner.

[0052] Trigger the multi-dimensional trajectory scoring mechanism of the public benchmark dataset, score each reference trajectory from preset dimensions, form paired data of reference trajectories and multi-dimensional scores and store them in the training buffer;

[0053] Using the reference trajectory in the paired data as the input feature and the corresponding multi-dimensional score mean as the true value label, a neural network model is constructed, and supervised learning training is performed using the mean squared error loss function and gradient descent optimization algorithm.

[0054] After each preset training epoch, a validation dataset is extracted and input into the model. Training ends when the error between the model's output score and the ground truth label is less than a preset threshold, thus obtaining the reward model. It is understood that during training, training stops when the average error on the dataset, or when the error on the dataset no longer decreases over multiple consecutive testing epochs.

[0055] The setup includes: a pre-defined simple planner (which can be understood as a simple algorithm used to generate a basic reference trajectory); a reference trajectory (which can be understood as a driving trajectory used as a reference standard); pre-defined dimensions (which can be understood as multiple aspects of the trajectory being scored, such as safety and comfort); a training buffer (which can be understood as a storage area used to temporarily store training data); input features (which can be understood as the reference trajectory data input to the model); ground truth labels (which can be understood as the true scores used for model learning corresponding to the input features); a neural network model (which can be understood as a network structure composed of multiple connected neurons); a mean squared error loss function (which can be understood as a function used to measure the difference between the model output and the ground truth labels); a gradient descent optimization algorithm (which can be understood as an optimization method to reduce the value of the loss function); and a validation dataset (which can be understood as a dataset used to evaluate the model's generalization ability during training).

[0056] Specifically, using the simulation framework, maps, and vehicle interaction records in the dataset, a simple planner generates reference trajectories for different scenarios, providing foundational data for training the reward model. The reference trajectories are scored according to preset dimensions, forming paired data of trajectories and scores, which are stored in a buffer. Using the reference trajectories as input and the mean score as the label, the model is trained using a specified loss function and optimization algorithm, enabling the model to learn to accurately score trajectories. After a certain number of training epochs, the model is validated using a validation set. If the error between the output score and the true value is small, the reward model is considered successfully trained and can be used for subsequent scoring of candidate trajectories.

[0057] S130. Determine the target route trajectory for autonomous driving execution based on multiple candidate route trajectories with confidence scores.

[0058] The target route trajectory can be understood as the driving route determined from the candidate route trajectories and ultimately used by the autonomous vehicle for actual execution.

[0059] Specifically, taking into account factors such as the confidence level of each candidate route, the most suitable route for the current scenario is selected from multiple candidate routes as the actual route for the vehicle to drive, ensuring safe and efficient vehicle operation.

[0060] In some possible implementations, the determination of the target based on multiple candidate route trajectories with confidence levels...

[0061] The route trajectory includes:

[0062] The reward model is invoked to score each candidate route trajectory to obtain a reward model score;

[0063] For each candidate route trajectory, a comprehensive score is determined based on the confidence level and the reward model score.

[0064] The target route trajectory for autonomous driving execution is determined based on the comprehensive scores from multiple sources.

[0065] Specifically, a pre-trained reward model is used to evaluate and score each candidate route trajectory generated by the model. A comprehensive score is calculated by combining the confidence level of each candidate route with the score given by the reward model. Based on the comprehensive scores of all candidate routes, the optimal route is selected as the actual route executed by the vehicle, ensuring the rationality and safety of the vehicle's driving.

[0066] Preferably, if there are at least two candidate route trajectories with the highest comprehensive score and the score difference is less than a preset threshold, the weights are further adjusted based on the current scenario type of the vehicle. Specifically, the weight of safety score is increased in urban congestion scenarios, and the weight of driving efficiency score is increased in highway scenarios. Finally, the route trajectory with the highest comprehensive score after adjustment is selected as the target route trajectory.

[0067] The technical solution of this invention acquires real-time environmental perception data, vehicle status data, and basic map data; inputs these data into an autonomous driving model to obtain multiple candidate route trajectories with confidence scores. The autonomous driving model includes a decision-making branch related to route planning, which is pre-trained using offline reinforcement learning and dual-source training data. Based on these multiple candidate route trajectories with confidence scores, a target route trajectory for autonomous driving execution is determined. This solves the technical problems of weak generalization ability, inaccurate trajectory evaluation, and high training complexity in traditional methods, achieving improved adaptability, accuracy, and training efficiency in route planning.

[0068] Figure 2a This is a flowchart of another method for determining the route trajectory of autonomous driving provided by an embodiment of the present invention. Based on the above embodiments, this embodiment is an optimization of the above embodiments, and its specific implementation can be found in the technical solution of this embodiment. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here. Figure 2a As shown, the method specifically includes the following steps:

[0069] S210. Collect environmental status information and trajectory data during the actual vehicle execution process, and call the reward model to score the trajectory data to form a new training dataset.

[0070] Specifically, the trained reward model is used to evaluate and score each candidate route trajectory generated by the model, providing a basis for subsequent route selection.

[0071] Among these, environmental state information can be understood as the surrounding environment data collected in real time by onboard sensors during the autonomous driving process. Trajectory data can be understood as the actual driving path data generated by the vehicle during driving. The new training dataset can be understood as the dataset used for secondary fine-tuning of the autonomous driving model.

[0072] Specifically, during normal autonomous driving, the vehicle continuously collects dynamic information about the surrounding environment and its own trajectory data through sensors. At the same time, the collected trajectory data is input into a pre-trained reward model, which scores the trajectory from dimensions such as safety and comfort. The environmental state information, trajectory data, and scores are linked and integrated to form a new training data set.

[0073] S220. Based on the new training dataset, the decision planning branch of the autonomous driving model is fine-tuned a second time to generate a parameter incremental update package containing only the optimization parameters of the decision planning branch.

[0074] The parameter incremental update package can be understood as a data package containing the optimized parameters of the decision-planning branches of the autonomous driving model.

[0075] Specifically, according to the offline reinforcement learning training process, the parameters of the model's decision planning branch are updated using the new training dataset, and only the optimized parameters of that branch are retained, without involving the frozen perception branch parameters, and packaged into a small-sized parameter incremental update package with high transmission efficiency.

[0076] S230. Perform a safety test on the parameter incremental update package. After the test is passed, update the autonomous driving model based on the parameter incremental update package.

[0077] Safety testing can be understood as testing to verify whether autonomous driving safety standards are met.

[0078] Specifically, a simulated test environment and a small-scale real-vehicle test scenario are set up. The incremental parameter update package is loaded into the test model to verify the driving safety of the updated model in various scenarios. If the test results meet the preset safety standards, the update package is pushed to the real-vehicle autonomous driving system to complete the incremental update of the model parameters. The preset safety standards can be pre-set based on experience, and this embodiment does not impose specific restrictions on them.

[0079] The technical solution of this invention addresses the issues of discrepancies between offline training models and real driving scenarios, high resource consumption during full model updates, and the risk of interrupting real vehicle services through a real vehicle incremental update process. It enables continuous optimization of autonomous driving models based on real vehicle data without interrupting autonomous driving services, thereby improving the model's adaptability to real-world scenarios, reducing resource consumption and safety risks associated with model updates, and ensuring the long-term stable evolution of the autonomous driving system.

[0080] Figure 2b A flowchart illustrating the incremental update data processing of the autonomous driving model for a real vehicle, as provided in another embodiment of the present invention for determining the route trajectory of autonomous driving; Figure 2b As shown, the method includes: evaluating trajectory performance (closed-loop performance test results) based on the actual vehicle execution effect, and collecting environmental state information (surround view image data), trajectory data, and trajectory evaluation values ​​(calculated using a reward model) as a new fine-tuning training dataset. Based on the new training dataset, offline model fine-tuning based on reinforcement learning (offline RL fine-tuner) is performed. After fine-tuning, safety testing is conducted. Once safety standards are met, an incremental parameter update package is generated and updated to the onboard end-to-end autonomous driving system for use in real-world autonomous driving.

[0081] In a preferred embodiment of the present invention, this embodiment is based on the existing end-to-end autonomous driving model—VADv2 model, which can output multiple trajectory information with confidence scores. Based on the pre-trained end-to-end autonomous driving model, offline reinforcement learning methods are used to further train and fine-tune the decision planning part using real-world vehicle datasets to optimize its performance on trajectory planning tasks.

[0082] Preferably, fine-tuning of offline reinforcement learning models based on real-vehicle datasets includes:

[0083] (1) Design the reinforcement learning objective function

[0084] The goal of reinforcement learning is to maximize the cumulative reward (i.e., the payoff). Mathematically, the objective function can be expressed as maximizing the expected cumulative reward, as shown in the following equation. The training process of a reinforcement learning model involves maximizing the following objective function. To optimize the policy network model.

[0085]

[0086] The KL divergence (non-negativity) of the last term is used to limit the step size of policy updates, ensuring that the new policy does not differ too much from the old policy. KL divergence calculation method:

[0087]

[0088] Coefficient of KL divergence term In this invention, the weights are optimized to be adaptive, ensuring that they automatically increase when gradient direction safety is low. To reduce the policy update step size:

[0089]

[0090] in, For safety losses (collision probability).

[0091] Preferably, the objective function involves three networks: the current policy network. Old strategy network and reference network The parameters of the current policy network are the latest model parameters after policy gradient optimization in the current step. The old policy network has the model parameters before this policy gradient optimization, and the reference network has the parameters of the pre-trained end-to-end autonomous driving model. Although three networks are mentioned, only one policy network model can actually be stored. The old policy network and the reference network are policy networks from different iteration optimization cycles, reducing memory usage. 's' represents the environment state information. G is a trajectory of one mode, and G is the number of all trajectory modes.

[0092] Preferred, This is the importance weight (trajectory importance sampling ratio), which represents the probability ratio of the same action under the old and new policies. The reason for using this weight is that when estimating the expected return of the current policy, only samples from the old policy are available, and this weight is used to correct for this distribution mismatch.

[0093] Advantage function The calculation method (the advantage function estimated under the old strategy):

[0094]

[0095] Where r is the reward value of G samples sampled from the old policy network.

[0096] (2) Offline training data sampling method

[0097] Offline real-vehicle training data (vehicle surround view image information and related label information) is read in batches. For each batch of data, it is input into a pre-trained end-to-end autonomous driving model, which outputs planned trajectories for multiple modalities. Then, the reward value is calculated based on the planned trajectories output by the model and the relevant label information in the dataset. Based on the above operations, a set of training data for the reinforcement learning algorithm is collected: state information s (vehicle surround view image information), action value a (multimodal planned trajectory), and reward value r. The above operations are repeated to collect the full training data for the reinforcement learning algorithm.

[0098] (3) Reward model

[0099] Understandably, considering that rule-based reward models are difficult to comprehensively evaluate the planned trajectory output of end-to-end autonomous driving systems, this invention proposes a reward model based on a neural network model. This model is trained under supervision on the public nuPlan dataset to obtain a neural network model for calculating reward values.

[0100] The nuPlan dataset is a large-scale benchmark dataset designed specifically for autonomous driving motion planning research, and it provides a closed-loop evaluation framework with an embedded multi-dimensional scoring mechanism for trajectory ground truth. The core data content of this dataset includes: map information, trajectory records (historical / future trajectories of the vehicle and surrounding vehicles), perception information, behavior labels (lane changes, parking, intersection recognition, etc.), and observed scene types.

[0101] In a preferred embodiment, the process of training the reward model is as follows:

[0102] 1. A closed-loop simulation framework based on nuPlan is used to generate a reference trajectory for the autonomous vehicle using a simple planner, which then interacts with surrounding vehicles. The reference trajectory is scored using nuPlan's scoring system (covering key aspects such as safety, human-likeness, and comfort), and the trajectory data and scoring information are then recorded. ), and store it in the training dataset buffer.

[0103] 2. Based on the obtained training dataset, supervised learning training is performed. The input is trajectory information, and the output is trajectory score. A mapping between trajectory information and trajectory score is established. After training is completed, a reward model can be obtained.

[0104] Specifically, in the application reasoning stage, the input of the reward model is the planned trajectory information output by the end-to-end autonomous driving system, and the output is the score value of the trajectory.

[0105] (4) Reinforcement learning fine-tuning model training process

[0106] Specifically, when fine-tuning the model, it is necessary to traverse the entire training dataset (the full dataset, each traversal is called completing 1 epoch). In order to facilitate memory management and introduce randomness, the dataset is divided into small batches (partial datasets, each traversal is called completing 1 batch).

[0107] In this embodiment, the current policy model is updated using single-sample or mini-batches of data. The old policy model is updated once after each batch is completed, and the reference model is updated once after each epoch. The specific process is as follows:

[0108] First, prepare a pre-trained end-to-end autonomous driving model as the policy network model. The perceptual branch parameters in the policy network model are frozen and do not participate in training updates. Then, during each epoch iteration, the reference model is updated once. Then, iterate through multiple batches of training data, updating the old model before each small batch of data. Assign to Using the old policy network model Multiple sets of trajectories with G modalities are generated. A reward model is then used to calculate the reward value for each trajectory in each set. Based on this reward value, the advantage function is calculated and used to calculate the objective function value. Finally, for each training sample in the mini-batch training data, the objective function value is calculated, and the parameter update amount is calculated using a policy gradient optimization algorithm to update the model. The process repeats until the specified number of iterations is reached.

[0109] (5) Optimize some model parameters

[0110] Specifically, when using reinforcement learning for fine-tuning training, the perception-related model structure needs to be frozen, and only branches related to the vehicle's features need to be adjusted, including the interaction modules between the vehicle and surrounding vehicles and the map, and the trajectory modality classification branch.

[0111] The technical solution of this invention improves the decision-making and planning capabilities of end-to-end autonomous driving systems and the overall generalization ability of the models through a reinforcement learning-based model fine-tuning method. The neural network-based reward model provides more comprehensive and accurate reward values, which helps improve the training speed and inference performance of the reinforcement learning model. By abandoning the evaluation network, algorithm complexity and memory usage are reduced. Incremental updates to the onboard model compensate for the performance deficiencies of offline reinforcement learning models, optimizing the real-world application effect of end-to-end autonomous driving systems. Furthermore, the elimination of full model retraining reduces resource consumption.

[0112] Figure 3 This is a schematic diagram of the structure of an autonomous driving route trajectory determination device provided in an embodiment of the present invention. Figure 3 As shown, the device includes: a data acquisition module 310, a candidate route output module 320, and a target route determination module 330.

[0113] The data acquisition module 310 is used to acquire real-time environmental perception data, vehicle status data, and basic map data of the actual vehicle; the candidate route output module 320 is used to input the real-time environmental perception data, vehicle status data, and basic map data of the actual vehicle into the autonomous driving model to obtain multiple candidate route trajectories with confidence scores output by the model; wherein, the autonomous driving model includes a decision planning branch related to route trajectory planning, and the decision planning branch is pre-trained based on offline reinforcement learning training method and dual-source training data; the target route determination module 330 is used to determine the target route trajectory for autonomous driving execution based on multiple candidate route trajectories with confidence scores.

[0114] The technical solution of this invention acquires real-time environmental perception data, vehicle status data, and basic map data; inputs these data into an autonomous driving model to obtain multiple candidate route trajectories with confidence scores. The autonomous driving model includes a decision-making branch related to route planning, which is pre-trained using offline reinforcement learning and dual-source training data. Based on these multiple candidate route trajectories with confidence scores, a target route trajectory for autonomous driving execution is determined. This solves the technical problems of weak generalization ability, inaccurate trajectory evaluation, and high training complexity in traditional methods, achieving improved adaptability, accuracy, and training efficiency in route planning.

[0115] In some alternative embodiments, the apparatus further includes:

[0116] The branching module is used to obtain a pre-trained end-to-end autonomous driving model before inputting the real-time environmental perception data, vehicle status data, and map basic data into the autonomous driving model, divide the environmental perception branch and the decision planning branch related to route trajectory planning in the autonomous driving model, and freeze the parameters of the environmental perception branch; the training data acquisition module is used to acquire dual-source training data and divide it into training set and test set based on the dual-source training data.

[0117] The training module is used to iteratively train the decision planning branch based on a preset number of training times and a training set. In each training session, the current policy network parameters are copied to the reference network. The training batches are traversed. In each batch, the old policy network is updated to the current policy network parameters first, and then real vehicle data is input to generate candidate training trajectories.

[0118] The optimization module is used to call a pre-trained reward model to calculate the reward value of candidate training trajectories, determine the advantage function value through a preset standardized formula, calculate the loss based on the objective function with adaptive KL divergence constraints, and update the decision planning branch parameters using mini-batch gradient descent to complete single-batch optimization.

[0119] The planning branch determination module is used to verify the current policy network with the test dataset after each iteration, score the output trajectory through the reward model, and determine the completed decision planning branch as the decision planning branch for route trajectory planning in the autonomous driving model if the trajectory score is not lower than a preset score threshold.

[0120] In some optional embodiments, the dual-source training data includes a real-vehicle historical driving dataset and a public benchmark dataset; the real-vehicle historical driving dataset contains historical surround view images, historical vehicle states, and corresponding historical driving trajectory labels under different scenarios; the public benchmark dataset contains a closed-loop simulation framework, a multi-dimensional trajectory scoring mechanism, map template data, and interaction trajectory records between the vehicle and surrounding vehicles.

[0121] In some alternative embodiments, the apparatus further includes:

[0122] The reference trajectory generation module is used to call the closed-loop simulation framework of the public benchmark dataset before calling the pre-trained reward model to calculate the reward value of the candidate training trajectory, obtain the map template data and the interaction trajectory records of surrounding vehicles in the dataset, and generate the reference trajectory of the vehicle in different scenarios based on the preset simple planner.

[0123] The paired data generation module is used to trigger the multi-dimensional trajectory scoring mechanism of the public benchmark dataset, score each reference trajectory from preset dimensions, form paired data of reference trajectory and multi-dimensional scores and store them in the training buffer.

[0124] The supervised training module is used to construct a neural network model with the reference trajectory in the paired data as the input feature and the corresponding multi-dimensional score mean as the true value label, and to perform supervised learning training using the mean squared error loss function and gradient descent optimization algorithm.

[0125] The reward model determination module is used to extract a validation dataset and input it into the model after each preset training round. If the error between the model output score and the true label is less than a preset threshold, the training ends to obtain the reward model.

[0126] In some optional embodiments, the offline reinforcement learning training method employs an objective function with adaptive constraints; the adaptive constraints are KL divergence terms, and the weights of the KL divergence terms are calculated based on the safety loss from the collision probability of the vehicle during the training process.

[0127] In some optional embodiments, the real-time environmental perception data of the vehicle includes surround view images collected by onboard sensors and dynamic information of surrounding vehicles; the vehicle status data includes the vehicle's current speed, position, and steering angle information; and the map base data includes lane line distribution, intersection signs, and road speed limit information.

[0128] In some optional embodiments, the target route determination module includes:

[0129] The model scoring unit is used to call the reward model to score each candidate route trajectory to obtain the reward model score;

[0130] The comprehensive score determination unit is used to determine the comprehensive score of each candidate route trajectory based on the confidence level and the reward model score.

[0131] The target route determination unit is used to determine the target route trajectory for autonomous driving execution based on multiple comprehensive scores.

[0132] In some alternative embodiments, the apparatus further includes:

[0133] The new training set generation module is used to collect environmental state information and trajectory data during the actual vehicle execution process, and call the reward model to score the trajectory data to form a new training dataset;

[0134] The update package generation module is used to perform secondary fine-tuning on the decision planning branch of the autonomous driving model based on the new training dataset, so as to generate a parameter incremental update package containing only the optimization parameters of the decision planning branch.

[0135] The model update module is used to perform safety testing on the parameter incremental update package, and after the test is passed, update the autonomous driving model based on the parameter incremental update package.

[0136] The autonomous driving route trajectory determination device provided in this embodiment of the invention can execute the autonomous driving route trajectory determination method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0137] Figure 4This is a schematic diagram of an electronic device for implementing the route trajectory determination method for autonomous driving according to embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0138] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0139] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0140] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the method of determining the route trajectory for autonomous driving.

[0141] In some embodiments, the method for autonomous driving route determination may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the method for autonomous driving route determination described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the method for autonomous driving route determination by any other suitable means (e.g., by means of firmware).

[0142] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0143] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0144] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0145] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0146] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0147] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0148] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0149] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for determining a route trajectory for autonomous driving, characterized in that The method comprises the following steps: acquiring real-time environment perception data of a real vehicle, vehicle state data and map basic data; inputting the real-time environment perception data of the real vehicle, the vehicle state data and the map basic data into an automatic driving model to obtain a plurality of candidate route trajectories with confidence degrees output by the model; wherein the automatic driving model comprises a decision-making and planning branch related to route trajectory planning, and the decision-making and planning branch is obtained by pre-training based on an offline reinforcement learning training method and double-source training data; determining a target route trajectory for automatic driving execution based on the plurality of candidate route trajectories with confidence degrees.

2. The method of claim 1, wherein, Before the real-time environment perception data of the real vehicle, the vehicle state data and the map basic data are input into the automatic driving model, the method further comprises the following steps: acquiring a pre-trained end-to-end automatic driving model, dividing an environment perception branch and a decision-making and planning branch related to route trajectory planning in the automatic driving model, and freezing parameters of the environment perception branch; acquiring double-source training data, and dividing the double-source training data into a training set and a test set based on the double-source training data; iteratively training the decision-making and planning branch based on a preset number of training times and the training set, copying current policy network parameters to a reference network each time, traversing each training batch, updating an old policy network to the current policy network parameters each time, and inputting real vehicle data to generate a candidate training trajectory; calling a pre-trained reward model to calculate a reward value of the candidate training trajectory, determining an advantage function value through a preset standardization formula, calculating a loss based on a target function containing an adaptive KL divergence constraint term, updating parameters of the decision-making and planning branch through a small-batch gradient descent method, and completing single-batch optimization; after each iteration is completed, verifying a current policy network with a test data set, scoring an output trajectory through a reward model, and determining the decision-making and planning branch that has completed training as a decision-making and planning branch for route trajectory planning in the automatic driving model when the trajectory score is not lower than a preset score threshold.

3. The method of claim 2, wherein, The double-source training data comprises a real vehicle historical driving data set and a public benchmark data set; the real vehicle historical driving data set contains historical surround view images, historical vehicle states and corresponding historical driving trajectory labels in different scenarios; and the public benchmark data set contains a closed-loop simulation framework, a multi-dimensional trajectory scoring mechanism, map template data and interactive trajectory records of a self vehicle and surrounding vehicles.

4. The method of claim 2, wherein, Before the pre-trained reward model is called to calculate the reward value of the candidate training trajectory, the method further comprises the following steps: calling a closed-loop simulation framework of the public benchmark data set to acquire map template data and interactive trajectory records of surrounding vehicles in the data set, and generating reference trajectories of the self vehicle in different scenarios based on a preset simple planner; triggering a multi-dimensional trajectory scoring mechanism of the public benchmark data set to score each reference trajectory from a preset dimension, forming paired data of reference trajectories and multi-dimensional scores and storing the paired data to a training buffer; inputting the reference trajectories in the paired data as input features and the corresponding multi-dimensional score averages as true value labels to construct a neural network model, and performing supervised learning training by using a mean square error loss function and a gradient descent optimization algorithm; After each preset training round is completed, the validation dataset is input into the model, and in a case where an error between a model output score and a true value label is less than a preset threshold, the training is ended to obtain the reward model.

5. The method of claim 1, wherein, The offline reinforcement learning training method adopts a target function containing an adaptive constraint term; the adaptive constraint term is a KL divergence term, and a weight of the KL divergence term is calculated by a safety loss of a collision probability of the vehicle in a training process.

6. The method of claim 1, wherein, The real vehicle real-time environment perception data includes surround view images and surrounding vehicle dynamic information collected by a vehicle-mounted sensor, the vehicle state data includes current vehicle speed, position, and steering angle information, and the map basic data includes lane line distribution, intersection identification, and road speed limit information.

7. The method of claim 4, wherein, The candidate route trajectory determination target route trajectory based on the multiple strip confidence levels includes: The reward model is called to score each candidate route trajectory to obtain a reward model score; For each candidate route trajectory, a comprehensive score of the candidate route trajectory is determined based on the confidence level and the reward model score; The target route trajectory for automatic driving execution is determined based on multiple comprehensive scores.

8. The method of claim 1, wherein, Further comprising: environmental state information and trajectory data in the execution process of the real vehicle are collected, and the reward model is called to score the trajectory data to form a new training dataset; Based on the new training dataset, the decision planning branch of the automatic driving model is fine-tuned again to generate a parameter incremental update package containing only decision planning branch optimization parameters; The parameter incremental update package is tested for safety, and after the test is qualified, the automatic driving model is updated based on the parameter incremental update package.

9. An automatic driving route trajectory determination device characterized by comprising: Comprising: a data acquisition module configured to acquire real vehicle real-time environment perception data, vehicle state data, and map basic data; a candidate route output module configured to input the real vehicle real-time environment perception data, the vehicle state data, and the map basic data into an automatic driving model to obtain multiple strip confidence level candidate route trajectories output by the model; wherein the automatic driving model includes a decision planning branch related to route trajectory planning, and the decision planning branch is obtained by pre-training based on an offline reinforcement learning training method and double-source training data; a target route determination module configured to determine a target route trajectory for automatic driving execution based on multiple strip confidence level candidate route trajectories.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the processor to implement the route trajectory determination method for automatic driving of any one of claims 1-8 when executed. The computer readable storage medium stores computer instructions for causing the processor to implement the route trajectory determination method for automatic driving of any one of claims 1-8 when executed.

Images