Automatic driving path planning model and training method thereof, vehicle and program product

Abstract

The invention relates to a training method of a path planning model for automatic driving, the path planning model is used for outputting path planning data at least under the condition of using perception input data, and the training method comprises the following steps: S1, carrying out first training on the path planning model based on a first training data set, the first training data set at least comprises first driving track data; s2, second training is carried out on the path planning model based on a second training data set, the second training data set at least comprises second driving track data and matched evaluation data, the evaluation data represents the user preference degree aiming at the second driving track data, and the evaluation data represents the user preference degree aiming at the second driving track data; and performing second training on the path planning model based on the user preference degree aiming at the second driving track data and / or based on the difference between different user preference degrees aiming at different second driving track data in the second driving track data. The application also relates to a path planning model for automatic driving, a vehicle and a computer program product.

Description

Technical Field

[0001] The present application relates to a training method for a path planning model for autonomous driving, a path planning model for autonomous driving, a vehicle, and a computer program product. Background Art

[0002] At present, the modes and driving styles of autonomous driving decision planning are relatively unified. For example, they can be divided into aggressive, normal, conservative, etc. However, this classification based on unified standards often cannot meet the personalized preferences of different users for driving styles. Summary of the invention

[0003] In order to at least partially solve the above problems, the present application proposes a training method for a path planning model for autonomous driving, a path planning model for autonomous driving, a vehicle, and a computer program product.

[0004] According to a first aspect of the present application, a training method for a path planning model for autonomous driving is provided, wherein the path planning model is used to output path planning data at least when using perception input data, and the training method comprises:

[0005] Step S1: performing a first training on the path planning model based on a first training data set, wherein the first training data set at least includes first driving trajectory data;

[0006] Step S2: performing a second training on the path planning model based on a second training data set, wherein the second training data set at least includes second driving trajectory data and corresponding evaluation data, wherein the evaluation data indicates a user preference for the second driving trajectory data.

[0007] The path planning model is secondly trained based on the user preference level for the second driving trajectory data and / or based on the difference between different user preference levels for different second driving trajectory data in the second driving trajectory data.

[0008] The core concept of this application is to introduce a user feedback mechanism into the path planning model so that the path planning model can provide customized driving styles and modes for different users. According to this application, by introducing a user feedback mechanism, the personalized preference needs of different users for driving styles and modes can be met, thereby improving the user experience.

[0009] In a preferred embodiment of the present application, it is provided that, in step S2, a reward model is trained based on the second training data set, the reward model is used to evaluate the second driving trajectory data, and the path planning model is secondly trained when using the reward model, wherein the reward model is trained based on the following loss function: the loss function depends on the difference between the corresponding different user preference levels for different second driving trajectory data for the same perceptual input data.

[0010] In a preferred embodiment of the present application, the training method further comprises:

[0011] Step S3: Obtain the user's historical driving trajectory data, and / or, for at least one driving scenario, provide at least one corresponding path planning data through the path planning model and obtain the corresponding evaluation given by the user, so as to perform a first update on the path planning model.

[0012] In a preferred embodiment of the present application, the training method further comprises:

[0013] Step S4: Acquire the user's real-time driving trajectory data and acquire the corresponding evaluation given by the user, so as to perform a second update on the path planning model.

[0014] In a preferred embodiment of the present application, it is provided that, in response to a negative evaluation given by a user, the negative evaluation is input into the path planning model so as to perform a second update on the path planning model and output updated path planning data.

[0015] In a preferred embodiment of the present application, it is provided that, in step S2, at least two reward models are trained based on the second training data set, and the at least two reward models are used to respectively evaluate the second driving trajectory data in at least two dimensions, and the path planning model is secondly trained using at least one of the at least two reward models.

[0016] In a preferred embodiment of the present application, it is provided that, in step S1, the path planning model is first trained based on the following loss function: the loss function depends on the difference between the path planning data output by the path planning model and the first driving trajectory data.

[0017] According to a second aspect of the present application, a path planning model for autonomous driving is proposed, and the path planning model is trained according to the training method described in the first aspect of the present application.

[0018] According to a third aspect of the present application, a vehicle is provided, in which the path planning model according to the second aspect of the present application is deployed.

[0019] According to a fourth aspect of the present application, a computer program product, such as a computer-readable program carrier, is proposed, wherein the computer program product includes or stores computer program instructions, and when the computer program instructions are executed by a processor, the computer program instructions can at least partially execute the steps of the method described in the first aspect of the present application.

[0020] Here, the features and details described in conjunction with the training method for a path planning model for autonomous driving according to the present application also apply to the path planning model for autonomous driving according to the present application, the vehicle according to the present application, and the computer program product according to the present application, and vice versa. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The present application is described in more detail below with reference to the accompanying drawings. The accompanying drawings show:

[0022] Figure 1 A schematic block diagram showing a training method for a path planning model for autonomous driving according to an exemplary embodiment of the present application is shown;

[0023] Figure 2 An exemplary flow chart showing a method for training a path planning model for autonomous driving according to an exemplary embodiment of the present application;

[0024] Figure 3 An exemplary flow chart showing a method for training a path planning model for autonomous driving according to an exemplary embodiment of the present application;

[0025] Figure 4 A schematic block diagram showing a path planning model for autonomous driving and an exemplary application thereof according to an exemplary embodiment of the present application;

[0026] Figure 5 A schematic block diagram of a vehicle according to an exemplary embodiment of the present application is shown;

[0027] Figure 6 A schematic block diagram of a computer program product according to an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] In order to make the technical problems, technical solutions and beneficial technical effects to be solved by the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and multiple exemplary embodiments. It should be understood that the exemplary embodiments described herein are only used for explanation and are not used to limit the scope of protection of the present application.

[0029] Figure 1 A principle block diagram of a training method 100 for a path planning model 200 for autonomous driving according to an exemplary embodiment of the present application is shown. The path planning model 200 (which can also be referred to as a decision planning model) is trained to output path planning data at least using perceptual input data. Here, the path planning model is based on machine learning (English: Machine Learning, ML), in particular, based on reinforcement learning (English: Reinforcement Learning, RL) and introduces human feedback or user feedback to implement human feedback reinforcement learning (English: Reinforcement Learning from Human Feedback, RLHF).

[0030] In step S1 of the training method 100 , the path planning model 200 is first trained based on a first training data set, where the first training data set at least includes first driving trajectory data.

[0031] In the scope of this application, the first training may also be referred to as pre-training or unsupervised pre-training. In the scope of this application, the first training data set may also be referred to as an unlabeled data set.

[0032] In the scope of the present application, the driving trajectory data (including the first driving trajectory data here and the second driving trajectory data to be mentioned below) should be broadly understood as relevant data that characterizes the driving state of a transportation vehicle, such as a vehicle. Exemplarily, the driving trajectory data can be represented as a series of frame data that are successive in time. Exemplarily, the frame data or the relevant data that characterizes the driving state contained in the frame data can include trajectory data, positioning data, and can also include driving operation data such as driving speed, driving acceleration, driving deceleration, and steering operations. After step S1 or the first training is completed, the path planning model 200 can predict, for example, the next driving state of the vehicle and can then infer a complete driving path or driving trajectory.

[0033] In step S2 of the training method 100, the path planning model 200 is secondly trained based on a second training data set, wherein the second training data set at least includes second driving trajectory data and corresponding evaluation data, wherein the evaluation data represents a user preference level for the second driving trajectory data, wherein the path planning model 200 is secondly trained based on a user preference level for the second driving trajectory data and / or based on a difference between different user preference levels for different second driving trajectory data in the second driving trajectory data.

[0034] Within the scope of the present application, the second training may also be referred to as fine-tuning training or supervised fine-tuning training. Within the scope of the present application, the second training data set may also be referred to as a labeled data set. Exemplarily, the second training data in the second training data set may be represented as paired labeled data, wherein the evaluation data representing the user's preference for the second driving trajectory data may also be referred to as annotations or labels. Exemplarily, the user's preference may be in the form of selection / rejection, rating score, sorting, preference level (significantly better, better, slightly better, seemingly better, etc.), etc., which is not limited here, and the content described below with reference to one form may be applied to other forms in a similar manner.

[0035] Here, human feedback or user feedback is introduced through evaluation data, so that the user's inclination or preference for different path planning can be taken into account in the training and / or application of the path planning model, and the path planning data output by the path planning model can be more in line with the user's personalized preference needs.

[0036] Figure 2 An exemplary flow chart of a training method 100 for a path planning model 200 for autonomous driving according to an exemplary embodiment of the present application is shown.

[0037] In step S1, the path planning model 200 is first trained (pre-trained) based on a first training data set, wherein the first training data set at least includes first driving trajectory data, and the path planning model obtained thereby may also be referred to as a pre-trained model. Here, exemplarily, the first training data set comes from a path trajectory actually collected by a human driver and the first driving trajectory data is continuous frame data representing the path trajectory. Here, the path planning model 200 is trained in such a way that the path planning model 200 can infer and predict the next frame data from one frame data and can further infer and predict multiple frame data or path trajectories that are successive in time from one starting frame data.

[0038] Here, in one embodiment, the path planning model 200 is first trained based on the following loss function loss_pretraining: the loss function loss_pretraining depends on the difference between the path planning data pred_trajectory output by the path planning model 200 and the first driving trajectory data gt_trajectory. Here, exemplarily, the loss function loss_pretraining for the first training or pre-training is defined as follows:

[0039] loss_pretraining=(pred_trajectory_t-gt_trajectory_t) 2

[0040] Wherein, pred_trajectory_t represents the trajectory position of, for example, the vehicle 300 at time point t, which is inferred and predicted by the path planning model 200, and gt_trajectory_t represents the trajectory position of the vehicle 300 at time point t, which is actually collected. Here, for example, the least square method is used for trajectory fitting or curve fitting, wherein the model parameters of the path planning model 200 are continuously adjusted and optimized so that the difference between the path planning data pred_trajectory output by the path planning model 200 and the actually collected (first) driving trajectory data gt_trajectory is minimized. Of course, it should be understood that other definitions and methods can also be used instead, which are not limited here.

[0041] In step S2, the path planning model 200 is subjected to a second training (supervised fine-tuning training) based on the second training data set, and the path planning model obtained is an optimized autonomous driving path planning model. As described above, the second training data in the second training data set can be represented as paired annotated data, such as a path trajectory (i.e., second driving trajectory data) together with a score annotated by an annotator that reflects the quality of the path trajectory or the tendency of the path trajectory (i.e., the corresponding evaluation data or label).

[0042] In an optional step SA, the path planning model 200 can be trained based on path trajectories labeled as “good” (or ranked high or scored high), for example.

[0043] In one embodiment, in step SB, the reward model RM is trained based on the second training data set (i.e., for example, a plurality of paired path trajectories and corresponding scores), so that the reward model RM can evaluate different path trajectories or can infer and predict the scores of different path trajectories. Here, the reward model RM is trained based on the following loss function loss_ranking: the loss function loss_ranking depends on the difference between the corresponding different user preference levels for different second driving trajectory data for the same perception input data. Here, the reward model RM is trained in this way so that the score given by the reward model RM can correspond to the score annotated by the annotator, especially so that the score given by the reward model RM is consistent with the score annotated by the annotator in terms of ranking.

[0044] Exemplarily, in training the reward model RM, a ranking loss based on cross entropy is used (as in InstructingGPT) or a margin is additionally used as a discrete function of the preference rating (as in Llama 2). Exemplarily, the loss function loss_ranking for the reward model RM is defined as follows:

[0045] loss_ranking=log(r(x,y_c)-r(x,y_r))

[0046] Where r(x,y) represents the score when the perceived input data is x and the output or inferred predicted path trajectory is y, y_c represents the path trajectory preferred or selected by the annotator, and y_r represents the path trajectory rejected by the annotator. It should be understood that the loss function definition listed here is only exemplary and not restrictive, and it can take any other suitable mathematical form.

[0047] Next, in step SC, the path planning model 200 is subjected to a second training (supervised fine-tuning training) using the reward model RM. Here, the path planning model 200 is trained and optimized by reinforcement learning based on the reward model RM. Here, the second training is implemented in such a way that the path planning model 200 can not only infer and predict, for example, the path trajectory of the vehicle, but also output inference prediction results that meet the user's preference requirements.

[0048] Optionally and / or preferably, in an embodiment not shown by an image, at least two reward models RM are trained based on the second training data set, and the at least two reward models RM are used to respectively evaluate the second driving trajectory data in at least two dimensions. Exemplarily, one dimension is usefulness and the other dimension is safety. Next, the path planning model 200 is subjected to a second training using at least one reward model RM of the at least two reward models RM, so that the path planning model 200 is improved in terms of usefulness and / or safety. However, it should be understood that the dimensions are not limited thereto, and the corresponding reward models of the corresponding dimensions can be used individually or in combination with each other.

[0049] Figure 3 An exemplary flow chart of a method for training a path planning model 200 for autonomous driving according to an exemplary embodiment of the present application is shown.

[0050] Here, steps S1 and S2 shown on the left correspond essentially to those of reference Figure 1 and Figure 2The steps S1 and S2 described above are not described in detail here. These training steps can be performed on the vehicle side or on the cloud side, and preferably on the cloud side.

[0051] Next, steps S3 and S4 shown on the right are described in detail. These training steps S3 and S4 (also referred to as calibration steps or alignment steps) can also be performed on the vehicle side, on the cloud side, and preferably on the vehicle side, and are usually performed after the path planning model 200 is deployed in the vehicle 300, preferably in the autonomous driving system of the vehicle 300.

[0052] In step S3, for example, after the model is deployed or after the model is started, one or more standardized driving scenarios are selected by the user or provided to the user, and the path planning model 200 accordingly outputs one or more corresponding path planning data for each driving scenario. Subsequently, the user gives a corresponding evaluation, such as the selection or rejection of a path planning data, the selection / rejection, sorting and / or scoring of multiple path planning data, and the evaluation can be input into the path planning model 200 as user feedback, so as to perform a first update (initialization) on the path planning model 200, so that the aligned and calibrated path planning model 200 can infer and predict path planning data that meets the user's expectations. Alternatively or additionally, the user's historical driving trajectory data can also be obtained in order to perform a first update on the path planning model 200. This can also be referred to as offline pre-calibration.

[0053] In step S4, for example, during the actual driving process of the vehicle 300, the user's real-time driving trajectory data is obtained and the corresponding evaluation or score given by the user is obtained, so as to perform a second update (real-time update) on the path planning model 200. Exemplarily, the path planning model 200 can output a path planning data (i.e., path trajectory) for user confirmation or can output multiple alternative path planning data for user selection. If the user feedback is "select" or "like", the vehicle 300 can perform automatic driving according to the output or selected path planning data. Conversely, if the user feedback is "reject" or "dislike", the path planning model 200 can obtain the user feedback as input data to adjust or update the model parameters and then output the path planning data based on the adjusted or updated path planning model for user use. In a preferred embodiment, in response to a negative evaluation given by a user (e.g., rejecting the output path planning data, and / or intervening in driving operations that do not meet user expectations, such as acceleration, deceleration, steering, etc.), the negative evaluation is input into the path planning model 200 so as to perform a second update on the path planning model 200 and output the updated path planning data.

[0054] Although the steps are shown in sequence here, it should be understood that the shown sequence is not limiting. For example, step S3 can also be performed again after step S4.

[0055] Figure 4 A schematic block diagram of a path planning model 200 for autonomous driving and an exemplary application thereof according to an exemplary embodiment of the present application is shown.

[0056] The path planning model 200 has been trained according to at least some of the method steps (eg, steps S1 and S2 ) in the training method 100 .

[0057] The path planning module 200 can obtain the perception input data related to the autonomous driving path planning from the perception input module M1. The perception input data includes, for example, external environment perception data, such as road data, traffic data (such as traffic density, traffic signs, traffic lights), and traffic participant data (such as pedestrians, cyclists and other vehicles). The perception input data also includes, for example, vehicle internal perception data, such as vehicle occupants, vehicle occupant instructions, etc. In addition, the perception input data also includes, for example, the positioning data of the vehicle 300 itself (such as GPS data) and optionally map data. The perception input module M1 can accordingly include various perception modules, especially various perception modules that already exist in the autonomous driving vehicle 200, such as cameras, video cameras, infrared cameras, radar sensors, lidar sensors, ultrasonic sensors, temperature sensors, humidity sensors, positioning sensors, microphones, communication modules, etc. The perception input results corresponding to the perception input data (such as environmental perception results, prediction results, positioning results) can be processed / fused into structured data on the perception input module M1 side or via an additional processing device, and can also be processed directly by the path planning model 200 on the path planning model 200 side, for example, in the form of intermediate features or tokens of the model, which is not limited here.

[0058] The path planning module 200 can also obtain user input data or user feedback data related to user preference parameters from the user input module M2, such as offline and online real-time calibrated user feedback. In one example, as described with reference to step S3, in the offline pre-calibration stage, the path planning module 200 obtains the user's evaluation (selection / rejection, scoring, sorting, etc.) of the path planning data output by the path planning model 200 for the standardized driving scene as user input data, so as to update / calibrate the path planning module 200, so that the updated / calibrated path planning model 200 can infer and predict the planned path that is more in line with the user's expectations. In one example, as described with reference to step S4, in the online real-time calibration stage, the path planning module 200 obtains the user's evaluation (especially rejection) of the planned path / driving operation output by the path planning model 200 for the actual real-time driving scene as user input data, so as to update / adjust the path planning module 200, so that the updated / adjusted path planning model 200 can infer and predict the planned path / driving operation that is more in line with the user's expectations.

[0059] The path planning module 200 can also be connected to the execution module M3. The execution module M3 receives the path planning data from the path planning module 200, and can display and automatically implement the path planning data, which will not be described in detail here.

[0060] Figure 5 A schematic block diagram of a vehicle 300 according to an exemplary embodiment of the present application is shown.

[0061] As in Figure 5 As shown in FIG, a path planning model 200 is deployed in the vehicle 300. Although not shown here, it is conceivable that other necessary modules are also deployed in the vehicle 300, including but not limited to reference Figure 4 The perception input module M1, user input module M2 and execution module M3 have been described.

[0062] Figure 6 A schematic block diagram of a computer program product 400 according to an exemplary embodiment of the present application is shown.

[0063] The computer program product 400 can be stored on a program carrier, in particular a computer-readable program carrier. The computer program product 400 includes or stores computer program instructions, which can at least partially perform the steps of the method described in the present application when the computer program instructions are executed by a processor.

[0064] In the present application, by introducing human feedback or user feedback, the user's inclination or preference for different path planning can be taken into account in the training and / or application of the path planning model, and the path planning data output by the path planning model can be more in line with the user's personalized preference needs.

[0065] Although specific embodiments of the present application are described in detail herein, they are provided for the purpose of explanation only and should not be considered to limit the scope of the present application. Various alternatives and modifications may be proposed without departing from the core and scope of the present application.

Claims

1. A method for training a path planning model for autonomous driving, the path planning model being configured to output path planning data at least using perception input data, the training method comprising: Step S1: performing a first training on the path planning model based on a first training data set, wherein the first training data set at least includes first driving trajectory data; Step S2: performing a second training on the path planning model based on a second training data set, wherein the second training data set at least includes second driving trajectory data and corresponding evaluation data, wherein the evaluation data indicates a user preference for the second driving trajectory data. The path planning model is secondly trained based on the user preference level for the second driving trajectory data and / or based on the difference between different user preference levels for different second driving trajectory data in the second driving trajectory data.

2. The training method according to claim 1, wherein: In step S2, a reward model is trained based on the second training data set, the reward model is used to evaluate the second driving trajectory data, and the path planning model is secondly trained using the reward model, wherein the reward model is trained based on the following loss function: the loss function depends on the difference between the corresponding different user preference levels of different second driving trajectory data for the same perception input data.

3. The training method according to claim 1 or 2, further comprising: Step S3: Obtain the user's historical driving trajectory data, and / or, for at least one driving scenario, provide at least one corresponding path planning data through the path planning model and obtain the corresponding evaluation given by the user, so as to perform a first update on the path planning model.

4. The training method according to any one of claims 1 to 3, further comprising: Step S4: Acquire the user's real-time driving trajectory data and acquire the corresponding evaluation given by the user, so as to perform a second update on the path planning model.

5. The training method according to claim 4, wherein: In response to a negative evaluation given by a user, the negative evaluation is input into the path planning model so as to perform a second update on the path planning model and output updated path planning data.

6. The training method according to any one of claims 1 to 5, wherein: In step S2, at least two reward models are trained based on the second training data set, and the at least two reward models are used to respectively evaluate the second driving trajectory data in at least two dimensions, and the path planning model is secondly trained using at least one of the at least two reward models.

7. The training method according to any one of claims 1 to 6, wherein: In step S1 , the path planning model is first trained based on the following loss function: the loss function depends on the difference between the path planning data output by the path planning model and the first driving trajectory data.

8. A path planning model for autonomous driving, wherein the path planning model is trained according to the training method according to any one of claims 1 to 7.

9. A vehicle, in which the path planning model according to claim 8 is deployed.

10. A computer program product, such as a computer-readable program carrier, comprising or storing computer program instructions, which are capable of at least partially executing the steps of the method according to any one of claims 1 to 7 when the computer program instructions are executed by a processor.

Images