Self-evolution, decision management method and device and system of an automatic driving model

Abstract

Embodiments of the present application disclose a self-evolution, decision management method and device and system of an automatic driving model. The method comprises: obtaining driving behaviors determined by a user and a current automatic driving model for a same driving task; generating a driving behavior loss function according to difference information between the driving behaviors determined by the user and the current automatic driving model; updating a loss function of the current automatic driving model according to the driving behavior loss function to obtain an updated loss function of the automatic driving model; and iteratively updating the current automatic driving model by using the updated loss function of the automatic driving model until convergence is achieved, to obtain an updated automatic driving model, thereby obtaining an automatic driving model with further reduced difference from the driving style of the user, so as to be closer to the personalized driving style of the user, effectively reduce the difficulty of the user in understanding the automatic driving behavior, and improve the trust of the user in the automatic driving technology.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of autonomous driving, and in particular to a self-evolution, decision management method and device and system of an autonomous driving model. BACKGROUND

[0002] Autonomous driving technology has the advantages of reducing the probability of traffic accidents, reducing energy consumption, and improving traffic efficiency, and has become a hot topic of widespread attention in the academic and industrial communities. After years of development, autonomous driving has achieved commercialization of high-level assisted driving and demonstration operation of L4-level autonomous driving taxis. The mainstream autonomous driving system is a hierarchical scheme based on a rule model, which realizes the rapid deployment of autonomous driving of vehicles through modules such as environment perception, intelligent decision, path planning, and control execution. However, since the modules are mainly based on the accumulation of if-else logic, it is impossible to exhaust all scenarios, and there are problems such as single mechanical processing method and difficulty in determining safety boundaries in handling complex scenarios.

[0003] With the rapid development of AI algorithms such as deep learning, reinforcement learning, and deep reinforcement learning, learning-based autonomous driving models can handle some boundary complex scenarios and are expected to obtain driving capabilities beyond human capabilities through autonomous learning. Among them, end-to-end autonomous driving models achieve output from the perception end to the control end, which revolutionizes the traditional rule module approach.

[0004] In actual applications, autonomous driving technology based on autonomous learning cannot meet the user's use requirements by implementing end-to-end or partial end-to-end autonomous driving to control vehicles such as cars. For example, the learning-based end-to-end autonomous driving cognitive decision method has low training efficiency and is difficult to achieve self-evolution through a large amount of data. Moreover, this scheme has a black box property, and the reliability of the output driving behavior is low and the explainability is poor, resulting in low user safety and acceptance. In addition, when the user starts autonomous driving, the user's experience of using autonomous driving is greatly reduced because the style of autonomous driving is different from the user's own style. Therefore, how to enable the autonomous driving system to make normal autonomous driving decision control of the vehicle while making the driving style closer to the user is a problem that needs to be solved. SUMMARY

[0005] To solve any of the above technical problems, embodiments of the present application provide a self-evolution, decision management method and device and system of an autonomous driving model.

[0006] To achieve the purpose of the embodiments of the present application, the embodiments of the present application provide a self-evolution method of an autonomous driving model, comprising:

[0007] Obtaining driving behaviors determined by a user and a current autonomous driving model for the same driving task;

[0008] generate a driving behavior loss function according to difference information between the user-determined driving behavior and the current automatic driving model-determined driving behavior;

[0009] update a loss function of the current automatic driving model according to the driving behavior loss function to obtain an updated loss function of the automatic driving model;

[0010] perform iteration on the current automatic driving model by using the updated loss function of the automatic driving model until convergence is reached to obtain an automatic driving model with updated parameters.

[0011] A decision management method of an automatic driving model, comprising:

[0012] determining a driving behavior corresponding to a driving task of a vehicle by using an automatic driving model, wherein the automatic driving model is obtained by self-evolution using the method described above;

[0013] detecting whether the driving behavior determined by the automatic driving model meets a preset safe driving condition;

[0014] outputting the driving behavior determined by the automatic driving model after detecting that the driving behavior determined by the automatic driving model meets the safe driving condition.

[0015] A self-evolution device of an automatic driving model, comprising:

[0016] an acquisition module configured to acquire driving behaviors determined by a user and a current automatic driving model for a same driving task;

[0017] a first determination module configured to generate a driving behavior loss function according to difference information between the user-determined driving behavior and the current automatic driving model-determined driving behavior;

[0018] an update module configured to update a loss function of the current automatic driving model according to the driving behavior loss function to obtain an updated loss function of the automatic driving model;

[0019] a self-evolution module configured to perform iteration on the current automatic driving model by using the updated loss function of the automatic driving model until convergence is reached to obtain an updated automatic driving model.

[0020] A decision management device of an automatic driving model, comprising:

[0021] a second determination module configured to determine a driving behavior corresponding to a driving task of a vehicle by using an automatic driving model, wherein the automatic driving model is obtained by self-evolution using the method described above;

[0022] The detection module is configured to detect whether the driving behavior determined by the automatic driving model meets a preset safe driving condition.

[0023] The output module is configured to output the driving behavior determined by the automatic driving model after detecting that the driving behavior determined by the automatic driving model meets the safe driving condition.

[0024] A self-evolution system of an automatic driving model, comprising:

[0025] At least one vehicle configured to provide raw data required by an automatic driving model to perform a self-evolution operation;

[0026] A data server configured to process the received raw data into training data required by the automatic driving model to perform the self-evolution operation;

[0027] A model server configured to use the training data to obtain the automatic driving model evolved by the method described above, and to update the automatic driving model deployed on the vehicle using the evolved automatic driving model.

[0028] A storage medium having a computer program stored therein, the computer program being configured to execute the method described above when running.

[0029] An electronic device comprising a memory having a computer program stored therein and a processor configured to execute the computer program to perform the method described above.

[0030] One of the above technical solutions has the following advantages or beneficial effects:

[0031] By obtaining the driving behavior determined by the user and the current automatic driving model for the same driving task, and generating a driving behavior loss function according to the difference information between the driving behavior determined by the user and the driving behavior determined by the current automatic driving model, the loss function of the current automatic driving model is updated according to the driving behavior loss function, the loss function of the updated automatic driving model is obtained, and the current automatic driving model is iterated until convergence using the loss function of the updated automatic driving model, thereby obtaining the updated automatic driving model, so as to obtain the automatic driving model with further reduced difference from the driving style of the user, so as to be closer to the personalized driving style of the user, effectively reduce the understanding difficulty of the user for the automatic driving behavior, and improve the trust degree of the user for the automatic driving technology.

[0032] The automatic driving model can be generated by using a pre-training model in addition to using the active exploration model. Through the pre-training model and the active exploration model, learning of the reward function, the loss function and the network parameters is realized in three different self-learning stages, i.e., a pre-training learning stage, an intermediate transition stage and a self-evolution learning exploration stage. After the learned parameter features are fused and processed, the automatic driving model is given, and the trained self-learning automatic driving model is used as the active exploration model in the environment for self-learning in the next stage until the model learns the optimal strategy. The database data can be effectively reused, and the problems of low learning and training efficiency and low sample utilization rate of the existing automatic driving model can be effectively solved.

[0033] By using the automatic driving model, the driving behavior corresponding to the driving task of the vehicle is determined, whether the driving behavior determined by the automatic driving model meets the preset safe driving condition is detected, and after it is detected that the driving behavior determined by the automatic driving model meets the safe driving condition, the driving behavior determined by the automatic driving model is output. The safety and rationality of the driving behavior output by the automatic driving model are ensured, the trust of users on the automatic driving technology is improved, and the problem of uninterpretable behavior output by the automatic driving model in the prior art is solved.

[0034] Other features and advantages of the embodiments of the present application will be described in the subsequent description, and some will become apparent from the description, or will be understood by those skilled in the art. The purpose and other advantages of the embodiments of the present application can be achieved and obtained by the structure specifically pointed out in the specification, claims and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0035] The accompanying drawings are used to provide a further understanding of the technical solutions of the embodiments of the present application, and constitute a part of the specification, and are used to explain the technical solutions of the embodiments of the present application together with the embodiments of the embodiments of the present application, and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0036] Figure 1 A flowchart of a self-evolution method of an automatic driving model provided by the embodiments of the present application is shown in the figure;

[0037] Figure 2 A schematic diagram of a self-evolution training mode of an automatic driving model provided by the embodiments of the present application is shown in the figure;

[0038] Figure 3 A flowchart of a decision management method of an automatic driving model provided by the embodiments of the present application is shown in the figure;

[0039] Figure 4 A schematic diagram of a running method of an automatic driving model provided by the embodiments of the present application is shown in the figure;

[0040] Figure 5A structural schematic diagram of a self-evolution device of an automatic driving model provided by an embodiment of the present application is shown in the figure.

[0041] Figure 6 A structural schematic diagram of a decision management device of an automatic driving model provided by an embodiment of the present application is shown in the figure.

[0042] Figure 7 A structural schematic diagram of a self-evolution system of an automatic driving model provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below with reference to the accompanying drawings. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other as long as they do not conflict.

[0044] In the process of implementing the present application, it is found that the learning-based automatic driving model still has the following problems, including:

[0045] Firstly, the generalization performance of the automatic driving model is insufficient when dealing with complex and variable dynamic scenes and driving tasks. For example, an automatic driving model trained for a specific high-speed automatic driving task cannot efficiently and accurately complete other similar tasks, such as urban automatic driving or automatic parking.

[0046] Secondly, the existing automatic driving model needs a large database for training and cannot effectively utilize the database data, lacks targeted utilization of data, causes data waste, has low learning efficiency, high cost, and large time consumption.

[0047] Finally, the existing technology does not fully consider the safety and comfort of the driver and passenger, the flexibility, and the low degree of personalization, especially the low acceptance and trust of the automatic driving system, making it difficult to realize the real unmanned and commercial landing of automatic driving.

[0048] In summary, in the self-evolution learning automatic driving scheme, there are problems such as how to improve the generalization of the model to the scene, reduce the invalid exploration and training process of the model, and consider the sustainable trust of the driver and passenger.

[0049] The corresponding solutions to the technical problems mentioned above are as follows, including:

[0050] Figure 1 A flowchart of a self-evolution method of an automatic driving model provided by an embodiment of the present application is shown in the figure.

[0051] As shown in the figure, the method comprises: Figure 1

[0052] ​Step 101, obtaining driving behavior determined by a user and a current automatic driving model for a same driving task;

[0053] The driving task record has a starting point and a destination of a trip of a vehicle, and can further include environmental perception information of the trip of the vehicle and vehicle driving information, wherein the environmental perception information includes information of a traffic participant, road environment information, traffic rule information, obstacle information, lane line information, distance to an obstacle, etc. The vehicle driving information includes vehicle chassis information and historical driving behavior information (such as driving speed, acceleration, steering information, number of lane changes, etc.) of a user (i.e., a driver or passenger of the vehicle).

[0054] The environmental perception information and the vehicle driving information can be obtained by using a camera, a millimeter wave radar, an ultrasonic radar, a laser radar, a GPS / inertial navigation system, a vehicle CANBUS, etc. installed on the automatic driving vehicle.

[0055] Specifically, the driving behavior of the user for the driving task can be obtained according to the historical driving behavior of the user, and the driving behavior of the current automatic driving model for the driving task can be obtained by using the current automatic driving model to process the driving task.

[0056] The current automatic driving model is an automatic driving model that has reached a model trial running condition, referred to as a transition stage automatic driving model; or an automatic driving model that has reached a formal running condition, referred to as a self-evolution automatic driving model.

[0057] The model performance of the transition stage automatic driving model is generally lower than that of the self-evolution automatic driving model.

[0058] Step 102, generating a driving behavior loss function according to difference information between the driving behavior determined by the user and the driving behavior determined by the current automatic driving model;

[0059] For the same driving task, the driving behavior determined by the user and the driving behavior determined by the current automatic driving model are different, mainly because the driving styles of the user and the current automatic driving model are different, so the difference information is obtained to represent the difference in driving styles between the two.

[0060] Specifically, a driving style corresponding to the driving behavior determined by the user is obtained to obtain first style data, and a driving style corresponding to the driving behavior determined by the current automatic driving model is obtained to obtain second style data; and a cross-entropy result between the first style data and the second style data is calculated to obtain the driving behavior loss function.

[0061] The calculation expression of the driving behavior loss function is as follows:

[0062]

[0063] wherein, represents a driving behavior loss function; represents a driving style of an automatic driving model; represents a driving style of a user; H represents a cross-entropy function.

[0064] Further, the driving style of the user is obtained by comprising:

[0065] Obtaining driving behavior data of the user, wherein the driving behavior data of the user comprises data of at least two parameters;

[0066] Determining a label corresponding to each parameter in the driving behavior data of the user to obtain label data;

[0067] According to the label data, determining the driving style of the user.

[0068] Specifically, the driving behavior is collected by a driving state sensor arranged on the vehicle, a driving behavior feature sequence B = {speed, acceleration, vehicle head time interval, lane changing frequency} is extracted, and the behavior feature sequence output value is input into a preset analysis model. The analysis model generates a corresponding label for the behavior feature sequence, thereby completing the recognition of the driving style of the user.

[0069] The above method can automatically complete the recognition of the driving style of the user, thereby effectively improving the data processing efficiency.

[0070] Step 103, updating the loss function of the current automatic driving model according to the driving behavior loss function to obtain an updated loss function of the automatic driving model;

[0071] By updating the driving behavior loss function to the loss function of the automatic driving model, the loss function of the automatic driving model can be more accurately represented, thereby providing support for the iteration of the automatic driving model to the driving style close to the user, so that the driving behavior output by the automatic driving model after iteration can be more consistent with the driving style of the user.

[0072] Further, the updated loss function of the automatic driving model is obtained by comprising:

[0073] Calculating the product between the driving behavior loss function and the weight corresponding to the driving behavior loss function to obtain a first calculation result;

[0074] Calculating the sum of the first calculation result and the loss function of the current automatic driving model to obtain the updated loss function of the automatic driving model.

[0075] Specifically, the expression of the loss function of the updated automatic driving model is as follows:

[0076]

[0077] wherein, denotes the loss function of the updated automatic driving model network; denotes the loss function of the automatic driving model network before updating; denotes the user driving behavior loss function; κ denotes the weight of the driving behavior loss function.

[0078] Step 104, using the loss function of the updated automatic driving model, iterates the current automatic driving model until convergence to obtain an updated automatic driving model;

[0079] The above method is used to iterate the current automatic driving model, so that the automatic driving model iterates towards the driving style close to the user, thereby obtaining an automatic driving model with further reduced difference from the driving style of the user, so as to be closer to the personalized driving style of the user, effectively reduce the difficulty of the user to understand the automatic driving behavior, and improve the trust degree of the user to the automatic driving technology.

[0080] The method provided by the embodiments of the present application obtains the driving behaviors determined by the user and the current automatic driving model for the same driving task, generates a driving behavior loss function according to the difference information between the driving behaviors determined by the user and the current automatic driving model, updates the loss function of the current automatic driving model according to the driving behavior loss function, obtains the loss function of the updated automatic driving model, and iterates the current automatic driving model using the loss function of the updated automatic driving model until convergence to obtain an updated automatic driving model, thereby obtaining an automatic driving model with further reduced difference from the driving style of the user, so as to be closer to the personalized driving style of the user, effectively reduce the difficulty of the user to understand the automatic driving behavior, and improve the trust degree of the user to the automatic driving technology.

[0081] The method provided by the embodiments of the present application is further described as follows:

[0082] Figure 2 The schematic diagram of the training method of the automatic driving model provided by the embodiments of the present application is shown in FIG. 1. Figure 2As shown, in order to solve the problem of low training efficiency of the automatic driving model in the prior art, the embodiment of the present application proposes that the automatic driving model can be generated by using a pre-training model in addition to using an active exploration model. Through the pre-training model and the active exploration model, learning of the reward function, the loss function and the network parameters is realized, the learned parameter features are fused and processed to be assigned to the automatic driving model, and the trained self-learning automatic driving model is used as the active exploration model in the next stage for self-learning in the environment until the model learns the optimal strategy. The database data can be effectively reused, and the trained model has higher learning efficiency than the original single model, and has the advantages of high training efficiency and high sample utilization rate.

[0083] In one example embodiment, the current automatic driving model is a transition stage automatic driving model based on a pre-training model and an active exploration model, or is a self-evolution automatic driving model based on an active exploration model and the transition stage automatic driving model, wherein the pre-training model is trained based on preset prior knowledge, and the active exploration model is learned by exploration in an actual driving environment.

[0084] Specifically, the training process of the automatic driving model sequentially passes through a pre-training learning stage, an intermediate transition stage and a self-evolution learning exploration stage. Among them:

[0085] During the exploration learning process of the automatic driving model, there are problems of low learning efficiency, low sample utilization rate, high time cost and insufficient automatic driving task processing capability, which can be solved by training the pre-training model based on the prior knowledge of the driving behavior trajectory library, learning the parameters of the reward function and the loss function, so as to generate the current automatic driving model by using the pre-training model in the pre-training learning stage;

[0086] In the intermediate transition stage, due to the problem of insufficient generalization of the automatic driving model trained completely based on expert experience to the scene, and the fact that the expert library cannot handle some scenes, an active exploration model can be set to explore the network parameters of the self-driving model while learning the expert experience, so as to improve the generalization and learning ability of the model. The network parameters of the pre-training model and the network parameters of the active exploration model are fused to update the automatic driving model and realize the self-evolution of the automatic driving model.

[0087] In the self-evolution learning exploration stage, since the deep learning of the automatic driving model has been completed, only the active exploration model can be used.

[0088] Further, the method for generating the current automatic driving model comprises:

[0089] based on the first loss function, the initial autonomous driving model is iteratively trained using the active exploration model and the pre-trained model until the model converges, obtaining a transition stage autonomous driving model, wherein the first loss function is obtained using the loss function of the pre-trained model and the loss function of the active exploration model;

[0090] based on the second loss function, the transition stage autonomous driving model is iteratively trained using the active exploration model until the model converges, obtaining a self-evolution autonomous driving model, wherein the second loss function is obtained using the loss function of the active exploration model;

[0091] wherein the autonomous driving model obtained by each iteration training operation is used as the active exploration model for the next iteration.

[0092] Specifically, in the pre-training learning stage, according to the loss function of the pre-trained model, the loss function of the autonomous driving model is determined, and the network parameters of the autonomous driving model are updated;

[0093] In the intermediate transition stage, according to the weighted sum of the loss function of the pre-trained model and the loss function of the active exploration model, the loss function of the autonomous driving model is determined, and the reward function of the autonomous driving model is determined according to the reward function of the active exploration model, and the network parameters of the autonomous driving model are updated;

[0094] In the self-evolution learning exploration stage, according to the reward function and the loss function of the active exploration model, the reward function and the loss function of the autonomous driving model are determined, and the network parameters of the autonomous driving model are updated.

[0095] On the basis of the pre-trained model and the active exploration model, the learned parameters are shared to the autonomous driving model, and the network parameters of the autonomous driving model are updated.

[0096] By fusing the pre-trained model and the active exploration model, the fused loss function calculation expression is as follows:

[0097]

[0098] wherein, represents the loss function of the active exploration model; represents the loss function of the pre-trained model; represents the fused loss function; λ is the loss weight.

[0099] According to the different values of λ, the autonomous driving model learning and training can be divided into three stages, wherein:

[0100] λ = 1, indicating that the autonomous driving model is in the pre-training learning stage;

[0101] 0 < λ < 1, indicating that the autonomous driving model is in the intermediate transition stage;

[0102] λ = 0, indicating that the autonomous driving model is in the self-evolution learning exploration stage.

[0103] It should be noted that, compared with the prior art which only uses the active exploration model to generate the autonomous driving model, the training method provided in the embodiments of the present application introduces a pre-training model trained using prior knowledge to generate the autonomous driving model in the pre-training learning stage, which can guide the iteration in the direction of the optimal driving behavior, and can greatly shorten the time required for training the autonomous driving model in the starting stage. In the intermediate transition stage, the pre-training model and the active exploration model are used to generate the autonomous driving model, so that the autonomous driving model can be iteratively optimized under the constraint condition of the determination of the driving behavior in the prior knowledge and the actual application, and the time required for training the autonomous driving model can be greatly shortened. Thus, the self-evolution learning exploration stage can be entered within a relatively short period of time.

[0104] Optionally, in the intermediate transition stage or the self-evolution learning exploration stage, the updated autonomous driving model is set as the active exploration model used next time.

[0105] Specifically, the network and parameters of the autonomous driving model are shared to the active exploration model for learning and re-optimization. This step is repeated until the average loss of the entire driving task is minimized, and the driving strategy obtained in different driving scenarios or tasks is optimal.

[0106] Further, the network parameters of the autonomous driving model can be shared to a new active exploration model, and the learning rate and other parameters can be updated.

[0107]

[0108]

[0109]

[0110] wherein θ = {θ pre , θ act , θ learn , α}, θ pre represents the pre-training model policy network parameters, θ act represents the active exploration model policy network parameters, and θ learn represents the self-evolution learning model network parameters. represents the relative entropy of the two distribution probabilities before and after the update of θ, and β represents a hyperparameter. an input state vector s of the shared policy network after updating θ t , and the output is the probability of driving behavior a t ; π θ (a t |s t ) represents an input state vector s of the shared policy network before updating θ t , and the output is the probability of driving behavior a t ; θ k+1 represents the parameters of the network after obtaining the minimum average loss.

[0111] In the prior art, the automatic driving model learns based on a reward function of fixed rules, which leads to insufficient generality of scenes and tasks. To solve the above problems, the embodiment of the application proposes that when various scenes and tasks are continuously explored through deep reinforcement learning, the excellent behavior characteristics of a large number of drivers in the driving behavior trajectory library are extracted, the reward function of the automatic driving model is fitted, so that the automatic driving model can learn the optimal driving behavior, and the strategy and the reward function are re-optimized based on the learning and exploration process of the active exploration model, and the above steps are repeated to realize the self-evolution iteration learning process, which can solve the problem of insufficient generality of the model.

[0112] The updating method of the network parameters of the pre-trained model will be described below:

[0113] The driving behavior trajectory library is extracted from the preset data server to obtain expert prior knowledge τ = {s 0, a 0, r 0 ; s 1, a 1, r 1 ; …; s t , a t , r t}, the trajectory library behavior and scene are data cleaned and feature mined, the state space and the action space are established, the pre-trained model is trained through the prior knowledge, the network parameters are learned, the reward function and the model parameters are updated, and good driving behavior is learned.

[0114] Wherein, the expression of the driving behavior action value function Q π (s, a) under the driving behavior π is as follows:

[0115]

[0116] Wherein, represents a weight matrix; μ π (s, a) represents the feature expectation of the driving behavior π.

[0117] Wherein, the expression of the reward function r(s, a) obtained by the driving behavior π is as follows:

[0118]

[0119] Wherein, h(s, a) represents the characteristic value of the environment perception state.

[0120] In one exemplary embodiment, after completing the training of the current pre-training model, the following operations are performed, and after the operations are completed, a new round of training of the pre-training model is started, including:

[0121] After completing the training of the current pre-training model, the difference between the feature expectation determined according to the prior knowledge and the feature expectation of the pre-training model after the training is completed is obtained to obtain a gradient value;

[0122] The current weight matrix and the reward function are updated using the gradient value, wherein the updated reward function is obtained using the updated weight matrix;

[0123] The loss function of the current pre-training model is updated using the updated weight matrix.

[0124] Specifically, the calculation expression of the gradient grad between the feature expectation of expert experience data and the feature expectation of the pre-training learning model is as follows:

[0125] grad=μ E -μ π ;

[0126] Wherein, μ E represents the feature expectation of expert experience data, and μ π represents the feature expectation of the pre-training model.

[0127] In the above calculation expression, the feature expectation μ E of expert experience data is the average value of the sum of the characteristic values of all collected driver trajectory data, and the calculation expression is as follows:

[0128]

[0129] Wherein, each of the N driver trajectories is composed of state-action pairs in total; γ represents the discount factor.

[0130] In the above calculation expression, the feature expectation μ π of the pre-training model is the sum of the trajectory characteristic values output by the pre-training model, and the calculation expression is as follows:

[0131]

[0132] Further, the updated weight matrix is obtained by the following way, including:

[0133] The product of the gradient value and the preset learning rate is calculated to obtain a second calculation result;

[0134] Obtaining the sum of the second calculation result and the current weight matrix to obtain an updated weight matrix.

[0135] Specifically, the updated weight matrix θ pre The calculation expression is as follows:

[0136] θ pre = θ pre + alpha * grad.

[0137] Further, by obtaining the optimal driving behavior and its corresponding loss function is:

[0138]

[0139] Where A t Indicates the model advantage function.

[0140] By training the pre-trained model through the expert prior knowledge of the driving behavior trajectory library, the parameters of the reward function and the loss function are learned, which can effectively solve the problems of low learning efficiency, low sample utilization, high time cost and insufficient model processing ability for automatic driving task of the automatic driving model at the beginning of exploration learning.

[0141] The updating method of the network parameters in the active exploration model will be described below:

[0142] Put the active exploration model into the actual driving environment for exploratory learning and training, complete the driving task in unknown scenarios, get different driving behaviors, and introduce a human (in this application, human refers to user, i.e. the driver of the vehicle) feedback mechanism in the disordered learning process to guide the learning of the autonomous driving vehicle, and update the reward function, loss function and learning rate parameters again.

[0143] Specifically, the training data N of the active exploration model is divided into a first part N1 and a second part N2, wherein the first part N1 is used to perform learning operation of driving behavior, and the second part N2 is used to learn human feedback mechanism, wherein N2=N-N1.

[0144] The calculation expression of the loss function of the Critic network based on the optimal value function is as follows:

[0145]

[0146] Preferably, considering the data of the guided learning strategy of the human feedback mechanism, the Actor network can maximize the value of the Critic, and learn from these demonstrations, wherein the calculation expression of the loss function of the Actor network is as follows:

[0147]

[0148] wherein ω represents a weight, used to measure the importance degree of the human feedback mechanism.

[0149] Further, the reward function designed by the penalty function can be quickly learned by human intervention and automatic exploration, and the reward function is as follows:

[0150]

[0151] wherein I represents a unit matrix, represents the action space of the driving behavior, represents the dimension of the action space of the driving behavior, r pen represents the penalty function, r t represents the current reward function of the active exploration model.

[0152] In addition, the network parameters and the learning rate can be updated according to the gradient value of the loss function with respect to the target learning rate; and the parameters of the penalty of the human feedback mechanism and the reward function of the learning model are updated.

[0153] Figure 3 A flowchart of a decision management method of an automatic driving model provided by an embodiment of the present application is shown in FIG. 3. Figure 3 As shown in FIG. 3, the method comprises the following steps.

[0154] Step 301, determining a driving behavior corresponding to a driving task of a vehicle by using an automatic driving model, wherein the automatic driving model is obtained by self-evolution using the method described above.

[0155] Step 302, detecting whether the driving behavior determined by the automatic driving model meets a preset safe driving condition.

[0156] The safe driving condition comprises at least one of the following:

[0157] Condition 1: the motion trajectory of the driving behavior determined by the automatic driving model does not overlap with the motion trajectory of the surrounding obstacles.

[0158] Condition 2: the lateral distance between the motion trajectory of the driving behavior determined by the automatic driving model and the preset reference trajectory is less than a preset threshold.

[0159] The condition 1 can be detected by using a preset collision detection model, and the condition 2 can be detected by using a preset deviation event model.

[0160] Step 303, outputting the driving behavior determined by the automatic driving model after detecting that the driving behavior determined by the automatic driving model meets the safe driving condition.

[0161] If the safe driving condition includes condition 1 and condition 2, the collision possibility can be determined by using the collision detection model first, if the driving behavior and the movement trajectory of the surrounding obstacles do not exist collision possibility in the same space-time, the lateral distance between the input driving behavior trajectory and the reference trajectory is determined, if the lateral distance is not more than 1.5 meters, the driving behavior determined by the automatic driving model is output.

[0162] Optionally, after detecting that the driving behavior determined by the automatic driving model meets the safe driving condition, a preset rule driving model is used to determine the driving behavior corresponding to the automatic driving task, and the driving behavior determined by the rule driving model is output.

[0163] The rule driving model can reuse the perception information (such as traffic participant state information, map information, traffic rule information, etc.) collected by the automatic driving model perception function, and use the obtained perception information to make cognition and decision on the driving scene of the vehicle, such as lane changing opportunity, left turn, right turn, acceleration and deceleration, etc., to help the interactive planning module to limit the convex space and perform motion planning based on the reference trajectory to generate an avoidance trajectory; the bottom control module mainly tracks the generated avoidance trajectory to make the vehicle travel according to the trajectory.

[0164] The method provided by the embodiment of the application uses the automatic driving model to determine the driving behavior corresponding to the driving task of the vehicle, detects whether the driving behavior determined by the automatic driving model meets the preset safe driving condition, and outputs the driving behavior determined by the automatic driving model after detecting that the driving behavior determined by the automatic driving model meets the safe driving condition, thereby ensuring the safety and rationality of the driving behavior output by the automatic driving model, improving the trust of users on the automatic driving technology, and solving the problem of unexplainable behavior output by the automatic driving model in the prior art.

[0165] The specific application process of the automatic driving model will be described below.

[0166] Figure 4 The schematic diagram of the running method of the automatic driving model provided by the embodiment of the application is shown in FIG. 1. Figure 4 As shown in FIG. 1, the method comprises:

[0167] S1, obtaining a driving task, wherein the driving task can include environment perception and driving style of a user;

[0168] S2, determining the driving behavior corresponding to the driving task by using an automatic driving model;

[0169] S3, judging the driving behavior determined by the automatic driving model according to the safe driving condition, if the safe driving condition is met, directly outputting the driving behavior determined by the automatic driving model, otherwise outputting the driving behavior determined by the rule-based driving model.

[0170] Wherein, the automatic driving model is iteratively optimized by repeatedly performing S1 to S3 until the automatic driving task is completed.

[0171] The above content proposes the process of the automatic driving model outputting the driving behavior, and in the process of determining the driving behavior, the automatic driving model also performs the following operations, including:

[0172] The pre-training model is trained using prior knowledge, and the automatic driving model is updated based on the trained pre-training model to improve the training efficiency of the automatic driving model; at the same time, the pre-training model is fused with the active exploration model using the human feedback mechanism to update the automatic driving model, and the generalization of the automatic driving model is improved.

[0173] In addition, by obtaining the driving behavior loss function corresponding to the driving behavior determined by the automatic driving model, the difference information between the user and the automatic driving model is determined, and the driving behavior loss function is fed back to the automatic driving model to promote the automatic driving model to update, so that the updated automatic driving model can output driving behavior more in line with the user's driving style in the future.

[0174] Figure 5 The structure schematic diagram of the self-evolution device of the automatic driving model provided by the embodiments of the present application.

[0175] As shown in Figure 5 The device comprises:

[0176] The acquisition module 501 is configured to acquire the driving behavior determined by the user and the current automatic driving model for the same driving task;

[0177] The first determination module 502 is configured to generate a driving behavior loss function according to the difference information between the driving behavior determined by the user and the driving behavior determined by the current automatic driving model;

[0178] The update module 503 is configured to update the loss function of the current automatic driving model according to the driving behavior loss function, to obtain the loss function of the updated automatic driving model;

[0179] The self-evolution module 504 is configured to iteratively update the current automatic driving model using the loss function of the updated automatic driving model until convergence, to obtain the updated automatic driving model.

[0180] The apparatus provided in this application acquires the driving behaviors determined by the user and the current autonomous driving model for the same driving task. Based on the difference information between the driving behaviors determined by the user and the driving behaviors determined by the current autonomous driving model, a driving behavior loss function is generated. Then, the loss function of the current autonomous driving model is updated based on the driving behavior loss function to obtain the updated loss function of the autonomous driving model. The updated loss function of the autonomous driving model is then used to iterate the current autonomous driving model until convergence to obtain the updated autonomous driving model. This results in an autonomous driving model that further reduces the difference from the user's driving style, making it closer to the user's personalized driving style. This effectively reduces the difficulty for users to understand autonomous driving behavior and increases users' trust in autonomous driving technology.

[0181] Figure 6 This is a schematic diagram of the structure of the decision management device for the autonomous driving model provided in an embodiment of this application. Figure 6 As shown, the device includes:

[0182] The second determining module 601 is configured to use an autonomous driving model to determine the driving behavior corresponding to the driving task of the vehicle, wherein the autonomous driving model is obtained by self-evolution using the method described above.

[0183] The detection module 602 is configured to detect whether the driving behavior determined by the autonomous driving model meets the preset safe driving conditions;

[0184] The output module 603 is configured to output the driving behavior determined by the autonomous driving model after detecting that the driving behavior determined by the autonomous driving model meets the safe driving conditions.

[0185] The device provided in this application uses an autonomous driving model to determine the driving behavior corresponding to the driving task of the vehicle, detects whether the driving behavior determined by the autonomous driving model meets the preset safe driving conditions, and outputs the driving behavior determined by the autonomous driving model after detecting that the driving behavior determined by the autonomous driving model meets the safe driving conditions. This ensures the safety and rationality of the driving behavior output by the autonomous driving model, improves the user's trust in autonomous driving technology, and solves the problem of the unexplainable behavior output by the autonomous driving model in the prior art.

[0186] Figure 7 This is a schematic diagram of the structure of the self-evolution system of the autonomous driving model provided in the embodiments of this application.

[0187] like Figure 7 As shown, the system includes:

[0188] at least one vehicle configured to provide raw data required for the self-evolution operation of the automatic driving model;

[0189] a data server configured to process the received raw data into training data required for the self-evolution operation of the automatic driving model;

[0190] a model server configured to obtain the automatic driving model self-evolved by the method described above by using the training data, and update the automatic driving model deployed on the vehicle by using the self-evolved automatic driving model.

[0191] Optionally, the model server is configured to perform lightweight processing on the self-evolved automatic driving model to obtain a lightweight processed model, and send the lightweight processed model to the vehicle.

[0192] The vehicle can be an automatic driving development vehicle, a data collection vehicle, or an automatic driving mass production vehicle. The vehicle collects data such as vehicle running scenarios, environmental states, vehicle driving behaviors, and interactive driving trajectories, and uploads the collected data to the data server. In addition, the lightweight processed model sent by the model server is deployed on the vehicle end and verified for feasibility, and the lightweight processed model is iteratively optimized and data recycled.

[0193] The data server can be a cloud data factory. After receiving the data collected by the vehicle, the data server stores, cleans, classifies, extracts features, and mines features, and stores the data for training of the lightweight processed model deployed on the vehicle end and training of the automatic driving model in the cloud.

[0194] The model server is usually deployed in the cloud. By using the computing power platform and data platform in the cloud, the model server performs annotation and processing of data, deployment and training of algorithm models, and finally performs lightweight and engineering processing on the trained automatic driving model, and deploys the model to the vehicle end for testing and verification.

[0195] The three devices described above in the system form an automatic driving data closed loop flywheel system, which realizes the self-evolution process of the automatic driving model. In addition, as the data flows, more and more scenarios are solved, and less and less data needs to be collected, which can effectively reduce the data processing pressure of the system.

[0196] The embodiment of the application provides a storage medium, and the storage medium stores a computer program. When the computer program is executed, the method described in any one of the above embodiments is performed.

[0197] The embodiment of the application provides an electronic device, which comprises a memory and a processor. The memory stores a computer program, and the processor is configured to execute the computer program to perform the method described in any one of the above embodiments.

[0198] Those of ordinary skill in the art will realize and understand that all or some of the steps in the methods disclosed above and the functional modules / units in the systems and devices can be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component can have multiple functions, or one function or step can be performed by several physical components in cooperation. Some or all of the components can be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on computer-readable media, which can include computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those of ordinary skill in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Furthermore, it is common and well understood by those of ordinary skill in the art that communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and can include any information delivery media.

Claims

1. A self-evolution method for an autonomous driving model, characterized in that, include: Acquire the driving behavior determined by the user and the current autonomous driving model for the same driving task; A driving behavior loss function is generated based on the difference between the driving behavior determined by the user and the driving behavior determined by the current autonomous driving model. Based on the driving behavior loss function, the loss function of the current autonomous driving model is updated to obtain the updated loss function of the autonomous driving model. Using the loss function of the updated autonomous driving model, the current autonomous driving model is iterated until convergence to obtain the updated autonomous driving model; The step of generating a driving behavior loss function based on the difference information between the user-determined driving behavior and the driving behavior determined by the current autonomous driving model includes: The driving style corresponding to the driving behavior determined by the user is obtained to obtain first style data; and the driving style corresponding to the driving behavior determined by the current autonomous driving model is obtained to obtain second style data. The cross-entropy between the first style data and the second style data is calculated to obtain the driving behavior loss function.

2. The method according to claim 1, characterized in that, The methods for obtaining the driving style include: Obtain the user's driving behavior data; Determine the tags corresponding to the user's driving behavior data to obtain tag data; Based on the tag data, the user's driving style is determined.

3. The method according to claim 1, characterized in that, The step of updating the loss function of the current autonomous driving model based on the driving behavior loss function to obtain the updated loss function of the autonomous driving model includes: Calculate the product between the driving behavior loss function and the weights corresponding to the driving behavior loss function to obtain the first calculation result; The sum of the first calculation result and the loss function of the current autonomous driving model is calculated to obtain the loss function of the updated autonomous driving model.

4. The method according to claim 1, characterized in that: The current autonomous driving model is a transitional autonomous driving model obtained based on a pre-trained model and an active exploration model; or... The current autonomous driving model is a self-evolving autonomous driving model obtained based on the active exploration model and the transitional stage autonomous driving model, wherein the transitional stage autonomous driving model is obtained based on the pre-trained model and the active exploration model. The pre-trained model is trained using pre-defined prior knowledge, while the active exploration model is learned through exploration in a real driving environment.

5. The method according to claim 4, characterized in that, The current method for generating autonomous driving models includes: Based on the first loss function, the initial autonomous driving model is iteratively trained using the active exploration model and the pre-trained model until the model converges, thus obtaining the transition stage autonomous driving model. The first loss function is obtained by using the loss function of the pre-trained model and the loss function of the active exploration model. Based on the second loss function, the active exploration model is used to iteratively train the transition stage autonomous driving model until the model converges, thereby obtaining a self-evolving autonomous driving model, wherein the second loss function is obtained using the loss function of the active exploration model; In this process, the autonomous driving model obtained from each iteration of training is used as the active exploration model for the next iteration of training.

6. The method according to claim 5, characterized in that, The process of iteratively operating the initial autonomous driving model based on the first loss function, using the active exploration model and the pre-trained model until the model converges, to obtain the transition phase autonomous driving model includes: In each iteration of training, the network parameters of the initial autonomous driving model are updated using the first loss function and the reward function of the initial autonomous driving model, wherein the reward function of the initial autonomous driving model is obtained using the reward function of the active exploration model.

7. The method according to claim 5, characterized in that, The step of iteratively operating the transition phase autonomous driving model based on the second loss function using an active exploration model until the model converges, to obtain a self-evolving autonomous driving model, includes: In each iteration, the network parameters of the transition phase autonomous driving model are updated using the second loss function and the reward function of the transition phase autonomous driving model, wherein the reward function of the transition phase autonomous driving model is obtained using the reward function of the active exploration model.

8. The method according to claim 5, characterized in that, The method for obtaining the loss function of the pre-trained model includes: After training the current pre-trained model is completed, the gradient value is obtained by the difference between the feature expectation determined by prior knowledge and the feature expectation of the pre-trained model. The gradient value is used to update the current weight matrix and reward function, wherein the updated reward function is obtained using the updated weight matrix; The updated weight matrix is ​​used to generate the loss function for the pre-trained model.

9. The method according to claim 8, characterized in that, The update method for the updated weight matrix includes: Calculate the product between the gradient value and the preset learning rate to obtain the second calculation result; The sum of the second calculation result and the current weight matrix is ​​obtained to get the updated weight matrix.

10. The method according to claim 4, characterized in that: The training data of the active exploration model is divided into a first part and a second part, wherein the first part is used to learn driving behavior, and the second part is used to learn human feedback mechanisms.

11. The method according to claim 10, characterized in that: The loss function of the active exploration model is represented by a deep reinforcement learning network based on the optimal value function, where: When representing the loss function of the active exploration model through a deep reinforcement learning network, the loss function of the active exploration model includes weights to represent the importance of human feedback mechanisms.

12. The method according to claim 11, characterized in that: The updated reward function of the active exploration model The expressions include: Where I represents the identity matrix, The action space representing driving behavior The dimension representing the action space of driving behavior. Represents the penalty function. This represents the current reward function of the active exploration model.

13. A decision management method for an autonomous driving model, characterized in that, include: Using an autonomous driving model, the driving behavior corresponding to the driving task of the vehicle is determined, wherein the autonomous driving model is obtained by self-evolution using the method described in any one of claims 1 to 12; The system detects whether the driving behavior determined by the autonomous driving model meets the preset safe driving conditions. After detecting that the driving behavior determined by the autonomous driving model meets the safe driving conditions, the driving behavior determined by the autonomous driving model is output.

14. The method according to claim 13, characterized in that, The method further includes: After detecting that the driving behavior determined by the autonomous driving model does not meet the safe driving conditions, the driving behavior corresponding to the autonomous driving task is determined using a preset rule driving model, and the driving behavior determined by the rule driving model is output.

15. The method according to claim 13, characterized in that, The safe driving conditions include at least one of the following: The trajectory of the driving behavior determined by the autonomous driving model does not overlap with the trajectory of the surrounding obstacles. The lateral distance between the driving trajectory determined by the autonomous driving model and the preset reference trajectory is less than a preset threshold.

16. A self-evolving device for an autonomous driving model, characterized in that, include: The acquisition module is configured to acquire the driving behavior determined by the user and the current autonomous driving model for the same driving task; The first determining module is configured to generate a driving behavior loss function based on the difference information between the driving behavior determined by the user and the driving behavior determined by the current autonomous driving model. The update module is configured to update the loss function of the current autonomous driving model based on the driving behavior loss function, so as to obtain the updated loss function of the autonomous driving model. The self-evolution module is configured to use the loss function of the updated autonomous driving model to iterate the current autonomous driving model until convergence, thereby obtaining the updated autonomous driving model. The step of generating a driving behavior loss function based on the difference information between the user-determined driving behavior and the driving behavior determined by the current autonomous driving model includes: The driving style corresponding to the driving behavior determined by the user is obtained to obtain first style data; and the driving style corresponding to the driving behavior determined by the current autonomous driving model is obtained to obtain second style data. The cross-entropy between the first style data and the second style data is calculated to obtain the driving behavior loss function.

17. A decision management device for an autonomous driving model, characterized in that, include: The second determining module is configured to use an autonomous driving model to determine the driving behavior corresponding to the driving task of the vehicle, wherein the autonomous driving model is obtained by self-evolution using the method described in any one of claims 1 to 12. The detection module is configured to detect whether the driving behavior determined by the autonomous driving model meets the preset safe driving conditions; The output module is configured to output the driving behavior determined by the autonomous driving model after detecting that the driving behavior determined by the autonomous driving model meets the safe driving conditions.

18. A self-evolving system for an autonomous driving model, characterized in that, include: At least one vehicle is configured to provide the raw data required for an autonomous driving model to perform self-evolving operations. The data server is configured to process the received raw data into training data required for the autonomous driving model to perform self-evolutionary operations. A model server is configured to use the training data to employ an autonomous driving model obtained through self-evolution as described in any one of claims 1 to 12, and to update the autonomous driving model deployed on the vehicle using the self-evolutionary autonomous driving model.

19. The system according to claim 18, characterized in that: The model server is configured to perform lightweight processing on the self-evolved autonomous driving model to obtain a lightweight model, and then send the lightweight model to the vehicle.

20. A storage medium storing a computer program, characterized in that, The computer program is configured to execute the method as described in any one of claims 1 to 12 or the method as described in any one of claims 13 to 15 when it is run.

21. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method as claimed in any one of claims 1 to 12 or the method as claimed in any one of claims 13 to 15.

Images