Safety reinforcement learning training method and device for end-to-end autonomous driving

Abstract

The application relates to the technical field of automatic driving, in particular to a safety reinforcement learning training method and device for end-to-end automatic driving, wherein the method comprises the following steps: detecting the safety state of a current decision and control strategy of a vehicle based on a preset driving environment model and an uncertainty constraint attenuation function; fitting the scattered random error of the preset driving environment model by using a preset integrated model; updating a value function according to a self-consistent condition of the value function, and evaluating the performance value of the current decision and control strategy; updating the uncertainty constraint attenuation function according to a self-consistent condition of the uncertainty constraint attenuation function, and expanding the corresponding feasible region; when the safety state reaches a preset level, iteratively updating the current decision and control strategy based on the random error, the performance value and the expanded feasible region until the average cumulative return of the current decision and control strategy reaches a preset threshold or reaches a maximum iteration number, and obtaining a final feasible decision and control strategy to strengthen a target automatic driving system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, and in particular to a safety reinforcement learning training method and device for end-to-end automatic driving. BACKGROUND

[0002] In the related art, an end-to-end automatic driving system neuralizes each module neural network of the automatic driving system, that is, all modules are differentiable, and then the driving performance of the system can be improved through the collection of a large amount of driving data and the backward gradient propagation. According to different deep learning algorithms, the decision and control modules of the end-to-end automatic driving system can be divided into two categories: supervised learning type decision and control modules and brain-like learning type decision and control modules. The supervised learning type decision and control modules are trained by a supervised learning algorithm, and need hundreds of billions of kilometers of artificial labeled driving data for training. However, it is difficult to collect high-quality driving data, and the labeling cost is high. The brain-like learning type decision and control modules are trained by a reinforcement learning algorithm, which collects driving data through exploration interaction with a real (or simulated) driving environment, and does not need artificial labeling. The brain-like learning type decision and control modules have the advantages of self-evolution and controllable cost, and have broad application prospects.

[0003] However, the training of the brain-like learning type decision and control modules in the related art is limited to exploration interaction in a simulated driving environment, and the performance degrades greatly when migrated to a real driving environment, which seriously affects the performance of the decision and control modules trained in the simulation in the real driving environment, cannot effectively guarantee the actual performance of the trained decision and control modules and the safety in the process of collecting data through exploration interaction in the real driving environment, and cannot meet the training needs of high-level end-to-end automatic driving system decision and control modules, which needs to be solved urgently. SUMMARY

[0004] The present application provides a safety reinforcement learning training method and device for end-to-end automatic driving to solve the problems that the training of the brain-like learning type decision and control modules in the related art is limited to exploration interaction in a simulated driving environment, and the performance degrades greatly when migrated to a real driving environment, which seriously affects the performance of the decision and control modules trained in the simulation in the real driving environment, cannot effectively guarantee the actual performance of the trained decision and control modules and the safety in the process of collecting data through exploration interaction in the real driving environment, and cannot meet the training needs of high-level end-to-end automatic driving system decision and control modules.

[0005] The first aspect embodiment of the application provides a safety reinforcement learning training method for end-to-end automatic driving, including the following steps: detecting a safety state of a current decision and control strategy of a vehicle based on a preset driving environment model and an uncertainty constraint attenuation function, so as to collect a predetermined amount of environment interaction data in the case that the safety state reaches a preset state; fitting a random error of the preset driving environment model by using a preset integrated model to obtain a fitted random error distribution; obtaining a value function of the current decision and control strategy, and updating the value function according to a self-consistent condition of the value function to obtain an updated value function, so as to evaluate a performance value of the current decision and control strategy by using the updated value function; updating an uncertainty constraint attenuation function of the current decision and control strategy according to a self-consistent condition of the uncertainty constraint attenuation function to obtain an updated uncertainty constraint attenuation function, so as to expand a feasible region corresponding to the updated uncertainty constraint attenuation function to obtain an expanded feasible region; iteratively updating the current decision and control strategy based on the environment interaction data, the random error distribution, the performance value and the expanded feasible region until an average cumulative return of the current decision and control strategy reaches a preset threshold or reaches a maximum iteration number, to obtain a final feasible decision and control strategy, so as to strengthen a target automatic driving system according to the final feasible decision and control strategy.

[0006] Optionally, in an embodiment of the application, the detection of the safety state of the current decision and control strategy of the vehicle based on the preset driving environment model and the uncertainty constraint attenuation function includes: generating at least one safe action of the vehicle by using the current decision and control strategy of the vehicle; adding exploration noise to the safe action to generate an exploration action; determining a safety state of the exploration action according to the preset driving environment model and the uncertainty constraint attenuation function, so as to detect the safety state of the current decision and control strategy of the vehicle.

[0007] Optionally, in an embodiment of the application, before fitting the random error of the preset driving environment model by using the preset integrated model, the method further includes: initializing a preset number of neural networks of the same structure to obtain initialized neural networks; training the initialized neural networks by using target data to obtain trained neural networks; training the current decision and control strategy, the value function and the uncertainty constraint attenuation function by using the trained neural networks.

[0008] Optionally, in an embodiment of the application, the updating of the value function according to the self-consistent condition of the value function to obtain the updated value function includes: generating virtual driving data by using the preset driving environment model; updating the value function by using the virtual driving data based on the self-consistent condition of the value function to obtain the updated value function.

[0009] Optionally, in an embodiment of the present application, the uncertainty constraint decay function is:

[0010]

[0011] wherein, is the number of state transitions required to first violate the state constraint starting from state x under the control of policy π, γ is a discount factor, and φ is a trainable parameter of the uncertainty constraint decay function.

[0012] The second aspect embodiment of the present application provides a safety reinforcement learning training device for end-to-end autonomous driving, comprising: a detection module configured to detect a safe state of a current decision and control policy of a vehicle based on a preset driving environment model and an uncertainty constraint decay function, so as to collect a predetermined amount of environment interaction data in the case that the safe state reaches a preset state; a fitting module configured to fit a random error of the preset driving environment model using a preset integrated model to obtain a fitted random error distribution; an evaluation module configured to obtain a value function of the current decision and control policy, and update the value function according to a self-consistent condition of the value function to obtain an updated value function, so as to evaluate a performance value of the current decision and control policy using the updated value function; an expansion module configured to update an uncertainty constraint decay function of the current decision and control policy according to a self-consistent condition of the uncertainty constraint decay function to obtain an updated uncertainty constraint decay function, so as to expand a feasible region corresponding to the updated uncertainty constraint decay function to obtain an expanded feasible region; and a reinforcement module configured to iteratively update the current decision and control policy based on the environment interaction data, the random error distribution, the performance value, and the expanded feasible region until an average cumulative return of the current decision and control policy reaches a preset threshold or reaches a maximum number of iterations, to obtain a final feasible decision and control policy, and to reinforce a target autonomous driving system according to the final feasible decision and control policy.

[0013] Optionally, in an embodiment of the present application, the detection module comprises: a first generation unit configured to generate at least one safe action of the vehicle using a current decision and control policy of the vehicle; a second generation unit configured to add exploration noise to the safe action to generate an exploration action; and a determination unit configured to determine a safe state of the exploration action according to the preset driving environment model and the uncertainty constraint decay function, so as to detect a safe state of the current decision and control policy of the vehicle.

[0014] Optionally, in an embodiment of the present application, the method further comprises: initializing a preset number of neural networks of the same structure to obtain initialized neural networks before fitting the random error of the preset driving environment model by the preset integrated model; training the initialized neural networks by using target data to obtain trained neural networks; and training the current decision and control policy, the value function and the uncertainty constraint attenuation function by using the trained neural networks.

[0015] Optionally, in an embodiment of the present application, the evaluation module comprises: a third generation unit configured to generate virtual driving data by using the preset driving environment model; and an update unit configured to update the value function by using the virtual driving data based on a self-consistent condition of the value function to obtain an updated value function.

[0016] Optionally, in an embodiment of the present application, the uncertainty constraint attenuation function is:

[0017]

[0018] wherein, is the number of state transitions required to first violate the state constraint under the control of the policy starting from state x, γ is a discount factor, and φ is a trainable parameter of the uncertainty constraint attenuation function.

[0019] The third aspect embodiment of the present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the safety reinforcement learning training method for end-to-end autonomous driving as described in the above embodiments.

[0020] The fourth aspect embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the safety reinforcement learning training method for end-to-end autonomous driving as described above.

[0021] The fifth aspect embodiment of the present application provides a computer program product, comprising a computer program, and the computer program is executed to implement the safety reinforcement learning training method for end-to-end autonomous driving as described above.

[0022] ​The embodiment of the application can limit the exploration interaction of the vehicle and the environment within the feasible region induced by the uncertainty constraint decay function, construct a safe reinforcement learning algorithm training method composed of safe exploration, model error estimation, standard policy evaluation, feasible region expansion and feasible policy improvement, obtain an improved performance decision and control strategy, and strengthen the end-to-end automatic driving system. Therefore, the iteration process of the algorithm training is optimized, the complexity of the algorithm training is reduced, and the performance of the decision and control strategy is effectively improved under the premise of ensuring the driving safety of the brain-like learning decision and control module in the real driving environment. Therefore, the problems in the related art that the brain-like learning decision and control module training is limited to the exploration interaction in the simulation driving environment, and the performance degrades greatly when migrated to the real driving environment, seriously affecting the performance of the decision and control module obtained by simulation training in the real driving environment, and being unable to effectively guarantee the actual performance of the trained decision and control module and the safety in the data collection process of the real driving environment exploration interaction, and being unable to meet the training requirements of the high-level end-to-end automatic driving system decision and control module are solved.

[0023] Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following and / or can be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0024] The above and / or additional aspects and advantages of the application will become apparent and be readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0025] Figure 1 A flowchart of a safe reinforcement learning training method for end-to-end automatic driving according to an embodiment of the application;

[0026] Figure 2 A flowchart of a safe exploration algorithm for an embodiment of the application;

[0027] Figure 3 A schematic diagram of an uncertainty constraint decay function definition method for an embodiment of the application;

[0028] Figure 4 A flowchart of a safe reinforcement learning training method for end-to-end automatic driving according to an embodiment of the application;

[0029] Figure 5 A structural schematic diagram of a safe reinforcement learning training device for end-to-end automatic driving according to an embodiment of the application;

[0030] Figure 6 A structural schematic diagram of an electronic device according to an embodiment of the application.

[0031] REFERENCE NUMERALS

[0032] 10 - Safety Reinforcement Learning Training Device for End-to-End Autonomous Driving: 100 - Detection Module, 200 - Fitting Module, 300 - Evaluation Module, 400 - Expansion Module, and 500 - Reinforcement Module; 601 - Memory, 602 - Processor, and 603 - Communication Interface. DETAILED DESCRIPTION

[0033] Embodiments of the present application are described below in detail, examples of which are shown in the accompanying drawings, in which the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation on the present application.

[0034] The safety reinforcement learning training method and device for end-to-end autonomous driving of the embodiments of the present application are described below with reference to the accompanying drawings. In view of the problem that the related art brain-like learning type decision and control module training is limited to exploration interaction in a simulation driving environment and greatly degrades in performance when migrated to a real driving environment, seriously affecting the performance of the decision and control module obtained by simulation training in a real driving environment, cannot effectively guarantee the actual performance of the trained decision and control module and the safety in the process of collecting data by exploration interaction in a real driving environment, and cannot meet the training requirements of a high-level end-to-end autonomous driving system decision and control module, the present application provides a safety reinforcement learning training method for end-to-end autonomous driving. In the method, the exploration interaction between the vehicle and the environment can be limited to a feasible region induced by an uncertainty constraint decay function, a safety reinforcement learning algorithm training method composed of safety exploration, model error estimation, standard policy evaluation, feasible region expansion, and feasible policy improvement is constructed, and a decision and control policy with improved performance is obtained to strengthen the end-to-end autonomous driving system. Thus, the iteration process of algorithm training is optimized under the premise of ensuring the driving safety of the brain-like learning type decision and control module when trained in a real driving environment, the complexity of algorithm training is reduced, and the performance of the decision and control policy is effectively improved. Thus, the problems such as the related art brain-like learning type decision and control module training being limited to exploration interaction in a simulation driving environment and greatly degrading in performance when migrated to a real driving environment, seriously affecting the performance of the decision and control module obtained by simulation training in a real driving environment, cannot effectively guarantee the actual performance of the trained decision and control module and the safety in the process of collecting data by exploration interaction in a real driving environment, and cannot meet the training requirements of a high-level end-to-end autonomous driving system decision and control module are solved.

[0035] Before explaining the safety reinforcement learning training method for end-to-end autonomous driving in the embodiments of the present application, the elements involved in the safety reinforcement learning training method for end-to-end autonomous driving in the embodiments of the present application and the end-to-end autonomous driving system are explained.

[0036] An autonomous vehicle refers to a new generation of vehicles that are equipped with advanced on-board sensors, controllers, actuators and other devices, have functions such as complex environment perception, intelligent decision-making and cooperative control, and can realize safe, efficient, comfortable and energy-saving driving, and ultimately replace human operation. The autonomous driving capability of a vehicle is divided into six levels, from level 0 to level 5, where level 0 is no automation, and level 5 is full automation. To achieve high-level autonomous driving capability of level 3 and above, an end-to-end autonomous driving system has great potential. The end-to-end autonomous driving system refers to a neural network of perception, prediction, planning, decision-making and control modules, which is trained by using artificially annotated (or unannotated) driving data through a deep learning algorithm to obtain an autonomous driving system.

[0037] The safety reinforcement learning training method for end-to-end autonomous driving in the embodiments of the present application includes a cycle process, which needs to initialize the following elements before starting: a real data pool a virtual data pool an initial decision-making and control policy π θ a value function and an uncertainty constraint attenuation function an integrated model of driving environment random errors where θ, w1, w2, φ, ψ = [ψ1, ψ2, …, ψ M ] are trainable parameters, and M is the number of sub-models of the integrated model.

[0038] Specifically, Figure 1 a flowchart of a safety reinforcement learning training method for end-to-end autonomous driving provided by the embodiments of the present application.

[0039] As shown in Figure 1 , the safety reinforcement learning training method for end-to-end autonomous driving includes the following steps:

[0040] In step S101, based on a preset driving environment model and an uncertainty constraint attenuation function, the safety state of the current decision-making and control policy of the vehicle is detected to collect a predetermined amount of environment interaction data in the case that the safety state reaches a preset state.

[0041] where the uncertainty constraint attenuation function can be expressed as:

[0042]

[0043] where is the number of state transitions required to first violate the state constraint under the control of the policy π from the state x under the control of the random error model ζ of the preset driving environment model, and if the current state has violated the constraint, If the future state will never violate the constraints. γ is the discount factor.

[0044] In some embodiments, the preset driving environment model can be understood as a partially known driving environment model trained using real driving data. The preset driving environment model in this application embodiment can be divided into a known part and an unknown part. The known part can be a dynamic model determined based on physical laws, identifying traffic participants (including motor vehicles, non-motor vehicles, pedestrians, etc.). The unknown part can be a stochastic error model describing the errors of the dynamic model and unmodeled dynamics. Let f represent the dynamic model and ζ represent the stochastic error model. The preset driving environment model can be defined as:

[0045] x t+1 =f(x) t ,u t )+ζ(x t ,u t (1)

[0046] Where t is the time variable, x is the state variable, u is the action variable, f is an analytical function, and ζ is a bounded random error model. In the embodiments of this application, r = r(x,u) can be additionally defined as the reward function, and h(x) ≤ 0 as the state constraint.

[0047] The uncertainty constraint attenuation function can be understood here as a constraint attenuation function with uncertainty, and can be expressed as follows:

[0048]

[0049] in, It is the number of state transitions required to first violate state constraints starting from state x under the control of the decision-making strategy π. If the current state has already violated the constraints... If the future state will never violate the constraints. γ is the discount factor, and φ is the trainable parameter of the uncertainty constraint decay function.

[0050] Based on a preset driving environment model and an uncertainty constraint decay function, this application embodiment can detect the safety of the vehicle's current decision-making strategy and determine the safety status of the current decision-making strategy, that is, the safety of performing a certain action under the current decision-making strategy, so as to collect a certain amount of environmental interaction data for subsequent steps under safe conditions.

[0051] It should be noted that the state in the embodiments of the present application refers to an environmental state, including but not limited to a self-vehicle state such as a vehicle position and a speed, and a surrounding driving environment state output by a perception module and a prediction module; the action can be understood as including but not limited to a control amount such as a vehicle accelerator pedal opening, a brake pedal opening, and a steering wheel rotation angle; the safety can be understood as no personal and property loss or other adverse consequences; and the exploration can be understood as a process in which a vehicle executor actually executes an action and causes interaction between the vehicle and a driving environment.

[0052] If the environmental state is safe, it is determined that the exploration action is safe, and the exploration action is actually executed to collect samples by the vehicle exploring interaction in the driving environment; if the environmental state is unsafe, it is determined that the exploration action is unsafe, the exploration action is reacquired until a safe action is found or a maximum number of times is reached. When the maximum number of times is reached, the safe action without the exploration noise is directly executed to collect samples by the vehicle exploring interaction in the driving environment.

[0053] The embodiments of the present application can use a preset driving environment model learned and a target constraint attenuation function with uncertainty to determine the safety of the exploration action, so as to maximize the exploration of the strategy on the premise of ensuring the safety of the environmental interaction.

[0054] Next, the process is further described.

[0055] Optionally, in an embodiment of the present application, based on the preset driving environment model and the uncertainty constraint attenuation function, the safety state of the current decision and control strategy of the vehicle is detected, including: generating at least one safe action of the vehicle by using the current decision and control strategy of the vehicle; adding exploration noise to the safe action to generate an exploration action; and determining the safety state of the exploration action according to the preset driving environment model and the uncertainty constraint attenuation function, to detect the safety state of the current decision and control strategy of the vehicle.

[0056] In actual execution, the preset driving environment model learned and the target constraint attenuation function with uncertainty can be used to determine the safety state of the current decision and control strategy.

[0057] Specifically, first, at least one safe action of the vehicle can be generated by using the current decision and control strategy, and an exploration action can be generated by adding exploration noise to the safe action; second, a next time state that can be transferred by using the current environmental state of the preset driving environment model and the exploration action is predicted, and the safety state of all next time states is determined by using the constraint attenuation function with uncertainty.

[0058] For example, Figure 2 A flowchart of a safety exploration algorithm of an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the safety exploration algorithm of the embodiment of the present application includes the following steps. Figure 2As shown, this application can repeatedly invoke the safety exploration algorithm to collect driving data in a real driving environment.

[0059] (1) The security exploration algorithm requires the current environment state x and the initial decision control strategy π as input. θ Uncertainty-constrained decay function Stochastic error model ζ for driving environment, safety threshold δ s and explore noise distribution d ∈ .

[0060] (2) Generate at least one safety action u based on the current state x. safe :

[0061] u safe =π θ (x), (2)

[0062] (3) Initialize the exploration action generation count variable i = 0.

[0063] (4) Determine if the count variable i generated by the exploration action is less than the maximum number of times K. If the count variable i generated by the exploration action is not less than the maximum number of times K, then change the action u. safe When applied to the vehicle, the environmental state transitions to x′, and a reward signal r is acquired.

[0064] (5) If the count variable i generated by the exploration action is less than the maximum number of times K, then according to the exploration noise distribution d ∈ Sampling yields exploration noise ∈ to obtain exploration action u explore :

[0065] u explore =u safe +∈,∈~d ε (3)

[0066] (6) Using the learned driving environment random error model ζ and uncertainty constraint decay function Determine the safety of the action and calculate the possible environmental state in the next moment. It should be noted that the next time-instance state here is predicted using the environmental model, not obtained from environmental feedback, and different environmental random error models with different parameters can yield different next time-instance states. Therefore, the subscript j is used to distinguish them.

[0067]

[0068] (7) Determine all possible environmental states for the next time step. Whether it is safe depends on whether the following formula is satisfied:

[0069]

[0070] determining whether all possible next-time environment states are safe if the exploration action u explore is safe.

[0071] (8) if the exploration action u explore is safe, the vehicle actually performs the exploration action u explore .

[0072] (9) the vehicle transitions to the next-time environment state and obtains the reward signal r.

[0073] (10) the driving data sample (x, u explore , x ′ , r) obtained in the process is added to the real data pool .

[0074] (11) if the exploration action u explore is unsafe, the above process is repeated until the maximum exploration action sampling number K is reached. If the exploration action u explore is still unsafe when the maximum sampling number K is reached, the vehicle actually performs the safe action u safe obtained from formula (3), the vehicle transitions to the next-time environment state and obtains the reward signal r. The driving data sample (x, u safe , x ′ , r) obtained in the process is added to the real data pool .

[0075] The embodiment of the present application can explore whether the next-time state of the preset driving environment model is safe, and ensure that the actions performed by the vehicle when exploring the environment in the reinforcement learning algorithm training process are all safe, i.e., do not violate the constraints and do not produce unsafe consequences.

[0076] In step S102, the preset integrated model is used to fit the random error of the preset driving environment model to obtain the fitted random error distribution.

[0077] In some embodiments, both the inaccurate deterministic dynamics modeling and the unmodeled dynamic characteristics in the preset driving environment model can produce certain errors. In order to reduce the inaccuracy in the final reinforcement training, the preset integrated model can be used to fit the random error part of the preset driving environment model, i.e., the errors produced by the inaccurate deterministic dynamics modeling and the unmodeled dynamic characteristics, to obtain the fitted usable random error distribution.

[0078] Next, the preset integrated model in the embodiment of the present application is further described.

[0079] Optionally, in one embodiment of this application, before fitting the dispersed random error of the preset driving environment model through the preset integrated model, the method further includes: initializing a preset number of neural networks with the same structure to obtain an initialized neural network; training the initialized neural network using target data to obtain a trained neural network; and training the current decision control strategy, value function, and uncertainty constraint decay function using the trained neural network.

[0080] In practical implementation, this application can use a preset ensemble model to fit the random error portion of a preset driving environment model. The preset ensemble model can, but is not limited to, initializing a preset number of neural networks with the same structure, and then training each neural network using target data. The trained neural networks can then form a preset ensemble model. Here, the preset number can be understood as a pre-set quantity, which can be set or adjusted by those skilled in the art according to actual circumstances or needs. This is merely an illustrative example and does not impose any specific limitations.

[0081] Under this training, the outputs of each neural network can form the atomic distribution estimate of the random error component of the preset driving environment model, i.e., the preset ensemble model. By fitting the random error component of the driving environment model using the preset ensemble model, it can be used to support subsequent processing.

[0082] For example, this application can use M neural networks with the same structure, each initialized under a different random seed, and in a data replay pool. Independent batches of data are collected for training. The loss function used in this process can be expressed as follows:

[0083]

[0084] The training of the preset driving environment model mainly involves the parameters ψ in the preset driving environment random error model. i =[ψ1,ψ2,…,ψ M The update of ] can be represented as follows:

[0085]

[0086] Where, α model It is the learning rate of the parameters of the random error model for the driving environment.

[0087] The embodiments of this application can use a preset ensemble model to simultaneously fit the errors caused by inaccurate dynamic modeling and the errors caused by unmodeled dynamic characteristics, and can fit the random error model of the driving environment into an atomic distribution, thereby avoiding the introduction of additional iterative optimization processes when updating the uncertainty constraint decay function and exploring the environment, and reducing the complexity of algorithm training.

[0088] In step S103, the value function of the current decision control policy is obtained, and the value function is updated according to the self-consistent condition of the value function to obtain an updated value function, so as to evaluate the performance value of the current decision control policy by using the updated value function.

[0089] In some embodiments, the application can obtain the value function of the current decision control policy, and update the value function of the current decision control policy using the self-consistent condition of the value function. The performance of the current decision control policy in a given state is evaluated by using the value function, which is further used in the subsequent feasible strategy improvement step.

[0090] The process is further described as follows.

[0091] Optionally, in an embodiment of the application, updating the value function according to the self-consistent condition of the value function to obtain an updated value function comprises: generating virtual driving data by using a preset driving environment model; and updating the value function by the virtual driving data based on the self-consistent condition of the value function to obtain the updated value function.

[0092] In actual implementation, when the application updates the value function of the strategy using the self-consistent condition of the value function of the current decision control policy, a large amount of virtual driving data can be generated by using the trained preset driving environment model, but is not limited to this.

[0093] Specifically, first, virtual driving data can be generated by using the preset driving environment model and stored in a virtual data pool to learn two value functions and The loss function used in the training can be expressed as follows:

[0094]

[0095] wherein, and are two value functions corresponding to the target value function, γ is a discount factor, α e is an entropy regularization coefficient, u ′ represents the action output by the policy at the next moment.

[0096] The embodiments of the application can generate a large amount of virtual driving data by using the preset driving environment model for value function estimation, feasible region expansion and strategy improvement, thereby deeply mining the information contained in the real driving data, improving the sample utilization efficiency of the algorithm, and reducing the demand for real driving data of the algorithm.

[0097] In step S104, the uncertainty constraint attenuation function of the current decision and control policy is updated according to the self-consistent condition of the uncertainty constraint attenuation function, to obtain an updated uncertainty constraint attenuation function, so as to expand the feasible region corresponding to the updated uncertainty constraint attenuation function, and obtain an expanded feasible region.

[0098] Based on the related description of other embodiments, it can be understood that the application can use a preset integrated model to fit the random error part of the preset driving environment model, and further, the random error obtained by fitting can be used to support the expansion of the strategy feasible region (i.e., the explorable region of the vehicle in the driving environment) learned by the proposed algorithm.

[0099] As a possible implementation manner, the embodiments of the application can use the self-consistent condition of the uncertainty constraint attenuation function to update the uncertainty constraint attenuation function corresponding to the current decision and control policy, and expand the corresponding feasible region, so as to expand the explorable region of the vehicle in the driving environment by using a larger feasible region, and support the performance improvement of the decision and control policy.

[0100] For example, the application can first use the data in the virtual data pool to identify the feasible region corresponding to the current decision and control policy. Figure 3 The schematic diagram of the uncertainty constraint attenuation function of an embodiment of the application for expanding the feasible region. As shown in Figure 3 , the uncertainty constraint attenuation function can be represented as follows:

[0101]

[0102] wherein, is the number of state transitions required to first violate the state constraint under the control of the policy π starting from state x, if the current state has already violated the constraint, if the future state will never violate the constraint, γ is a discount factor, and φ represents a trainable parameter of the uncertainty constraint attenuation function.

[0103] The self-consistent condition of the uncertainty constraint attenuation function can be represented as:

[0104]

[0105] wherein, the definition of the function c(x) can be represented as follows:

[0106]

[0107] wherein, the definition of the constraint state set X cstr can be represented as follows:

[0108]

[0109] wherein, is the state space, h(x) is the state constraint function, and the set is the constraint state set X cstr , the complement set of X

[0110]

[0111] The self-consistent condition shown in formula (9) is a compression mapping, that is, the trainable parameter φ of the uncertainty constraint attenuation function is iterated according to the self-consistent condition and can converge to the uncertainty constraint attenuation function The loss function used in training is the cross-entropy loss, and its definition can be represented as follows:

[0112]

[0113] wherein,

[0114]

[0115] The uncertainty constraint attenuation function induced by the feasible region can be represented as follows:

[0116]

[0117] wherein, δ s is a safety threshold, which can be taken as zero in theory, but due to the existence of learning errors in the actual training process, it can be selected as a small positive number according to the actual situation. For example, when the maximum value of the preset single state trajectory length is 200, δ s = 0.1 can be set. are the uncertainty constraint attenuation function values corresponding to the current state and the next state, respectively.

[0118] The self-consistent condition of the uncertainty constraint attenuation function in the embodiments of the present application can induce a compression mapping, only need to initialize any parameterized function with sufficient expression ability, and use the self-consistent condition to iterate repeatedly, and the function will converge to the uncertainty constraint attenuation function corresponding to the decision control policy. It should be noted that the feasible region induced by the uncertainty constraint attenuation function in the above steps is monotonically expanded.

[0119] Step S105, based on the environmental interaction data, the random error distribution, the performance value and the extended feasible region, the current decision and control policy is iteratively updated until the average cumulative return of the current decision and control policy reaches a preset threshold or reaches a maximum iteration number, and a final feasible decision and control policy is obtained, so as to strengthen the target automatic driving system according to the final feasible decision and control policy.

[0120] In some other embodiments, in the case that the safety state of the current decision and control policy of the vehicle reaches a preset state, i.e., the current decision and control policy is safe, the current decision and control policy can be iteratively updated based on the environmental interaction data, the random error distribution, the performance value and the extended feasible region, the performance of the current decision and control policy is improved, and then the target automatic driving system to be strengthened is trained using the improved decision and control policy.

[0121] Further, in the embodiments of the present application, when the current decision and control policy is iteratively updated, a regional policy updating mechanism can be used, i.e., different policy improvement methods are used inside and outside the feasible region.

[0122] In the feasible region indicated by the uncertainty constraint attenuation function, in addition to improving the performance of the policy, the feasibility of the policy should also be ensured, so the policy improvement is defined as a constrained optimization problem, after being converted into an unconstrained problem using the penalty function method, the policy gradient method is used to update the policy.

[0123] Among them, the updating method in the feasible region can be defined as the following constrained optimization problem:

[0124]

[0125] Outside the feasible region, only the maximization of the policy feasibility in the worst case is considered, i.e., the minimization of the constraint violation, and the policy gradient method is used to update the policy. Thus, the performance of the decision and control policy is improved under the premise of ensuring the safety of the decision and control policy. Its definition can be expressed as follows:

[0126]

[0127] In the actual training of the algorithm, the loss function of the policy is defined as follows:

[0128]

[0129] Among them,

[0130]

[0131] Among them, λ is a coefficient for balancing the loss function inside and outside the feasible region, η(x ′ ) is an indicator function, which can be defined as:

[0132]

[0133] In the examples of the present application, the formula (19) is may be defined as follows:

[0134]

[0135] where, α e is the same parameter as in formula (8), i.e., the entropy regularization coefficient.

[0136] This step takes a regional strategy improvement mechanism, which uses different optimization objectives inside and outside the feasible region. This mechanism ensures that the performance of the strategy is monotonically improved within the feasible region. Outside the feasible region, maximizing the feasibility of the strategy can bring potential feasible states into the feasible region.

[0137] The above steps are repeatedly run, and the algorithm terminates when any of the following conditions is met: (1) the average cumulative return reaches a set threshold, i.e., the algorithm converges; (2) the number of algorithm iterations reaches a set maximum value.

[0138] The present application will be described in detail below with a specific embodiment.

[0139] Step S401, start.

[0140] Step S402, initialize parameters, such as real data pool virtual data pool initial strategy π θ value function and uncertainty-constrained decay function integrated model ζ of random errors of driving environment ψ .

[0141] Step S403, use the learned preset driving environment model and the uncertainty-constrained decay function to explore whether the next time environment state is safe.

[0142] Step S404, use the integrated model to fit the random errors of the driving environment model.

[0143] Step S405, use the self-consistent condition of the value function to update the value function of the strategy, and evaluate the performance of the current decision control strategy under a given state.

[0144] Step S406, use the self-consistent condition of the uncertainty-constrained decay function to update the uncertainty-constrained decay function corresponding to the current decision control strategy, and expand the corresponding feasible region.

[0145] Step S407, under the premise of ensuring the safety of the current decision control policy, the performance of the current decision control policy is improved.

[0146] Step S408, it is judged whether the iterative update of the current decision control policy converges, if converges, the final decision control policy is obtained, thereby strengthening the automatic driving system.

[0147] Step S409, end.

[0148] According to the safety reinforcement learning training method for end-to-end automatic driving proposed in the embodiments of the present application, the exploration interaction between the vehicle and the environment is limited in the feasible region induced by the uncertainty constraint decay function, a safety reinforcement learning algorithm training method composed of safety exploration, model error estimation, standard policy evaluation, feasible region expansion and feasible policy improvement is constructed, a decision control policy with improved performance is obtained, and the end-to-end automatic driving system is strengthened. Therefore, the iterative process of algorithm training is optimized under the premise of ensuring the driving safety of the brain-like learning decision control module in the real driving environment, the complexity of algorithm training is reduced, and the performance of the decision control policy is effectively improved. Therefore, the problems in the related art that the brain-like learning decision control module training is limited to exploration interaction in the simulation driving environment, and the performance degrades greatly when migrated to the real driving environment, seriously affecting the performance of the simulation training decision control module in the real driving environment, cannot effectively guarantee the actual performance of the trained decision control module and the safety in the real driving environment exploration interaction data collection process, and cannot meet the training requirements of high-level end-to-end automatic driving system decision control module are solved.

[0149] Secondly, the safety reinforcement learning training device for end-to-end automatic driving proposed in the embodiments of the present application is described with reference to the accompanying drawings.

[0150] Figure 5 is a structural schematic diagram of the safety reinforcement learning training device for end-to-end automatic driving in the embodiments of the present application.

[0151] As Figure 5 shown, the safety reinforcement learning training device for end-to-end automatic driving 10 includes a detection module 100, a fitting module 200, an evaluation module 300, an expansion module 400 and a strengthening module 500.

[0152] The detection module 100 is configured to detect the safety state of the current decision control policy of the vehicle based on the preset driving environment model and the uncertainty constraint decay function, so as to collect a predetermined amount of environment interaction data in the case that the safety state reaches the preset state.

[0153] The fitting module 200 is configured to fit the random error of the preset driving environment model by using the preset integrated model, to obtain the distribution of the fitted random error.

[0154] The evaluation module 300 is configured to obtain a value function of the current decision control policy, and update the value function according to a self-consistent condition of the value function to obtain an updated value function, so as to evaluate a performance value of the current decision control policy by using the updated value function.

[0155] The expansion module 400 is configured to update the uncertainty constraint attenuation function of the current decision control policy according to a self-consistent condition of the uncertainty constraint attenuation function, to obtain an updated uncertainty constraint attenuation function, so as to expand a feasible region corresponding to the updated uncertainty constraint attenuation function to obtain an expanded feasible region.

[0156] The reinforcement module 500 is configured to iteratively update the current decision control policy based on the environment interaction data, the random error distribution, the performance value and the expanded feasible region until an average cumulative return of the current decision control policy reaches a preset threshold or reaches a maximum number of iterations, to obtain a final feasible decision control policy, so as to reinforce the target autonomous driving system according to the final feasible decision control policy.

[0157] Optionally, in an embodiment of the present application, the detection module 100 comprises a first generation unit, a second generation unit and a determination unit.

[0158] The first generation unit is configured to generate at least one safe action of the vehicle by using a current decision control policy of the vehicle.

[0159] The second generation unit is configured to add exploration noise to the safe action to generate an exploration action.

[0160] The determination unit is configured to determine a safe state of the exploration action according to a preset driving environment model and an uncertainty constraint attenuation function, to detect a safe state of the current decision control policy of the vehicle.

[0161] Optionally, in an embodiment of the present application, the detection module 100 further comprises an initialization module, a first training module and a second training module.

[0162] The initialization module is configured to initialize a preset number of neural networks of the same structure before fitting a random error of the preset driving environment model by using a preset integrated model, to obtain an initialized neural network.

[0163] The first training module is configured to train the initialized neural network by using target data to obtain a trained neural network.

[0164] The second training module is configured to train the current decision control policy, the value function and the uncertainty constraint attenuation function by using the trained neural network.

[0165] Optionally, in an embodiment of the present application, the evaluation module 300 comprises a third generation unit and an update unit.

[0166] The third generation unit is configured to generate virtual driving data by using a preset driving environment model.

[0167] The updating unit is configured to update the value function by the virtual driving data based on a self-consistent condition of the value function, to obtain an updated value function.

[0168] Optionally, in an embodiment of the present application, the uncertainty constraint attenuation function can be expressed as:

[0169]

[0170] wherein, is the number of state transitions required to first violate the state constraint starting from state x under the control of policy π, γ is a discount factor, and φ is a trainable parameter of the uncertainty constraint attenuation function.

[0171] It should be noted that the foregoing explanation and description of the embodiment of the method for training the safety reinforcement learning of the end-to-end automatic driving also applies to the embodiment of the device for training the safety reinforcement learning of the end-to-end automatic driving, which will not be described here.

[0172] The device for training the safety reinforcement learning of the end-to-end automatic driving according to the embodiment of the present application can limit the exploration interaction of the vehicle and the environment within the feasible region induced by the uncertainty constraint attenuation function, construct a safety reinforcement learning algorithm training method composed of safe exploration, model error estimation, standard policy evaluation, feasible region expansion and feasible policy improvement, obtain an improved performance decision and control policy, and strengthen the end-to-end automatic driving system. Thus, the iteration process of the algorithm training is optimized under the premise of ensuring the driving safety of the brain-like learning decision and control module when training in the real driving environment, the complexity of the algorithm training is reduced, and the performance of the decision and control policy is effectively improved. Thus, the problems in the related art that the brain-like learning decision and control module training is limited to the exploration interaction in the simulation driving environment, and the performance degrades greatly when migrating to the real driving environment, seriously affecting the performance of the decision and control module obtained by simulation training in the real driving environment, failing to effectively guarantee the actual performance of the trained decision and control module and the safety in the data collection process of the exploration interaction in the real driving environment, and failing to meet the training requirements of the decision and control module of the high-level end-to-end automatic driving system are solved.

[0173] Figure 6 The structure schematic diagram of the electronic device provided by the embodiment of the present application is shown in the figure. The electronic device can include:

[0174] The memory 601, the processor 602, and the computer program stored in the memory 601 and executable on the processor 602.

[0175] The processor 602 implements the end-to-end autonomous driving oriented security reinforcement learning training method provided in the above embodiments when executing a program.

[0176] Further, the electronic device further comprises:

[0177] The communication interface 603 is configured to communicate between the memory 601 and the processor 602.

[0178] The memory 601 is configured to store a computer program executable on the processor 602.

[0179] The memory 601 can include a high-speed RAM memory, and can also include a non-volatile memory, for example, at least one disk memory.

[0180] If the memory 601, the processor 602 and the communication interface 603 are implemented independently, the communication interface 603, the memory 601 and the processor 602 can be connected to each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 6 In the figure, only one thick line is used to represent, but it does not mean that there is only one bus or one type of bus.

[0181] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can complete communication between each other through an internal interface.

[0182] The processor 602 can be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0183] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the end-to-end autonomous driving oriented security reinforcement learning training method as above.

[0184] The embodiment of the present application further provides a computer program product comprising a computer program which can run computer instructions, and the computer instructions are executed by a processor to realize the method for safety reinforcement learning training of end-to-end automatic driving provided by the embodiment of the present application.

[0185] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or N embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples without contradiction.

[0186] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "N" is at least two, for example, two, three, etc., unless otherwise explicitly and specifically limited.

[0187] Any process or method descriptions in flow charts or otherwise described herein represent embodiments which can be understood as a module, segment, or portion of code which comprises one or N executable instructions for implementing the specified logical function or process, and the scope of the preferred embodiments of the present application includes additional implementation in which the functions described are performed in a different order, including substantially simultaneously, or in reverse order, as will be understood by those skilled in the art of the embodiments to which the present application pertains.

[0188] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered as a sequence of executable instructions stored in a computer readable medium, which can be executed by an instruction execution system, apparatus or device, such as a computer-based system, a processor-based system, or other system that can fetch the instructions from the instruction execution system, apparatus or device and execute the instructions, or a combination of the above. For the purposes of this specification, a "computer readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus or device. The computer readable medium can be a computer readable storage medium or a computer readable signal medium. The computer readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a propagation medium. The computer readable signal medium can include, but is not limited to, a computer readable medium that facilitates transfer of the program from one place to another. A specific example of a computer readable medium is a non-transitory computer-readable storage medium. A specific example of a computer readable signal medium is a source or destination of the computer readable medium. Another specific example of a computer readable signal medium is a computer readable signal travelling through space. Thus, a computer readable medium can take many forms of hardware to carry out the program for use by or in connection with the instruction execution system, apparatus or device.

[0189] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented in hardware and in another embodiment, the hardware can be implemented using any or a combination of the following technologies, which are each well known in the art: a discrete logic circuit(s) having logic gates for implementing logic functions upon an application of data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array(s) (PGA), a field programmable gate array (FPGA), etc.

[0190] Those of skill in the art would understand that the steps of the methods carried out above can be carried out wholly or partly by a program instructing relevant hardware, and the program can be stored in a computer readable storage medium, and when executed, includes one or a combination of the steps of the method embodiments.

[0191] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing module, or each of the units can be physically present separately, or two or more units can be integrated in one module. The integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0192] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A safety reinforcement learning training method for end-to-end autonomous driving, characterized in that, Includes the following steps: Based on a preset driving environment model and an uncertainty constraint decay function, the safety status of the vehicle's current decision-making strategy is detected, and a predetermined amount of environmental interaction data is collected when the safety status reaches a preset state. The environmental interaction data is the data collected when the action is the safety status under the current decision-making strategy. The status refers to the environmental state, which includes the vehicle's own position, speed, the surrounding driving environment state output by the perception module, and the surrounding driving environment state output by the prediction module. The actions include the vehicle's accelerator pedal opening, brake pedal opening, and steering wheel angle. The random error of the preset driving environment model is fitted using a preset ensemble model to obtain the fitted random error distribution. The preset ensemble model is a neural network model obtained by training multiple initialized neural networks with the same structure. Obtain the value function of the current decision-making strategy, and update the value function according to its self-consistency condition to obtain the updated value function. Use the updated value function to evaluate the performance value of the current decision-making strategy. The expression for the self-consistency condition of the value function is: in, For loss function, It is a discount factor. It is the entropy regularity coefficient. For virtual data pool, It is a state variable. The environmental state after the transfer. It is an action variable. This indicates the action to be output by the strategy at the next moment. As a reward signal, For the initial decision-making strategy; Based on the self-consistency condition of the uncertainty constraint decay function, the uncertainty constraint decay function of the current decision-making strategy is updated to obtain the updated uncertainty constraint decay function, thereby expanding the feasible region corresponding to the updated uncertainty constraint decay function, resulting in the expanded feasible region. The uncertainty constraint decay function is: in, A random error model under a certain driving environment model Under the control of the state Start in strategy The number of state transitions required for the first violation of state constraints under control, if the current state has already violated the constraints. If the future state will never violate the constraints, , It is a discount factor. The trainable parameters represent the uncertainty constraint decay function; The expression for the self-consistency condition of the uncertainty constraint attenuation function is as follows: in, For the current decision-making strategy The corresponding uncertainty constraint decay function, Uncertainty-constrained decay function Trainable parameters, It is a state space. It is a state constraint function, a set It is a set of constraint states The complement set; Based on the environmental interaction data, the random error distribution, the performance value, and the expanded feasible region, the current decision control strategy is iteratively updated until the average cumulative return of the current decision control strategy reaches a preset threshold or the maximum number of iterations is reached, to obtain the final feasible decision control strategy, so as to enhance the target autonomous driving system according to the final feasible decision control strategy.

2. The method according to claim 1, characterized in that, The method of detecting the safety status of the vehicle's current decision-making strategy based on a preset driving environment model and an uncertainty constraint attenuation function includes: At least one safety action of the vehicle is generated using the vehicle's current decision-making strategy; Add exploration noise to the safety action to generate the exploration action; Based on the preset driving environment model and the uncertainty constraint decay function, the safety status of the exploration action is determined to detect the safety status of the vehicle's current decision-making strategy.

3. The method according to claim 1, characterized in that, Before fitting the random error of the preset driving environment model to the preset integrated model, the method further includes: A preset number of neural networks with the same structure are initialized to obtain an initialized neural network. The initial neural network is trained using the target data to obtain the trained neural network. The trained neural network is used to train the current decision control strategy, the value function, and the uncertainty constraint decay function.

4. The method according to claim 1, characterized in that, The step of updating the value function according to the self-consistency condition of the value function to obtain the updated value function includes: Virtual driving data is generated using the preset driving environment model; Based on the self-consistency condition of the value function, the value function is updated using the virtual driving data to obtain the updated value function.

5. A safety reinforcement learning training device for end-to-end autonomous driving, characterized in that, include: The detection module is used to detect the safety status of the vehicle's current decision-making strategy based on a preset driving environment model and an uncertainty constraint decay function, so as to collect a predetermined amount of environmental interaction data when the safety status reaches a preset state; wherein, the environmental interaction data is the data collected when the action is the safety status under the current decision-making strategy, wherein the state is the environmental state, the environmental state includes the vehicle's own position, speed, the surrounding driving environment state output by the perception module, and the surrounding driving environment state output by the prediction module, and the action includes the vehicle's accelerator pedal opening, brake pedal opening, and steering wheel angle; The fitting module is used to fit the random error of the preset driving environment model using a preset ensemble model to obtain the fitted random error distribution. The preset ensemble model is a neural network model obtained by training multiple initialized neural networks with the same structure. An evaluation module is used to obtain the value function of the current decision-making strategy, and update the value function according to the self-consistency condition of the value function to obtain the updated value function, so as to evaluate the performance value of the current decision-making strategy using the updated value function, wherein the expression of the self-consistency condition of the value function is: in, For loss function, It is a discount factor. It is the entropy regularity coefficient. For virtual data pool, It is a state variable. The environmental state after the transfer. It is an action variable. This indicates the action to be output by the strategy at the next moment. As a reward signal, For the initial decision-making strategy; The extension module is used to update the uncertainty constraint decay function of the current decision-making strategy according to the self-consistency condition of the uncertainty constraint decay function, so as to obtain the updated uncertainty constraint decay function, thereby expanding the feasible region corresponding to the updated uncertainty constraint decay function, and obtaining the expanded feasible region. The uncertainty constraint decay function is: in, From state Start in strategy The number of state transitions required for the first violation of state constraints under control, if the current state has already violated the constraints. If the future state will never violate the constraints, , It is a discount factor. The trainable parameters represent the uncertainty constraint decay function; The expression for the self-consistency condition of the uncertainty constraint attenuation function is as follows: in, For the current decision-making strategy The corresponding uncertainty constraint decay function, Uncertainty-constrained decay function Trainable parameters, It is a state space. It is a state constraint function, a set It is a set of constraint states The complement set; The enhancement module is used to iteratively update the current decision control strategy based on the environmental interaction data, the random error distribution, the performance value, and the expanded feasible region, until the average cumulative return of the current decision control strategy reaches a preset threshold or the maximum number of iterations is reached, so as to obtain the final feasible decision control strategy and enhance the target autonomous driving system according to the final feasible decision control strategy.

6. The apparatus according to claim 5, characterized in that, The detection module includes: The first generation unit is used to generate at least one safety action of the vehicle using the vehicle's current decision-making strategy; The second generation unit is used to add exploration noise to the safety action to generate the exploration action; The determining unit is used to determine the safety status of the exploration action based on the preset driving environment model and the uncertainty constraint decay function, so as to detect the safety status of the vehicle's current decision control strategy.

7. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the safety reinforcement learning training method for end-to-end autonomous driving as described in any one of claims 1-4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the safety reinforcement learning training method for end-to-end autonomous driving as described in any one of claims 1-4.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed, it is used to implement the safety reinforcement learning training method for end-to-end autonomous driving as described in any one of claims 1-4.

Images