Safety reinforcement learning training method and device for end-to-end automatic driving

Abstract

The invention relates to the technical field of automatic driving, in particular to a safety reinforcement learning training method and device for end-to-end automatic driving, and the method comprises the steps: detecting the safety state of a current decision control strategy of a vehicle based on a preset driving environment model and an uncertainty constraint attenuation function; using the preset integrated model to fit the scattered random error of the preset driving environment model; updating the value function according to the self-consistent condition of the value function, and evaluating the performance value of the current decision control strategy; according to the self-consistent condition of the uncertainty constraint attenuation function, updating the uncertainty constraint attenuation function, and expanding a corresponding feasible region; and when the safety state reaches a preset level, performing iterative updating on the current decision control strategy based on the random error, the performance value and the expanded feasible region until the average accumulated return of the current decision control strategy reaches a preset threshold value or reaches the maximum number of iterations, and obtaining a final feasible decision control strategy so as to strengthen the target automatic driving system.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a safety reinforcement learning training method and device for end-to-end autonomous driving. Background Art

[0002] In the related technology, the end-to-end autonomous driving system neuralizes each module of the autonomous driving system, that is, all modules are differentiable, and then the driving performance of the system can be improved by collecting a large amount of driving data and using reverse gradient propagation. According to different deep learning algorithms, the decision-making and control modules of the end-to-end autonomous driving system can be divided into two categories: supervised learning decision-making and control modules and brain-like learning decision-making and control modules. The supervised learning decision-making and control modules are trained by supervised learning algorithms and require tens of billions of kilometers of manually labeled driving data for training, but high-quality driving data collection is difficult and the labeling cost is high. The brain-like learning decision-making and control modules are trained by reinforcement learning algorithms. They autonomously collect driving data by exploring and interacting with the real (or simulated) driving environment, and do not require manual labeling. They have the advantages of self-evolution and controllable costs, and have broad application prospects.

[0003] However, the brain-like learning decision-making and control module training in related technologies is limited to the exploration and interaction of simulated driving environments, and the performance degrades significantly when transferred to real driving environments, which seriously affects the performance of the decision-making and control modules obtained through simulation training in real driving environments. It cannot effectively guarantee the actual performance of the decision-making and control modules obtained through training and the safety of the decision-making and control modules during the data collection process of exploring and interacting in real driving environments, and cannot meet the training needs of decision-making and control modules of high-level end-to-end autonomous driving systems, and needs to be urgently addressed. Summary of the invention

[0004] The present application provides a safety reinforcement learning training method and device for end-to-end autonomous driving to solve the problems in the related art that the training of brain-like learning decision-making and control modules is limited to the exploration and interaction of simulated driving environments, and the performance is greatly degraded when migrating to real driving environments, which seriously affects the performance of the decision-making and control modules obtained through simulation training in real driving environments, cannot effectively guarantee the actual performance of the decision-making and control modules obtained through training and the safety of the data collected during the exploration and interaction of real driving environments, and cannot meet the training requirements of decision-making and control modules of high-level end-to-end autonomous driving systems.

[0005] The first aspect of the present application provides a safety reinforcement learning training method for end-to-end autonomous driving, comprising the following steps: based on a preset driving environment model and an uncertainty constraint attenuation function, detecting the safety state of the current decision-making control strategy of the vehicle, so as to collect a predetermined amount of environmental interaction data when the safety state reaches a preset state; fitting the random error of the preset driving environment model using a preset integrated model to obtain a fitted random error distribution; obtaining a value function of the current decision-making control strategy, and updating the value function according to the self-consistent condition of the value function to obtain an updated value function, so as to evaluate the performance value of the current decision-making control strategy using the updated value function; updating the uncertainty constraint attenuation function of the current decision-making control strategy according to the self-consistent condition of the uncertainty constraint attenuation function to obtain an updated uncertainty constraint attenuation function, so as to expand the feasible region corresponding to the updated uncertainty constraint attenuation function to obtain an expanded feasible region; iteratively updating the current decision-making control strategy based on the environmental interaction data, the random error distribution, the performance value and the expanded feasible region until the average cumulative reward of the current decision-making control strategy reaches a preset threshold or reaches a maximum number of iterations, so as to obtain a final feasible decision-making control strategy, so as to strengthen the target autonomous driving system according to the final feasible decision-making control strategy.

[0006] Optionally, in one embodiment of the present application, the safety status of the vehicle's current decision-making and control strategy is detected based on a preset driving environment model and an uncertainty constraint attenuation function, including: using the vehicle's current decision-making and control strategy to generate at least one safety action of the vehicle; adding exploration noise to the safety action to generate an exploration action; and determining the safety status of the exploration action based on the preset driving environment model and the uncertainty constraint attenuation function to detect the safety status of the vehicle's current decision-making and control strategy.

[0007] Optionally, in one embodiment of the present application, before fitting the random error of the preset driving environment model through the preset integrated model, it also includes: initializing a preset number of neural networks with the same structure to obtain an initialized neural network; training the initialized neural network with target data to obtain a trained neural network; and training the current decision-making control strategy, the value function and the uncertainty constraint attenuation function with the trained neural network.

[0008] Optionally, in one embodiment of the present application, updating the value function according to the self-consistent condition of the value function to obtain an updated value function includes: generating virtual driving data using the preset driving environment model; and updating the value function through the virtual driving data based on the self-consistent condition of the value function to obtain an updated value function.

[0009] Optionally, in one embodiment of the present application, the uncertainty constraint attenuation function is:

[0010]

[0011] in, is the number of state transitions required to violate the state constraint for the first time starting from state x under the control of policy π, γ is the discount factor, and φ is a trainable parameter of the uncertainty constraint decay function.

[0012] The second aspect of the present application provides a safety reinforcement learning training device for end-to-end autonomous driving, including: a detection module, which is used to detect the safety state of the current decision-making and control strategy of the vehicle based on a preset driving environment model and an uncertainty constraint attenuation function, so as to collect a predetermined amount of environmental interaction data when the safety state reaches a preset state; a fitting module, which is used to fit the random error of the preset driving environment model using a preset integrated model to obtain a distribution of random errors after fitting; an evaluation module, which is used to obtain a value function of the current decision-making and control strategy, and update the value function according to the self-consistent condition of the value function to obtain an updated value function, so as to evaluate the current decision-making and control strategy using the updated value function. an expansion module, used to update the uncertainty constraint attenuation function of the current decision-making control strategy according to the self-consistent condition of the uncertainty constraint attenuation function, to obtain an updated uncertainty constraint attenuation function, so as to expand the feasible region corresponding to the updated uncertainty constraint attenuation function, to obtain an expanded feasible region; a strengthening module, used to iteratively update the current decision-making control strategy based on the environmental interaction data, the random error distribution, the performance value and the expanded feasible region, until the average cumulative reward of the current decision-making control strategy reaches a preset threshold or reaches a maximum number of iterations, to obtain a final feasible decision-making control strategy, so as to strengthen the target autonomous driving system according to the final feasible decision-making control strategy.

[0013] Optionally, in one embodiment of the present application, the detection module includes: a first generation unit, used to generate at least one safety action of the vehicle using the current decision-making and control strategy of the vehicle; a second generation unit, used to add exploration noise to the safety action to generate an exploration action; and a determination unit, used to determine the safety state of the exploration action based on the preset driving environment model and the uncertainty constraint attenuation function to detect the safety state of the current decision-making and control strategy of the vehicle.

[0014] Optionally, in one embodiment of the present application, it also includes: an initialization module, which is used to initialize a preset number of neural networks with the same structure to obtain an initialized neural network before fitting the random error of the preset driving environment model through a preset integrated model; a first training module, which is used to train the initialized neural network with target data to obtain a trained neural network; and a second training module, which is used to train the current decision-making control strategy, the value function and the uncertainty constraint attenuation function with the trained neural network.

[0015] Optionally, in one embodiment of the present application, the evaluation module includes: a third generation unit, used to generate virtual driving data using the preset driving environment model; an updating unit, used to update the value function through the virtual driving data based on the self-consistent condition of the value function to obtain an updated value function.

[0016] Optionally, in one embodiment of the present application, the uncertainty constraint attenuation function is:

[0017]

[0018] in, is the number of state transitions required to violate the state constraint for the first time starting from state x under the control of policy π, γ is the discount factor, and φ is a trainable parameter of the uncertainty constraint decay function.

[0019] The third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the safety reinforcement learning training method for end-to-end autonomous driving as described in the above embodiment.

[0020] The fourth aspect embodiment of the present application provides a computer-readable storage medium, which stores a computer program, which, when executed by a processor, implements the above-mentioned safety reinforcement learning training method for end-to-end autonomous driving.

[0021] The fifth aspect embodiment of the present application provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned safety reinforcement learning training method for end-to-end autonomous driving.

[0022] The embodiment of the present application can limit the exploration interaction between the vehicle and the environment to the feasible region induced by the uncertainty constraint attenuation function, construct a safety reinforcement learning algorithm training method consisting of safety exploration, model error estimation, standard strategy evaluation, feasible region expansion and feasible strategy improvement, and obtain a decision-making control strategy with improved performance to strengthen the end-to-end autonomous driving system. Thus, it is achieved that the iterative process of algorithm training is optimized, the complexity of algorithm training is reduced, and the performance of the decision-making control strategy is effectively improved under the premise of ensuring the driving safety of the car when the brain-like learning decision-making control module is trained in a real driving environment. Therefore, the problems that the training of the brain-like learning decision-making control module in the related technology is limited to the exploration interaction of the simulated driving environment and the performance degradation is large when it is transferred to the real driving environment, which seriously affects the performance of the decision-making control module obtained by simulation training in the real driving environment, cannot effectively guarantee the actual performance of the decision-making control module obtained by training and the safety of the data collected during the exploration interaction of the real driving environment, and cannot meet the training requirements of the decision-making control module of the high-level end-to-end autonomous driving system.

[0023] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0025] Figure 1 A flowchart of a safety reinforcement learning training method for end-to-end autonomous driving provided according to an embodiment of the present application;

[0026] Figure 2 A flowchart of a security exploration algorithm according to an embodiment of the present application;

[0027] Figure 3 A schematic diagram of a method for defining an uncertainty constraint attenuation function according to an embodiment of the present application;

[0028] Figure 4 This is a flowchart of a safety reinforcement learning training method for end-to-end autonomous driving according to an embodiment of the present application;

[0029] Figure 5 A schematic diagram of the structure of a safety reinforcement learning training device for end-to-end autonomous driving provided according to an embodiment of the present application;

[0030] Figure 6 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application.

[0031] Reference numerals:

[0032] 10-Safety reinforcement learning training device for end-to-end autonomous driving: 100-detection module, 200-fitting module, 300-evaluation module, 400-extension module and 500-enhancement module; 601-memory, 602-processor and 603-communication interface. DETAILED DESCRIPTION

[0033] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0034] The following describes the safety reinforcement learning training method and device for end-to-end autonomous driving in the embodiment of the present application with reference to the accompanying drawings. In view of the fact that the training of the brain-like learning decision-making control module in the related art mentioned in the above background technology is limited to the exploration interaction of the simulated driving environment, and the performance is greatly degraded when it is transferred to the real driving environment, which seriously affects the performance of the decision-making control module obtained by simulation training in the real driving environment, cannot effectively guarantee the actual performance of the decision-making control module obtained by training and the safety of the data collection process of the real driving environment exploration interaction, and cannot meet the training requirements of the decision-making control module of the high-level end-to-end autonomous driving system, the present application provides a safety reinforcement learning training method for end-to-end autonomous driving, in which the exploration interaction between the vehicle and the environment can be limited to the feasible region induced by the uncertainty constraint attenuation function, and a safety reinforcement learning algorithm training method consisting of safety exploration, model error estimation, standard strategy evaluation, feasible region expansion and feasible strategy improvement can be constructed to obtain a decision-making control strategy with improved performance to strengthen the end-to-end autonomous driving system, thereby achieving the optimization of the iterative process of algorithm training, reducing the complexity of algorithm training, and effectively improving the performance of the decision-making control strategy under the premise of ensuring the driving safety of the brain-like learning decision-making control module when training in the real driving environment. This solves the problem that the training of brain-like learning decision-making and control modules in related technologies is limited to exploration and interaction in simulated driving environments, and the performance degrades significantly when transferred to real driving environments, seriously affecting the performance of decision-making and control modules obtained through simulation training in real driving environments, and failing to effectively guarantee the actual performance of the decision-making and control modules obtained through training and the safety of data collected during exploration and interaction in real driving environments, and failing to meet the training requirements of decision-making and control modules of high-level end-to-end autonomous driving systems.

[0035] Before explaining the safety reinforcement learning training method for end-to-end autonomous driving in the embodiment of the present application, the elements involved in the safety reinforcement learning training method for end-to-end autonomous driving in the embodiment of the present application and the end-to-end autonomous driving system are first explained.

[0036] Self-driving cars are cars that are equipped with advanced on-board sensors, controllers, actuators and other devices, and have functions such as complex environment perception, intelligent decision-making, and collaborative control. They can achieve "safe, efficient, comfortable, and energy-saving" driving, and can eventually replace human operation. They are an important direction of technological innovation in the automotive industry. The self-driving capabilities of cars are divided into six levels from level 0 to level 5, of which level 0 is no automation and level 5 is full automation. In order to achieve high-level self-driving capabilities of level 3 and above, end-to-end self-driving systems have great potential. End-to-end self-driving systems refer to self-driving systems that are trained by deep learning algorithms using neural networks for modules such as perception, prediction, planning, decision-making, and control, using manually labeled (or unlabeled) driving data.

[0037] The safety reinforcement learning training method for end-to-end autonomous driving in the embodiment of the present application includes a cyclic process, and the following elements need to be initialized before starting: real data pool Virtual Data Pool Initial decision-making strategy π θ , value function and Uncertainty Constraint Decay Function Integrated model of random errors in driving environment Where θ,w1,w2,φ,ψ=[ψ1,ψ2,…,ψ M ] is a trainable parameter, and M is the number of sub-models in the integrated model.

[0038] Specifically, Figure 1 A flowchart of a safety reinforcement learning training method for end-to-end autonomous driving provided in an embodiment of the present application.

[0039] like Figure 1 As shown, the safety reinforcement learning training method for end-to-end autonomous driving includes the following steps:

[0040] In step S101, based on a preset driving environment model and an uncertainty constraint attenuation function, the safety state of the current decision-making control strategy of the vehicle is detected to collect a predetermined amount of environmental interaction data when the safety state reaches a preset state.

[0041] Among them, the uncertainty constraint attenuation function can be expressed as:

[0042]

[0043] in, It is the number of state transitions required to violate the state constraint for the first time under the control of the random error model ζ of the preset driving environment model, starting from state x and under the control of strategy π. If the current state has violated the constraint, If future states will never violate the constraints, γ is the discount factor.

[0044] In some embodiments, the preset driving environment model here can be understood as a partially known driving environment model learned and trained using real driving data. The preset driving environment model in the embodiment of the present application can be divided into a known part and an unknown part, wherein the known part can be a dynamic model determined by traffic participants (including motor vehicles, non-motor vehicles, pedestrians, etc.) established according to physical laws, and the unknown part can be a random error model that describes the dynamic model error and unmodeled dynamics. With f representing the dynamic model and ζ representing the random error model, the preset driving environment model can be defined as:

[0045] x t+1 =f(x t ,u t )+ζ(x t ,u t ), (1)

[0046] Wherein, t is a time variable, x is a state variable, u is an action variable, f is an analytical function, ζ is a random error model, and the model is bounded. In the embodiment of the present application, r=r(x,u) may be additionally defined as a reward function, and h(x)≤0 may be defined as a state constraint.

[0047] The uncertainty constraint attenuation function here can be understood as a constraint attenuation function with uncertainty, which can be expressed as follows:

[0048]

[0049] in, is the number of state transitions required to violate the state constraint for the first time under the control of the decision-making strategy π starting from state x. If the current state has violated the constraint, If future states will never violate the constraints, γ is the discount factor and φ is a trainable parameter of the uncertainty constraint decay function.

[0050] Based on the preset driving environment model and the uncertainty constraint attenuation function, the embodiment of the present application can detect the safety of the vehicle's current decision-making and control strategy and determine the safety status of the current decision-making and control strategy, that is, the safety of performing a certain action under the current decision-making and control strategy, so as to collect a certain amount of environmental interaction data in a safe manner for use in subsequent steps.

[0051] It should be noted that the state in the embodiments of the present application refers to the environmental state, including but not limited to the vehicle's own position, speed and other vehicle states and the surrounding driving environment states output by perception, prediction and other modules; action here can be understood as control quantities including but not limited to the vehicle's accelerator pedal opening, brake pedal opening, steering wheel angle and other control quantities; safety can be understood as no personal injury, property loss or other adverse consequences; exploration here can be understood as the process in which the vehicle actuator actually executes an action and causes the vehicle to interact with the driving environment.

[0052] If the environment state is safe, the exploration action is determined to be safe, and the action is actually executed to allow the vehicle to explore interactive samples in the driving environment; if the environment state is unsafe, the exploration action is determined to be unsafe, and the exploration action is re-acquired until a safe action is found or the maximum number of times is reached. When the maximum number of times is reached, the safe action without adding exploration noise is directly executed to allow the vehicle to explore interactive samples in the driving environment.

[0053] The embodiment of the present application can use the learned preset driving environment model and the target constraint attenuation function with uncertainty to judge the safety of the exploration action, so as to maximize the exploratory nature of the strategy while ensuring the safety of the environmental interaction.

[0054] The process is further described below.

[0055] Optionally, in one embodiment of the present application, based on a preset driving environment model and an uncertainty constraint attenuation function, the safety status of the vehicle's current decision-making and control strategy is detected, including: using the vehicle's current decision-making and control strategy to generate at least one safety action of the vehicle; adding exploration noise to the safety action to generate an exploration action; and determining the safety status of the exploration action based on the preset driving environment model and the uncertainty constraint attenuation function to detect the safety status of the vehicle's current decision-making and control strategy.

[0056] In the actual implementation process, the present application can use the learned preset driving environment model and the target constraint attenuation function with uncertainty to determine the safety status of the current decision-making and control strategy.

[0057] Specifically, first, the current decision-making and control strategy can be used to generate at least one safe action of the vehicle, and an exploratory action can be generated by adding exploratory noise to the safe action; secondly, the current environmental state of the preset driving environment model and the exploratory action are used to predict the next state that may be transferred to, and a constraint attenuation function with uncertainty is used to judge the safety state of all next-moment states.

[0058] For example, Figure 2 This is a flowchart of a security exploration algorithm according to an embodiment of the present application. Figure 2As shown, the present application can repeatedly call the safety exploration algorithm to collect driving data in a real driving environment.

[0059] (1) The safety exploration algorithm needs to input the current environment state x and the initial decision-making control strategy π θ , uncertainty constraint decay function Driving environment random error model ζ, safety threshold δ s and explore the noise distribution d ∈ .

[0060] (2) Generate at least one safe action u based on the current state x safe :

[0061] u safe =π θ (x), (2)

[0062] (3) Initialize the exploration action generation counter variable i=0.

[0063] (4) Determine whether the exploration action generation count variable i is less than the maximum number of times K. If the exploration action generation count variable i is not less than the maximum number of times K, action u safe Acting on the vehicle, the environment state is transferred to x′, and the reward signal r is obtained.

[0064] (5) If the exploration action generation count variable i is less than the maximum number K, then according to the exploration noise distribution d ∈ Sample the exploration noise ∈ to obtain the exploration action u explore :

[0065] u explore =u safe +∈,∈~d ε , (3)

[0066] (6) Using the learned driving environment random error model ζ and uncertainty constraint attenuation function Determine the safety of the action and calculate the possible state of the environment at the next moment It should be noted that the next moment state here is predicted by the environmental model, not obtained by environmental feedback, and environmental random error models with different parameters can obtain different next moment states, so the subscript j is used to distinguish them:

[0067]

[0068] (7) Determine all possible environmental states at the next moment Is it safe when the following formula is met:

[0069]

[0070] Determine all possible environmental states at the next moment are all safe, then the exploration action u explore It is safe.

[0071] (8) If the exploration action u explore is safe, then the vehicle actually performs the exploration action u explore .

[0072] (9) The vehicle transfers to the next environmental state And obtain the reward signal r.

[0073] (10) The driving data samples (x, u) obtained in this process are explore ,x ′ ,r)Add to the real data pool middle.

[0074] (11) If the exploration action u explore is unsafe, repeat the above process until the maximum number of exploration action sampling times K is reached. If the exploration action u explore If it is still unsafe, the vehicle actually executes the safety action u obtained by formula (3) safe , the vehicle transfers to the next moment's environment state And obtain the reward signal r. The driving data samples (x,u safe ,x ′ ,r) will be added to the real data pool middle.

[0075] The embodiment of the present application can explore whether the next moment state of the preset driving environment model is safe, ensuring that the actions performed by the vehicle when exploring the environment during the reinforcement learning algorithm training process are safe, that is, they do not violate constraints and do not produce unsafe consequences.

[0076] Step S102, using a preset integrated model to fit the random error of a preset driving environment model to obtain a fitted random error distribution.

[0077] In some embodiments, inaccurate deterministic dynamic modeling and unmodeled dynamic characteristics in the preset driving environment model may produce certain errors. In order to reduce the inaccuracy during the final intensive training, the present application can use a preset integrated model to fit the random error part of the preset driving environment model, that is, the error caused by inaccurate deterministic dynamic modeling and unmodeled dynamic characteristics, to obtain a usable random error distribution after fitting.

[0078] Next, the preset integration model in the embodiment of the present application is further explained.

[0079] Optionally, in one embodiment of the present application, before fitting the dispersed random errors of a preset driving environment model through a preset integrated model, it also includes: initializing a preset number of neural networks of the same structure to obtain an initialized neural network; training the initialized neural network with target data to obtain a trained neural network; and training the current decision-making control strategy, value function, and uncertainty constraint attenuation function with the trained neural network.

[0080] In the actual implementation process, the present application can use a preset integrated model to fit the random error part of the preset driving environment model. Among them, the preset integrated model can be, but is not limited to, initializing a preset number of multiple neural networks of the same structure, and then using the target data to train each neural network. The trained neural network can form a preset integrated model. Among them, the preset number here can be understood as a pre-set number, which can be set or adjusted by professional and technical personnel in this field according to actual conditions or actual needs. This is only an exemplary description without specific limitation.

[0081] Under this training, the output of each neural network can constitute the random error part of the preset driving environment model, that is, the atomic distribution estimate of the preset integrated model. The random error part of the driving environment model is fitted by the preset integrated model, which can be used to support the subsequent processing process.

[0082] For example, the present application may use M neural networks of the same structure, each of which is initialized with a different random seed and in the data playback pool. In the process, batches of data are collected independently for training. The loss function used in this process can be expressed as follows:

[0083]

[0084] The training of the preset driving environment model is mainly to set the parameter ψ in the random error model of the preset driving environment i =[ψ1,ψ2,…,ψ M ] can be expressed as follows:

[0085]

[0086] Among them, α model is the learning rate of the parameters of the random error model of the driving environment.

[0087] The embodiments of the present application can use a preset integrated model to simultaneously fit the errors caused by inaccurate dynamic modeling and the errors caused by unmodeled dynamic characteristics, and can fit the random error model of the driving environment into an atomic distribution, thereby avoiding the introduction of additional iterative optimization processes when updating the uncertainty constraint attenuation function and exploring the environment, and reducing the complexity of algorithm training.

[0088] Step S103, obtaining the value function of the current decision-making control strategy, and updating the value function according to the self-consistent condition of the value function to obtain an updated value function, so as to use the updated value function to evaluate the performance value of the current decision-making control strategy.

[0089] In some embodiments, the present application can obtain the value function of the current decision-making control strategy, and use the self-consistent conditions of the value function to update the value function of the current decision-making control strategy. The value function can be used to evaluate the performance of the current decision-making control strategy under a given state, and then used in subsequent feasible strategy improvement steps.

[0090] The process is further described below.

[0091] Optionally, in one embodiment of the present application, the value function is updated according to the self-consistent condition of the value function to obtain an updated value function, including: generating virtual driving data using a preset driving environment model; based on the self-consistent condition of the value function, updating the value function through virtual driving data to obtain an updated value function.

[0092] In the actual implementation process, when the present application uses the self-consistent condition of the value function of the current decision-making control strategy to update the value function of the strategy, it can be implemented by, but is not limited to, using a preset driving environment model obtained through training to generate a large amount of virtual driving data.

[0093] Specifically, first, the preset driving environment model can be used to generate virtual driving data and store it in the virtual data pool. To learn two value functions and The loss function used in training can be expressed as follows:

[0094]

[0095] in, and is the target value function corresponding to the two value functions, γ is the discount factor, α e is the entropy regularization coefficient, u ′ Indicates the action of the strategy output at the next moment.

[0096] The embodiments of the present application can generate a large amount of virtual driving data through a preset driving environment model for value function estimation, feasible area expansion and strategy improvement, and then deeply mine the information contained in the real driving data, improve the sample utilization efficiency of the algorithm, and reduce the algorithm's demand for real driving data.

[0097] Step S104, based on the self-consistent condition of the uncertainty constraint attenuation function, the uncertainty constraint attenuation function of the current decision-making control strategy is updated to obtain an updated uncertainty constraint attenuation function, so as to expand the feasible area corresponding to the updated uncertainty constraint attenuation function to obtain an expanded feasible area.

[0098] Based on the relevant descriptions of other embodiments, it can be understood that the present application can use a preset integrated model to fit the random error part of the preset driving environment model. Furthermore, the random error obtained by fitting can be used to support the expansion of the feasible area of ​​the strategy obtained by learning the proposed algorithm (i.e., the explorable area of ​​the vehicle in the driving environment).

[0099] As a possible implementation method, the embodiment of the present application can use the self-consistent conditions of the uncertainty constraint attenuation function to update the uncertainty constraint attenuation function corresponding to the current decision-making control strategy, and expand the corresponding feasible area to utilize the larger feasible area to expand the explorable area of ​​the vehicle in the driving environment, thereby supporting the performance improvement of the decision-making control strategy.

[0100] For example, the application may first use a virtual data pool The data within identifies the feasible area corresponding to the current decision-making control strategy. Figure 3 This is a schematic diagram of the expansion of the feasible region of the uncertainty constraint attenuation function according to an embodiment of the present application. Figure 3 As shown, the uncertainty constraint attenuation function As shown above, it can be expressed as follows:

[0101]

[0102] in, is the number of state transitions required to violate the state constraint for the first time under the control of strategy π starting from state x. If the current state has violated the constraint, If future states will never violate the constraints, γ is the discount factor and φ represents the trainable parameter of the uncertainty constraint decay function.

[0103] Uncertainty Constraint Decay Function The self-consistency condition can be expressed as:

[0104]

[0105] Among them, the definition of function c(x) can be expressed as follows:

[0106]

[0107] Among them, the constraint state set X cstr The definition of can be expressed as follows:

[0108]

[0109] in, is the state space, h(x) is the state constraint function, and the set is the set of constraint states X cstr The complement set of can be expressed as follows:

[0110]

[0111] The self-consistent condition shown in formula (9) is a compression mapping, that is, the randomly initialized uncertainty constraint attenuation function The trainable parameter φ, which is iterated according to the self-consistent condition, can converge to the uncertainty constraint attenuation function corresponding to the current decision-making control strategy π The loss function used in training is cross entropy loss, which can be defined as follows:

[0112]

[0113] in,

[0114]

[0115] Uncertainty Constraint Decay Function The definition of the induced feasible domain can be expressed as follows:

[0116]

[0117] Among them, δ s is a safety threshold. In theory, its value can be zero. However, in the actual training process, due to the existence of learning errors, it can be selected as a smaller positive number according to the actual situation. For example, when the maximum value of the preset single state trajectory length is 200, δ s =0.1; are the uncertainty constraint attenuation function values ​​corresponding to the current state and the next state respectively.

[0118] The self-consistent condition of the uncertainty constraint attenuation function in the embodiment of the present application can induce a compression mapping. It only needs to initialize any parameterized function with sufficient expressive power and iterate repeatedly using the self-consistent condition. The function will converge to the uncertainty constraint attenuation function corresponding to the decision-making control strategy. It should be noted that the feasible region induced by the uncertainty constraint attenuation function in the above steps is monotonically expanded.

[0119] Step S105, based on the environmental interaction data, random error distribution, performance value and expanded feasible area, the current decision-making control strategy is iteratively updated until the average cumulative reward of the current decision-making control strategy reaches a preset threshold or reaches the maximum number of iterations, and the final feasible decision-making control strategy is obtained to strengthen the target autonomous driving system according to the final feasible decision-making control strategy.

[0120] In other embodiments, when the safety state of the vehicle's current decision-making and control strategy reaches a preset state, that is, the current decision-making and control strategy is safe, the present application can iteratively update the current decision-making and control strategy based on environmental interaction data, random error distribution, performance value and expanded feasible area, improve the performance of the current decision-making and control strategy, and then use the improved decision-making and control strategy to intensively train the target autonomous driving system, that is, the autonomous driving system that needs intensive training.

[0121] Furthermore, in the embodiments of the present application, when iteratively updating the current decision-making control strategy, a regional strategy update mechanism can be used, that is, different strategy improvement methods are used inside and outside the feasible domain.

[0122] In the feasible region indicated by the uncertainty constraint attenuation function, in addition to improving the policy performance, the feasibility of the policy should also be guaranteed. Therefore, the policy improvement is defined as a constrained optimization problem. After being transformed into an unconstrained problem using the penalty function method, the policy gradient method is used to update the policy.

[0123] Among them, the update method in the feasible region can be defined as the following constrained optimization problem:

[0124]

[0125] Outside the feasible region, only the feasibility of the strategy under the worst case is considered, that is, the constraint violation is minimized, and the strategy is updated using the policy gradient method. This improves the performance of the decision control strategy while ensuring its security. Its definition can be expressed as follows:

[0126]

[0127] In the actual training of the algorithm, the loss function of the strategy is defined as follows:

[0128]

[0129] in,

[0130]

[0131] Among them, λ is the coefficient used to balance the loss function inside and outside the feasible region, η(x ′ ) is an indicator function, which can be defined as:

[0132]

[0133] In the present invention, the formula (19) It can be defined as follows:

[0134]

[0135] Among them, α e It is the same parameter as in formula (8), namely the entropy regularization coefficient.

[0136] This step adopts a regional strategy improvement mechanism, using different optimization objectives inside and outside the feasible region. This mechanism ensures that the performance of the strategy is monotonically improved within the feasible region. Outside the feasible region, maximizing the feasibility of the strategy can bring potential feasible states into the feasible region.

[0137] The above steps are repeated, and the algorithm terminates when any of the following conditions is met: (1) the average cumulative return reaches the set threshold, that is, the algorithm converges; (2) the number of algorithm iterations reaches the set maximum value.

[0138] The present application is described in detail below with reference to a specific embodiment.

[0139] Step S401, start.

[0140] Step S402, initializing parameters, such as real data pool Virtual Data Pool Initial strategy π θ , value function and Uncertainty Constraint Decay Function Integrated model of random errors in driving environmentζ ψ .

[0141] Step S403, using the learned preset driving environment model and the constraint attenuation function with uncertainty to explore whether the environmental state at the next moment is safe.

[0142] Step S404: Use the integrated model to fit the random error of the driving environment model.

[0143] Step S405, using the self-consistent condition of the value function, update the value function of the strategy and evaluate the performance of the current decision-making control strategy under a given state.

[0144] Step S406: using the self-consistent condition of the constraint decay function with uncertainty, update the constraint decay function with uncertainty corresponding to the current decision-making strategy, and expand the corresponding feasible region.

[0145] Step S407: This step improves the performance of the current decision-making control strategy while ensuring the security of the current decision-making control strategy.

[0146] Step S408, determining whether the iterative update of the current decision-making control strategy has converged, and if so, obtaining the final decision-making control strategy, thereby strengthening the autonomous driving system.

[0147] Step S409, end.

[0148] According to the safety reinforcement learning training method for end-to-end autonomous driving proposed in the embodiment of the present application, the exploration interaction between the vehicle and the environment can be limited to the feasible region induced by the uncertainty constraint attenuation function, and a safety reinforcement learning algorithm training method consisting of safety exploration, model error estimation, standard strategy evaluation, feasible region expansion and feasible strategy improvement can be constructed to obtain a decision-making control strategy with improved performance to strengthen the end-to-end autonomous driving system. Thus, under the premise of ensuring the driving safety of the car when the brain-like learning decision-making control module is trained in a real driving environment, the iterative process of algorithm training is optimized, the complexity of algorithm training is reduced, and the performance of the decision-making control strategy is effectively improved. Thus, the problems that the training of the brain-like learning decision-making control module in the related art is limited to the exploration interaction of the simulated driving environment and the performance degradation is large when it is transferred to the real driving environment is solved, which seriously affects the performance of the decision-making control module obtained by simulation training in the real driving environment, cannot effectively guarantee the actual performance of the decision-making control module obtained by training and the safety of the data collected during the exploration interaction of the real driving environment, and cannot meet the training requirements of the decision-making control module of the high-level end-to-end autonomous driving system.

[0149] Next, a safety reinforcement learning training device for end-to-end autonomous driving proposed according to an embodiment of the present application is described with reference to the accompanying drawings.

[0150] Figure 5 It is a structural diagram of a safety reinforcement learning training device for end-to-end autonomous driving according to an embodiment of the present application.

[0151] like Figure 5 As shown, the safety reinforcement learning training device 10 for end-to-end autonomous driving includes: a detection module 100, a fitting module 200, an evaluation module 300, an expansion module 400 and a reinforcement module 500.

[0152] Among them, the detection module 100 is used to detect the safety status of the current decision-making and control strategy of the vehicle based on a preset driving environment model and an uncertainty constraint attenuation function, so as to collect a predetermined amount of environmental interaction data when the safety status reaches a preset state.

[0153] The fitting module 200 is used to fit the random error of the preset driving environment model using the preset integrated model to obtain the fitted random error distribution.

[0154] The evaluation module 300 is used to obtain the value function of the current decision-making control strategy, and update the value function according to the self-consistent condition of the value function to obtain an updated value function, so as to use the updated value function to evaluate the performance value of the current decision-making control strategy.

[0155] The expansion module 400 is used to update the uncertainty constraint attenuation function of the current decision-making control strategy according to the self-consistent condition of the uncertainty constraint attenuation function to obtain an updated uncertainty constraint attenuation function, so as to expand the feasible area corresponding to the updated uncertainty constraint attenuation function to obtain an expanded feasible area.

[0156] The strengthening module 500 is used to iteratively update the current decision-making control strategy based on the environmental interaction data, random error distribution, performance value and expanded feasible area, until the average cumulative return of the current decision-making control strategy reaches a preset threshold or reaches the maximum number of iterations, so as to obtain the final feasible decision-making control strategy, so as to strengthen the target autonomous driving system according to the final feasible decision-making control strategy.

[0157] Optionally, in one embodiment of the present application, the detection module 100 includes: a first generation unit, a second generation unit and a determination unit.

[0158] The first generating unit is used to generate at least one safety action of the vehicle by using the current decision-making and control strategy of the vehicle.

[0159] The second generating unit is used to add exploration noise to the safety action to generate an exploration action.

[0160] The determination unit is used to determine the safety state of the exploration action according to a preset driving environment model and an uncertainty constraint attenuation function to detect the safety state of the current decision-making and control strategy of the vehicle.

[0161] Optionally, in one embodiment of the present application, it also includes: an initialization module, a first training module and a second training module.

[0162] The initialization module is used to initialize a preset number of neural networks with the same structure before fitting the random error of the preset driving environment model through the preset integrated model to obtain an initialized neural network.

[0163] The first training module is used to train the initialization neural network using the target data to obtain the trained neural network.

[0164] The second training module is used to train the current decision-making control strategy, value function and uncertainty constraint attenuation function using the trained neural network.

[0165] Optionally, in one embodiment of the present application, the evaluation module 300 includes: a third generation unit and an update unit.

[0166] The third generating unit is used to generate virtual driving data using a preset driving environment model.

[0167] The updating unit is used to update the value function through virtual driving data based on the self-consistent condition of the value function to obtain an updated value function.

[0168] Optionally, in one embodiment of the present application, the uncertainty constraint attenuation function can be expressed as:

[0169]

[0170] in, is the number of state transitions required to violate the state constraint for the first time starting from state x under the control of policy π, γ is the discount factor, and φ is a trainable parameter of the uncertainty constraint decay function.

[0171] It should be noted that the aforementioned explanation of the embodiment of the safety reinforcement learning training method for end-to-end autonomous driving is also applicable to the safety reinforcement learning training device for end-to-end autonomous driving of this embodiment, and will not be repeated here.

[0172] According to the safety reinforcement learning training device for end-to-end autonomous driving proposed in the embodiment of the present application, the exploration interaction between the vehicle and the environment can be limited to the feasible region induced by the uncertainty constraint attenuation function, and a safety reinforcement learning algorithm training method consisting of safety exploration, model error estimation, standard strategy evaluation, feasible region expansion and feasible strategy improvement can be constructed to obtain a decision-making control strategy with improved performance to strengthen the end-to-end autonomous driving system. Thus, under the premise of ensuring the driving safety of the car when the brain-like learning decision-making control module is trained in a real driving environment, the iterative process of algorithm training is optimized, the complexity of algorithm training is reduced, and the performance of the decision-making control strategy is effectively improved. Thus, the problems that the training of the brain-like learning decision-making control module in the related art is limited to the exploration interaction of the simulated driving environment and the performance degradation is large when it is transferred to the real driving environment is solved, which seriously affects the performance of the decision-making control module obtained by simulation training in the real driving environment, cannot effectively guarantee the actual performance of the decision-making control module obtained by training and the safety of the data collected during the exploration interaction of the real driving environment, and cannot meet the training requirements of the decision-making control module of the high-level end-to-end autonomous driving system.

[0173] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0174] A memory 601 , a processor 602 , and a computer program stored in the memory 601 and executable on the processor 602 .

[0175] When the processor 602 executes the program, the safety reinforcement learning training method for end-to-end autonomous driving provided in the above embodiment is implemented.

[0176] Furthermore, the electronic device further comprises:

[0177] The communication interface 603 is used for communication between the memory 601 and the processor 602 .

[0178] The memory 601 is used to store computer programs that can be executed on the processor 602 .

[0179] The memory 601 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0180] If the memory 601, the processor 602 and the communication interface 603 are implemented independently, the communication interface 603, the memory 601 and the processor 602 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0181] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can communicate with each other through an internal interface.

[0182] The processor 602 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0183] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned safety reinforcement learning training method for end-to-end autonomous driving.

[0184] An embodiment of the present application also provides a computer program product, including a computer program, which can run computer instructions. When the computer instructions are executed by a processor, the safety reinforcement learning training method for end-to-end autonomous driving provided by the embodiment of the present application is implemented.

[0185] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0186] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0187] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0188] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0189] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of multiple of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0190] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0191] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0192] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A safety reinforcement learning training method for end-to-end autonomous driving, characterized in that: The following steps are involved: Based on a preset driving environment model and an uncertainty constraint attenuation function, detecting a safety state of a current decision-making and control strategy of the vehicle, so as to collect a predetermined amount of environmental interaction data when the safety state reaches a preset state; Using a preset integrated model to fit the random error of the preset driving environment model, to obtain a fitted random error distribution; Acquire the value function of the current decision control strategy, and update the value function according to the self-consistent condition of the value function to obtain an updated value function, so as to evaluate the performance value of the current decision control strategy by using the updated value function; According to the self-consistent condition of the uncertainty constraint attenuation function, the uncertainty constraint attenuation function of the current decision-making control strategy is updated to obtain an updated uncertainty constraint attenuation function, so as to expand the feasible region corresponding to the updated uncertainty constraint attenuation function to obtain an expanded feasible region; Based on the environmental interaction data, the random error distribution, the performance value and the expanded feasible area, the current decision-making control strategy is iteratively updated until the average cumulative reward of the current decision-making control strategy reaches a preset threshold or reaches a maximum number of iterations, so as to obtain a final feasible decision-making control strategy to strengthen the target autonomous driving system according to the final feasible decision-making control strategy.

2. The method according to claim 1, characterized in that: The method of detecting the safety state of the current decision-making and control strategy of the vehicle based on the preset driving environment model and the uncertainty constraint attenuation function includes: generating at least one safety action of the vehicle using a current decision-making control strategy of the vehicle; adding exploration noise to the safety action to generate an exploration action; The safety state of the exploration action is determined according to the preset driving environment model and the uncertainty constraint attenuation function to detect the safety state of the current decision-making and control strategy of the vehicle.

3. The method according to claim 1, characterized in that: Before fitting the random error of the preset driving environment model through the preset integrated model, the method further includes: Initializing a preset number of neural networks with the same structure to obtain an initialized neural network; Using target data to train the initialized neural network to obtain a trained neural network; The trained neural network is used to train the current decision-making control strategy, the value function and the uncertainty constraint attenuation function.

4. The method according to claim 1, characterized in that The updating of the value function according to the self-consistent condition of the value function to obtain an updated value function includes: generating virtual driving data using the preset driving environment model; Based on the self-consistent condition of the value function, the value function is updated by using the virtual driving data to obtain an updated value function.

5. The method according to claim 1, characterized in that The uncertainty constraint attenuation function is: in, is the number of state transitions required to violate the state constraint for the first time starting from state x under the control of policy π, γ is the discount factor, and φ is a trainable parameter of the uncertainty constraint decay function.

6. A safety reinforcement learning training device for end-to-end autonomous driving, characterized in that: include: A detection module, for detecting a safety state of a current decision-making control strategy of the vehicle based on a preset driving environment model and an uncertainty constraint attenuation function, so as to collect a predetermined amount of environmental interaction data when the safety state reaches a preset state; A fitting module, used for fitting the random error of the preset driving environment model using a preset integrated model to obtain a fitted random error distribution; An evaluation module, used to obtain a value function of the current decision-making control strategy, and update the value function according to a self-consistent condition of the value function to obtain an updated value function, so as to evaluate the performance value of the current decision-making control strategy using the updated value function; An expansion module is used to update the uncertainty constraint attenuation function of the current decision-making control strategy according to the self-consistent condition of the uncertainty constraint attenuation function to obtain an updated uncertainty constraint attenuation function, so as to expand the feasible region corresponding to the updated uncertainty constraint attenuation function to obtain an expanded feasible region; The strengthening module is used to iteratively update the current decision-making control strategy based on the environmental interaction data, the random error distribution, the performance value and the expanded feasible area until the average cumulative reward of the current decision-making control strategy reaches a preset threshold or reaches a maximum number of iterations, so as to obtain a final feasible decision-making control strategy, so as to strengthen the target autonomous driving system according to the final feasible decision-making control strategy.

7. The device according to claim 6, characterized in that The detection module comprises: A first generating unit, configured to generate at least one safety action of the vehicle using a current decision-making control strategy of the vehicle; a second generating unit, configured to add exploration noise to the safety action to generate an exploration action; A determination unit is used to determine the safety state of the exploration action according to the preset driving environment model and the uncertainty constraint attenuation function to detect the safety state of the current decision-making and control strategy of the vehicle.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the safety reinforcement learning training method for end-to-end autonomous driving as described in any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the safety reinforcement learning training method for end-to-end autonomous driving as described in any one of claims 1-5.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed, it is used to implement the safety reinforcement learning training method for end-to-end autonomous driving as described in any one of claims 1-5.

Images