Autonomous driving decision-making method, system and medium based on safety curriculum reinforcement learning

Abstract

The present invention relates to a method, system, and medium for autonomous driving decision-making based on safety curriculum reinforcement learning. The method comprises the following steps: perceiving information about surrounding vehicles and the vehicle itself as environmental information; employing a curriculum reinforcement learning framework to divide the autonomous driving task into a safety phase and a performance phase, wherein the safety phase trains a safety agent based on a reward function for the safety phase; deploying a safety agent and a performance agent in the performance phase, fixing the safety agent trained in the safety phase, training the performance agent based on a reward function for the performance phase, and switching between the safety agent and the performance agent using a switching function; and interacting with the environment, updating network parameters based on a loss function, and controlling the vehicle based on the safety actions output by the network in the performance phase, thereby achieving safe autonomous driving decisions. Compared with existing technologies, the present invention has the advantages of efficient training and safe interaction.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to an autonomous driving decision-making method, system, and medium based on safety course reinforcement learning. Background Art

[0002] With the rapid development of autonomous driving technology, its importance is becoming increasingly prominent. This technology has the potential to revolutionize transportation, improve road safety, enhance mobility, and transform all sectors of the economy. Within this field, reinforcement learning (RL) has emerged as a key decision-making method. Through RL, intelligent agents can learn optimal actions through interaction with the environment, playing a key role in solving complex decision-making problems. However, despite RL's powerful decision-making capabilities, its trial-and-error nature can lead to safety challenges. In addition, the autonomous driving environment is extremely complex, with characteristics such as a high-dimensional state space, a continuous action space, and long-delay feedback. This poses significant challenges to reinforcement learning, resulting in a very time-consuming learning process.

[0003] On the one hand, to ensure the safety of the agent during reinforcement learning training and after final convergence, current methods include changing the optimization objective, modifying the exploration process, and intervention-based reinforcement learning. However, these methods still have shortcomings in practical applications. Changing the optimization objective and modifying the exploration process are essentially soft constraint processing methods, which cannot fully guarantee safety. Intervention-based reinforcement learning uses intervention mechanisms to directly and effectively correct dangerous reinforcement learning actions, which can greatly improve the safety of the agent and meet the safety requirements of autonomous driving. However, the design of the intervention subject and intervention mechanism is very difficult.

[0004] On the other hand, the complex traffic environments involved in autonomous driving, encompassing multiple factors and potential interactions, make decision-making a difficult and time-consuming task. Curriculum learning, a reinforcement learning approach, is an effective method for helping intelligent agents gradually adapt to such complex environments. By breaking the driving process into manageable steps, curriculum learning enables drivers to gradually learn and master skills based on previously learned knowledge. This incremental approach allows drivers to gradually navigate increasingly complex situations, thereby improving performance and adaptability, particularly in challenging autonomous driving scenarios. However, this approach often requires extensive modification of the environment during implementation, which is not only time-consuming and labor-intensive but can also be complex and difficult in practice. Furthermore, curriculum learning simply breaks down the driving task into a series of smaller tasks and lacks specific mechanisms to ensure that previously learned tasks are implemented and retained as the agent enters subsequent stages. Therefore, in the field of autonomous driving, more efficient and reliable methods are needed to address complex driving decision-making. Summary of the Invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide an autonomous driving decision-making method, system and medium based on safety course reinforcement learning, so as to achieve higher driving safety and accelerate the learning process of the intelligent agent.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] According to a first aspect of the present invention, a method for autonomous driving decision-making based on safety curriculum reinforcement learning is provided, the method comprising the following steps:

[0008] Perceive information about surrounding vehicles and the vehicle itself as environmental information;

[0009] The curriculum reinforcement learning framework is used to divide the autonomous driving task into a safety phase and a performance phase. In the safety phase, the safety agent is trained based on the reward function of the safety phase. In the performance phase, the safety agent and the performance agent are deployed. The safety agent trained in the safety phase is fixed, and the performance agent is trained based on the reward function of the performance phase. A switching function is used to switch between the safety agent and the performance agent.

[0010] The intelligent agent interacts with the environment, updates its own network parameters according to the loss function of the safety stage and the loss function of the performance stage, controls the vehicle based on the safe actions output by the network in the performance stage, and realizes the safety decision of autonomous driving.

[0011] The reward function R in the safety phase s for:

[0012]

[0013] where ρ i (i=1,2,3) is the weight coefficient;

[0014]

[0015] Among them, s t represents the state at time t, a t represents the action at time t, n is the total number of observed vehicles, represents the speed of the i-th vehicle in the x direction, i=0 represents the vehicle itself; x 0 , x i ,y 0 ,y i Represents the x-coordinate and y-coordinate of the vehicle and the i-th vehicle respectively, x ref represents the minimum safe distance, L represents the lane width, m is the number of lanes, r accident is the negative feedback given when the vehicle encounters an accident, S accident A collection of unexpected states.

[0016] The reward function R of the performance stage p for:

[0017]

[0018] where ω i (i=1, 2, 3, 4) is the weight coefficient;

[0019]

[0020] Among them, s t represents the state at time t, a t represents the action at time t, represents the speed of the i-th vehicle in the x direction, i=0 represents the vehicle itself; v ref is the reference speed affected by the surrounding environment; m is the number of lanes; y 0 ,y i denote the y coordinates of the vehicle and the i-th vehicle respectively; h is the vehicle heading; a is the action taken by the agent; It is the positive incentive given when the car overtakes other cars.

[0021] The state space of reinforcement learning is defined as a two-dimensional tensor of size n×m, where n represents the number of vehicles to be observed and m represents the number of states to be observed for each vehicle. Specifically, it is expressed as:

[0022] s=[s[0], s[1], s[2],…, s[n-1]],

[0023]

[0024] Among them, x i ,y i represents the x and y coordinates of the i-th vehicle, It represents the speed of the i-th vehicle in the x and y directions; h i represents the heading angle of the vehicle; n is the total number of observed vehicles.

[0025] The action taken by the agent is represented as Among them, A represents the action space, σ represents the longitudinal acceleration and steering angle respectively.

[0026] The switching function Δ is:

[0027] Δ P (s t )~B(1,p(s t )),

[0028]

[0029] Among them, s t represents the state at time t, ε P >0,ζ P ∈[0, 1], is a hyperparameter; are the actions of the performance agent and the safety agent, respectively; Represents the state action value based on security policy, represents the state-action value based on the performance strategy, E is the expectation; B represents the Bernoulli distribution; clip is the clipping function; π p is the performance strategy, π s is the security strategy; Δ = 1 means that the security agent will take over the performance agent to interact with the environment, and Δ = 0 means that the performance agent interacts with the environment.

[0030] The loss function of the network parameter update in the security phase is:

[0031]

[0032] Among them, φ s represents the network parameters of the critic in the security phase, θ s represents the network parameters of the actors in the security phase, E represents the expectation, For safety rewards, represents the state action value in the safety stage, α is the temperature factor, γ is the discount factor, s t , s t+1 Indicates the current and next state, a t , a t+1Indicates the current action and the next action, D is the cache playback, Strategies for security agents.

[0033] To evaluate the switching function Q network The parameters used are The critic network evaluates the Q value of the state; in order to learn The following objective function is optimized:

[0034]

[0035] in, For performance rewards, Indicates the state action value of the performance stage.

[0036] The loss function for network parameter update in the performance phase is:

[0037]

[0038] Among them, φ p is the Q network parameter of the performance agent, θ p is the policy network parameter of the performance agent, Δ is the switching function, s t represents the state at time t, a t represents the action at time t, λ is the weight parameter, D is the cache playback, The strategy for the performance agent.

[0039] According to a second aspect of the present invention, an autonomous driving decision-making system based on safety curriculum reinforcement learning includes:

[0040] The perception subsystem is used to perceive information about surrounding vehicles and the vehicle itself as environmental information;

[0041] The decision-making subsystem is used to receive the perception information about the interactive environment from the perception subsystem, make decisions on the control amount of the self-driving car, and send it to the execution subsystem to enable the self-driving car to achieve autonomous driving: the curriculum reinforcement learning framework is used to divide the autonomous driving task into a safety stage and a performance stage. In the safety stage, the safety agent is trained based on the reward function of the safety stage; in the performance stage, the safety agent and the performance agent are deployed, the safety agent trained in the safety stage is fixed, and the performance agent is trained based on the reward function of the performance stage, and a switching function is used to switch between the safety agent and the performance agent; the agent interacts with the environment, updates its own network parameters according to the loss function of the safety stage and the loss function of the performance stage, and outputs the safety action control amount based on the network output of the performance stage;

[0042] The execution subsystem is used to control the vehicle according to the control quantity output by the decision subsystem.

[0043] According to a third aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] The present invention has the ability of efficient convergence and safe interaction, which can effectively improve the safety of autonomous driving. Specifically, the present invention divides the training process into a safety stage and a performance stage. In the safety stage, the intelligent body will give priority to meeting the safety specifications to ensure that the safety of the system is fully guaranteed before improving the overall performance, thereby improving the efficiency of training. Subsequently, in the performance stage, a safety assurance mechanism is introduced through the switching function, that is, the safety intelligent body continuously monitors the intelligent body to ensure the driving safety of the autonomous vehicle. This method can effectively improve the training effect, allowing the intelligent body to gradually improve its performance under the guidance of safety specifications. This method provides an innovative way to solve the safety problem of reinforcement learning in autonomous driving. Through the effective division of the training process and the introduction of a safety assurance mechanism, this method can achieve efficient convergence and safe interaction, bringing a more reliable and safe prospect to the development of autonomous driving technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a system structure diagram of the present invention;

[0047] Figure 2 Schematic diagram of the safety stage and performance stage of the present invention;

[0048] Figure 3 Schematic diagram of the process for implementing the method of the present invention;

[0049] Figure 4 This is a schematic diagram of the safety switching control of the present invention;

[0050] Figure 5 A comparison diagram of driving strategy results obtained by the present invention and a traditional reinforcement learning method in one embodiment;

[0051] Figure 6 A comparison chart of driving strategy results obtained by the present invention and the traditional course learning method in one embodiment. DETAILED DESCRIPTION

[0052] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0053] Obviously, the drawings described below are merely examples or embodiments of the present application. Those skilled in the art can, without inventive effort, apply the present application to other similar scenarios based on these drawings. Furthermore, it is also understood that, although the effort involved in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, changes in design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as an insufficiency of the content disclosed in this application.

[0054] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent.

[0055] This embodiment first provides an autonomous driving decision-making system based on safety curriculum reinforcement learning, including:

[0056] The perception subsystem is used to perceive information about surrounding vehicles and the vehicle itself as environmental information;

[0057] The decision-making subsystem receives the perception information about the interactive environment from the perception subsystem, determines the control variables of the vehicle (acceleration and front wheel angle), and sends them to the execution subsystem to enable the vehicle to achieve autonomous driving. The decision-making subsystem consists of a safety agent, a performance agent, and a safety switching function. The safety switching function determines safe actions based on the actions of the agents, such as Figure 1 As shown in the figure, the curriculum reinforcement learning framework is used to divide the autonomous driving task into a safety phase and a performance phase. In the safety phase, the safety agent is trained based on the reward function of the safety phase. In the performance phase, the safety agent and the performance agent are deployed. The safety agent trained in the safety phase is fixed, and the performance agent is trained based on the reward function of the performance phase. A switching function is used to switch between the safety agent and the performance agent. The agent interacts with the environment, updates its own network parameters according to the loss function of the safety phase and the loss function of the performance phase, and outputs the safety action control quantity based on the network output in the performance phase.

[0058] The execution subsystem is used to control the vehicle according to the control quantity output by the decision subsystem.

[0059] Based on the above system, this embodiment provides an autonomous driving decision-making method based on safety curriculum reinforcement learning, which includes the following steps:

[0060] Perceive information about surrounding vehicles and the vehicle itself as environmental information;

[0061] like Figure 2As shown in the figure, the curriculum reinforcement learning framework is used to divide the autonomous driving task into a safety phase and a performance phase, so as to gradually improve driving safety and performance. In the safety phase, the safety agent is trained based on the reward function of the safety phase. In the performance phase, the safety agent and the performance agent are deployed. The safety agent trained in the safety phase is fixed, and the performance agent is trained based on the reward function of the performance phase. A switching function is used to switch between the safety agent and the performance agent.

[0062] The intelligent agent interacts with the environment, updates its own network parameters according to the loss function of the safety stage and the loss function of the performance stage, controls the vehicle based on the safe actions output by the network in the performance stage, and realizes the safety decision of autonomous driving.

[0063] Safety course reinforcement learning divides the learning of driving tasks into a safety stage and a performance stage, so as to gradually improve driving safety and performance. In the safety stage, the driving agent focuses on keeping the lane clear and ensuring the safety of the driving process. In the performance stage, the driving agent improves driving performance through behaviors such as overtaking, following, and changing lanes, while maintaining a certain level of safety. By introducing safety courses and a phased training strategy, the present invention can speed up learning and focus on safety during driving. This method can effectively balance the requirements of safety and performance in driving decisions, and provides a reliable autonomous driving decision-making solution. It should be noted that although this embodiment assumes that the driving scenario is a two-lane straight highway, this method can be extended to other driving scenarios to meet the needs of different driving environments.

[0064] The specific implementation process of the above method is as follows Figure 3 As shown in the figure, it is mainly divided into three stages: design of reinforcement learning related elements, design of security mechanism, and design of reinforcement learning update mechanism.

[0065] 1. Design of reinforcement learning related elements

[0066] Compared with traditional curriculum learning methods, the method proposed in this paper focuses on task decomposition and simplifies the design of each stage by modifying only the reward function, providing convenience and flexibility for the design of each stage. On this basis, the reinforcement learning training process can be summarized as follows:

[0067] Γ=<M s , M p >,

[0068] M s =<S, A, P, R s ,Y,d0>,

[0069] M p =<S, A, P, R p,γ,d0>.

[0070] Where S is the reinforcement learning state space, A is the action space, P is the state transition probability, and R s 、R p are the reward functions for the safety and performance stages respectively, γ is the discount factor, and d0 is the initial state.

[0071] Furthermore, to further enhance training safety, the performance phase utilizes a hybrid agent approach, rather than a single agent. This hybrid agent comprises a safety agent and a performance agent. A switching function is introduced to regulate the interaction between the safety and performance agents, ensuring they are appropriately protected as they interact with the environment. This design effectively balances safety and performance optimization, enabling the safety agent to intervene when necessary to maintain driving safety.

[0072] (1) Reward function R in the safety phase s

[0073] In the safety phase, ensuring safety during driving is the primary goal of autonomous driving decision-making. The training of the safety phase aims to enable the driving agent to complete the journey without any accidents within a relatively short training time. In the two-lane application scenario, the specific safety phase behavior requirement is to maintain the same lane when decelerating and not to overtake. Therefore, the reward function R in the safety phase is s Designed to:

[0074]

[0075] where ρ i (i=1, 2, 3) is the weight coefficient;

[0076]

[0077] Among them, s t represents the state at time t, a t represents the action at time t, n is the total number of observed vehicles, represents the speed of the i-th vehicle in the x direction, i=0 represents the vehicle itself; x 0 , x i ,y 0 ,y i Represents the x-coordinate and y-coordinate of the vehicle and the i-th vehicle respectively, x ref represents the minimum safe distance, L represents the lane width, m is the number of lanes, in this embodiment, m=2, r accident is the negative feedback given when the vehicle encounters an accident, S accident A collection of unexpected states. The purpose of is to keep the agent at a low speed, consistent with the minimum observed speed of surrounding vehicles; The purpose of r is to make the agent always keep a safe distance from the vehicle in front; centreline The purpose is to prevent the vehicle from deviating too far from the center line.

[0078] (2) Reward function R in the performance stage p

[0079] Under the premise of ensuring a certain level of safety for the agent, the goal of the performance phase is to achieve a higher level of performance. In a two-lane application scenario, this means that the agent needs to perform actions such as overtaking and following. In order to fully consider these factors, the reward function R in the performance phase is p Designed to:

[0080]

[0081] where ω i (i=1, 2, 3, 4) is the weight coefficient;

[0082]

[0083] Among them, s t represents the state at time t, a t represents the action at time t, represents the speed of the i-th vehicle in the x direction, i=0 represents the vehicle itself; v ref is the reference speed affected by the surrounding environment; m is the number of lanes; y 0 ,y i denote the y coordinates of the vehicle and the i-th vehicle respectively; h is the vehicle heading; a is the action taken by the agent; It is the positive incentive given when the vehicle overtakes another vehicle. The default value is 1. and The goals are similar, but The agent is required to maintain a high speed during the performance phase; The purpose is to improve the vehicle's smooth and comfortable driving experience.

[0084] (3) State space S

[0085] In order to include complete and accurate environmental information, the state space of reinforcement learning is defined as a two-dimensional tensor of size n×m, where n represents the number of vehicles to be observed and m represents the number of states to be observed for each vehicle. Specifically, it is expressed as:

[0086] s=[s[0], s[1], s[2],…, s[n-1]],

[0087]

[0088] Among them, x i ,y i represents the x and y coordinates of the i-th vehicle, It represents the speed of the i-th vehicle in the x and y directions; h i represents the heading angle of the vehicle; n is the total number of observed vehicles.

[0089] (4) Action space A

[0090] The action taken by the agent is represented as in, σ represents the longitudinal acceleration and steering angle respectively. To ensure driving safety, physical constraints need to be imposed on the action:

[0091]

[0092] 2. Security assurance mechanism design

[0093] In order to ensure the security of the agent during the performance phase, the present invention introduces a protection mechanism called Safe Switching Control (SSC).

[0094] The control of SSC includes deploying safety agents and performance agents in the performance phase. s is trained and fixed as a safety agent in the performance phase. Meanwhile, the performance agent is trained according to R s Perform training. Performance strategy π p and security policy π s The combination of actions forms a mixed strategy π b The core of SSC is the switching between the safety agent and the performance agent. Under normal circumstances, the performance agent interacts with the environment, but when the performance agent takes dangerous actions, the safety agent needs to take over in time. Specifically, Figure 4 As shown, the performance agent takes safe actions and does not require the intervention of the safety agent; however, when the performance agent takes dangerous actions, the safety agent intervenes. Therefore, the present invention uses a switching function Δ to control the switching of agents. Δ = 1 means that the safety agent will take over the performance agent, and Δ = 0 means that the performance agent will interact with the environment. Then, the mixed strategy π b It can be expressed as follows:

[0095] π b =Δπ s +(1-Δ)π p .

[0096] The design of Δ needs to ensure the timely intervention of the security agent, so the switching function is designed as follows:

[0097] Δ P(s t )~B(1,p(s t )),

[0098]

[0099] Among them, s t represents the state at time t; ε P >0,ζ P ∈[0, 1], is a hyperparameter; are the actions of the performance agent and the safety agent, respectively; Represents the state action value based on security policy, It represents the state-action value based on the performance strategy, E is the expectation; B represents the Bernoulli distribution; clip is the clipping function. When the first term of the function is less than the second term, the second term is taken; when the first term of the function is greater than the third term, the third term is taken; otherwise, the first term is taken.

[0100] The switching function randomly determines whether to intervene based on the difference between the actions taken by the safety agent and the performance agent. By using this formula, the safety agent can intervene promptly when training encounters a dangerous situation without sacrificing training speed.

[0101] 3. Update mechanism design

[0102] The proposed safe curriculum learning method adopts an actor-critic framework, which includes two main neural networks and an additional auxiliary network. The actor network, also known as the policy network, uses the parameters θ to select the appropriate action based on the current state. The critic network, also known as the Q network, uses the parameters To estimate the state-action value, the present invention uses two Q networks to reduce the error of the Q network when calculating the Q function.

[0103] Through the actor-critic framework, the actor network is responsible for generating action strategies, while the critic network evaluates the value of these actions. The actor network uses the current state as input and selects an action based on the learned policy parameters θ. The critic network uses the learned parameters θ based on the current state and the selected action. To estimate the state v action value. The advantage of this actor-critic framework is that the performance of the entire system can be optimized by improving the strategy of the actor network.

[0104] (1) Updates during the security phase

[0105] The loss function for network parameter update in the security phase is:

[0106]

[0107] Among them, φs represents the network parameters of the critic in the security phase, θ s represents the network parameters of the actors in the security phase, E represents the expectation, For safety rewards, represents the state action value in the safety stage, α is the temperature factor, γ is the discount factor, s t , s t+1 Indicates the current and next state, a t , a t+1 Indicates the current action and the next action, D is the cache playback, Strategies for security agents.

[0108] To obtain the evaluation Q network of the switching function in SSC Another parameter used is The critic network evaluates the Q value of the state. In order to learn The following objective functions are optimized:

[0109]

[0110] in, For performance rewards, Indicates the state action value of the performance stage.

[0111] (2) Performance phase updates

[0112] During the performance phase, there are two agents: the safety agent and the performance agent. The safety agent is trained only during the safety phase and remains fixed throughout the performance phase. While the safety agent's intervention can ensure the performance agent's safety, there is a risk of over-reliance on this intervention, which may impair the performance agent's independent decision-making ability. To reduce the performance agent's over-reliance on the safety agent's intervention, this method uses negative rewards to discourage frequent intervention. Therefore, the following loss function is used to update the performance agent's Q network and policy network:

[0113]

[0114] Among them, φp is the Q network parameter of the performance agent, θp is the policy network parameter of the performance agent, Δ is the switching function, st represents the state at time t, at represents the action at time t, λ is the weight parameter, D is the cache playback, The strategy for the performance agent.

[0115] Taking into account the diversity of scenarios, this embodiment only gives the experimental verification results of the comparison with the benchmark algorithm on a two-lane highway. However, the present invention is applicable to all interactive scenarios of autonomous vehicles and other traffic participants. According to the actual road and vehicle dimensions, the lane width is set to 4m and the vehicle size is set to 4m by 1.8m. These specifications are selected to be close to the actual road and vehicle dimensions for accurate simulation and analysis. Following the general reinforcement learning task, the maximum step size of each round is set to 100. In addition, the critic network and actor network involved in the algorithm are both multi-layer perceptrons (MLP) with three hidden layers and 256 hidden units in each layer.

[0116] In order to evaluate the effectiveness of the proposed algorithm compared with traditional reinforcement learning algorithms, the proposed algorithm uses auxiliary baseline algorithms such as SAC and PPO for comparative analysis. To ensure fairness in the comparison, the network parameters in these baseline algorithms are set to be the same as those in the proposed algorithm, the reward function uses Rp, and other parameters are kept consistent with the proposed algorithm. The experimental results are shown in Figure 2. Figure 5 As shown. According to the image, the round length of the algorithm MCL of the present invention during training reached a high point in the safety stage and remained at a high point in the performance stage. This shows that the algorithm of the present invention improved safety in the safety stage, and steadily improved performance while maintaining safety in the performance stage. In addition, the reward rising speed of the algorithm of the present invention in the performance stage is also higher than that of SAC and PPO, which shows that it improves the training speed while improving safety. By comparison with traditional reinforcement learning algorithms, the experimental results show that the algorithm of the present invention has advantages in improving safety and performance. The high point of the round length and the increase in the reward rising speed during the training process of the algorithm of the present invention both show its effectiveness and superiority in a two-lane environment. These results verify the effectiveness of the algorithm of the present invention in the experiment and show its improvements in safety and performance.

[0117] In addition, in order to verify the effectiveness of the proposed security mechanism SSC, this embodiment uses ordinary curriculum learning (CL) for comparison. In ordinary curriculum learning, the second stage lacks a protection mechanism and only includes an intelligent agent that interacts with the environment. The rest of the settings are the same as the algorithm of the present invention. Figure 6As shown. The results show that CL still maintains a certain degree of security in the second stage, but the algorithm MCL of the present invention shows higher stability and security. In addition, the performance improvement speed and the speed of final convergence to a higher level of the algorithm of the present invention both exceed CL. This phenomenon can be attributed to the inherent security mechanism provided by the algorithm of the present invention, which provides a favorable guarantee for performance improvement. By comparing with ordinary course learning, the experimental results effectively verified the effectiveness of the proposed security mechanism SSC. Compared with CL, the algorithm of the present invention shows significant improvements in both security and performance. The stability, security improvement and rapid performance improvement of the algorithm of the present invention fully demonstrate that SSC has a positive impact on performance improvement while improving system security.

[0118] In summary, the method of the present invention demonstrates faster learning speed and higher decision-making safety compared to traditional reinforcement learning methods. By introducing a hybrid agent consisting of a safety agent and a performance agent, and by flexibly adjusting switching functions, the method of the present invention can actively intervene in safety during driving, thereby ensuring the safety of decisions. Overall, the decision-making system proposed in this invention has significant advantages over other methods in terms of learning speed and decision-making safety. This discovery provides strong support for the realization of reliable, safe, and efficient autonomous driving technology.

[0119] In the context of the present invention, machine-readable medium can be a tangible medium that can contain or store a program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0120] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. An autonomous driving decision-making method based on safety curriculum reinforcement learning, characterized in that: The method comprises the following steps: Perceive information about surrounding vehicles and the vehicle itself as environmental information; The curriculum reinforcement learning framework is used to divide the autonomous driving task into a safety phase and a performance phase. In the safety phase, the safety agent is trained based on the reward function of the safety phase. In the performance phase, the safety agent and the performance agent are deployed. The safety agent trained in the safety phase is fixed, and the performance agent is trained based on the reward function of the performance phase. A switching function is used to switch between the safety agent and the performance agent. The agent interacts with the environment, updates its own network parameters based on the loss function of the safety phase and the loss function of the performance phase, and controls the vehicle based on the safe actions output by the performance phase network to achieve safe decision-making for autonomous driving. The switching function Δ is: D P (s t )~B(1,p(s t )), Among them, s t represents the state at time t, ε P >0,ζ P ∈[0,1], is a hyperparameter; are the actions of the performance agent and the safety agent, respectively; Represents the state action value based on security policy, represents the state-action value based on the performance strategy, E is the expectation; B represents the Bernoulli distribution; clip is the clipping function; π p is the performance strategy, π s is the security strategy; Δ = 1 means that the security agent will take over the performance agent to interact with the environment, and Δ = 0 means that the performance agent interacts with the environment.

2. The autonomous driving decision-making method based on safety curriculum reinforcement learning according to claim 1, characterized in that: The reward function R in the safety phase s for: where ρ i (i=1,2,3) is the weight coefficient; Among them, s t represents the state at time t, a t represents the action at time t, n is the total number of observed vehicles, represents the speed of the i-th vehicle in the x direction, i=0 represents the vehicle itself; x 0 ,x i ,y 0 ,y i Represents the x-coordinate and y-coordinate of the vehicle and the i-th vehicle respectively, x ref represents the minimum safe distance, L represents the lane width, m is the number of lanes, r accident is the negative feedback given when the vehicle encounters an accident, S accident A collection of unexpected states.

3. The autonomous driving decision-making method based on safety curriculum reinforcement learning according to claim 1, characterized in that: The reward function R of the performance stage p for: where ω i (i=1,2,3,4) is the weight coefficient; Among them, s t represents the state at time t, a t represents the action at time t, represents the speed of the i-th vehicle in the x direction, i=0 represents the vehicle itself; v ref is the reference speed affected by the surrounding environment; m is the number of lanes; y 0 ,y i denote the y coordinates of the vehicle and the i-th vehicle respectively; h is the vehicle heading; a is the action taken by the agent; is the positive incentive given when the vehicle overtakes the other vehicle; r accident is the negative feedback given when the vehicle encounters an accident, S accident A collection of unexpected states.

4. The autonomous driving decision-making method based on safety curriculum reinforcement learning according to claim 1, characterized in that: The state space of reinforcement learning is defined as a two-dimensional tensor of size n×m, where n represents the number of vehicles to be observed and m represents the number of states to be observed for each vehicle. Specifically, it is expressed as: s=[s[0],s[1],s[2],…,s[n-1]], Among them, x i ,y i represents the x and y coordinates of the i-th vehicle, It represents the speed of the i-th vehicle in the x and y directions; h i represents the heading angle of the vehicle; n is the total number of observed vehicles.

5. The autonomous driving decision-making method based on safety curriculum reinforcement learning according to claim 1, characterized in that: The action taken by the agent is represented as Among them, A represents the action space, σ represents the longitudinal acceleration and steering angle respectively.

6. The autonomous driving decision-making method based on safety curriculum reinforcement learning according to claim 1, characterized in that: The loss function of the network parameter update in the security phase is: Among them, φ s represents the network parameters of the critic in the security phase, θ s represents the network parameters of the actors in the security phase, E represents the expectation, For safety rewards, represents the state action value in the safety stage, α is the temperature factor, γ is the discount factor, s t ,s t+1 Indicates the current and next state, a t ,a t+1 Indicates the current action and the next action, D is the cache playback, strategies for security agents; To evaluate the switching function Q network The parameters used are The critic network evaluates the Q value of the state; in order to learn The following objective function is optimized: in, For performance rewards, Indicates the state action value of the performance stage.

7. The autonomous driving decision-making method based on safety curriculum reinforcement learning according to claim 1, characterized in that: The loss function for network parameter update in the performance phase is: Among them, φ p is the Q network parameter of the performance agent, θ p is the policy network parameter of the performance agent, Δ is the switching function, s t represents the state at time t, a t represents the action at time t, For performance rewards, Indicates the state action value of the security stage, represents the state action value of the performance stage, α is the temperature factor, γ is the discount factor, λ is the weight parameter, D is the cache playback, The strategy for the performance agent.

8. An autonomous driving decision-making system based on safety curriculum reinforcement learning, characterized in that: include: The perception subsystem is used to perceive information about surrounding vehicles and the vehicle itself as environmental information; The decision-making subsystem is used to receive the perception information about the interactive environment from the perception subsystem, make decisions on the control amount of the self-driving car, and send it to the execution subsystem to enable the self-driving car to achieve autonomous driving: the curriculum reinforcement learning framework is used to divide the autonomous driving task into a safety stage and a performance stage. In the safety stage, the safety agent is trained based on the reward function of the safety stage; in the performance stage, the safety agent and the performance agent are deployed, the safety agent trained in the safety stage is fixed, and the performance agent is trained based on the reward function of the performance stage, and a switching function is used to switch between the safety agent and the performance agent; the agent interacts with the environment, updates its own network parameters according to the loss function of the safety stage and the loss function of the performance stage, and outputs the safety action control amount based on the network output of the performance stage; The execution subsystem is used to control the vehicle according to the control quantity output by the decision subsystem; The switching function Δ is: D P (s t )~B(1,p(s t )), Among them, s t represents the state at time t, ε P >0,ζ P ∈[0,1], is a hyperparameter; are the actions of the performance agent and the safety agent, respectively; Represents the state action value based on security policy, represents the state-action value based on the performance strategy, E is the expectation; B represents the Bernoulli distribution; clip is the clipping function; π p is the performance strategy, π s is the security strategy; Δ = 1 means that the security agent will take over the performance agent to interact with the environment, and Δ = 0 means that the performance agent interacts with the environment.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images