Intelligent agent reinforcement learning method and device for controlling vehicle driving
By introducing expert supervision and incentives in a real-world environment, the learning process of autonomous driving agents is optimized, solving the problems of insufficient applicability of simulation environments and low training efficiency, and enabling agents to learn efficiently in real-world environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 709TH RESEARCH INSTITUTE CHINA STATE SHIPBUILDING CORP LTD
- Filing Date
- 2023-02-22
- Publication Date
- 2026-05-19
AI Technical Summary
Existing autonomous driving agents are not well-suited for training in simulated environments, and the large solution space and sparse reward acquisition during the learning process lead to low training efficiency.
This paper proposes a reinforcement learning approach that integrates expert supervision and incentives. By interacting with expert experience in a real environment, a Markov real training dataset is constructed, and the reward is modified using expert approval values to optimize the network parameters of the Actor-Critic framework.
It improves the applicability and learning efficiency of autonomous driving agents in real-world environments and reduces policy search costs.
Smart Images
Figure CN116176606B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of reinforcement learning technology, and more specifically, relates to a reinforcement learning method and apparatus for controlling vehicle driving. Background Technology
[0002] In recent years, deep reinforcement learning technology has achieved groundbreaking progress by combining the perception capabilities of deep learning with the decision-making capabilities of reinforcement learning, and has been widely applied in fields such as chess, game AI, autonomous driving, and robot control. Currently, most intelligent algorithms, represented by reinforcement learning, train agents under the support of simulation software. The credibility of the simulation environment directly affects the applicability of the agent to actual combat. At the same time, due to the complexity and diversity of application environments, the learning process of the agent has characteristics such as a large solution space and sparse reward acquisition, making training difficult to converge, exploration costs high, and learning efficiency low.
[0003] Therefore, the following problems exist in the construction of intelligent agents in the current field of autonomous driving: (1) For autonomous driving intelligent agents trained under the support of simulation environment, the credibility and authenticity of simulation software directly affect the applicability of intelligent agents to actual driving control; (2) Due to the complexity and diversity of actual driving environment, the learning process of intelligent agents has characteristics such as large solution space and sparse acquisition of key rewards, which makes the policy training search cost high and the learning efficiency low. Summary of the Invention
[0004] In response to the above-mentioned deficiencies or improvement needs of existing technologies, this invention mainly addresses the problems of difficult and inefficient exploration by agents in complex environment reinforcement learning and the difficulty in guiding training with high-value human decision-making experience. It proposes a reinforcement learning method that integrates expert supervision and incentives, incorporating the supervision information of expert experience and behavior and its quantitative incentive results into the training process, thereby reducing the exploration cost of the agent in the environment and improving the learning convergence efficiency.
[0005] To achieve the above objectives, according to one aspect of the present invention, an agent reinforcement learning method for controlling vehicle driving is provided, comprising the following steps:
[0006] (1) Construct a stochastic policy agent;
[0007] (1-1) Implement the learning and training process of the agent under the Actor-Critic framework. The output of the agent is the probability distribution of the execution data of the steering wheel angle, throttle control force, and brake control force parameters.
[0008] (1-2) Initialize the agent decision network parameters θ, i.e., the Actor network. All node values in the network are sampled using a Gaussian distribution with a mean of 0 and a standard deviation of 0.01. Policy model π θIn vehicle observation information data t Under the given conditions, the output results constitute the Gaussian probability distribution of the relevant parameters for the decision-making vehicle control parameters. and in and These are the mean and covariance matrix of the Gaussian model of the decision-making action, namely the mean of the parameters of steering wheel angle, throttle control force, and brake control force, and the matrix elements that constitute the covariance matrix. The interaction process obtains the executed actions (a1, a2, a3) through sampling and inputs them into the environment to generate state transitions, where a1, a2, and a3 represent the steering wheel angle value, throttle control force value, and brake control force value during the actual driving process, respectively.
[0009] (1-3) Initialize the agent evaluation network parameters φ, i.e. the Critic network. All node values of the network are obtained by sampling through a Gaussian distribution with a mean of 0 and a standard deviation of 0.01.
[0010] (2) The agent generates a Markov real training dataset by interacting directly with the real environment, and generates the original constructed training dataset by the expert experience decision dataset in the training environment.
[0011] (3) For the original constructed training dataset obtained in (2), calculate the expert approval value by the decision behavior similarity comparison method, and determine whether each decision result is an expert-approved decision result;
[0012] (4) Modify the original constructed training dataset based on the expert approval decision results, recalculate the reward based on the expert approval value, and finally obtain the expert incentive training dataset. This dataset, together with the real training dataset, is used to update the agent network.
[0013] (5) Repeat steps (2)-(4) until the agent converges.
[0014] In one embodiment of the present invention, step (2) specifically includes:
[0015] (2-1) By combining the relevant information acquired by the agent within a single time step through a real driving interaction process, a Markov real training dataset D is generated. r <S,A,R,next_S> Where: S represents the driving environment state observed by the agent; A represents the action selected by the agent through sampling after obtaining the vehicle control parameter action probability distribution through the decision network; R represents the reward feedback obtained by the agent directly from the driving environment according to the rule setting; next_S represents the driving environment state at the next time step;
[0016] (2-2) Generate the original constructed training dataset by using the expert experience decision dataset in the training environment. This expert experience decision dataset is the driver's historical decision data.
[0017] In one embodiment of the present invention, step (2-2) specifically includes:
[0018] (2-2-1) Obtain the expert experience decision-making dataset E e A e >, its data entries (s e ,a e ) represents the state of environment s e Results of expert driver experience-based decision-making actions based on experience analysis (a) e , where a e The expert's steering wheel angle value a 1e Expert throttle control force value a 2e The expert brake control force value consists of a 3e ;
[0019] (2-2-2) Utilizing expert experience to make decisions on the set of states in dataset E e Initialize the environment, and sequentially set the value s within it. e The agent's policy model π is input as the observed state. θ The probability distribution of actions is obtained. Where N a The Gaussian probability distribution of action a has the following probability density function:
[0020]
[0021] Where A is the set of actions (a1, a2, a3) consisting of steering wheel angle value, throttle control force value, and brake control force value.
[0022] (2-2-3) Selecting agent decision-making action a through sampling a Interact with the environment and record the reward feedback from the environment. o Next environment state (next_s) e Action probability density function f a and the corresponding expert driver experience decision-making results a e The original training data entries (s) are constructed. e ,a a ,r o ,next_s e ,f a ,a e ), generate the original constructed training dataset D o e A a ,Ro ,next_S e ,F a A e >
[0023] In one embodiment of the present invention, step (3) specifically includes: sequentially traversing D o Data entries (s) e ,a a ,r o ,next_s e ,f a ,a e Based on the judgment results, expert incentive information is introduced into the training data.
[0024] In one embodiment of the present invention, step (3) specifically includes:
[0025] (3-1) Action a is decided by the agent a probability density function f a (a1,a2,a3) and the result of expert driver's experience-based decision-making actions a e Calculate the probability:
[0026]
[0027] This probability value p is considered to be the expert-approved value;
[0028] (3-2) Define the expert approval threshold p based on human experience. e ;
[0029] (3-3) If p>p e If the decision is consistent with expert experience, it is recognized by the supervising expert and receives expert incentive feedback.
[0030] In one embodiment of the present invention, step (3-2-3) specifically includes:
[0031] (3-3-1) Calculate expert incentive feedback r e =αp, where α is the expert incentive control coefficient;
[0032] (3-3-2) Update data entries (s) e ,a a ,r o ,next_s e ,f a ,a e ) is (s e ,a a ,r o ,next_s e ,r e ).
[0033] In one embodiment of the present invention, step (4) specifically includes:
[0034] (4-1) Traverse the original constructed training data entries (s) e ,a a ,r o ,next_s e ,r e ), generate new expert-incentivized data training entries;
[0035] (4-2) Form a new expert incentive training dataset D by using expert incentive training data entries. e ;
[0036] (4-3) The real training dataset and the expert-incentivized training dataset together constitute the total training dataset D. According to the Actor-Critic framework, the network parameters φ of the agent Critic are updated through D.
[0037] (4-4) Update the Actor decision network parameters θ through the agent Critic according to the Actor-Critic framework.
[0038] In one embodiment of the present invention, step (4-1) specifically includes:
[0039] (4-1-1) Calculate the new expert reward r a =r o +r e ;
[0040] (4-1-2) Modify the original constructed training data entries to expert incentive training data entries (s e ,a a ,r a ,next_s e ).
[0041] According to another aspect of the present invention, an intelligent agent reinforcement learning device for controlling vehicle driving is also provided, including at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor, wherein the instructions, after being executed by the processor, are used to complete the intelligent agent reinforcement learning method for controlling vehicle driving.
[0042] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:
[0043] This invention presents an optimization scheme for autonomous driving agent networks in a human-machine hybrid mode. To address the aforementioned problems, expert supervision and incentives can be introduced during the learning and training process. Expert incentive information serves as guidance to assist the agent in learning, leveraging human experience to guide the optimization direction of the agent network. By influencing the update of the agent's evaluation network (Critic) through expert incentives, human supervision information is introduced into the interaction process between the agent and the simulation environment. This makes the decision-making methods of the autonomous driving agent more consistent with human experience and behavior, thereby increasing the model's applicability to real-world environments. Furthermore, it reduces policy search costs and improves learning efficiency. Attached Figure Description
[0044] Figure 1 This is a flowchart illustrating the intelligent agent reinforcement learning method for controlling vehicle driving in an embodiment of the present invention. Detailed Implementation
[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.
[0046] For the field of autonomous driving, a control agent oriented towards vehicle driving is trained. The agent receives real-time image data, radar data, and motion data observed during vehicle movement as state input, calculates and outputs steering wheel angle, throttle control force, and brake control force parameters as action output, and obtains corresponding reward values based on the vehicle's motion. The training objective is to enable the vehicle to drive smoothly while ensuring driving safety, maintaining a normal speed as much as possible, and avoiding collisions with other vehicles or objects. Various factors such as driving speed, driving wear and tear, and the coordination of steering wheel angle / throttle / brake operations all affect the reward feedback. The agent needs to find a suitable strategy to maximize the driving score. Figure 1 As shown, the intelligent agent reinforcement learning method for controlling vehicle driving provided by the present invention includes the following steps:
[0047] (1) Construct a stochastic policy agent;
[0048] (1-1) Implement the learning and training process of the agent under the Actor-Critic framework. The output of the agent is the probability distribution of the execution data of the steering wheel angle, throttle control force, and brake control force parameters.
[0049] (1-2) Initialize the agent decision network parameters θ, i.e., the Actor network. All node values in the network are sampled using a Gaussian distribution with a mean of 0 and a standard deviation of 0.01. Policy model π θ In vehicle observation information data t Under the given conditions, the output results constitute the Gaussian probability distribution of the relevant parameters for the decision-making vehicle control parameters. and in and These are the mean and covariance matrix of the Gaussian model of the decision-making action, namely the mean of the parameters of steering wheel angle, throttle control force, and brake control force, and the matrix elements that constitute the covariance matrix. The interaction process obtains the executed actions (a1, a2, a3) through sampling and inputs them into the environment to generate state transitions, where a1, a2, and a3 represent the steering wheel angle value, throttle control force value, and brake control force value during the actual driving process, respectively.
[0050] (1-3) Initialize the agent evaluation network parameters φ, i.e. the Critic network. All node values of the network are obtained by sampling through a Gaussian distribution with a mean of 0 and a standard deviation of 0.01.
[0051] (2) The agent generates a Markov real training dataset by interacting directly with the real environment, and generates the original constructed training dataset by the expert experience decision dataset in the training environment.
[0052] (2-1) By combining the relevant information acquired by the agent within a single time step through a real driving interaction process, a Markov real training dataset D is generated. r <S,A,R,next_S> Where: S represents the driving environment state observed by the agent; A represents the action selected by the agent through sampling after obtaining the vehicle control parameter action probability distribution through the decision network; R represents the reward feedback obtained by the agent directly from the driving environment according to the rule setting; next_S represents the driving environment state at the next time step;
[0053] (2-2) Generate the original constructed training dataset by using the expert experience decision dataset in the training environment. This expert experience decision dataset is the driver's historical decision data.
[0054] (2-2-1) Obtain the expert experience decision-making dataset E e A e >. Its data entries (s e ,a e ) represents the state of environment s e Results of expert driver experience-based decision-making actions based on experience analysis (a) e , where a e The expert's steering wheel angle value a1e Expert throttle control force value a 2e The expert brake control force value consists of a 3e ;
[0055] (2-2-2) Utilizing expert experience to make decisions on the set of states in dataset E e Initialize the environment, and sequentially set the value s within it. e The agent's policy model π is input as the observed state. θ The probability distribution of actions is obtained. Where N a The Gaussian probability distribution of action a has the following probability density function:
[0056]
[0057] Where A is the set of actions (a1, a2, a3) consisting of steering wheel angle value, throttle control force value, and brake control force value;
[0058] (2-2-3) Selecting agent decision-making action a through sampling a Interact with the environment and record the reward feedback from the environment. o Next environment state (next_s) e Action probability density function f a and the corresponding expert driver experience decision-making results a e The original training data entries (s) are constructed. e ,a a ,r o ,next_s e ,f a ,a e ), generate the original constructed training dataset D o e A a ,R o ,next_S e ,F a A e >
[0059] (3) For the original constructed training dataset obtained in (2), calculate the expert approval value by the decision behavior similarity comparison method, and determine whether each decision result is an expert-approved decision result;
[0060] Specifically, traverse D sequentially. o Data entries (s) e ,a a ,r o ,next_s e ,f a ,a e Based on the judgment results, expert incentive information is introduced into the training data; specifically including:
[0061] (3-1) Action a is decided by the agent a probability density function f a (a1,a2,a3) and the result of expert driver's experience-based decision-making actions a e Calculate the probability:
[0062]
[0063] This probability value p is considered to be the expert-approved value;
[0064] (3-2) Define the expert approval threshold p based on human experience. e ;
[0065] (3-3) If p>p e If so, it is considered that the agent's decision-making results conform to expert experience, are recognized by the supervising expert, and receive expert incentive feedback;
[0066] (3-3-1) Calculate expert incentive feedback r e =αp, where α is the expert incentive control coefficient;
[0067] (3-3-2) Update data entries (s) e ,a a ,r o ,next_s e ,f a ,a e ) is (s e ,a a ,r o ,next_s e ,r e );
[0068] (4) Modify the original construction training dataset based on the expert approval decision results, recalculate the reward return through the expert approval value, and finally obtain the expert incentive training dataset. This dataset is used together with the real training dataset to update the agent network.
[0069] (4-1) Traverse the original constructed training data entries (s) e ,a a ,r o ,next_s e ,r e ), generate new expert-incentivized data training entries;
[0070] (4-1-1) Calculate the new expert reward r a =r o +r e ;
[0071] (4-1-2) Modify the original constructed training data entries to expert incentive training data entries (s e ,a a ,r a ,next_s e );
[0072] (4-2) Form a new expert incentive training dataset D by using expert incentive training data entries. e ;
[0073] (4-3) The real training dataset and the expert-incentivized training dataset together constitute the total training dataset D. According to the Actor-Critic framework, the network parameters φ of the agent Critic are updated through D.
[0074] (4-4) Update the Actor decision network parameters θ through the agent Critic according to the Actor-Critic framework;
[0075] (5) Repeat steps (2)-(4) until the agent converges.
[0076] Furthermore, the present invention also provides an intelligent agent reinforcement learning device for controlling vehicle driving, including at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor, wherein the instructions, after being executed by the processor, are used to complete the intelligent agent reinforcement learning method for controlling vehicle driving.
[0077] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. An agent-based reinforcement learning method for controlling vehicle driving, characterized in that, Includes the following steps: (1) Construct a stochastic policy agent; (1-1) Implement the learning and training process of the agent under the Actor-Critic framework. The output of the agent is the probability distribution of the execution data of the steering wheel angle, throttle control force, and brake control force parameters. (1-2) Initialize the agent decision network parameters This refers to the Actor network, where all node values are sampled using a Gaussian distribution with a mean of 0 and a standard deviation of 0.
01. The policy model... Vehicle observation information data Under the given conditions, the output results constitute the Gaussian probability distribution of the relevant parameters for the decision-making vehicle control parameters. and ,in and These represent the mean and covariance matrix of the Gaussian model of the decision-making action, namely the mean of the parameters of steering wheel angle, throttle control force, and brake control force, and the matrix elements that constitute the covariance matrix. The interaction process obtains the executed actions through sampling. The input to the environment causes a state transition, where These represent the steering wheel angle, accelerator control force, and brake control force values respectively during actual driving. (1-3) Initialize the agent to evaluate network parameters This refers to the Critic network, where all node values are sampled using a Gaussian distribution with a mean of 0 and a standard deviation of 0.
01. (2) The agent generates a Markov real training dataset by directly interacting with the real environment, and generates the original constructed training dataset by using the expert experience decision dataset in the training environment. Step (2) specifically includes: (2-1) combining the relevant information acquired by the agent within a single time step into a Markov real training dataset through a real driving interaction process. ,in: This represents the driving environment state observed by the intelligent agent; This means that the intelligent agent obtains the probability distribution of vehicle control parameters through a decision network and selects the appropriate action by sampling. This refers to the reward feedback that the intelligent agent obtains directly from the driving environment according to the rules set; Indicates the driving environment state at the next time step; (2-2) Generate the original constructed training dataset through the expert experience decision dataset in the training environment, which is the driver's historical decision data; (3) For the original constructed training dataset obtained in (2), calculate the expert approval value by the decision behavior similarity comparison method, and determine whether each decision result is an expert-approved decision result; (4) Modify the original construction training dataset based on the expert approval decision results, recalculate the reward return based on the expert approval value, and finally obtain the expert incentive training dataset. This expert incentive training dataset is used together with the real training dataset to update the agent network. (5) Repeat steps (2)-(4) until the agent converges.
2. The agent reinforcement learning method for controlling vehicle driving as described in claim 1, characterized in that, The specific steps (2-2) include: (2-2-1) Obtaining expert experience decision-making dataset Its data entries Indicated for environmental conditions Results of expert driver experience-based decision-making actions based on experience analysis ,in Steering wheel angle value by expert Expert throttle control value Composition of expert braking control force values ; (2-2-2) Utilizing expert experience to make decisions using datasets The set of states in Initialize the environment, and sequentially set the values within it. As the observed state input, the agent's policy model The probability distribution of actions is obtained. ,in Indicates action The Gaussian probability distribution of is given by: Where A is the set of actions consisting of steering wheel angle value, accelerator control force value, and brake control force value. (2-2-3) Selecting agent decision-making actions through sampling Interact with the environment and record its reward feedback. Next environmental state Action probability density function and the corresponding expert driver experience decision-making results , forming the original training data entries Generate the original constructed training dataset .
3. The agent reinforcement learning method for controlling vehicle driving as described in claim 2, characterized in that, Step (3) specifically includes: traversing sequentially Data entries in Based on the judgment results, expert incentive information is introduced into the training data.
4. The agent reinforcement learning method for controlling vehicle driving as described in claim 3, characterized in that, Step (3) specifically includes: (3-1) Actions are made by the agent probability density function Results of expert driver experience-based decision-making Calculate the probability: This probability value Considered as an expert-approved value; (3-2) Define the expert approval threshold based on human experience. ; (3-3) If If the decision is consistent with expert experience, it is recognized by the supervising expert and receives expert incentive feedback.
5. The agent reinforcement learning method for controlling vehicle driving as described in claim 4, characterized in that, The specific steps (3-2-3) include: (3-3-1) Calculate expert incentive feedback ,in This is the expert incentive control coefficient; (3-3-2) Update data entries for .
6. The agent reinforcement learning method for controlling vehicle driving as described in claim 2, characterized in that, Step (4) specifically includes: (4-1) Traverse the original constructed training data entries Generate new expert-incentive data training entries; (4-2) Form a new expert incentive training dataset by using expert incentive training data entries. ; (4-3) The real training dataset and the expert-incentivized training dataset together constitute the total training dataset. According to the Actor-Critic framework, through Update agent Critic evaluation network parameters ; (4-4) Update the Actor decision network parameters through the agent Critic according to the Actor-Critic framework. .
7. The agent reinforcement learning method for controlling vehicle driving as described in claim 6, characterized in that, The specific steps (4-1) include: (4-1-1) Calculate the new expert reward payout ; (4-1-2) Modify the original constructed training data entries to expert-incentivized training data entries. .
8. An intelligent agent reinforcement learning device for controlling vehicle driving, characterized in that: It includes at least one processor and a memory, which are connected via a data bus. The memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to complete the agent reinforcement learning method for controlling vehicle driving as described in any one of claims 1-7.