Automatic driving reinforcement learning method based on automatic manual guidance

Abstract

The invention discloses an automatic driving reinforcement learning method based on automatic manual guidance, and relates to the field of reinforcement learning methods of automatic driving technologies. The problems that an existing reinforcement learning method is low in efficiency and low in model accuracy are solved by constructing an efficient manual guidance framework and developing a corresponding automatic manual guidance algorithm. The method comprises the steps that a complex road scene, a large-scale traffic training environment and a dense traffic testing environment are established; setting an automatic manual guidance algorithm starting condition and an automatic manual guidance action; constructing a deep reinforcement learning network, and establishing an automatic manual guidance A2C algorithm model; and circularly simulating and training the action probability model, and testing a model training effect in a dense traffic flow test environment after training is ended. According to the method, the training time of reinforcement learning can be shortened, the workload of a human driver during monitoring and operating an automatic driving vehicle is reduced, and a model with human driving style characteristics is trained.

Description

Technical Field

[0001] The present invention proposes an autonomous driving reinforcement learning method based on automatic manual guidance, which relates to the field of reinforcement learning methods of autonomous driving technology. Background Art

[0002] Self-driving cars are gradually becoming a key technology to improve the efficiency and quality of human life. They can perform personnel transportation, cargo delivery and various driving tasks. In the research and development of self-driving technology, reinforcement learning, as a core method of machine learning, significantly improves the action control ability of intelligent agents in complex environments through an exploratory trial-and-error mechanism. In particular, reinforcement learning combined with deep neural networks has become a research hotspot in the field of self-driving due to its advantages in handling high-dimensional environmental state representation and generalization capabilities.

[0003] Reinforcement learning algorithms face some challenges in practical applications. First, such algorithms usually require a lot of interactions with the environment to explore and learn effective strategies, which leads to low learning efficiency. Second, in order to achieve optimization and control in complex tasks, reinforcement learning algorithms often need to be carefully designed and utilize a lot of computing resources, which limits their feasibility in practical applications. Therefore, a new algorithm is urgently needed to improve the learning efficiency of reinforcement learning algorithms and the safety and reliability of self-driving cars in practical applications. Summary of the invention

[0004] In order to solve the above problems, the present invention aims to propose an automatic manual guidance-based reinforcement learning method for autonomous driving, which solves the efficiency and accuracy problems in the prior art by integrating the automatic manual guidance mechanism. The core of the present invention is to build an efficient manual guidance framework and develop a corresponding automatic manual guidance algorithm. The design goal of the framework and algorithm is to reduce the burden of monitoring and intervention of human operators in complex traffic environments, and significantly improve the performance of autonomous driving systems in decision-making and operation.

[0005] According to the present invention, an automatic driving reinforcement learning method based on automatic manual guidance includes the following specific related steps:

[0006] Step S1: Establishing a complex road scene with roads with different numbers, as well as a large-scale traffic training environment and a dense traffic test environment containing multiple lanes and intersections;

[0007] Step S2: setting the conditions for starting the automatic manual guidance algorithm and the automatic manual guidance action;

[0008] Step S3: construct a deep reinforcement learning network and establish an automatic human-guided A2C algorithm model;

[0009] Step S4: loop simulation and train the action probability model, and after the training is completed, test the training effect of the automatic manual guidance model in a dense traffic flow test environment with multiple lanes and intersections.

[0010] Step S1 specifically includes the following contents:

[0011] Step S11: creating a complex road scene of an intersection without traffic lights to increase the complexity of the driving trajectories of traffic participants and intelligent agents;

[0012] Step S12: construct a large-scale traffic training environment, where traffic participants include cars, bicycles and pedestrians. Each traffic participant sets a driving trajectory with its starting point and end point located on different numbered roads. The behavior patterns of cars, bicycles and pedestrians are defined so that they can make autonomous decisions in the absence of traffic lights. There is at least one traffic participant at the starting position of each lane to simulate real traffic density.

[0013] Step S13: construct a dense traffic test environment, increase the number of traffic participants in the dynamic traffic flow, make the number of traffic flows in the test environment greater than the number of traffic flows in the training scenario, and simulate dense traffic flows.

[0014] Step S2 specifically includes the following contents:

[0015] Step S21: Actions acting on the agent during training Using the action probability distribution output by the neural network , The expression is as follows:

[0016]

[0017] The neural network adjusts the policy function To output the action, the policy function takes into account the current state and related parameters. To determine at the current moment Which action to take? To promote exploration and avoid local optimal solutions, the neural network sends a policy Add noise to the output of , is sampled from a predefined normal distribution. Add to based on current state and parameterized action outputs to generate exploratory actions , the intensity of exploration is determined by the training-dependent variable control, Adjust as your training progresses. As training progresses, the neural network gradually reduces the variables The value of The magnitude of the strategy Able to execute more stably, The real number space in which the variable Definition, its dimension determines the size of the action space. During the entire training process, the system continuously updates the strategy and variables , so that the action is balanced between exploration and exploitation;

[0018] Step S22: Automatic manual guidance of actions when a human participant decides to participate in a reinforcement learning training cycle Will replace , The expression is as follows:

[0019]

[0020] in Representation and Action The identity matrix of the same spatial dimensions, represents the demonstration mask, which is used to control whether to adopt the action provided by humans;

[0021] Step S23: Steering action in It is limited to a certain angle range, the unit is degree, the angle limit range is as follows:

[0022]

[0023] Torque action in It is limited to a certain amplitude range, the unit is Newton meter, the amplitude limit range is as follows:

[0024]

[0025] Step S24: The agent starts from the starting point On the way to the destination , there is a reference trajectory , the trajectory is a third-order Bezier curve whose control points and parameters define the precise shape and direction of the trajectory. The coordinates of the time are ,in is the horizontal coordinate, is the longitudinal coordinate in meters, and each point of the trajectory is calculated and Euclidean distance , find the minimum Euclidean distance , the expression is as follows:

[0026]

[0027] correspond The minimum trajectory coordinate point is ;

[0028] Step S25: Setting and The target distance between , when the conditions are met , When , the manual guidance algorithm is started, where Calibrate the deviation parameters for lateral motion, , Calibrate the deviation parameters for longitudinal motion, , according to the agent at the current moment The coordinates of the minimum trajectory coordinate point The vector relationship between Use in case of ,exist Choose the appropriate and , let the agent approach the reference trajectory to achieve artificial guidance action Better action probability distribution than neural network output effect.

[0029] Step S3 specifically includes the following contents:

[0030] Step S31: Establish the Actor_Human and Critic_Human neural network architecture, where the input of the Actor_Human network is the state , state space is a 16-dimensional space vector, and the output is the probability distribution of the action , action space is a 2-dimensional vector, and As the input of the Critic_Human network, to estimate the state of the agent at the next moment Q value;

[0031] Step S32: Setting is the simulation time step , setting a smaller simulation step size allows artificial guidance of the action For precise control, each step in the training process will generate a new data according to the current moment and Computing Reinforcement Learning Rewards , when a new data is generated during training, the new data Stored in the experience buffer, the new data will be saved to the first position of the experience buffer. The formula is as follows:

[0032]

[0033] The definitions of each sub-reward are as follows: For speed reward, r vx ∈ [ − 1 , 1 ] , For torque rewards, r T ∈ [ − 1 , 1 ] , Rewards for lane keeping, r keep ∈ [ − 1 , 1 ] , To redirect the reward, r delta ∈ [ − 1 , 1 ] , For collision rewards, r collision ∈ [ − 1 , 1 ] , It is a reward for manual guidance. When the manual guidance flag is triggered, A positive small-scale incentive, r H ∈ [ 0 , 0 . 2 ] ;

[0034] Step S33: Establishing advantage estimation function , Discount factor , is a hyperparameter , during initialization , as the training progresses, The value of gradually decreases to reduce dependence on future rewards and improve the exploration efficiency of the algorithm. Is to determine the next state of the intelligent agent Is it the flag bit of the state triggering the termination condition? If the termination condition is triggered, , start a new round of training, otherwise , the formula is as follows:

[0035]

[0036] in is the time difference function, and the formula is as follows:

[0037]

[0038] Step S34: Calculate the loss function. The policy gradient loss function is , the formula is as follows:

[0039] Loss policy =  [ − A π ( s t , a ) ⋅ log π ( a t RL | s t ; θ ) + Loss imitation ]

[0040] The value loss function is , the formula is as follows:

[0041] Loss value =  [ ( R ( s t ) − V ( s t ) ) 2 ]

[0042] The entropy loss is , the formula is as follows:

[0043] Loss entropy =  [ log π ( a t RL | s t ) ]

[0044] The imitation loss is , the formula is as follows:

[0045]

[0046] The total loss is , the formula is as follows:

[0047]

[0048] Step S35: When the number of training rounds reaches the maximum number of rounds, the training process ends. is the optimal strategy, and then Return to step S32 and continue training until The training ends when convergence occurs.

[0049] Step S4 specifically includes the following contents:

[0050] Step S41: When the state triggers the termination condition, when the lateral and longitudinal positions of the vehicle are less than or equal to 1 meter from the end point, the current model parameters are saved. After convergence, deploy the test environment, which includes multi-lane dynamic traffic simulation, generating virtual vehicles with random lane changes, acceleration and deceleration, intersection conflict scenarios, complex road conditions with pedestrians crossing illegally and high-density traffic flow, setting a continuous traffic queue with a distance of less than or equal to 5 meters, and testing the automatic manual guidance model in a dense traffic flow test environment, which can better detect the effect of model training;

[0051] Step S42: During the test, the flag of the manual guidance algorithm needs to be turned off to test the accuracy of the model. The following comparative experiment is performed under the same test scenario to compare the action selection and whether the end point can be reached when the manual guidance algorithm is turned on and off during the training process as evaluation criteria to complete the test of the automatic manual guidance model.

[0052] The beneficial effects of the present invention are:

[0053] 1. By introducing an automatic manual guidance mechanism, the training efficiency of the autonomous driving system is improved. Compared with the traditional reinforcement learning method, the present invention uses the prior knowledge and experience of human experts to provide an optimized learning path for the intelligent agent. This guidance reduces the blindness of the intelligent agent in exploring the environment, speeds up the learning process, and enables the autonomous driving system to master driving skills in complex traffic environments more quickly. In this way, the present invention not only shortens the training time, but also improves the resource utilization efficiency in the learning process.

[0054] 2. The automatic manual guidance algorithm can analyze the surrounding environment status information in real time and automatically modify the vehicle's actions according to the status, avoiding the need for manual intervention by the driver, reducing the workload of human drivers in monitoring and operating autonomous vehicles, and realizing a model with human driving style characteristics. The system can autonomously handle complex driving tasks and more effectively maintain the stability of the vehicle control system and the ability to handle complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Flowchart of the reinforcement learning method for automated human-guided autonomous driving;

[0056] Figure 2 A flowchart of the neural network autonomous driving model training process;

[0057] Figure 3 This is the strategy loss change curve of the neural network training process;

[0058] Figure 4 It is the torque change comparison curve of the model test;

[0059] Figure 5 Steering wheel angle change comparison curve for model testing;

[0060] Figure 6 This is the displacement comparison curve of the model test. DETAILED DESCRIPTION

[0061] In order to further understand the content of the present invention, the present invention is described in detail with reference to the accompanying drawings and embodiments. It should be understood that the embodiments are merely for explaining the present invention but not for limiting it.

[0062] The specific implementation steps are as follows:

[0063] Step S1: Establishing a complex road scene with roads with different numbers, as well as a large-scale traffic training environment and a dense traffic test environment containing multiple lanes and intersections;

[0064] Step S2: setting the conditions for starting the automatic manual guidance algorithm and the automatic manual guidance action;

[0065] Step S3: construct a deep reinforcement learning network and establish an automatic human-guided A2C algorithm model;

[0066] Step S4: loop simulation and train the action probability model, and after the training is completed, test the training effect of the automatic manual guidance model in a dense traffic flow test environment with multiple lanes and intersections.

[0067] In step S1, if Figure 1 As shown in step S1, in order to obtain the required traffic training environment, the following steps need to be performed:

[0068] Step S11: creating a complex road scene of an intersection without traffic lights to increase the complexity of the driving trajectories of traffic participants and intelligent agents;

[0069] Step S12: construct a large-scale traffic training environment, where traffic participants include cars, bicycles and pedestrians. Each traffic participant sets a driving trajectory with its starting point and end point located on different numbered roads. The behavior patterns of cars, bicycles and pedestrians are defined so that they can make autonomous decisions in the absence of traffic lights. There is at least one traffic participant at the starting position of each lane to simulate real traffic density.

[0070] Step S13: construct a dense traffic test environment, increase the number of traffic participants in the dynamic traffic flow, make the number of traffic flows in the test environment greater than the number of traffic flows in the training scenario, and simulate dense traffic flows.

[0071] In step S2, if Figure 1 As shown in step S2, setting the conditions for starting the automatic manual guidance algorithm and the automatic manual guidance action specifically includes the following steps:

[0072] Step S21: Actions acting on the agent during training Using the action probability distribution output by the neural network , The expression is as follows:

[0073]

[0074] The neural network adjusts the policy function To output the action, the policy function takes into account the current state and related parameters. To determine at the current moment Which action to take? To promote exploration and avoid local optimal solutions, the neural network sends a policy Add noise to the output of , is sampled from a predefined normal distribution. Add to based on current state and parameterized action outputs to generate exploratory actions , the intensity of exploration is determined by the training-dependent variable control, Adjust as your training progresses. As training progresses, the neural network gradually reduces the variables The value of The magnitude of the strategy Able to execute more stably, The real number space in which the variable Definition, its dimension determines the size of the action space. During the entire training process, the system continuously updates the strategy and variables , so that the action is balanced between exploration and exploitation;

[0075] Step S22: Automatic manual guidance of actions when a human participant decides to participate in a reinforcement learning training cycle Will replace , The expression is as follows:

[0076]

[0077] in Representation and Action The identity matrix of the same spatial dimensions, represents the demonstration mask, which is used to control whether to adopt the action provided by humans;

[0078] Step S23: Steering action in It is limited to a certain angle range, the unit is degree, the angle limit range is as follows:

[0079]

[0080] Torque action in It is limited to a certain amplitude range, the unit is Newton meter, the amplitude limit range is as follows:

[0081]

[0082] Step S24: The agent starts from the starting point On the way to the destination , there is a reference trajectory , the trajectory is a third-order Bezier curve whose control points and parameters define the precise shape and direction of the trajectory. The coordinates of the time are ,in is the horizontal coordinate, is the longitudinal coordinate in meters, and each point of the trajectory is calculated and Euclidean distance , find the minimum Euclidean distance , the expression is as follows:

[0083]

[0084] correspond The minimum trajectory coordinate point is ;

[0085] Step S25: Figure 2 As shown, set and The target distance between , when the conditions are met , When , the manual guidance algorithm is started, where Calibrate the deviation parameters for lateral motion, , Calibrate the deviation parameters for longitudinal motion, , according to the agent at the current moment The coordinates of the minimum trajectory coordinate point The vector relationship between Use in case of ,exist Choose the appropriate and , let the agent approach the reference trajectory to achieve artificial guidance action Better action probability distribution than neural network output effect.

[0086] In step S3, if Figure 1 As shown in step S3, a deep reinforcement learning network is constructed to establish an automatic manual guidance A2C algorithm model, which specifically includes the following steps:

[0087] Step S31: Establish the Actor_Human and Critic_Human neural network architecture, where the input of the Actor_Human network is the state , state space is a 16-dimensional space vector, and the output is the probability distribution of the action , action space is a 2-dimensional vector, and As the input of the Critic_Human network, to estimate the state of the agent at the next moment Q value;

[0088] Step S32: Figure 2 As shown, set is the simulation time step , setting a smaller simulation step size allows artificial guidance of the action For precise control, each step in the training process will generate a new data according to the current moment and Computing Reinforcement Learning Rewards , when a new data is generated during training, the new data Stored in the experience buffer, the new data will be saved to the first position of the experience buffer. The formula is as follows:

[0089]

[0090] The definitions of each sub-reward are as follows: For speed reward, r vx ∈ [ − 1 , 1 ] , For torque rewards, r T ∈ [ − 1 , 1 ] , Rewards for lane keeping, r keep ∈ [ − 1 , 1 ] , To redirect the reward, r delta ∈ [ − 1 , 1 ] , For collision rewards, r collision ∈ [ − 1 , 1 ] , It is a reward for manual guidance. When the manual guidance flag is triggered, A positive small-scale incentive, r H ∈ [ 0 , 0 . 2 ] ;

[0091] Step S33: Establishing advantage estimation function , Discount factor , is a hyperparameter , during initialization , as the training progresses, The value of gradually decreases to reduce dependence on future rewards and improve the exploration efficiency of the algorithm. Is to determine the next state of the intelligent agent Is it the flag bit of the state triggering the termination condition? If the termination condition is triggered, , start a new round of training, otherwise , the formula is as follows:

[0092]

[0093] in is the time difference function, and the formula is as follows:

[0094]

[0095] Step S34: Figure 2 As shown, the loss function needs to be calculated to measure the difference between the model prediction and the actual result. In the above example, we need to calculate the policy gradient loss, value loss, entropy loss, and imitation loss, where the policy gradient loss function is: , the policy loss uses the advantage estimation function To measure the difference between the action probability distribution output by the Actor_Human network and the actual optimal strategy, the formula is as follows:

[0096] Loss policy =  [ − A π ( s t , a ) ⋅ log π ( a t RL | s t ; θ ) + Loss imitation ]

[0097] The value loss function is , which is used to measure the difference between the state estimated by the Critic_Human network, the value of the action and the actual return. The formula is as follows:

[0098] Loss value =  [ ( R ( s t ) − V ( s t ) ) 2 ]

[0099] The entropy loss is , so that the strategy has diversity, the formula is as follows:

[0100] Loss entropy =  [ log π ( a t RL | s t ) ]

[0101] The imitation loss is ,make Alternative , To simulate the loss weight, used to determine imitate degree, among which , the formula is as follows:

[0102]

[0103] The total loss is , the formula is as follows:

[0104]

[0105] Step S35: When the number of training rounds reaches the maximum number of rounds, the training process ends. is the optimal strategy, and then Return to step S32 and continue training until The training ends when convergence occurs, e.g. Figure 3 As shown, the loss of the manually guided policy converges faster than the loss of the reinforcement learning policy without manual guidance.

[0106] In step S4, Figure 1 As shown in step S4, the action probability model is simulated and trained in a loop. After the training is completed, the training effect of the automatic manual guidance model is tested in a dense traffic flow test environment with multiple lanes and intersections. Specifically, the steps include:

[0107] Step S41: When the state triggers the termination condition, when the lateral and longitudinal positions of the vehicle are less than or equal to 1 meter from the end point, the current model parameters are saved. After convergence, deploy the test environment, which includes multi-lane dynamic traffic simulation, generating virtual vehicles with random lane changes, acceleration and deceleration, intersection conflict scenarios, complex road conditions with pedestrians crossing illegally and high-density traffic flow, setting a continuous traffic queue with a distance of less than or equal to 5 meters, and testing the automatic manual guidance model in a dense traffic flow test environment, which can better detect the effect of model training;

[0108] Step S42: During the test, the manual guidance algorithm flag needs to be turned off to test the accuracy of the model. The following comparative experiment is performed in the same test scenario to compare the action selection and whether the end point can be reached when the manual guidance algorithm is turned on and off during the training process as the evaluation criteria to complete the test of the automatic manual guidance model. The test process results are compared as follows: Figure 4 Figure 5 Figure 6 As shown, the manually guided model actions are smoother and can reach the end point than the reinforcement learning model actions without manual guidance.

Claims

1. A reinforcement learning method for autonomous driving based on automatic manual guidance, characterized in that: The method comprises the following steps: Step S1: Establishing a complex road scene with roads with different numbers, as well as a large-scale traffic training environment and a dense traffic test environment containing multiple lanes and intersections; Step S2: setting the conditions for starting the automatic manual guidance algorithm and the automatic manual guidance action; Step S3: construct a deep reinforcement learning network and establish an automatic human-guided A2C algorithm model; Step S4: loop simulation and train the action probability model, and after the training is completed, test the training effect of the automatic manual guidance model in a dense traffic flow test environment with multiple lanes and intersections.

2. The automatic driving reinforcement learning method based on automatic manual guidance according to claim 1, characterized in that: The complex road scene, large-scale traffic training environment and dense traffic test environment in step S1 include the following steps: Step S11: creating a complex road scene of an intersection without traffic lights to increase the complexity of the driving trajectories of traffic participants and intelligent agents; Step S12: construct a large-scale traffic training environment, where traffic participants include cars, bicycles and pedestrians. Each traffic participant sets a driving trajectory with its starting point and end point located on different numbered roads. The behavior patterns of cars, bicycles and pedestrians are defined so that they can make autonomous decisions in the absence of traffic lights. There is at least one traffic participant at the starting position of each lane to simulate real traffic density. Step S13: construct a dense traffic test environment, increase the number of traffic participants in the dynamic traffic flow, make the number of traffic flows in the test environment greater than the number of traffic flows in the training scenario, and simulate dense traffic flows.

3. The automatic driving reinforcement learning method based on automatic manual guidance according to claim 1, characterized in that: The setting of starting the automatic manual guidance algorithm conditions and the automatic manual guidance actions in step S2 includes the following steps: Step S21: Actions acting on the agent during training Using the action probability distribution output by the neural network , The expression is as follows: The neural network adjusts the policy function To output the action, the policy function takes into account the current state and related parameters. To determine at the current moment Which action to take? To promote exploration and avoid local optimal solutions, the neural network sends a policy Add noise to the output of , is sampled from a predefined normal distribution. Add to based on current state and parameterized action outputs to generate exploratory actions , the intensity of exploration is determined by the training-dependent variable control, Adjust as your training progresses. As training progresses, the neural network gradually reduces the variables The value of The magnitude of the strategy Able to execute more stably, The real number space in which the variable Definition, its dimension determines the size of the action space. During the entire training process, the system continuously updates the strategy and variables , so that the action is balanced between exploration and exploitation; Step S22: Automatic manual guidance of actions when a human participant decides to participate in a reinforcement learning training cycle Will replace , The expression is as follows: in Representation and Action The identity matrix of the same spatial dimensions, represents the demonstration mask, which is used to control whether to adopt the action provided by humans; Step S23: Steering action in It is limited to a certain angle range, the unit is degree, the angle limit range is as follows: Torque action in It is limited to a certain amplitude range, the unit is Newton meter, the amplitude limit range is as follows: Step S24: The agent starts from the starting point On the way to the destination , there is a reference trajectory , the trajectory is a third-order Bezier curve whose control points and parameters define the precise shape and direction of the trajectory. The coordinates of the time are ,in is the horizontal coordinate, is the longitudinal coordinate in meters, and each point of the trajectory is calculated and Euclidean distance , find the minimum Euclidean distance , the expression is as follows: correspond The minimum trajectory coordinate point is ; Step S25: Setting and The target distance between , when the conditions are met , When , the manual guidance algorithm is started, where Calibrate the deviation parameters for lateral motion, , Calibrate the deviation parameters for longitudinal motion, , according to the agent at the current moment The coordinates of the minimum trajectory coordinate point The vector relationship between Use in case of ,exist Choose the appropriate and , let the agent approach the reference trajectory to achieve artificial guidance action Better action probability distribution than neural network output effect.

4. The method for automatic driving reinforcement learning based on automatic manual guidance according to claim 1, characterized in that: The step S3 of constructing a deep reinforcement learning network and establishing an automatic manual guidance A2C algorithm model includes the following steps: Step S31: Establish the Actor_Human and Critic_Human neural network architecture, where the input of the Actor_Human network is the state , state space is a 16-dimensional space vector, and the output is the probability distribution of the action , action space is a 2-dimensional vector, and As the input of the Critic_Human network, to estimate the state of the agent at the next moment Q value; Step S32: Setting is the simulation time step , setting a smaller simulation step size allows artificial guidance of the action For precise control, each step in the training process will generate a new data according to the current moment and Computing Reinforcement Learning Rewards , when a new data is generated during training, the new data Stored in the experience buffer, the new data will be saved to the first position of the experience buffer. The formula is as follows: The definitions of each sub-reward are as follows: For speed reward, , For torque rewards, , Rewards for lane keeping, , To redirect the reward, , For collision rewards, , It is a reward for manual guidance. When the manual guidance flag is triggered, A positive small-scale incentive, ; Step S33: Establishing the advantage estimation function , Discount factor , is a hyperparameter , during initialization , as the training progresses, The value of gradually decreases to reduce dependence on future rewards and improve the exploration efficiency of the algorithm. Is to determine the next state of the intelligent agent Is it the flag bit of the state triggering the termination condition? If the termination condition is triggered, , start a new round of training, otherwise , the formula is as follows: in is the time difference function, and the formula is as follows: Step S34: Calculate the loss function. The policy gradient loss function is , the formula is as follows: The value loss function is , the formula is as follows: The entropy loss is , the formula is as follows: The imitation loss is , the formula is as follows: The total loss is , the formula is as follows: Step S35: When the number of training rounds reaches the maximum number of rounds, the training process ends. is the optimal strategy, and then Return to step S32 and continue training until The training ends when convergence occurs.

5. The method for automatic driving reinforcement learning based on automatic manual guidance according to claim 1, characterized in that: In step S4, the action probability model is simulated and trained in a loop, and after the training is completed, the training effect of the automatic manual guidance model is tested in a dense traffic flow test environment with multiple lanes and intersections, including the following steps: Step S41: When the state triggers the termination condition, when the lateral and longitudinal positions of the vehicle are less than or equal to 1 meter from the end point, the current model parameters are saved. After convergence, deploy the test environment, which includes multi-lane dynamic traffic simulation, generating virtual vehicles with random lane changes, acceleration and deceleration, intersection conflict scenarios, complex road conditions with pedestrians crossing illegally and high-density traffic flow, setting a continuous traffic queue with a distance of less than or equal to 5 meters, and testing the automatic manual guidance model in a dense traffic flow test environment, which can better detect the effect of model training; Step S42: During the test, the flag of the manual guidance algorithm needs to be turned off to test the accuracy of the model. The following comparative experiment is performed under the same test scenario to compare the action selection and whether the end point can be reached when the manual guidance algorithm is turned on and off during the training process as evaluation criteria to complete the test of the automatic manual guidance model.

Images