An intelligent driving decision learning method based on reinforcement learning

Abstract

The application provides an intelligent driving decision learning method based on reinforcement learning, the gate output is used for shielding irrelevant continuous parameters, the overall error of training is reduced, the interaction relationship between different parameters is considered in the whole model, first, the gate output discrete parameters, then the corresponding continuous parameter network is activated, the correlation degree of the parameters is more accurately increased, the traditional exploration method uses random exploration, and a special exploration parameter needs to be set, the model directly explores according to the value output by the gate network, and the exploration and future benefits can be well balanced. The application can quickly output mixed parameters, the model can respond to the change of the environment in time, when the parameters of the model need to be migrated, only part of the network model parameters need to be changed, and the whole network does not need to be retrained.

Description

TECHNICAL FIELD

[0001] The present application relates to a reinforcement learning-based intelligent driving decision learning method. BACKGROUND

[0002] Decision plays an important role in the development of human history, and has a significant influence on the historical process of mankind. In the historical story, we see the role of military strategist and counselor, which is actually the role of decision-making. However, with the development of science and technology, part of the decision-making process has been taken over by machines, such as the frequently discussed Go, intelligent driving, and recommendation system, which have been decided by artificial intelligence algorithms.

[0003] Traditional decision-making methods require the environment to be static during the decision-making process, and the decision-making action will not affect the change of the environment. However, the environment in the real world is basically changing, such as Go, StarCraft, and autonomous driving. When an action is made, the environment will change accordingly. The previous decision for a specific environment is no longer meaningful. Reinforcement learning algorithm only needs the agent to interact with the environment continuously after defining the reward function, and according to the given reward value, it can find the relatively optimal algorithm. It is very suitable for dynamic environment and is currently the most popular artificial intelligence algorithm in decision-making algorithm.

[0004] In the reinforcement learning decision-making process, when the strategy is complex, it is easy to have a mixed action space, which has both discrete parameter selection and continuous parameter selection. For example, in a soccer robot, a character needs to be selected to pass the ball, which is a discrete parameter. If the size of the force of the foot when passing the ball is selected, it is a continuous parameter. Current reinforcement learning algorithms are basically only for pure discrete actions or pure contact actions, and rarely involve mixed actions.

[0005] The mixed expert system is a relatively popular model at present, which mainly uses the idea of ensemble learning method, uses a gating network to specialize the sub-network, reduces the coupling relationship between single model parameters, and improves the performance under complex tasks and speeds up the operation.

[0006] The current mainstream mixed action space output technical scheme first inputs the state into a model to output all parameters, and then inputs the state and all output parameters into another network to output all future returns. The discrete parameter with the maximum return and the corresponding continuous parameter will be output as the output of this mixed action. This model mainly uses the actor-critic algorithm model structure, and combines DDPG (deep deterministic policy gradient) and DQN (Deep Q-Network) to handle mixed parameter problems.

[0007] The first paper "Deep Reinforcement Learning in Parametrized Action Space" uses a model to output continuous parameters and discrete parameters from state simultaneously, and then concatenates the two with state to output the expected return value of all parameters through another model.

[0008] The second paper "Parametrized Deep Q-Networks Learning: Reinforcement Learning with Discrete-Continuous Hybrid Action Space" uses a model to output all continuous parameters from state, and then combines state with continuous parameters to output the future return value of each discrete action through another model, which is called PDQN for short.

[0009] The third paper "Hybrid actor-critic reinforcement learning in parameterized action space" believes that outputting actions by two networks in series is too slow, and the training time will be greatly lengthened, so it changes to output continuous parameters and discrete parameters by two networks respectively, reducing the time of model output parameters, which is called HPPO for short.

[0010] The most traditional method of mixed parameters is to segment the continuous action parameters into discrete space, and then use discrete action algorithms to solve mixed parameters. Another approach is to map all discrete actions to continuous action space through some transformation algorithms, and then use continuous action parameter algorithms. However, these methods have many problems, such as losing the relationship between parameters when directly continuous or discrete, and also losing the original parameter structure. The first paper borrows the actor-critic algorithm structure in DDPG algorithm, which first uses a network to output continuous parameters and discrete parameters simultaneously, and then concatenates all parameters to input into another network to output the expected return value. This model simply treats all parameters equally, and there is no interaction between parameters, resulting in low accuracy of the output parameters of the model.

[0011] To solve this problem, the second paper believes that the model output process is a sequential process, and when humans mix parameters, they will first estimate the size of each continuous parameter, and then determine the optimal discrete action based on all continuous parameters. Therefore, the model will first output all continuous parameters through a network, and then input all continuous parameters into another network to output the expected return value of each action, and the action with the maximum expected return value is the optimal action. This model considers the interaction between parameters, but due to its serial structure design, the parameter output rate of the model is slow.

[0012] The third paper mainly solves the slow parameter problem of the second paper, and outputs continuous parameters and discrete parameters separately from two networks to speed up the parameter output rate of the model, but directly outputting them will lack the interaction between parameters. Therefore, the paper will pass through a model sharing layer to extract shared representations before outputting the mixed parameters.

[0013] The third paper still outputs all continuous parameters from a network when outputting model parameters, which still causes the problem of parameter oscillation back and forth when updating the model. The present application uses a Mixture-of-Experts (MoE) structure to output continuous parameters under each discrete parameter using an independent network, avoiding interference between different parameters, and only outputting the continuous parameters of one network during testing to speed up the parameter output rate of the model. SUMMARY

[0014] The technical problem to be solved by the present application is how to quickly output suitable parameters from mixed parameters. By calculating the discrete action under the state, the corresponding continuous parameter network is started without the need to output all parameters, which can reduce the time of network parameter output and also consider the interaction between continuous parameters and discrete parameters. At the same time, when migrating model parameters, only part of the network needs to be modified, without the need to retrain the entire network, greatly improving the efficiency of migration.

[0015] The present application specifically provides an intelligent driving decision learning method based on reinforcement learning, comprising the following steps:

[0016] Step 1, building a mixed parameter algorithm neural network actor based on an expert gating network,

[0017] Step 2, merging the current road information received by the vehicle-mounted camera and the sensors on both sides of the vehicle body as the current state information into the mixed parameter algorithm neural network actor;

[0018] Step 3, controlling the steering wheel, accelerator, and brake of the vehicle to execute the mixed parameters (a dis,acon) t , where a dis a represents the discrete action parameters of the output. con This represents the continuous action parameters output, and a reward rating (r) is given based on the quality of the driving control in this instance. t And after a certain time (usually 200ms), the vehicle receives the next step of the onboard camera and the current road information received by the sensors on both sides of the vehicle. t+1 ;

[0019] Step 4, transfer the current vehicle's interaction data [s] t ,(a dis ,a con ) t ,r t ,s t+1 The samples are stored in the training sample pool buffer. Training begins when the number of samples in the buffer exceeds the amount of data required for one training session (Batch).

[0020] Step 5: Use the trained model for intelligent driving.

[0021] In step 1, the hybrid parameter algorithm neural network actor includes a gate network (gate) and four expert networks (expert). The state information (state) is input into the gate network (gate), and the output parameter discrete action (a) is generated. dis There are four values ​​representing the probability values ​​of four discrete actions: left turn, hold, right turn, and brake. The state information includes image information captured by the vehicle's camera, distance information received by the vehicle's radar sensors, and the current state data of the steering wheel, accelerator, and brake collected by the vehicle's sensors. Since only one of the four actions can actually be executed, one of the four action probability values ​​is set to 1, and the others are set to 0, such as a. dis = (0, 1, 0, 0) indicates that the holding action is performed in the discrete parameter left turn, hold, right turn, and brake actions;

[0022] In step 1, there are four expert networks, corresponding to four discrete actions. The input of each expert network is the state, and the output is the continuous parameters under the corresponding discrete action, denoted as a. con .

[0023] In step 1, if a dis If the expression is (1, 0, 0, 0), then the first expert network will output the continuous action parameters a. con = (a, b), representing an angle of a (e.g., 30°) and a velocity of b (e.g., 60°); if a dis If the expression is (0, 1, 0, 0), then the second expert network (expert) will output parameter a. con= c, indicating the speed is c (such as 55); if a dis = (0, 1, 0, 0), the fourth expert network outputs the continuous action parameter a con = d, indicating the brake force is d (such as 0.7); the above angle value range is 0-60 (unit: degree, the maximum turning angle is generally 60 degrees), the speed range is 0-100 (unit: km / h), and the brake force is 0-1; here, two representations are mixed, there will be four expert networks, but for simplicity, only some examples are given, and the complete corresponding values are:

[0024] The output (1, 0, 0, 0) indicates the first action;

[0025] The output (0, 1, 0, 0) indicates the second action;

[0026] The output (0, 0, 1, 0) indicates the third action;

[0027] The output (0, 0, 0, 1) indicates the fourth action;

[0028] In step 1, after the model outputs the discrete parameters and the continuous parameters, an additional network critic is needed to evaluate the revenue effect of the current action, the input is state and a con , and the output is 1 value, indicating the good or bad of the control action parameter on the driving experience under the current vehicle state, the range is -1-1, such as the output is 1, indicating that the current action output probability of the model should be increased, if the output is -1, indicating that the action output probability of the current model needs to be reduced.

[0029] In step 4, the model training process includes the following steps:

[0030] Step 4-1, setting the model training parameters;

[0031] Step 4-2, splicing to get the mixed parameters;

[0032] Step 4-3, splicing state s t+1_batch and (a dis_batch , a con_batch ) t+1 , input the spliced data into the critic network to get the future expected revenue Q t+1 at t+1;

[0033] Step 4-4, splicing state s t_batch and (a dis_batch , a con_batch ) t , input the spliced data into the critic network to get the future expected revenue Q t at t, and Qt +γr batch With Q t+1 The gap between Q critic , where r batch represents the loss function loss dis_batch , a con_batch ) t The evaluation of the driving experience after the mixed action is updated using loss critic in the opposite direction to update the parameters of the critic network;

[0034] Step 4-5, the state s t_batch After passing through the same gate network gate and the corresponding expert network as step 4-2, the mixed action parameters (a dis_batch ,a con_batch ) t are obtained. t_batch The state s dis_batch is spliced with (a con_batch ,a t )' and input into the critic network, and the loss function loss actor is obtained by adding the outputs of all critic networks. actor After using -loss t_batch for backpropagation, the parameters of the gate network gate and all expert networks are updated.

[0035] Step 4-6, deploy the trained mixed parameter algorithm neural network actor model in the vehicle to verify the performance of the model, and artificially evaluate the driving experience of the model on a road.

[0036] Step 4-7, determine whether the mixed parameter algorithm neural network actor model improves the driving experience, if it improves, return to step 4-2 for further iteration training, otherwise execute step 4-8.

[0037] Step 4-8, end the training model.

[0038] Step 4-1 includes: setting the Batch size of the training, generally set to 128, the discount factor γ is generally set to 0.99, and a Batch of training data [s t_batch ,(a dis_batch ,a con_batch ) t ,r batch ,s t+1_batch ] is taken from the collected training data buffer for training, where s t_batch represents the state of the car at time t, (a dis_batch ,a con_batch ) t represents the discrete parameters and continuous parameters output by the network at time t, and rbatch representing the car adopting (a dis_batch ,a con_batch ) t the obtained income, s t+1_batch representing adopting the mixed parameter (a dis_batch ,a con_batch ) t the state of the car after.

[0039] Step 4-2 comprises: collecting the common state s t+1_ Through the gate network gate, the Gumbel Sofmax operation is performed on the output of the gate network gate, and the Gumbel Sofmax operation can adopt a discrete action according to the probability of the gate output, and output (a dis_batch ) t+1 is a one-hot representation, (a dis_batch ) t+1 representing whether the next state is steering, keeping or braking; (a dis_batch ) t+1 The output (1, 0, 0, 0) represents the first action; the output (0, 1, 0, 0) represents the second action; the output (0, 0, 1, 0) represents the third action; and the output (0, 0, 0, 1) represents the fourth action;

[0040] The state s t+1_batch According to the value of (a dis_batch ) t+1 The output continuous parameter is (a con_batch ) t+1 , (a con_batch ) t+1 representing the vehicle speed, angle and braking force controlled by the next state, (a dis_batch ) t+1 and (a con_batch ) t+1 are spliced to obtain the mixed parameter (a dis_batch ,a con_batch ) t+1 .

[0041] Beneficial effects: from the technical point of view, the present application can reduce the overall error of training by shielding irrelevant continuous parameters through the output of the gate, and the whole model considers the interaction between different parameters, first outputs discrete parameters by the gate, and then activates the corresponding continuous parameter network, which more accurately increases the correlation of parameters, and the traditional exploration method uses random exploration, which needs to set special exploration parameters, the model directly explores according to the value output by the gate network, and can well balance exploration and future income.

[0042] From the application level, the application can quickly output the mixed parameters, the model can respond to the changes of the environment in time, and when the parameters of the model need to be migrated, only part of the network model parameters need to be changed, and the entire network does not need to be retrained. BRIEF DESCRIPTION OF DRAWINGS

[0043] The above and / or other aspects of the present application will become more apparent by describing in detail the preferred embodiments thereof with reference to the attached drawings.

[0044] Figure 1 It is a mixed parameter algorithm model diagram based on an expert gate network.

[0045] Figure 2 It is a mixed parameter algorithm collection training data flow diagram based on an expert gate network.

[0046] Figure 3 It is a model training process diagram.

[0047] Figure 4 It is an automatic driving decision process diagram. DETAILED DESCRIPTION

[0048] The application provides an intelligent driving decision learning method based on reinforcement learning, and the specific structure of the provided model comprises a gate network gate and a plurality of expert networks, the specific number of the expert networks is determined by the number of discrete parameters, and the structure of the model is as shown in Figure 1 .

[0049] The mixed parameter algorithm collection training data flow based on the expert gate network is as shown in Figure 2 .

[0050] Step 101, initialize the parameters of the gate network, all expert networks and the parameters of the critic network, set the training buffer size L.

[0051] Step 102, input the current state s t of the environment into the model network, first input s t into the gate network, then perform sofmax on the output of the gate network, perform one-hot representation on the output according to the probability of sofmax, and finally input the state into the corresponding expert network according to the one-hot value, splice the discrete parameter a dis and the continuous parameter a con , and output (a dis , a con )

[0052] Step 103, execute (a disa con ), receive the reward r t and the next state s t+1 of the environment.

[0053] Step 104, store [s t , (a dis , a con ) t , r t , s t+1 ] in the training buffer, and start training when the data meeting the batch size is collected.

[0054] The whole training process of the model is as follows Figure 3 .

[0055] Step 201, take out a batch of training data [s t_batch , (a dis_batch , a con_batch ) t , r batch , s t+1_batch ] from the collected data buffer for training.

[0056] Step 202, pass s t+1_batch through the gate network, perform Gumbel Sofmax on the output of the network, perform one-hot representation on the output according to the probability of Gumbel Sofmax, pass the state through all the expert networks at the same time, output all the continuous parameters, then multiply the one-hot value with the expert network, set the irrelevant continuous parameters to 0, and obtain (a dis_batch , a con_batch ) t+1 .

[0057] Step 203, splice the state s t+1_batch and (a dis_batch , a con_batch ) t+1 , input the spliced data into the critic network, and obtain the future expected return Q t+1 at t+1.

[0058] Step 204, splice the state s t_batch and (a dis_batch , a con_batch ) t , input the spliced data into the critic network, and obtain the future expected return Q t at t, and perform mean square difference operation on Q t +γr batch and Q t+1 to obtain losscritic , using loss critic to update the parameters of critic.

[0059] Step 205, input s t_batch to the critic network after the same processing as step 202 to get (a dis_batch , a con_batch ) ' t , and input s t_batch and (a dis_batch , a con_batch ) ' t to the critic network after splicing, and take the average of all values to get loss actor , update the parameters of the gate network and all expert networks after back propagation using -loss actor .

[0060] Step 206, verify the performance of the model in the environment. Here, the cumulative return of the model on a trajectory is evaluated.

[0061] Step 207, determine whether the model performance has improved. If it has, return to step 302 to continue iterative training. Otherwise, perform step 208.

[0062] Step 208, end the training of the model.

[0063] The above decision-making technology can be applied to autonomous driving. For real autonomous driving scenarios, especially urban traffic scenarios, the shape of the road is highly complex, with types of road conditions such as overpasses, crossroads, entrance and exit ramps, etc. Traffic participants are also diverse, including passenger cars, buses, trucks, bicycles, electric vehicles, pedestrians, etc. At the same time, the controller resources on the vehicle are very limited, and the contradiction between energy consumption / cost constraints and high computational burden is more prominent.

[0064] In autonomous driving, vehicles need to complete various necessary basic operations such as speed adjustment, distance maintenance, lane changing, and overtaking. To complete these operations, the vehicle's throttle, brake, and steering wheel need to be controlled. The above operations can be easily abstracted into the following decision sequence Figure 4 , which is very similar to the hybrid parameter decision-making framework in the present application.

[0065] The overall process is as follows:

[0066] Step 1, build Figure 1The hybrid parameter algorithm neural network actor based on the expert gate network includes a gate network and two or more expert networks. The vehicle-mounted camera and the sensors on both sides of the vehicle body receive the current road information as state information input into the hybrid parameter algorithm neural network actor. The gate network is initialized with the state as input and a as output. dis , which is in one-hot form, because only one of the four discrete parameters, left turn, keep, right turn, and brake, is actually selected. a dis =(0, 1, 0, 0) represents the selection of the keep action. There are four expert networks in total, and the actual execution will select a corresponding expert network according to the value of a dis . The input of the expert network is state, and the output is a con . If a dis =0, the first expert network outputs the parameter a con , where a con =(30, 60) represents an angle of 30 and a speed of 60 (km / h). If a dis =1, the parameter output by the second expert network is a con =55, which represents a speed of 55, where the angle value ranges from 0 to 60 (units of degrees, and the maximum turning angle is generally 60 degrees), the speed ranges from 0 to 100, and the brake force ranges from 0 to 1 (if the brake is fully pressed, the brake force at this time is 1). The Critic network needs to be additionally set to evaluate the driving effect of the current action, with state, a con , and a dis as input and 1 value as output, representing the good or bad of the current action for vehicle driving, with a range of -1 to 1. The size L of the training sample pool buffer is set.

[0067] Step 2: The image information collected by the vehicle-mounted camera, the distance information received by the vehicle radar sensor, and the current state data of the steering wheel, accelerator, and brake collected by the vehicle sensor are combined as s t (i.e., the state at time t) input into the hybrid parameter algorithm neural network actor. First, s t is input into the gate network gate, and then the Gumbel Sofmax operation is performed on the output of the gate network gate. The Gumbel Sofmax operation normalizes the output, and the probability sum of all actions is 1. According to the probability value, one of the four actions is taken, which is a dis , representing whether to turn, keep, or brake at the current time. According to the value of a dis , the state is input into a disThe output of one corresponding expert network is a con , representing the steering angle, speed, brake force that needs to be output, and the discrete parameter a dis and the continuous parameter a con are spliced and output (a dis , a con );

[0068] Step 3, control the steering wheel, accelerator, brake of the vehicle to execute the network output parameters (a dis , a con ), and give a reward evaluation r t according to the good or bad of this driving control, and the current road information s t+1 received by the vehicle-mounted camera and the sensors on both sides of the vehicle body.

[0069] Step 4, store the interaction data [s t , (a dis , a con ) t , r t , s t+1 ] of the current vehicle in the training sample pool buffer, and start training when the collected Batch data meet the training requirements, otherwise the model collection process is executed.

[0070] Step 4-1, set the Batch size of training as B and the discount factor as γ, and take out a B training data [s t_batch , (a dis_batch , a con_batch ) t , r batch , s t+1_batch ] from the collected data buffer for training.

[0071] Step 4-2, pass the collected vehicle-mounted camera and sensors on both sides of the vehicle body s t+1_batch through the gate network, perform Gumbel Sofmax on the output of the network, and perform one-hot representation (a dis_batch ) t+1 on the output according to the probability of Gumbel Sofmax, representing whether to turn, keep or brake in the next state. At the same time, pass s t+1_batch through all the expert networks to output all the continuous parameters (a con_batch ) t+1 , representing the speed, angle, brake force of the vehicle in the next state control. Then multiply the one-hot value with the expert network, set the irrelevant continuous parameters to 0, and get (a dis_batch , a con_batch ) t+1 .

[0072] Step 4-3, set state s t+1_batch With (a) dis_batch ,a con_batch ) t+1 The concatenated data is then input into the CRT network to obtain the expected future return Q at time t+1. t+1 .

[0073] Step 4-4, change state s t_batch With (a) dis_batch ,a con_batch ) t The concatenated data is then input into the CRT network to obtain the expected future return Q at time t. t Q t +γr batch With Q t+1 The loss is obtained by performing mean square error calculation. critic Using loss critic Perform reverse propagation to update the parameters of the ctrtic.

[0074] Steps 4-5, s t_batch After the same processing as in step 4-2, we obtain (a) dis_batch ,a con_batch )′ t , will state s t_batch With (a) dis_batch ,a con_batch )′ t After concatenation, the data is input into the critic network, and the loss is obtained by averaging all the values. actor Using -loss actor After backpropagation, update the parameters of the gate network and all expert networks.

[0075] Steps 4-6 involve validating and evaluating the model's performance in an environment. This involves evaluating the model's cumulative reward over a single trajectory.

[0076] Step 4-7: Determine if the model performance has improved. If it has improved, return to step 4-2 to continue iterative training; otherwise, proceed to step 4-8.

[0077] Steps 4-8 complete the model training.

[0078] Step 5, intelligent driving using the trained model. Due to the high cost of actual vehicles, this embodiment uses an AWS DeepRacer Evo race car, which contains a 400 million pixel camera module, two scanning LiDAR radar sensors, and receives data every 200 ms. The image size in the data is 240*240, the data size of a LiDAR sensor is 86, and the current state data size of the steering wheel, accelerator, and brake collected by the vehicle sensor is 3. Therefore, the size of the input state state is 50387. The control system can control the speed size, steering angle, and brake force of the current race car, where the speed size is 0-4 m / s, the steering angle is 0-30°, and the brake force is 0-1, which can fit the actual vehicle control process. The reward function is set to -1 if it hits the yellow line, runs out of the edge line, or hits the obstacle, 1 if it is in the middle of the yellow line and the white line, and other rewards are divided into -1-1 according to the distance between the vehicle and the edge line. A complete data processing process is as follows:

[0079] (1) The data collected by the camera module and the radar sensor is merged and transmitted to the actor network as state

[0080] (2) First, the state is transmitted to the gate network of the actor, and one of the four actions of left turn, keep, right turn, and brake is output;

[0081] (3) According to the action output by the gate, the state is transmitted to the corresponding expert network, and the continuous parameters of steering angle, speed, and brake force are output;

[0082] (4) The vehicle executes the above mixed parameters, and gives the reward according to the driving situation after 200 ms of execution.

[0083] After hundreds of episodes are performed according to the above manner, a relatively good strategy can be obtained, the probability of obtaining a reward of +1 is 92.15%, and the vehicle can drive at the maximum speed at the straight line and automatically slow down to turn the corner at the corner. Compared with the previous method, the effect is obviously better. When the discrete continuous parameters are used and the classic DQN algorithm is used for training, due to too many parameters, too large action space and too large exploration space, the training is slow, and after 8000 episodes of training, the vehicle still runs off the runway at the straight line. After training by using the P-DQN method, the convergence is started after 2000 episodes, and the probability of the final reward being +1 is 35.14%. By using the algorithm of the embodiment of the application, the convergence is started after about 500 episodes, and the probability of the reward being +1 is 92.15%. Compared with the previous algorithm, the algorithm of the application is more stable, and the convergence speed is relatively faster. It can be seen that compared with the previous method, the application can obviously improve the result on the problem of mixed parameter action space.

[0084] In the specific implementation, the application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium can store a computer program, and the computer program can run the invention content of the intelligent driving decision learning method based on reinforcement learning and part or all steps in each embodiment of the application when executed by the data processing unit. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), or the like.

[0085] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the application can be realized by means of a computer program and a corresponding general hardware platform. Based on such understanding, the technical solutions in the embodiments of the application can be embodied in the form of a computer program, i.e., a software product, which can be stored in a storage medium and includes a plurality of instructions for causing a device (which can be a personal computer, a server, a single-chip microcomputer, a MCU, or a network device) including a data processing unit to execute the method described in each embodiment or some parts of the embodiments of the application.

[0086] The application provides an intelligent driving decision learning method based on reinforcement learning. There are many methods and ways to implement the technical solutions, and the above description is only the preferred embodiment of the application. It should be pointed out that, for ordinary skilled persons in the technical field, some improvements and refinements can be made without departing from the principles of the application, and these improvements and refinements should be regarded as the protection scope of the application. The components not explicitly described in the embodiments can be implemented by using the existing technology.

Claims

1. A method for learning intelligent driving decisions based on reinforcement learning, characterized in that, The method comprises the following steps: Step 1, building a hybrid parameter algorithm neural network actor based on an expert gating network; Step 2, merging the current road information received by the vehicle camera and the sensors on both sides of the vehicle body as the current state information input into the hybrid parameter algorithm neural network actor; Step 3, control the vehicle's steering wheel, accelerator, and brakes at the execution time t, and output the mixed parameters (a) of the network. dis ,a con ) t , where a dis a represents the discrete action parameters of the output. con This represents the continuous action parameters output, and a reward rating (r) is given based on the quality of the driving control in this instance. t And after a certain period of time, the vehicle receives the next step from the onboard camera and the sensors on both sides of the vehicle to receive the current road information. t+1 ; Step 4, the interaction data [s t ,(a dis ,a con ) t ,r t ,s t+1 ] of the current vehicle is stored in a training sample pool buffer, and when the number of samples in the buffer is greater than the data amount Batch required for one training, the training is started; Step 5, using the trained model for intelligent driving; In step 1, the mixed parameter algorithm neural network actor includes a gating network gate and four expert networks expert, and state information state is input into the gating network gate to output a parameter discrete action a dis There are four, representing the probability values of four discrete actions left turn, keep, right turn and brake; the state information state includes image information collected by a vehicle-mounted camera, distance information received by a vehicle radar sensor, and state data of a steering wheel, an accelerator and a brake collected by a vehicle-mounted sensor at the current moment; In step 1, the expert network experts share four, corresponding to four discrete actions, and the input of each expert network expert is state, and the output is the continuous parameter under the corresponding discrete action, denoted as a con ; In step 1, if a dis = (1, 0, 0, 0), the first expert network expert outputs the continuous action parameter a con = (a, b) indicating an angle of a and a velocity of b; if a dis = (0, 1, 0, 0), the second expert network expert outputs the parameter a con = c indicating a velocity of c; if a dis = (0, 0, 0, 1), the fourth expert network expert outputs the continuous action parameter a con = d indicating a braking force of d; In step 1, after the model outputs discrete parameters and continuous parameters, an additional network critic is needed to evaluate the revenue effect of the current action, the input is state and a con , and the output is 1 value, which represents the good or bad of the control action parameter on the driving experience under the current vehicle state, and the range is -1~1, such as output 1, which means that the current action output probability of the model should be increased, and if the output is -1, it means that the action output probability of the current model needs to be reduced; In step 4, the model training process comprises the following steps: Step 4-1, setting the model training parameters; Step 4-2, splicing to obtain hybrid parameters; Step 4-3, state s t+1_batch and (a dis_batch , a con_batch ) t+1 Concatenation, input the concatenated data into the critic network to obtain the future expected return Q t+1 at time t+1 Step 4-4, update the state s t_batch with (a dis_batch ,a con_batch ) t Concatenate, input the concatenated data into the critic network to obtain the future expected return Q t at time t t +γr batch The gap between Q t+1 and Q critic is taken as the loss function loss batch , where r dis_batch represents the evaluation of the driving experience after taking (a con_batch ,a t Mixed action; use loss critic to update the parameters of the critic network in the opposite direction; Step 4-5, input state s t_batch After passing through the same gating network gate and corresponding expert networks as step 4-2, the mixed action parameters (a dis_batch ,a con_batch )′ t are obtained t_batch After splicing state s dis_batch and (a con_batch ,a t )′ t , the critic network is input, and the loss function loss actor is obtained by adding the outputs of all critic networks. After using-loss actor for back propagation, the parameters of the gating network gate and all expert networks are updated; Step 4-6, deploying the trained hybrid parameter algorithm neural network actor model on the vehicle to verify the performance of the model and evaluate the driving experience of the model on a section of road; Step 4-7, determining whether the hybrid parameter algorithm neural network actor model improves the driving experience, if it improves, returning to step 4-2 for iterative training, otherwise executing step 4-8; Step 4-8, ending the model training; Step 4-1 includes: setting the Batch size of training, the discount factor γ, taking out a Batch size of training data from the collected training data buffer [s t_batch , (a dis_batch , a con_batch ) t , r batch , s t+1_batch ] for training, wherein s t_batch represents the state of the car at time t, (a dis_batch , a con_batch ) t represents the discrete parameters and continuous parameters output by the network at time t, r batch represents the income obtained by the car adopting (a dis_batch , a con_batch ) t , and s t+1_batch represents the state of the car after adopting the mixed parameters (a dis_batch , a con_batch ) t ; Step 4-2 includes: collecting the common state s of the vehicle-mounted camera and the sensors on both sides of the vehicle body in the training data t+1_batch Through the gating network gate, the Gumbel Sofmax operation is performed on the output of the gating network gate, and the Gumbel Sofmax operation can take a discrete action according to the probability of the gate output, and output (a dis_batch ) t+1 is a one-hot representation, (a dis_batch ) t+1 represents whether the next state is steering, keeping or braking; State s t+1_batch According to (a dis_batch ) t+1 The output continuous parameters are (a con_batch ) t+1 , (a con_batch ) t+1 , which represent the vehicle speed, angle, and brake force of the next state control. The mixed parameters (a dis_batch ) t+1 and (a con_batch ) t+1 are spliced to obtain the next state (a dis_batch ,a con_batch ) t+1 .

Images