A vehicle automatic tracking driving method based on deep reinforcement learning
Through deep reinforcement learning methods, a dual deep Q network and usage experience playback solution was built, combined with the RMSprop algorithm, the problems of low tracking and driving efficiency and high data labeling cost of autonomous driving are solved, and efficient and accurate automatic tracking and driving are achieved. Suitable for complex road environments.
Patent Information
- Application Number
- CN202110954229.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-08-19
- Publication Date
- 2025-01-10
- Estimated Expiration
- 2041-08-19
AI Technical Summary
The existing autonomous vehicle tracking driving methods are inefficient and require a large amount of data labeling, which leads to high costs and is difficult to effectively comply with traffic laws and regulations in complex road environments, which may lead to traffic order disorder and safety accidents.
Using a method based on deep reinforcement learning, a dual deep Q network is built for image feature extraction, an experience replay scheme is used for environmental interaction training, and a network training is optimized with the RMSprop algorithm to realize end-to-end prediction from image to vehicle control, avoiding traditional methods Complex PID control and large amounts of data annotations in it.
It realizes efficient automatic tracking driving, reduces data labeling costs, improves the universality and applicability of the system, reduces the maximum deviation in a single deep reinforcement learning network, and improves training efficiency and vehicle control accuracy.
Smart Images
Figure CN113657292B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence and autonomous driving technology, and in particular relates to a vehicle automatic tracking driving method based on deep reinforcement learning. Background Art
[0002] In recent years, autonomous driving has developed rapidly, and traffic will enter a special situation where autonomous vehicles and traditional vehicles will merge in the foreseeable future. Therefore, how to make autonomous vehicles better comply with existing traffic regulations is an increasingly prominent traffic issue.
[0003] The present invention mainly focuses on the tracking driving of autonomous driving vehicles. If this problem cannot be solved, it may cause traffic disorder and even cause traffic safety accidents.
[0004] Research on automatic vehicle tracking driving at home and abroad currently mainly involves the field of supervised learning. The research purpose is to enable the vehicle to reproduce the operational control actions of the human driver for the input real-time image. This method involves complex road extraction, centerline prediction, PID parameter integration, and vehicle state machine design. The use of deep reinforcement learning can achieve end-to-end control from vehicle camera input to steering servo.
[0005] Deep reinforcement learning is the product of the combination of reinforcement learning and deep learning. It integrates the powerful understanding ability of deep learning in perceptual problems such as vision and the decision-making ability of reinforcement learning, realizing end-to-end learning.
[0006] At present, deep reinforcement learning has been put into use and can solve many complex problems in real-life scenarios. The most famous example is AlphaGo, which uses deep reinforcement learning to defeat the world Go champion Lee Sedol. In addition, it is also widely used in robot control, real-time battle games, etc.
[0007] Existing vehicle automatic tracking driving methods are mostly based on multiple links such as lane line extraction, driving area prediction and vehicle decision-making control. The method based on deep reinforcement learning realizes end-to-end control from input image to steering servo, breaking the control ideas of existing methods; and traditional artificial intelligence methods also involve a large amount of data labeling work. Faced with complex road traffic environments, labeling work inevitably brings huge time and cost consumption. Summary of the invention
[0008] The present invention aims to provide a vehicle automatic tracking driving method based on deep reinforcement learning to solve the problem that traditional methods are too complicated and inefficient and the cost problem caused by a large amount of data labeling work.
[0009] In order to solve the above technical problems, the specific technical solutions of the present invention are as follows:
[0010] A vehicle automatic tracking driving method based on deep reinforcement learning specifically comprises the following steps:
[0011] Step 1: Construct a dual deep Q network for image feature extraction;
[0012] Step 2: Use the experience replay scheme to collect environment interaction training samples and perform dual network training;
[0013] Step 3: Use the RMSprop algorithm to optimize the network training process;
[0014] Step 4: Use the trained dual-depth Q network for prediction to achieve automatic tracking driving.
[0015] Furthermore, in step 1:
[0016] The input pixel size of the dual deep Q network accepts an RGB color image of size 224*224 as the input of the network, that is, the structure of the input data is 3*224*224;
[0017] The dual deep Q network passes through a fully connected layer to obtain a 19*1*1 network output.
[0018] Furthermore, the first layer of the structure of the input data uses a 7*7 convolution operation, and the obtained feature dimension output is 64*112*112, and then a maximum pooling operation is performed, and the obtained output feature is 64*56*56;
[0019] The second layer of the input data structure performs residual module feature extraction, and the extracted image feature dimension is 64*56*56;
[0020] The third layer of the input data structure performs residual module feature extraction, and the obtained feature dimension output is 128*28*28;
[0021] The fourth layer of the input data structure performs residual module feature extraction, and the obtained feature output dimension is 256*14*14;
[0022] The fifth layer of the input data structure performs residual module feature extraction, and the output dimension obtained is 512*7*7;
[0023] The sixth layer of the input data structure performs an average pooling operation to obtain a final feature vector of 512*1*1.
[0024] Furthermore, in step 2: the experience replay scheme can synchronously generate corresponding training data and obtain training data sequences during the network training process, and each training data sequence is stored in a storage space and the training data sequences are equally likely extracted during the dual network training process to perform dual network training;
[0025] Further, the training data sequence is labeled as (P1, A, P2, R);
[0026] Among them: P1 represents the image obtained by the front camera of the current vehicle, A represents the action performed according to the dual network prediction, P2 represents the image obtained by the front camera of the vehicle at the next moment after the action is performed, and R represents the effect feedback of the vehicle driving environment after the current action is performed.
[0027] Furthermore, in step 2, when performing dual network training: only one of the two networks is updated with weight coefficients each time, and one of them is randomly selected with equal probability as the network to be updated Q π , then the other one is used as an auxiliary update network
[0028] Furthermore, the objective function for updating the parameters of the updated network is:
[0029]
[0030] Among them: α represents the learning rate, γ represents the discount rate, Q π (P1, A) is the value reward output of the network to be updated corresponding to the current picture P1 and the execution action A, R is the immediate benefit of the current action, Indicates that the auxiliary update network is used to adjust the parameters of the network to be updated based on the value reward output obtained by receiving the image P2 at the next moment and the corresponding action, where argmax a (Q π (P2, A)) indicates that the corresponding action that makes the network to be updated obtain the maximum value in the action value sequence for the image P2 is selected as a reference for selecting the auxiliary update network action.
[0031] Furthermore, in step 3, the RMSprop algorithm is used to adaptively adjust the network learning rate during the training and fitting process of the dual network. The objective function of the algorithm during the training process is:
[0032]
[0033] Where: Q π and According to the equal probability selection criterion in the training process, two deep neural networks Q1 and Q2 are represented, r trepresents the immediate benefit of the action at time t, P t+1 represents the received image at time t+1, a is the selected action, and θ is the parameter to be trained in the network.
[0034] Furthermore, in step 3, the RMSprop algorithm is used to adaptively adjust the network learning rate during the training and fitting process of the dual network. The objective function of adaptively adjusting the learning rate during the training process is:
[0035]
[0036] Where: θ t represents the network parameters obtained after the tth adjustment, ρ represents the weight coefficient of the gradient, α represents the initial learning rate, ε represents an arbitrarily small positive number, G t Represents the gradient of the current objective function, δ t Tuning parameters for adaptive learning rate.
[0037] Furthermore, in step 4, when the trained dual-depth Q network is used for real-time image prediction:
[0038] Input the image into the two networks respectively, and obtain the corresponding action value function values output by the two networks;
[0039] The two groups of function value vectors are added together, and the largest vector component is taken as the action output corresponding to the current input image, thereby realizing automatic tracking driving.
[0040] The vehicle automatic tracking driving method based on deep reinforcement learning of the present invention has the following advantages:
[0041] 1. The present invention adopts a deep reinforcement learning method to directly realize the end-to-end network establishment from image acquisition to vehicle control, avoiding the complex PID control link in the traditional method;
[0042] 2. The present invention adopts an improved residual network structure to extract image features, taking into account the balance between network complexity and feature effectiveness;
[0043] 3. The present invention adopts the experience playback method, which avoids the process of labeling a large number of data samples in traditional supervised learning, and effectively improves the convenience and versatility of the invention;
[0044] 4. The present invention adopts a dual deep network for action prediction, which can effectively reduce the performance loss caused by the maximization deviation in a single deep reinforcement learning network;
[0045] 5. The present invention adopts the RMSprop algorithm to perform adaptive adjustment of the network learning rate, which can accelerate the convergence speed of the training process and improve the training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 The schematic diagram of the vehicle automatic tracking driving method based on deep reinforcement learning of the present invention;
[0047] Figure 2 A schematic diagram of the neural network structure used in the present invention; DETAILED DESCRIPTION
[0048] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of a vehicle automatic tracking driving method based on deep reinforcement learning of the present invention in conjunction with the accompanying drawings.
[0049] The present invention provides Figure 1-2 A vehicle automatic tracking driving method based on deep reinforcement learning is shown in, wherein: specifically comprising the following steps:
[0050] Step 1: Construct a dual deep Q network for image feature extraction;
[0051] In this embodiment, a dual-depth Q network structure is adopted to construct a deep neural network, taking into account the balance between the complexity of the network and the effectiveness of the output features. The dual-depth Q network is used for the prediction of multidimensional vector numerical regression. The use of the dual-depth Q network solves the problems of gradient diffusion, gradient explosion and network degradation that occur during network training.
[0052] The specific operation of the dual deep Q network is to downsample the input feature data to retain its original low-dimensional features, and to perform multi-core convolution with a step size of two to further extract low-dimensional features, which to a certain extent takes into account the fusion of low-dimensional features and high-dimensional features.
[0053] The input pixel size of the dual deep Q network accepts an RGB color image of size 224*224 as the input of the network, that is, the structure of the input data is 3*224*224;
[0054] The first layer of the input data structure uses a 7*7 convolution operation, and the resulting feature dimension output is 64*112*112, and then a maximum pooling operation is performed, and the resulting output feature is 64*56*56;
[0055] The second layer of the input data structure performs residual module feature extraction, and the extracted image feature dimension is 64*56*56;
[0056] The third layer of the input data structure performs residual module feature extraction, and the obtained feature dimension output is 128*28*28;
[0057] The fourth layer of the input data structure performs residual module feature extraction, and the obtained feature output dimension is 256*14*14;
[0058] The fifth layer of the input data structure performs residual module feature extraction, and the output dimension obtained is 512*7*7;
[0059] The sixth layer of the input data structure performs an average pooling operation to obtain a final feature vector of 512*1*1, and then passes through a fully connected layer to obtain the final required 19*1*1 network output.
[0060] The output of the network represents the expected value return of taking different actions. The structure of the dual network is exactly the same, both using the above network structure, the difference lies in the difference in the adjustment of weight parameters during training.
[0061] Step 2: Use the experience replay scheme to collect environment interaction training samples and perform dual network training;
[0062] The key point of the adopted experience replay training scheme is to automatically generate corresponding training data during the training process.
[0063] The training phase is divided into two steps: obtaining the training data sequence and training the network.
[0064] The training data sequence records the current image P1 captured by the front camera of the vehicle, the action A predicted by the dual network, the image P2 captured by the front camera of the vehicle at the next moment after the action is executed, and the effect feedback R of the vehicle driving environment after the current action is executed. The training data sample sequence is marked as (P1, A, P2, R).
[0065] The process of obtaining training data is to input the image P1 of the front camera of the vehicle into the neural network in training, obtain the immediate network prediction result, and determine the final execution action A according to the prediction result of the network according to the ε-greedy strategy, where ε is the flexibility index set in advance, |A| is the total number of established actions, and π(a|P) is the probability of selecting a certain action based on the current image.
[0066]
[0067] where q π (P|a) represents the action value obtained by the network to be updated based on the current image input and action; Represents the action value obtained by the auxiliary update network based on the current image input and action, and then records the real-time data P2 of the vehicle's front camera at the next moment after the action is executed, as well as the immediate reward R obtained after the current action is executed. Thus, a training data sequence (P1, A, P2, R) is obtained.
[0068] The training data sequence obtained each time is stored in the storage space for use in the subsequent stage of network training.
[0069] During the training process, the experience replay scheme is used to select training data, that is, a training data sequence is randomly selected with equal probability in the storage space for network training. The reason for this is that neural network training requires that each data sample is independent of each other, but the training sample sequences obtained by the above method have a strong correlation, and the experience replay scheme can largely ensure the independence between samples.
[0070] If the agreed termination state occurs during the training process, the vehicle simulation environment is reset to continue collecting the training sample sequence.
[0071] In this embodiment, two deep neural networks with exactly the same structure are initialized, which avoids the maximization deviation when using a single deep neural network to predict action value, and can obtain more accurate prediction results.
[0072] For a selected training sample sequence (P1, A, P2, R), the objective function of the network update is:
[0073]
[0074] Among them: α represents the learning rate, γ represents the discount rate, Q π (P1, A) is the value reward output of the network to be updated corresponding to the current picture P1 and the execution action A, R is the immediate benefit of the current action, Indicates that the auxiliary update network is used to adjust the parameters of the network to be updated based on the value reward output obtained by receiving the image P2 at the next moment and the corresponding action, where argmax a (Q π (P2, A)) indicates that the corresponding action that makes the network to be updated obtain the maximum value in the action value sequence for the image P2 is selected as a reference for selecting the auxiliary update network action.
[0075] When performing dual network training, only one of the two networks is updated at a time, and one of them is randomly selected with equal probability as the network to be updated. π , then the other one is used as an auxiliary update network The selection of the network to be updated is completely random, that is, both networks have a probability of 0.5 to be designated as the network to be updated.
[0076] According to the training sequence randomly selected in the training sample storage space, the parameters of the network to be updated are updated, that is, it is executed with a probability of 0.5, then,
[0077] Q1(P1,A)←Q1(P1,A)+α[R+γQ2(P2,argmax a (Q1(P2, A)))-Q1(P1, A)]
[0078] Among them: α represents the learning rate, γ represents the reward discount rate, and Q(P, A) is the value reward output obtained by the network corresponding to the current picture and the executed action.
[0079] Step 3: Use the RMSprop algorithm to optimize the network training process;
[0080] Specifically, in step 3:
[0081] In this embodiment, the RMSprop algorithm is used to adaptively adjust the network learning rate during the training and fitting process of the dual network. The objective function of the algorithm during the training process is:
[0082]
[0083] Where: Q π and According to the equal probability selection criterion in the training process, two deep neural networks Q1 and Q2 are represented, r t represents the immediate benefit of the action at time t, P t+1 represents the received image at time t+1, a is the selected action, and θ is the parameter to be trained in the network.
[0084] According to the traditional fixed learning rate gradient descent method, it can be expressed as θ t+1 =θ t -aG t ,
[0085] where θ t+1 =θ t -aG t is the gradient of the current objective function. However, this method adopts the RMSprop algorithm to adaptively adjust the network learning rate. The objective function of adaptively adjusting the learning rate during training is:
[0086]
[0087] Where: θ t represents the network parameters obtained after the tth adjustment, ρ represents the weight coefficient of the gradient, α represents the initial learning rate, ε represents an arbitrarily small positive number to ensure that the denominator is not zero, G t Represents the gradient of the current objective function, δ t Tuning parameters for adaptive learning rates.
[0088] Step 4: Use the trained dual-depth network to make predictions and achieve automatic tracking driving.
[0089] When using the trained dual-depth network for real-time image prediction:
[0090] In this embodiment, the image is input into two networks respectively to obtain the corresponding action value function values output by the two networks, the two groups of function value vectors are added together, and the largest vector component is taken as the action output corresponding to the current input image, thereby realizing automatic tracking driving.
[0091] The dual deep neural network performs the final vehicle steering control, and the 19-dimensional output of the two neural networks obtained based on the image input The sum is performed, and the action corresponding to the maximum component of the new vector is selected as the final operation performed by the steering servo.
[0092] Compared with traditional supervised learning, deep reinforcement learning does not require labeled data. The training of the target network is completed during the continuous interaction between the target and the environment. Therefore, the present invention can be applied to complex road environments and has better applicability and higher training efficiency.
[0093] Working principle: This vehicle automatic tracking driving method based on deep reinforcement learning is an end-to-end network prediction between the implementation input from the vehicle's front camera and the value function obtained from the execution action of the vehicle's steering servo, and the corresponding execution action is selected according to the final network prediction result.
[0094] The vehicle's steering gear is controlled by the joint prediction results of the real-time input image data of the road based on the dual deep neural network. The reinforcement learning training concept is adopted in the training process to simulate the autonomous interaction between the vehicle and the environment to generate a training sample sequence, thereby achieving the autonomous realization of vehicle automatic tracking driving from training to application.
[0095] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.
Claims
1. A vehicle automatic tracking driving method based on deep reinforcement learning, characterized in that: The specific steps include: Step 1: Construct a dual deep Q network for image feature extraction; Step 2: Use the experience replay scheme to collect environment interaction training samples and perform dual network training; Step 3: Use the RMSprop algorithm to optimize the network training process; Step 4: Use the trained dual-depth Q network to make predictions, thereby achieving automatic tracking driving; In step 1: The input pixel size of the dual deep Q network accepts an RGB color image of size 224*224 as the input of the network, that is, the structure of the input data is 3*224*224; The dual deep Q network passes through a fully connected layer to obtain a 19*1*1 network output; The first layer of the input data structure uses a 7*7 convolution operation, and the resulting feature dimension output is 64*112*112, and then a maximum pooling operation is performed, and the resulting output feature is 64*56*56; The second layer of the input data structure performs residual module feature extraction, and the extracted image feature dimension is 64*56*56; The third layer of the input data structure performs residual module feature extraction, and the obtained feature dimension output is 128*28*28; The fourth layer of the input data structure performs residual module feature extraction, and the feature output dimension obtained is 256*14*14; The fifth layer of the input data structure performs residual module feature extraction, and the output dimension obtained is 512*7*7; The sixth layer of the input data structure performs an average pooling operation to obtain a final feature vector of 512*1*1; In step 4, when using the trained dual-depth Q network for real-time image prediction: Input the image into the two networks respectively, and obtain the corresponding action value function values output by the two networks; The function value vectors of the two groups are added together, and the largest vector component is taken as the action output corresponding to the current input image, thereby realizing automatic tracking driving; The dual deep neural network performs the final vehicle steering control, and the 19-dimensional output of the two neural networks obtained based on the image input The sum is performed, and the action corresponding to the maximum component of the new vector is selected as the final operation performed by the steering servo.
2. The vehicle automatic tracking driving method based on deep reinforcement learning according to claim 1 is characterized in that: In step 2: the experience replay scheme can synchronously generate corresponding training data and obtain training data sequences during the network training process, and each training data sequence is stored in a storage space and the training data sequence is extracted with equal probability during the dual network training process for dual network training.
3. The vehicle automatic tracking driving method based on deep reinforcement learning according to claim 2 is characterized in that: The training data sequence is labeled as (P1, A, P2, R); Among them: P1 represents the image obtained by the front camera of the current vehicle, A represents the action performed according to the dual network prediction, P2 represents the image obtained by the front camera of the vehicle at the next moment after the action is performed, and R represents the effect feedback of the vehicle driving environment after the current action is performed.
4. The vehicle automatic tracking driving method based on deep reinforcement learning according to claim 1 is characterized in that: In step 2, when performing dual network training: only one of the two networks is updated with weight coefficients each time, and one of them is randomly selected with equal probability as the network to be updated Q π , then the other one is used as an auxiliary update network 5. The vehicle automatic tracking driving method based on deep reinforcement learning according to claim 4 is characterized in that: The objective function for updating the parameters of the network to be updated is: Among them: α represents the learning rate, γ represents the discount rate, Q π (P1, A) is the value reward output of the network to be updated corresponding to the current picture P1 and the execution action A, R is the immediate benefit of the current action, Indicates that the auxiliary update network is used to adjust the parameters of the network to be updated based on the value reward output obtained by receiving the image P2 at the next moment and the corresponding action, where argmax a (Q π (P2, A)) indicates that the corresponding action that makes the network to be updated obtain the maximum value in the action value sequence for the image P2 is selected as a reference for selecting the auxiliary update network action.
6. The vehicle automatic tracking driving method based on deep reinforcement learning according to claim 1 is characterized in that: In step 3, the RMSprop algorithm is used to adaptively adjust the network learning rate during the training and fitting process of the dual network. The objective function of the algorithm during the training process is: Where: Q π and According to the equal probability selection criterion in the training process, two deep neural networks Q1 and Q2 are represented, r t represents the immediate benefit of the action at time t, P t+1 represents the received image at time t+1, a is the selected action, and θ is the parameter to be trained in the network.
7. The vehicle automatic tracking driving method based on deep reinforcement learning according to claim 1 is characterized in that: In step 3, the RMSprop algorithm is used to adaptively adjust the network learning rate during the training and fitting process of the dual network. The objective function of adaptively adjusting the learning rate during the training process is: Where: θ t represents the network parameters obtained after the tth adjustment, ρ represents the weight coefficient of the gradient, α represents the initial learning rate, ε represents an arbitrarily small positive number, G t Represents the gradient of the current objective function, δ t Tuning parameters for adaptive learning rates.