Lane keeping method based on deep reinforcement learning

Through the deep reinforcement learning method, the design and implementation of lane keeping tasks are simplified, the problems of complex and costly rule design in the existing technology are solved, and the stable driving and rapid experiment of small cars in the lane are realized.

CN113844448BActive Publication Date: 2024-11-26GUANGDONG SONGKE INTELLIGENT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111111167.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-18
Publication Date
2024-11-26
Estimated Expiration
2041-09-18

AI Technical Summary

Technical Problem

In the existing intelligent driving technology, the manual design of rules is complex and difficult to cope with environmental variability, resulting in high design and implementation costs, and high hardware equipment complexity and experimental costs, making it difficult to quickly conduct experiments and improvements in lane keeping tasks.

Method used

Adopt a lane keeping method based on deep reinforcement learning, collecting environmental and vehicle status data, using neural networks for self-learning, simplifying lane detection-process-control process, realizing end-to-end command output control, and using ε-greedy strategy and experience cache The pool performs model update until the model converges.

Benefits of technology

It realizes self-learning of lane maintenance tasks for small cars, simplifies traditional processes, reduces design and implementation costs, improves the stable driving ability of vehicles in the lane, and reduces experimental costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113844448B_ABST
    Figure CN113844448B_ABST
Patent Text Reader

Abstract

The present invention discloses a lane keeping method based on deep reinforcement learning, which includes collecting environment-agent state observation values, namely, environment state data and vehicle state data, and pre-processing the data before transferring them into a network model and initializing the network. The network learns the driving task of lane keeping in a reinforcement learning manner based on the interaction data, during which the network is continuously updated until the model converges, and then the optimal strategy output control for task execution is performed based on the converged network. The lane keeping method based on deep reinforcement learning provided by the present invention realizes self-learning of the lane keeping task of a small car, simplifies the traditional lane detection-processing-control process, and realizes end-to-end command output control.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle decision control, and in particular to a lane keeping method based on deep reinforcement learning. Background Art

[0002] As the beginning of the emerging technological revolution and one of the largest-scale application scenarios of artificial intelligence, driverless technology is of great significance to improving urban traffic and environmental friendliness. At the same time, using competition experimental vehicles as prototypes for driverless technology research has greatly reduced experimental costs and testing cycles, and has progressive significance in accelerating the development of related technologies.

[0003] The current intelligent driving / assistance technology solution is still a perception-decision-control pipeline process module. There are still many shortcomings in the decision-making block of this solution: first, the manual design and formulation of rules, although sophisticated, are extremely complex and costly to design and implement; second, the designed rule strategies are generally difficult to cope with the variability of the environment; third, due to the inherent complexity of hardware equipment and the high experimental costs, the expenses for improvements and experiments on actual vehicles are high.

[0004] Therefore, how to provide a lane keeping method based on deep reinforcement learning that is easy to experiment with quickly is an urgent problem that researchers in this technical field need to solve. Summary of the invention

[0005] The purpose of the present invention is to provide a lane keeping method based on deep reinforcement learning, which realizes self-learning of the lane keeping task of a small vehicle, simplifies the traditional lane detection-processing-control process, realizes end-to-end command output control, and can solve the problems raised in the above-mentioned background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] The lane keeping method based on deep reinforcement learning includes the following steps:

[0008] S1: Collecting the state observation values ​​of the environment-agent (Env-Agent), including the environment state data and the vehicle state data;

[0009] S2: Initialize the model network first, then start the interaction. After each interaction, feed the data s into the neural network, get the corresponding output a, and then continue to interact with the environment according to the action selected by the output to get the complete data of one interaction <s, a, s', r>. Then, use this method to interact with the environment multiple times, get multiple interaction data, and store them in the experience buffer pool (Replay Buffer, RB);

[0010] S3: Sample a batch of interaction data from RB and perform a network update; repeat S2 and update the network multiple times until the final model converges. The update formula of the Q network is:

[0011]

[0012] In the formula, s t is the observed value of the environment, a t For t The action selected after feeding into the neural network, Q(s t ,a t ) is in state s t Take action a t The expected return, When the step is t+1, always choose to make Q(s t+1 ,a t+1 )The value corresponding to the largest action a, r t+1 When the observation value s is obtained at step t t Then make an action t The reward obtained, γ is the reward discount factor, between [0,1], and α represents the magnitude of the network update;

[0013] S4: Perform strategy control based on the converged model. The strategy is:

[0014]

[0015] Among them, π * Represents the optimal strategy learned. For any t-th step, under the environmental observation value s, the agent strategy π * The expected return of the action taken is the largest compared to other actions, that is, the optimal action. Such a mapping s→a * That is, an optimal strategy π * .

[0016] Furthermore, S1 specifically includes the following steps:

[0017] Step S101: The environmental state data includes the world coordinates Position (x, y, z) of the vehicle relative to a predefined origin, the vehicle's cross-track error CTE (i.e., the distance between the vehicle and the track center in the lane it is in), and the vehicle's collision relationship Hit (i.e., whether there is a collision).

[0018] Furthermore, the vehicle status data includes the absolute speed Speed ​​of the vehicle, the steering value Steering of the vehicle, the throttle value Throttle of the vehicle, and the camera image data Image returned by the camera in front of the vehicle. The preprocessing process is:

[0019] The camera image data Image returned by the camera directly in front of the vehicle is cropped and standardized. First, the pixels of each image are normalized to the range [0,1], and then the image size is cropped to the set size, here (height=120,width=160). Then, data storage is constructed based on the above vehicle status data, and the data written includes:

[0020]

[0021] Among them, mod represents the current data source mode. When data is collected manually before training, mod=user; when data is collected automatically by the vehicle during training, mod=car; i represents the current data storage id number in a batch of collected data.

[0022] Furthermore, S2 specifically includes the following steps:

[0023] S201: Initialize the model network first, and then start the interaction. The initialization of the model network means that the network obtains a more reasonable initialization weight, reduces unnecessary and invalid exploration, speeds up the convergence of the model, and reduces the training time. The vehicle state data described in step 11 is preprocessed and stored as initialization data to initialize the network. The input image data (120, 160, 3) is flattened after 5 convolution activations (Convolution2D)-drop operations, and then passes through two fully connected activation layers. Finally, a one-dimensional vector of length 2 is output, which represents the Q value corresponding to the output [steering, throttle] action vector. The initial interaction refers to the exploration strategy of the vehicle in the process of interacting with the environment. The ε-greedy strategy is used to interact and explore the environment. According to the probability ε, the action with the largest Q value corresponding to the output action vector is executed. There is a 1-ε probability to randomly execute an action from the action space. The action space is 21 values ​​between [-1,1], and the value interval is 0.1. The actual value mapped to the actual vehicle control / throttle is multiplied by the value range of the corresponding control / throttle. The probability ε is a hyperparameter set by humans. The value is initially set to 0.6. After a given number of training rounds epoch, it continues to increase during the training process and linearly increases to 0.9 to the last round;

[0024] S202: After the interaction starts, at time t (step) the data s t The corresponding returned camera image data is fed into the network model to obtain the corresponding action output, and the action a is obtained after the action is selected according to the ε-greedy strategy t , and then execute the action on the environment to obtain the new data s returned by the environment at the next moment (step)t+1 and the last time we took action a t Rewards received t+1 , and then get a complete interaction data <s t ,a t ,s t+1 ,r t+1 >, stored in the RB experience pool. Then, similarly, the new environmental observation data s t+1 As the next round of t Continue to interact with the environment and obtain new interaction data, that is, the new <s t ,a t ,s t+1 ,r t+1 >, and is stored in RB again. Repeat this process multiple times until the round exit condition is triggered, such as running out of the road, crossing the lane boundary, RB experience pool storage space is full, etc.;

[0025] Furthermore, it is characterized in that S3 specifically includes the following steps:

[0026] S301: Sample a batch of interaction data from RB and perform a network update. The update formula is as described in formula (1). After the update, retain 1 / 4 of the interaction data with the highest reward for the next round of updates. Until the final model converges. The convergence of the model is reflected in the data that the single-step reward and round reward gradually increase and eventually stabilize with slight fluctuations. In practice, it is reflected in the fact that the vehicle can stably stay in the lane and can drive a complete section of the road.

[0027] Furthermore, S4 specifically includes the following steps:

[0028] S401: Carrying out policy control according to the converged model means that the trained model can be loaded onto the vehicle for testing in the training environment without storing interaction data and updating the network; at the same time, the vehicle is tested in a new test environment.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The present invention discloses a lane keeping method based on deep reinforcement learning, including collecting environment-agent state observation values, namely, environment state data and vehicle state data, and pre-processing the data before transferring them into a network model and initializing the network. The network learns the driving task of lane keeping in a reinforcement learning manner based on the interactive data, during which the network is continuously updated until the model converges, and then the optimal strategy output control for task execution is performed based on the converged network. The lane keeping method based on deep reinforcement learning provided by the present invention realizes self-learning of the lane keeping task of a small car, simplifies the traditional lane detection-processing-control process, and realizes end-to-end command output control. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a schematic diagram of a complete circular road without cross intersections;

[0032] Figure 2 This is a schematic diagram of the vehicle lanes on a complete circular road without cross intersections;

[0033] Figure 3 A logical diagram of the lane keeping strategy;

[0034] Figure 4 This is a schematic diagram of the internal calculation process of a neural network;

[0035] Figure 5 Schematic diagram of the neural network training process;

[0036] Figure 6 It is a program module interaction diagram. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] like Figure 1 As shown in the figure, a complete circular road without crossroads in the simulation environment is taken as an example. Assuming that the lane where the vehicle is located is a single lane on the right, the starting position of the vehicle is the starting origin of the world coordinate system, which is also the end position of the vehicle.

[0039] After the vehicle starts from the starting position, it begins to obtain environmental status data, including the world coordinates Position (x, y, z) of the vehicle relative to the predefined origin, the vehicle's cross-track error CTE (that is, the distance between the vehicle and the center of the track in the lane) and the vehicle's collision relationship Hit (that is, whether there is a collision), as well as its own vehicle status data, including the vehicle's absolute speed Speed, the vehicle's steering value Steering, the vehicle's throttle value Throttle and the camera image data Image returned by the camera directly in front of the vehicle, etc. These data are then transmitted to the model training in the core processing unit for use. The model outputs control commands based on the transmitted data, and then outputs control actions.

[0040] The following example illustrates the lane keeping method based on deep reinforcement learning. Assuming the task scenario is as follows Figure 1-6 The figure shows a complete ring road without crossroads. The following steps are included:

[0041] Step 1: Collect the state observations of the environment-agent (Env-Agent), including the environment state data and the vehicle state data, as the input of the network model. At the same time, the data needs to be preprocessed. The specific process of data collection and preprocessing is: the vehicle camera returns the image data, and the camera image data Image returned by the camera directly in front of the vehicle is cropped and standardized. First, the pixels of each image are normalized to the range of [0,1], and then the image size is cropped to the set size, here (height=120,width=160).

[0042] Then, data storage is built based on the above vehicle status data, and the written data is:

[0043]

[0044] Among them, mod represents the current data source mode. When data is collected manually before training, mod=user; when data is collected automatically by the vehicle during training, mod=car; i represents the current data storage id number in a batch of collected data.

[0045] Step 2: The vehicle command control strategy is output by the model network after processing the input data, denoted as π θ , indicating that the strategy π is determined by the network parameter θ. Specifically, the network model represents the Q-value function of the output control action, and the network parameter θ is used to adjust the value corresponding to the update action, thereby adjusting the output strategy π; and the optimal strategy π is obtained * That is, the adjustment parameter θ is continuously updated to the optimal value, that is, the model tends to converge. The data of model convergence shows that the single-step reward and round reward gradually increase and eventually stabilize and fluctuate slightly. In practice, the vehicle can stably stay in the lane and complete a complete road. The Q value is defined as the network in state s t Take action a t The maximum future expected reward that can be obtained. The network input is the camera image data (120, 160, 3) obtained from the vehicle state information data, and the output is the network action vector Q value.

[0046] The neural network structure is as follows Figure 3 As shown in the figure, the input image data (120, 160, 3) is flattened after 5 convolution activations (Convolution2D)-drop operations, and then passes through two fully connected activation layers. Finally, a one-dimensional vector of length 2 is output, representing the Q value corresponding to the output action vector [steering, throttle].

[0047] The specific process of neural network model training is as follows:

[0048] Step 21: Initialization. In order to allow the network to obtain a more reasonable initialization weight, reduce unnecessary invalid exploration, speed up the convergence of the model, and thus reduce the training time, the pre-processed storage data of the vehicle state data is used as the initialization data to update the network and obtain the initial weight parameters.

[0049] Step 22: Start interaction to obtain interaction data. The start of interaction refers to the exploration strategy of the vehicle in the process of interacting with the environment. The ε-greedy strategy is used to interact and explore the environment. The action with the largest Q value corresponding to the output action vector is executed according to the probability ε. There is a 1-ε probability to randomly execute an action from the action space. The action space is 21 values ​​between [-1,1], and the value interval is 0.1. The actual value mapped to the actual vehicle control / throttle is multiplied by the value range of the corresponding control / throttle. After the interaction starts, at time t (step), the data s t The corresponding returned camera image data is fed into the network model to obtain the corresponding action output, and the action a is obtained after the action is selected according to the ε-greedy strategy t , and then execute the action on the environment to obtain the new data s returned by the environment at the next moment (step) t+1 and the last time we took action a t Rewards received t+1 , and then get a complete interaction data <s t ,a t ,s t+1 ,r t+1 >, stored in the RB experience pool. Then, similarly, the new environmental observation data s t+1 As the next round of t Continue to interact with the environment and obtain new interaction data, that is, the new <s t ,a t ,s t+1 ,r t+1 >, and is stored in RB again. This process is repeated many times until the round exit condition is triggered, such as running out of the road, crossing the lane boundary, RB experience pool storage space is full, etc.

[0050] Step 23: Update the network in multiple single rounds. As described in step 22, all the interaction data RB of a round can be obtained. Sample a batch of interaction data from RB and perform a network update. The update formula is as described in formula (1). After the update, retain the 1 / 4 interaction data with the highest reward for the next round of updates. After this round, repeat step 22 and update the network multiple times until the final model converges. The convergence of the model is reflected in the data that the single-step reward and round reward gradually increase and eventually stabilize with small fluctuations. In practice, it is reflected that the vehicle can stably stay in the lane and can drive a complete section of the road.

[0051] Step 24: Optimal policy control test. Policy control based on the converged model means that the trained model can be loaded onto the vehicle for testing in the training environment without storing interaction data and updating the network; at the same time, the vehicle is tested in a new test environment.

[0052] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A lane keeping method based on deep reinforcement learning, characterized in that: The steps include: S1: Collecting environment-agent state observations, including environment state data and vehicle state data. The specific steps include: Step S101: The environmental state data includes the world coordinates Position (x, y, z) of the vehicle relative to a predefined origin, the cross-track error CTE of the vehicle, and the collision relation Hit of the vehicle; The vehicle status data includes the absolute speed of the vehicle, the steering value of the vehicle, the throttle value of the vehicle, and the camera image data returned by the camera in front of the vehicle. The preprocessing process is: The camera image data returned by the camera directly in front of the vehicle Perform cropping and standardization. First, normalize the pixels of each image to the range [0, 1]. Then crop the image to the set size, here (height = 120, width = 160). Then, build a data storage based on the above vehicle status data. The data written is: Among them, mod represents the current data source mode. Before training, when data is collected manually, mod = user. During training, when the vehicle automatically collects data, mod = user. i represents the current data storage id number in a batch of collected data; S2: Initialize the model network first, then start the interaction, use the preprocessed storage data of the vehicle status data described in step S101 as the initialization data to initialize the network, and flatten the input image data (120, 160, 3) after 5 convolution activation-discard operations, and then pass through two fully connected activation layers, and finally output a one-dimensional vector with a length of 2; After the interaction starts, at time t, the data s t The corresponding returned camera image data is fed into the neural network to obtain the corresponding action output, and the action is selected according to the ε-greedy strategy to obtain the action Then execute the action on the environment to obtain the new data s returned by the environment at the next moment. t+1 Take action at the last moment Rewards received t+1 , and then get a complete interaction data Stored in the experience buffer pool, then similarly, the new environmental observation data s t+1 As the next round of t Continue to interact with the environment, obtain new interaction data, and store it in the experience cache pool again. Repeat this process multiple times until the exit condition of the round is triggered; S3: Sample a batch of interaction data from the experience buffer pool and perform a network update; repeat S2 and update the network multiple times until the final model converges. The update formula of the model network is: In the formula, s t is the observed value of the environment, For t The action selected by the result after feeding into the neural network, For state s t Take action The expected return, When the step is t+1, always choose The biggest move The corresponding value, r t+1 When the observation value s is obtained at step t t After taking action The reward obtained, γ is the reward discount factor, between [0,1], and α represents the magnitude of the network update; S4: Perform strategy control based on the converged model. The strategy is: Among them, π * Represents the optimal strategy learned. For any t-th step, under the environmental observation value s, the optimal strategy π * The expected return of the action taken is the largest compared to other actions. That is, an optimal strategy π * .