An obstacle avoidance method for autonomous vehicles based on LSTM-DQN

Abstract

This invention discloses an obstacle avoidance method for autonomous vehicles based on LSTM-DQN, comprising: detecting surrounding environmental information by the autonomous vehicle and inputting the environmental information into an LSTM-DQN model; extracting features from the environmental information using the Long Short-Term Memory (LSTM-DQN) network in the LSTM-DQN model and comparing them with a database to determine the type of obstacle and predict the obstacle; inputting the obstacle prediction result into the DQN part trained in the LSTM-DQN model to provide the optimal obstacle avoidance action, and the autonomous vehicle executing the current optimal obstacle avoidance strategy. This invention can utilize the LSTM model to extract features from the surrounding environment in advance, predict obstacles, and perceive obstacles ahead of time.

Description

Technical Field

[0001] This invention belongs to the field of intelligent logistics, specifically an obstacle avoidance method for unmanned vehicles based on LSTM-DQN. Background Technology

[0002] With the continuous and in-depth development of social informatization, factory logistics workshops are gradually becoming more automated and mechanized, with unmanned vehicles (UAVs) handling goods in the workshops greatly improving labor efficiency and reducing costs. However, this also brings problems such as UAVs colliding with walls, colliding with each other, and crashing. These problems are usually caused by malfunctioning detection sensors on the UAVs or mismatches between the external environment recognition and the database they possess. Improving the obstacle avoidance capabilities and detection accuracy of UAVs and providing solutions can help improve workshop efficiency, reduce maintenance costs, and enhance user trust.

[0003] Improving the obstacle avoidance capabilities of unmanned vehicles (UGVs) in workshops has always been a crucial research area for software engineers. UGV obstacle avoidance typically consists of three stages: detecting moving obstacles, tracking moving obstacles, and obstacle avoidance actions. In obstacle detection, LiDAR-based detection is a hot research topic. Mainstream methods include map difference, entity clustering, and target tracking. Map difference analyzes the distribution of obstacles based on their varying states at different times to obtain motion information. Entity clustering classifies the data detected by radar sensors and then describes and detects their states. Target tracking tracks obstacle trajectories to obtain obstacle motion information. With the development of deep learning, tracking moving obstacles often employs CNN neural networks to identify targets, followed by the MOT algorithm based on Markov Decision Processes (MDPs) to solve for the target's dynamic position. Obstacle avoidance actions mainly involve path planning, and commonly used methods include potential field methods, fuzzy logic methods, neural network methods, occupancy grid methods, spatial search methods, and direct planning methods based on data fusion. Currently, most obstacle avoidance systems in workshops rely on a specific sensor for real-time obstacle avoidance, without utilizing the vehicle's learning capabilities or the contextual information detected by the sensor. This results in low efficiency and certain deficiencies in obstacle avoidance capabilities. Summary of the Invention

[0004] To address the aforementioned technical deficiencies in existing technologies, this invention proposes an obstacle avoidance method for unmanned vehicles based on LSTM-DQN.

[0005] The technical solution to achieve the purpose of this invention is: an obstacle avoidance method for unmanned vehicles based on LSTM-DQN, the specific steps of which are as follows:

[0006] The autonomous vehicle detects information about the surrounding environment and inputs this information into the LSTM-DQN model.

[0007] The Long Short-Term Memory network in the LSTM-DQN model is used to extract features from environmental information and compare them with the database to determine the type of obstacle and complete the prediction of obstacles.

[0008] The obstacle prediction results are input into the DQN part trained in the LSTM-DQN model to provide the optimal obstacle avoidance action, and the autonomous vehicle executes the current optimal obstacle avoidance strategy.

[0009] Preferably, the Long Short-Term Memory (LSTM-DQN) network in the LSTM-DQN model includes a forget gate, a memory gate, and an output gate. The specific process of using the LSTM-DQN model's LSTM-DQN network to extract features from environmental information is as follows:

[0010] The environmental information x input at time t t Compared with the output h of the Long Short-Term Memory network at the previous time step t-1 Combined into a single vector [x] t ,h t-1 The forgetting factor f is obtained by compressing the vector to the (0,1) interval using the sigmoid activation function. t , will long-term memory C t-1 Multiply by the forgetting factor f t Complete C t-1 The forgetting process;

[0011] Multiply the memory factor by the short-term memory at the current time t, and then multiply by C, which represents the completion of the forgetting process. t-1 Adding them together yields a new long-term memory C. t ;

[0012] The output factor O is calculated using a sigmoid activation function. t For new long-term memory C t Using the tanh activation function, the output factor O t The result h is obtained by adding the result to the long-term memory processed using the tanh activation function. t .

[0013] Preferably, the forgetting factor f t The calculation formula is:

[0014] f t =σ(W f ·[h t-1 ,x t ]+b f )

[0015] In the above formula, σ is the sigmoid activation function, and W f b represents the forgetting weight of the forgetting gate. f This is a bias term.

[0016] Preferably, memory factor i t Specifically:

[0017] i t =σ(W i ·[h t-1 ,x t ]+b i )

[0018] In the above formula, σ is the sigmoid activation function, and W i For memory gate memory weights, b i This is a memory factor bias term;

[0019] New short-term memory Specifically:

[0020]

[0021] In the above formula, tanh is the activation function, and W c For memory retrieval weights, b c This is a short-term memory bias term;

[0022] New long-term memory C t Specifically:

[0023]

[0024] In the formula f t C is the forgetting factor. t-1 For long-term memory at time t-1, i t For memory factors, For new short-term memory.

[0025] Preferably, the output factor O t Specifically:

[0026] O t =σ(W o [h t-1 ,x t ]+b o )

[0027] In the formula, σ is the sigmoid activation function, and W o b represents the output weight of the output gate. o This is the output factor bias term.

[0028] Preferably, the training process for the DQN part is as follows:

[0029] 1) Randomly select an initial state of the autonomous vehicle, i.e., the autonomous vehicle position (x0, y0), and adopt ε-greedy action selection based on the initial state of the autonomous vehicle, i.e. select the optimal action under the current strategy with a probability of ε, and select a random action with a probability of 1-ε.

[0030] 2) The unmanned vehicle performs the action in the simulation environment, obtains the artificially designed environmental reward value r and enters a new state (x1, y1), and stores the quadruple (the previous unmanned vehicle position s, the action performed by the unmanned vehicle a, the reward value given by the environment r, and the new position of the unmanned vehicle s') into the experience pool.

[0031] The autonomous vehicle continuously interacts with the environment, storing the quadruple data in the experience pool.

[0032] 3) Once the experience pool is full, randomly sample from the experience pool to update the network parameters;

[0033] 4) Repeat step 3) until the DQN partially converges.

[0034] Compared with the prior art, the significant advantages of this invention are:

[0035] (1) The present invention can use the LSTM model to extract features of the surrounding environment in advance, predict obstacles, and perceive obstacles in advance.

[0036] (2) The present invention uses a trained algorithm model, which enables unmanned vehicles to complete workshop tasks in a fixed workshop environment without using cameras to take pictures, thereby reducing resource consumption.

[0037] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0038] Figure 1 This is a flowchart of an obstacle avoidance method for autonomous vehicles based on LSTM-DQN.

[0039] Figure 2 It is the network structure of the LSTM part in LSTM-DQN. Detailed Implementation

[0040] like Figure 1 As shown, an obstacle avoidance method for autonomous vehicles based on LSTM-DQN includes the following steps:

[0041] Step 1: The unmanned vehicle carries three cameras to detect the surrounding environment and inputs the information into the LSTM-DQN model.

[0042] Step 2: The LSTM-DQN model first uses a Long Short-Term Memory (LSTM) network to extract features from the environment. This LSTM network includes three gates: a forget gate, a remember gate, and an output gate. After the external environment is input, it is processed through each gate, and the extracted features are output and compared with a database to determine the type of obstacle, thus enabling the autonomous vehicle to predict obstacles. The network structure is as follows: Figure 2 As shown in the figure. C in the figure t-1 The long-term memory retained at time t-1 serves to preserve the memory information from time t-1; h t-1 The memory output at time t-1; x t This represents the input information, specifically the surrounding environment information captured by the camera; h t This is the output at the current time t, which is the prediction result for the obstacle.

[0043] 1) Forget Gate. First, the input x at time t is... t and the output h of the previous time step t-1 Combined into a single vector [x t ,h t-1 Then, the vector is compressed to the (0,1) interval using the sigmoid activation function to obtain the forgetting factor f. t This forgetting factor is used to control long-term memory C. t-1 The forgetting of certain memories, and then C t-1 Multiply by the forgetting factor f t Complete C t-1 The forgetting process. Forgetting factor f t The calculation formula is as follows:

[0044] f t =σ(W f ·[h t-1 ,x t ]+b f )

[0045] 2) Memory gates. Used to update long-term memory C. t-1 The memory gate contains two activation functions: the sigmoid activation function, which determines which values ​​to update, and the tanh activation function, which updates values ​​from [x...]. t ,h t-1 New short-term memories are generated in [the process]. The specific method is to first calculate the memory factor i. t Then, the memory factor is multiplied by the short-term memory at the current time t, and then multiplied by C, which represents the completion of the forgetting process. t-1 The update is completed by adding them together, resulting in the new long-term memory C. t i t , and Ct The calculation formula is as follows:

[0046] i t =σ(W i ·[h t-1 ,x t ]+b i )

[0047]

[0048]

[0049] In the formula, b i For the memory gate bias term, b c W is the memory retrieval bias term. C For memory retrieval weights, W i For memory gates, the memory weights are used.

[0050] 3) Output gate. Used to output the environmental features at the current moment, i.e., obstacle prediction. First, the output factor O is calculated using a sigmoid activation function. t Used to determine long-term memory C t Which features need to be output, and then for C t The tanh activation function is used for processing, and the results are finally summed to obtain the output h. t The output factor O t Output h t The calculation formula is as follows:

[0051] O t =σ(W o [h t-1 ,x t ]+b o )

[0052] h t =O t *tanh(C t )

[0053] In the formula, σ is the sigmoid activation function, and W o b represents the output weight of the output gate. o This is a bias term.

[0054] Step 3: Input the obstacle prediction results into the DQN part of the LSTM-DQN model to execute the current optimal obstacle avoidance strategy. For example, if a wall is predicted to appear within 5 meters, the DQN will output a steering action to inform the autonomous vehicle that steering is the best obstacle avoidance action. The autonomous vehicle executes this action to complete obstacle prediction and avoidance. DQN uses a deep neural network to simulate the quality of a certain action taken by the autonomous vehicle in the current state, thereby continuously updating the obstacle avoidance strategy π. Once the model converges, the optimal obstacle avoidance strategy is obtained. The DQN model is trained through a simulation system, and the training process is as follows:

[0055] 1) First, randomly select an initial state for the autonomous vehicle, such as the center of the autonomous vehicle being located at (x0, y0). Based on this state, select an action, such as moving forward at a certain constant speed. The action selection adopts the ε-greedy method, that is, select the optimal action under the current strategy with a probability of ε, and select a random action with a probability of 1-ε.

[0056] 2) After the action selection is completed, the autonomous vehicle executes the action in the simulation environment, then obtains the manually designed environmental reward value r and enters a new state (x1, y1). The quadruple (the previous position of the autonomous vehicle s, the action performed by the autonomous vehicle a, the reward value r given by the environment, and the new position of the autonomous vehicle s') is stored in the experience pool. Then, the autonomous vehicle is continuously allowed to interact with the environment, and the quadruple is stored in the experience pool.

[0057] 3) Once the experience pool is full, random samples are taken from the experience pool to update the network parameters.

[0058] 4) Repeat step 3) until the network converges.

[0059] The trained LSTM-DQN model is loaded onto the autonomous vehicle. The vehicle captures information about the surrounding environment through a camera and then inputs it into the LSTM-DQN model. The model can then output obstacle prediction information and provide the optimal obstacle avoidance action.

[0060] Example

[0061] This embodiment is based on a real-world test of an unmanned vehicle in a workshop. First, an LSTM-DQN model is trained on a physical simulation system for the camera-equipped unmanned vehicle. Then, the trained LSTM-DQN model is loaded into the unmanned vehicle's chip. The training metrics of this model are compared with those of classical algorithms, and the results show that this method has superior performance, as shown in the table below:

[0062]

[0063]

[0064] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

[0065] It should be understood that, in order to simplify the present invention and help those skilled in the art understand its various aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes described in a single embodiment or with reference to a single figure. However, the present invention should not be construed as including all features in the exemplary embodiments as essential technical features of the claims of this patent.

[0066] It should be understood that the modules, units, components, etc., included in the device of one embodiment of the present invention can be adaptively changed to be placed in a device different from that embodiment. Different modules, units, or components included in the device of the embodiment can be combined into a single module, unit, or component, or they can be divided into multiple sub-modules, sub-units, or sub-components.

Claims

1. An obstacle avoidance method for unmanned vehicles based on LSTM-DQN, characterized in that, The specific steps are as follows: An autonomous vehicle detects surrounding environmental information and inputs this information into an LSTM-DQN model. The LSTM-DQN model's Long Short-Term Memory (LSTM-DQN) network includes a forget gate, a memory gate, and an output gate. The specific process of feature extraction from the environmental information using the LSTM-DQN model's LSTM-DQN network is as follows: Will Environmental information input at all times Compared with the output of the Long Short-Term Memory network at the previous time step Integrate into a single vector The forgetting factor is obtained by compressing the vector to the (0,1) interval using the sigmoid activation function. , will long-term memory Multiply by the forgetting factor ,Finish The forgetting process; Compare memory factors with the current moment Multiply by short-term memory, and by the time the forgetting process is completed. Adding them together creates new long-term memories. ; The output factor is calculated using a sigmoid activation function. For new long-term memories use Activation function, output factor With use The result is obtained by multiplying the long-term memory after activation function processing. ; Memory Factors Specifically: In the above formula, It is the sigmoid activation function. For memory gate memory weights, This is a memory factor bias term; New short-term memory Specifically: In the above formula, For activation function, For memory extraction weights, This is a short-term memory bias term; New long-term memory Specifically: In the formula Forgetting factor, for Long-term memory of moments For memory factors, For new short-term memory; The Long Short-Term Memory network in the LSTM-DQN model is used to extract features from environmental information and compare them with the database to determine the type of obstacle and complete the prediction of obstacles. The obstacle prediction results are input into the trained DQN part of the LSTM-DQN model to provide the optimal obstacle avoidance action, and the autonomous vehicle executes the current optimal obstacle avoidance action. The training process for the DQN part is as follows: 1) Randomly select an initial state for the driverless vehicle, i.e., the driverless vehicle's position. And based on the initial state of the driverless car, adopt Choose an action, that is, use The probability of choosing the optimal action under the current strategy is used to determine the optimal action. The probability of choosing a random action; 2) The unmanned vehicle performs the aforementioned actions in a simulated environment and obtains a human-designed environmental reward value. And enter a new state The quaternion (the previous step's autonomous vehicle position) Actions performed by driverless vehicles Rewards from the environment The new position of driverless cars Stored in the experience pool; The autonomous vehicle continuously interacts with the environment, storing the quadruple data in the experience pool. 3) Once the experience pool is full, randomly sample from the experience pool to update the network parameters; 4) Repeat step 3) until the DQN partially converges.

2. The obstacle avoidance method for unmanned vehicles based on LSTM-DQN according to claim 1, characterized in that, Forgetting factor The calculation formula is: In the above formula, It is the sigmoid activation function. For the forgetting weight of the forgetting gate, This is a bias term.

3. The obstacle avoidance method for unmanned vehicles based on LSTM-DQN according to claim 1, characterized in that, Output factor Specifically: In the formula, It is the sigmoid activation function. The output weights of the output gate, This is the output factor bias term.

Images