Underground coal mine robot path planning method based on deep reinforcement learning

Abstract

The invention relates to an underground coal mine robot path planning method based on deep reinforcement learning, and the method comprises the steps: obtaining to-be-predicted state information, designing a bimodal reward function, and driving a robot to move towards a target position to avoid an obstacle; inputting the to-be-predicted state information into the path planning model to obtain a path planning result; the path planning model is obtained by training the improved DDPG double-layer network model by using a training set; the step of improving the DDPG double-layer network model comprises the steps that an LSTM layer is added in front of an Actor network module of the DDPG double-layer network model and used for making full use of environment information, a full connection layer in a hidden layer of the Actor network module adopts a ReLU activation function, a Sigmoid activation function is introduced into the last layer of the Actor network module, it is ensured that output values are all non-negative values, and the LSTM layer is used for making full use of environment information. And replacing the last full connection layer of the Critic network module in the DDPG double-layer network model with the improved crevasse network.

Description

Technical Field

[0001] The present invention relates to the technical field of robot path planning, and particularly to a method for path planning of a coal mine underground robot based on deep reinforcement learning. Background Art

[0002] A coal mine underground robot is an intelligent device used in dangerous environments such as coal mines, aiming to perform underground operation tasks, reduce human input, and improve work efficiency and safety. Path planning, as one of the core problems of mobile robot technology, is mainly used to design an optimal or sub-optimal obstacle avoidance path from the starting point to the target point for the robot in a complex underground environment. However, traditional path planning methods often have high computational complexity and limited effects when facing complex underground terrains and dynamic obstacles, making it difficult to meet actual requirements. In recent years, with the development of artificial intelligence technology, path planning methods based on deep reinforcement learning have gradually become important solutions.

[0003] The Q-Learning method in reinforcement learning helps the robot find the optimal path by updating the Q-value table of state-action. However, in a high-dimensional state and action space, the Q-value table will become too large, resulting in a significant increase in memory and computational requirements and prone to the "curse of dimensionality". For this reason, Google DeepMind proposed the Deep Q-Network (DQN) in 2013, which approximated the Q-value function through a neural network, significantly alleviating the curse of dimensionality problem and achieving end-to-end path planning from perception to action. However, the DQN algorithm is only applicable to a discrete action space and cannot meet the robot's demand for continuous actions.

[0004] To solve this problem, Google DeepMind launched the Deep Deterministic Policy Gradient (DDPG) algorithm in 2016, which combines the Actor-Critic framework to make it suitable for a continuous action space. However, due to problems such as low efficiency and difficult convergence in the training process of the deep deterministic policy gradient algorithm, it is very difficult to play a role when using the DDPG algorithm in practical problems. Due to the complexity of the coal mine underground environment, this algorithm is prone to phenomena such as low path planning efficiency and unsatisfactory effect of avoiding dynamic obstacles. Summary of the Invention

[0005] In order to solve the problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a method for path planning of a coal mine underground robot based on deep reinforcement learning, aiming to solve the problems of efficient path planning and dynamic obstacle avoidance in the coal mine underground environment.

[0006] To achieve the above purpose, the present invention provides the following solutions:

[0007] A path planning method for a coal mine underground robot based on deep reinforcement learning, comprising:

[0008] Obtain the state information to be predicted, where the state information to be predicted includes: the linear velocity and angular velocity of the robot, environmental information, the relative position of the target, and the relative position of obstacles;

[0009] Design a dual-modal reward function, and according to the dual-modal reward function, drive the robot to avoid dynamic and static obstacles during the process of moving towards the target position;

[0010] Input the state information to be predicted into a path planning model to obtain a path planning result; the path planning model is obtained by training an improved DDPG double-layer network model using a training set, and the training set includes: current state information, the action to be executed currently and its corresponding reward value, and the state information at the next moment obtained after executing the action;

[0011] Improving the DDPG double-layer network model includes: adding an LSTM layer in front of the Actor network module of the DDPG double-layer network model to make full use of the environmental information, using a ReLU activation function for the fully connected layer in the hidden layer of the Actor network module, and introducing a Sigmoid activation function in the last layer of the Actor network module to ensure that the output values are all non-negative values, and replacing the last fully connected layer of the Critic network module in the DDPG double-layer network model with an improved dueling network.

[0012] Optionally, after obtaining the state information to be predicted, it includes:

[0013] Obtain the state information to be predicted, incorporate the state information to be predicted into the state space, introduce Poisson encoding to represent the randomness of the state space, convert the state space into Poisson encoding, and convert the Poisson encoding into a new state:

[0014]

[0015] where λ is the pulse rate, Δt is the time window size of the Poisson encoding, t is the time step, t! is the factorial of the time step, s p is the converted new state, and P(t, s i ) is the probability function based on the Poisson distribution.

[0016] Optionally, designing the dual-modal reward function includes:

[0017] r = r s + βr d + r w + r v + r o

[0018] Among them, r is the bimodal reward function, and r s gives a positive reward when approaching and reaching the target, and a negative reward when colliding with a static obstacle. β is the adaptive adjustment factor, and r d is the reward for guiding the robot to avoid dynamic obstacles, and r w is the reward function for the angular velocity, and r v is the reward for accelerating to reach the target, and r o is the penalty imposed when oscillation occurs.

[0019] Optionally, the reward r s has the following expression:

[0020]

[0021] Among them, 400 and -200 respectively represent the rewards for reaching the target and colliding with a static obstacle. G th and Obs th respectively represent the distance thresholds from the robot to the destination and the static obstacle. SO dis is the relative distance between the robot and the nearest static obstacle that can be observed by the robot. Dis goal (t) is the relative distance between the robot and the destination at time t, and A is the amplification factor;

[0022] The reward r d has the following expression:

[0023] r d = A * (DO dis (t) - DO dis (t - 1)), if DO dis < DO th

[0024] Among them, t is the time t, and DO dis is the relative distance between the robot and the nearest dynamic obstacle, and DO th is the distance at which the dynamic obstacle affects the robot;

[0025] The reward r w has the following expression:

[0026]

[0027] Among them, w t is the angular velocity of the robot at time t;

[0028] The reward r v has the following expression:

[0029]

[0030] Among them, v t is the linear velocity of the robot at time t;

[0031] The reward r o has the following expression:

[0032]

[0033] Among them, Δw is the angular velocity change of the robot at adjacent times.

[0034] Optionally, the improved DDPG double-layer network model includes:

[0035] The first Actor network module is used to obtain the current action to be executed according to the current state information;

[0036] The first Critic network module is used to obtain the evaluation value of the current action to be executed based on the current state information and the current action to be executed;

[0037] The second Actor network module is used to obtain the best action to be executed in the next moment state according to the next moment state information obtained after executing the action;

[0038] The second Critic network module is used to obtain the evaluation value of the best action based on the next moment state information and the best action.

[0039] Optionally, improving the dueling network includes:

[0040] On the basis of the dueling network, add the average value of the action advantage function as a regulating factor:

[0041]

[0042] Among them, Q(s,a) is the evaluation value calculated by the dueling network, V(s) is the state value, A(s,z a ) is the advantage of taking z a in the action interval in state s, n is the number of action intervals, and z is the action interval set.

[0043] Optionally, training the improved DDPG double-layer network model using the training set includes:

[0044] Sort the sample data in real-time in descending order according to the reward value, divide the data with the top target rankings into the priority sampling pool, and divide the remaining sample data into the uniform sampling pool. And in each iteration, judge the reward value of the new data sample with the preset priority sampling pool threshold. When the reward value of the new data sample is greater than the priority sampling pool threshold, store the new data sample in the priority sampling pool;

[0045] Set a sampling weight between the priority sampling pool and the uniform sampling pool, and collect sample data from the priority sampling pool and the uniform sampling pool as the training set according to the sampling weight, and use the training set to train the improved DDPG double-layer network model.

[0046] The beneficial effects of the present invention are as follows:

[0047] The present invention incorporates dynamic obstacle information into the state input, and introduces Poisson coding to represent the randomness of the state space, optimizing the state input of the network.

[0048] By designing a bimodal reward function, the present invention can effectively guide the robot to avoid dynamic and static obstacles during movement, thereby improving the efficiency of path planning. In addition, based on the ROS Move Base package, the robot is guided to collect key data for training in real time, and the present invention realizes the autonomous navigation of the robot in a complex environment.

[0049] The present invention also introduces LSTM and duel networks into the DDPG network structure, alleviating the local optimum problem while improving the evaluation accuracy of the Q value and the stability of the algorithm. At the same time, the experience replay mechanism is optimized, and the convergence speed of the model is accelerated by combining priority sampling and uniform sampling.

[0050] Combining the above effects, the present invention significantly improves the safety, robustness and navigation efficiency of the robot path planning, and is particularly suitable for complex and dynamic environments such as underground coal mines. Description of the Drawings

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0052] Figure 1 It is a flowchart of a method for path planning of an underground coal mine robot based on deep reinforcement learning according to an embodiment of the present invention;

[0053] Figure 2 It is a schematic diagram of data collection according to an embodiment of the present invention;

[0054] Figure 3 It is a schematic diagram of the DDPG algorithm network structure according to an embodiment of the present invention;

[0055] Figure 4 It is a schematic diagram of the Actor network after introducing the LSTM network according to an embodiment of the present invention;

[0056] Figure 5 Schematic diagram of the Critic network after introducing the dueling network in the embodiment of the present invention;

[0057] Figure 6 Schematic diagram of the experience library data processing and sampling in the embodiment of the present invention;

[0058] Figure 7 Schematic diagram of the coal mine underground simulation environment in the embodiment of the present invention; wherein, (a) is the coal mine underground training environment built for the robot using Gazebo, and (b) is the unknown coal mine underground test environment built for the robot using Gazebo. Detailed implementation manners

[0059] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0060] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0061] This embodiment discloses a path planning method for a coal mine underground robot based on deep reinforcement learning, including: obtaining the state information to be predicted, where the state information to be predicted includes: the linear velocity and angular velocity of the robot, environmental information, the relative position of the target, and the relative position of the obstacle; designing a dual-modal reward function, and according to the dual-modal reward function, driving the robot to avoid dynamic and static obstacles during the process of moving towards the target position; inputting the state information to be predicted into the path planning model to obtain the path planning result; the path planning model is obtained by training the improved DDPG double-layer network model using a training set, and the training set includes: the current state information, the action to be executed currently and its corresponding reward value, and the state information at the next moment obtained after executing the action; the improved DDPG double-layer network model includes: adding an LSTM layer in front of the Actor network module of the DDPG double-layer network model to make full use of the environmental information, using the ReLU activation function for the fully connected layer in the hidden layer of the Actor network module, and introducing a Sigmoid activation function in the last layer of the Actor network module to ensure that the output values are all non-negative values, and replacing the last fully connected layer of the Critic network module in the DDPG double-layer network model with an improved dueling network.

[0062] Specifically: This embodiment discloses a path planning method for a coal mine underground robot based on deep reinforcement learning, including the following steps:

[0063] Step 1: The mobile robot obtains environmental information through the lidar it carries, and obtains and processes the robot's position information and target point information in real time through the built-in plug-in of the robot operating system. The dynamic obstacle information is incorporated into the state input, and Poisson coding is introduced to represent the randomness of the state space.

[0064] Step 2: A dual-modal (dynamic and static mode) reward function is designed to drive the robot to avoid dynamic and static obstacles during the process of moving towards the target position.

[0065] Step 3: The Move base function package is used to achieve autonomous data collection. The robot can navigate autonomously in various complex environments and collect key navigation data in real time.

[0066] Step 4: Design and improve the neural network structure model of the DDPG algorithm: On the basis of the dual-network structure of the DDPG algorithm, the LSTM network and the dueling network are introduced into the DDPG algorithm. The past state and the current state of the robot are comprehensively considered to jointly determine the robot's actions, ensuring the correlation between the robot's previous and subsequent actions. At the same time, the evaluation accuracy of the Q value and the stability of the algorithm are improved.

[0067] Step 5: In the model training stage, the pre-collected data is stored in a common experience pool, and the experience replay mechanism of DDPG is improved; based on the reward function, priority sampling and uniform sampling are combined according to a predefined ratio to accelerate the convergence speed of DDPG.

[0068] Step 6: Use the trained model to perform path planning for the coal mine underground robot.

[0069] Further, after obtaining the state information to be predicted, it includes: obtaining the state information to be predicted, incorporating the state information to be predicted into the state space, introducing Poisson coding to represent the randomness of the state space, converting the state space into Poisson coding, and converting the Poisson coding into a new state:

[0070]

[0071] where λ is the pulse rate, Δt is the time window size of the Poisson coding, t is the time step, t! is the factorial of the time step, s p is the new state after conversion, and P(t, s i ) is the probability function based on the Poisson distribution.

[0072] Specifically: Step 1, the input state is an important factor in achieving the optimal path. Incorporate dynamic obstacle information into the state space s, where the state space: s = {l, d, v, w, p}. Here, l represents lidar data, d is the relative position of the dynamic obstacles observable by the robot. When no dynamic obstacles are detected, the value of d is set to 0. In addition, v and w are the linear velocity and angular velocity of the robot respectively, and p represents the relative position of the target. In path planning, due to the existence of random dynamic obstacles, the environment is relatively complex. When there are complex non-linear relationships in the state space, the Poisson distribution can better reflect these non-linear relationships.

[0073] Introduce Poisson coding to represent the randomness of the state space, convert the state space s into Poisson coding, and then convert the Poisson coding into a new state s p .

[0074]

[0075] where t represents the time step, Δt represents the time window size of the Poisson coding, and λ is the pulse rate. For each feature s p in the state vector s i , a corresponding Poisson distribution parameter p(t, s i ) will be selected.

[0076] Furthermore, design a bimodal reward function including:

[0077] r = r s + βr d + r w + r v + r o

[0078] where r is the bimodal reward function, r s gives a positive reward when approaching and reaching the target, and a negative reward when colliding with static obstacles. β is an adaptive adjustment factor, r d is the reward for guiding the robot to avoid dynamic obstacles, r w is the reward function for the angular velocity, r v is the reward for accelerating to reach the target, r o is the penalty imposed when oscillation occurs.

[0079] Furthermore, the expression of the reward r s is:

[0080]

[0081] where 400 and -200 represent the rewards for reaching the target and colliding with static obstacles respectively, G th and Obs thRepresent the distance thresholds from the robot to the destination and the static obstacle, respectively, SO dis Is the relative distance to the nearest static obstacle observable by the robot, Dis goal (t) is the relative distance between the robot and the destination at time t, and A is the amplification factor;

[0082] Reward r d The expression of is:

[0083] r d = A * (DO dis (t) - DO dis (t - 1)), if DO dis < DO th

[0084] Among them, t is the time t, and DO dis Is the relative distance to the nearest dynamic obstacle of the robot, DO th Is the distance at which the dynamic obstacle affects the robot;

[0085] Reward r w The expression of is:

[0086]

[0087] Among them, w t Is the angular velocity of the robot at time t;

[0088] Reward r v The expression of is:

[0089]

[0090] Among them, v t Is the linear velocity of the robot at time t;

[0091] Reward r o The expression of is:

[0092]

[0093] Among them, Δw is the change in angular velocity of the robot at adjacent times. Specifically: Step 2, the reward function is set as follows, and β is the adaptive adjustment factor.

[0094] r = r s + βr d + r w + r v + r o

[0095] r sRewards drive the robot to move towards the target. The rewards increase when approaching the target and decrease when moving away. A positive reward is given when the target is reached, and a negative reward is received when colliding with a static obstacle. The reward r s is expressed as follows:

[0096]

[0097] where 400 and -200 represent the rewards for reaching the target and colliding with a static obstacle respectively, G th and Obs th represent the distance thresholds from the robot to the destination and the static obstacle respectively, used to judge whether it reaches the destination and collides with the static obstacle. SO dis is the relative distance between the robot and the nearest static obstacle that can be observed by the robot, Dis goal (t) is the relative distance between the robot and the destination at time t, and A is an amplification factor.

[0098] In a dynamic environment, r d the reward guides the robot to avoid dynamic obstacles. When the robot moves away from a dynamic obstacle, r d the reward increases, which helps to improve the success rate of path planning and cope with complex environments. The reward r d is expressed as follows:

[0099] r d = A * (DO dis (t) - DO dis (t - 1)), if DO dis < DO th

[0100] To maintain the stability of the robot during movement, a reward function is designed for the angular velocity in this embodiment. When the angular velocity exceeds a certain limit, a penalty is imposed. The reward r w is expressed as follows:

[0101]

[0102] To accelerate the arrival of the robot, the reward r v is defined as follows:

[0103]

[0104] To improve the flexibility and smoothness of the robot, a penalty is imposed when oscillation occurs. The reward r o is defined as follows:

[0105]

[0106] Furthermore, the improved DDPG double-layer network model includes: a first Actor network module for obtaining the current action to be executed according to the current state information; a first Critic network module for obtaining the evaluation value of the current action to be executed based on the current state information and the current action to be executed; a second Actor network module for obtaining the best action to be executed in the next state according to the next-state information obtained after executing the action; and a second Critic network module for obtaining the evaluation value of the best action to be executed in the next state based on the next-state information and the best action to be executed.

[0107] Furthermore, the improved dueling network includes, on the basis of the dueling network, adding the average value of the action advantage function as a regulating factor:

[0108]

[0109] where Q(s,a) is the evaluation value calculated by the dueling network, V(s) is the state value, and A(s,z a ) is the advantage of taking z a in the action interval in state s, n is the number of action intervals, and z is the set of action intervals.

[0110] Specifically: Step 4, the DDPG algorithm consists of four network modules, including two Actor networks and two Critic networks, and the two Actor networks have the same structure, and the two Critic networks have the same structure. One of the Actor networks and one of the Critic networks form the main network, and the remaining two networks form the target network. The main Actor network receives the state information of the robot and outputs the current action to be executed; the target Actor network receives the next-state information obtained after executing the action and outputs the best action to be executed in that state, rather than the actually executed action. The main Critic network inputs the current state and the current action to be executed and outputs the evaluation value of this action; the target Critic network inputs the next state and the predicted action and outputs the evaluation value of the predicted action in that state. In addition, the LSTM network is combined with the Actor networks of the main network and the target network, and an LSTM layer is added in front of each Actor network to make full use of the environmental information collected previously. In the hidden layer, the fully connected layer uses the ReLU activation function. In order to prevent the mobile robot from making backward movements, a Sigmoid activation function is introduced in the last layer of the network to ensure that the output is always non-negative. At the same time, the last fully connected layer of the Critic networks of the main network and the target network is replaced with an improved dueling network, and through the continuous learning and updating of the state value function and the action advantage function, the evaluation of the Q value is made more accurate, thereby improving the learning efficiency of the intelligent robot.

[0111] In the dueling network architecture, the output is divided into two parts: the state-value function and the action-advantage function. The state-value function focuses on evaluating the quality of the current state, independent of specific actions, and only pays attention to the state itself; while the action-advantage function focuses on the advantage of taking a certain action in a given state. By adding these two together, the Q-value for performing a specific action in a specific state can be calculated. The advantage of this is that it can clearly distinguish the differences in the effects of different actions for the current state, and can clearly distinguish the importance of each state under different states. The formula for the dueling network to calculate the Q-value is as follows:

[0112] Q(s,a) = V(s) + A(s,a)

[0113] Among them, V(s) represents the state value, and A(s,a) represents the advantage of taking action a in state s.

[0114] In this work, this embodiment focuses on a continuous action space. Therefore, according to its experimental environment, the action space is uniformly divided into n intervals, and z is used to represent the set of action intervals. The improved dueling network refers to adding the average value of the action-advantage function as a regulatory factor on the basis of the original dueling network, which helps to improve the stability of the algorithm and the evaluation accuracy of the Q-value. The formula for the improved dueling network to calculate the Q-value is as follows:

[0115]

[0116] Furthermore, training the improved DDPG double-layer network model using the training set includes: storing all the data pre-collected using the Move base function package in a common experience pool, sorting them in real-time in descending order according to the reward value, dividing the data with the top target rankings into a prioritized sampling pool, and dividing the remaining sample data into a uniform sampling pool. And in each iteration, judge the reward value of the new data sample with the preset prioritized sampling pool threshold. When the reward value of the new data sample is greater than the prioritized sampling pool threshold, store the new data sample in the prioritized sampling pool; set sampling weights between the prioritized sampling pool and the uniform sampling pool, and collect sample data from the prioritized sampling pool and the uniform sampling pool according to the sampling weights as the training set, and use the training set to train the improved DDPG double-layer network model.

[0117] Specifically: Step 5, the specific steps of combining prioritized sampling and uniform sampling are as follows:

[0118] First, sort the sample data in real-time in descending order according to the size of their reward values, and then remove the data with smaller reward values at the bottom of the experience pool. Divide the data with the top P% of the reward values into a prioritized sampling pool, and the remaining data into a uniform sampling pool. In the process of data selection, select samples from the sorted experience pool.

[0119] Next, in each iteration, the agent generates new data k. If the reward value in this data exceeds the set priority sampling pool threshold, it is added to the pool.

[0120] Finally, for each training process, a total of N datasets are selected. Between the priority sampling pool and the uniform sampling pool, the sampling weight w is set, and N*w and N*(1 - w) data are obtained from them respectively, and then these are combined for training.

[0121] This sampling strategy combines the advantages of priority sampling and uniform sampling, preferentially selects high-quality data with large reward values, and eliminates some low-quality data, which is beneficial to accelerating network training and promoting algorithm convergence.

[0122] The present invention improves the safety, robustness, and navigation efficiency of robot path planning by incorporating dynamic obstacle information into the state input, designing a bimodal reward function, using pre-collected key data for training, and improving the experience replay mechanism. As Figure 1 shown, it is the flowchart of the method of the present invention, and the specific steps are as follows:

[0123] Step 1, the mobile robot obtains environmental information through the lidar carried by itself, obtains the robot position information and target point information in real time through the built-in plug-in of the robot operating system and processes them, incorporates the dynamic obstacle information into the state input, and introduces Poisson coding to represent the randomness of the state space.

[0124] The robot uses the lidar to obtain the distance information from the surrounding obstacles. The lidar detects the distance information in 20 directions, and its survey range is from -90° to 90° (with the robot's front as 0°). Thus, it can be obtained that the lidar collects a beam of data every 9° as the input information. As Figure 7 (a) and Figure 7 (b) shown, the point reached by the straight line emitted in front of the robot model is the detection area of the lidar.

[0125] The input state is an important factor for achieving the optimal path. Incorporate the dynamic obstacle information into the state space s, state space: s = {l, d, v, w, p}. Where l represents the lidar data, and d is the relative position of the dynamic obstacle observable by the robot. When no dynamic obstacle is detected, the value of d is set to 0. In addition, v and w are the linear velocity and angular velocity of the robot respectively, and p represents the relative position of the target. In path planning, due to the existence of random dynamic obstacles, the environment is relatively complex. When there are complex non-linear relationships in the state space, the Poisson distribution can better reflect these non-linear relationships.

[0126] Poisson coding is introduced to represent the randomness of the state space. The state space s is converted into Poisson coding, and then the Poisson coding is converted into a new state s p .

[0127]

[0128] t represents the time step, Δt represents the time window size of Poisson coding, and λ is the pulse rate. For each feature s p in the state vector s i , a corresponding Poisson distribution probability function value p(t, s i ) is selected.

[0129] Step 2, a bimodal (dynamic and static modes) reward function is designed to drive the robot to move towards the target position while avoiding dynamic and static obstacles.

[0130] The reward function is set as intrinsic reward and extrinsic reward respectively. In a dynamic environment, the intrinsic reward guides the robot to avoid dynamic obstacles. When the robot moves away from dynamic obstacles, the intrinsic reward increases, which helps to improve the success rate of path planning and cope with complex environments. In a static environment, the influence of the intrinsic reward weakens, and the extrinsic reward drives the robot to move stably towards the target.

[0131] Step 2, the reward function is set as follows, where β is an adaptive adjustment factor.

[0132] r = r s + βr d + r w + r v + r o

[0133] r s The reward drives the robot to move towards the target. The reward increases when approaching the target and decreases when moving away. A positive reward is given when reaching the target, and a negative reward is received when colliding with a static obstacle. The reward r s is expressed as follows:

[0134]

[0135] Among them, 400 and -200 represent the rewards for reaching the target and colliding with a static obstacle respectively. G th and Obs th represent the distance thresholds from the robot to the destination and the static obstacle respectively, which are used to judge whether it reaches the destination and collides with the static obstacle. SO dis is the relative distance between the robot and the nearest static obstacle that can be observed, and Dis goal (t) is the relative distance between the robot and the destination at time t, and A is an amplification factor.

[0136] In a dynamic environment, r d The reward guides the robot to avoid dynamic obstacles. When the robot moves away from a dynamic obstacle, r d the reward increases, which helps to improve the success rate of path planning and cope with complex environments. The reward r d is expressed as follows:

[0137] r d = A * (DO dis (t) - DO dis (t - 1)), if DO dis < DO th

[0138] To maintain the stability of the robot during movement, in this embodiment, a reward function is designed for the angular velocity. When the angular velocity exceeds a certain limit, a penalty is imposed. The reward r w is expressed as follows:

[0139]

[0140] To accelerate the robot's arrival, the reward r v is defined as follows:

[0141]

[0142] To improve the flexibility and smoothness of the robot, a penalty is imposed when oscillation occurs. The reward r o is defined as follows:

[0143]

[0144] Step 3: Use the Move base package to implement autonomous data collection. The robot can autonomously navigate in various complex environments and collect key navigation data in real time.

[0145] Traditional methods of experience collection usually rely on manual settings. Although they work well in controlled environments, they have obvious deficiencies in practical applications. First, manual settings are subjective and may lead to insufficient adaptability and flexibility of the data in complex or unknown scenarios. Second, fixed experience settings often limit the exploration scope, resulting in insufficient diversity of the dataset, which in turn affects the generalization ability of the learning strategy under more complex conditions.

[0146] To solve these problems, this embodiment introduces the Move Base package to achieve autonomous data collection. As Figure 2 shown, the robot can autonomously navigate in multiple environments and collect key data in real time. The autonomously collected data includes the linear velocity v and angular velocity of the robot w, laser scan data l, relative position p of the target, relative position d of the dynamic obstacles observable by the robot, reward function value r.

[0147] All the collected data will be stored in the experience pool, providing better support for the agent's autonomous navigation in complex unknown environments. This autonomous collection method provides a solid foundation for subsequent policy optimization, not only shortening the training time but also enhancing the generalization ability of the policy.

[0148] Step 4, design a neural network structure model for improving the DDPG algorithm: Based on the dual-network structure of the DDPG algorithm, introduce the LSTM network and the dueling network into the DDPG algorithm to jointly determine the robot's actions by integrating the robot's past state and current state, ensuring the correlation between the robot's successive actions, while improving the evaluation accuracy of the Q value and the stability of the algorithm.

[0149] The DDPG algorithm consists of four network modules, as Figure 3 shown, including two Actor networks and two Critic networks, and the two Actor networks have the same structure, and the two Critic networks have the same structure. One Actor network and one Critic network form the main network, and the remaining two networks form the target network. The main Actor network receives the state information of the robot and outputs the action to be executed currently; the target Actor network receives the next state information obtained after executing the action and outputs the best action predicted to be executed in that state, rather than the actual executed action. The main Critic network inputs the current state and the action to be executed currently and outputs the evaluation value of this action; the target Critic network inputs the next state and the predicted action and outputs the evaluation value of the predicted action in that state.

[0150] Regarding the time series characteristics of the state vector, introduce the LSTM network in the DDPG algorithm to process the state information to enhance the long-term dependence relationship between the current state and the historical state. Add an LSTM layer in front of each Actor network to make full use of the environmental information collected previously. The DDPG network framework with the introduction of LSTM is as Figure 4 shown. Specifically, the Actor network consists of 3 fully connected layers and 3 activation functions, receives the state information of the robot, and outputs the action information, that is, the linear velocity and the angular velocity. In the hidden layer, the fully connected layer uses the ReLU activation function. To avoid the mobile robot from making backward movements, a Sigmoid activation function is introduced in the last layer of the network to ensure that the output is always non-negative.

[0151] The Critic network receives the state of the robot and the actions to be performed, and outputs the evaluation value of performing the actions in that state. Replace the last fully connected layer of the Critic network with an improved dueling network, and the structure is as Figure 5 shown. The state and actions are input into the Critic network, and fully connected layers with ReLU activation functions process the actions and the state respectively. Then, the processing results of the two layers pass through the dueling network to calculate the final Q value.

[0152] Step 5, in the model training stage, store all the pre-collected data in a common experience pool, and improve the experience replay mechanism of DDPG; based on the reward function, combine priority sampling and uniform sampling according to a predefined ratio to accelerate the convergence speed of DDPG.

[0153] The combination of priority sampling and uniform sampling is as Figure 6 shown, and the specific steps are as follows:

[0154] First, sort the sample data in real-time in descending order according to their reward values, and then remove the data with smaller reward values at the bottom of the experience pool. Divide the data with the top P% of the reward values into the priority sampling pool, and the remaining data into the uniform sampling pool. During the data selection process, select samples from the sorted experience pool.

[0155] Next, in each iteration, the agent generates new data k. If the reward value in this data exceeds the set priority sampling pool threshold, add it to the pool.

[0156] Finally, for each training process, a total of N data sets are selected. Set the sampling weight w between the priority sampling pool and the uniform sampling pool, and obtain N*w and N*(1 - w) data from them respectively, and then combine these for training.

[0157] This sampling strategy combines the advantages of priority sampling and uniform sampling, preferentially selects high-quality data with large reward values, and eliminates some low-quality data, which is beneficial to accelerating network training and promoting algorithm convergence.

[0158] Step 6, use the trained model for path planning of the coal mine underground robot.

[0159] Use the model trained in the Figure 7 (a) environment, in the Figure 7(b) Robot path planning in a coal mine environment. By loading the trained Actor and Critic network models, the robot will obtain the environmental state input by the sensor in real time and generate the next action through the Actor network. The Critic network evaluates the value of the generated action to ensure the optimality and safety of path planning. The generated action is converted into a robot control instruction to drive it to move autonomously in the coal mine underground environment, dynamically avoid obstacles, and continuously adjust the path according to the real-time updated environmental data, and finally reach the target point safely and efficiently.

[0160] The embodiments described above are only descriptions of the preferred embodiments of the present invention and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, various deformations and improvements made by those of ordinary skill in the art to the technical solutions of the present invention shall fall within the protection scope determined by the claims of the present invention.

Claims

1. A coal mine robot path planning method based on deep reinforcement learning, characterized in that: include: Acquire state information to be predicted, the state information to be predicted including: linear velocity and angular velocity of the robot, environmental information, relative position of the target and relative position of the obstacle; Designing a bimodal reward function, and driving the robot to avoid dynamic and static obstacles in the process of moving to a target position according to the bimodal reward function; The state information to be predicted is input into the path planning model to obtain the path planning result; the path planning model is obtained by training the improved DDPG two-layer network model using the training set, and the training set includes: current state information, the action to be performed and its corresponding reward value, and the next state information obtained after the action is performed; Improving the DDPG two-layer network model includes: adding an LSTM layer before the Actor network module of the DDPG two-layer network model to make full use of the environmental information, the fully connected layer in the hidden layer of the Actor network module adopts the ReLU activation function, and the Sigmoid activation function is introduced in the last layer of the Actor network module to ensure that the output values ​​are all non-negative values, and the last fully connected layer of the Critic network module in the DDPG two-layer network model is replaced by the improved duel network.

2. The coal mine robot path planning method based on deep reinforcement learning according to claim 1 is characterized in that: After obtaining the state information to be predicted, the following steps are included: The state information to be predicted is obtained, the state information to be predicted is incorporated into the state space, Poisson coding is introduced to represent the randomness of the state space, the state space is converted into Poisson coding, and the Poisson coding is converted into a new state: Where λ is the pulse rate, Δt is the time window size of Poisson coding, t is the time step, t! is the factorial of the time step, and s p is the new state after conversion, P(t,s i ) is a probability function based on Poisson distribution.

3. The coal mine robot path planning method based on deep reinforcement learning according to claim 1 is characterized in that: Designing the bimodal reward function includes: r＝r s +βr d +r w +r v +r o Among them, r is the bimodal reward function, r s Positive rewards are given when approaching and reaching the target, and negative rewards are given when colliding with static obstacles. β is the adaptive adjustment factor, and r d The reward for guiding the robot to avoid dynamic obstacles, r w is the reward function of angular velocity, r v is the reward for accelerating to the target, r o is the penalty imposed when oscillation occurs.

4. The coal mine underground robot path planning method based on deep reinforcement learning according to claim 3 is characterized in that: Reward s The expression is: Among them, 400 and -200 represent the rewards for reaching the target and colliding with static obstacles, respectively. th and Obs th Respectively represent the distance thresholds from the robot to the destination and static obstacles, SO dis is the relative distance to the nearest static obstacle that the robot can observe, Dis goal (t) is the relative distance between the robot and the destination at time t, and A is the amplification factor; Reward d The expression is: r d =A*(DO dis (t)-DO dis (t-1)),if DO dis <DO th Where t is time t, DO dis is the relative distance between the robot and the nearest dynamic obstacle, DO th The distance at which dynamic obstacles affect the robot; Reward w The expression is: Among them, w t is the angular velocity of the robot at time t; Reward v The expression is: Among them, v t is the linear velocity of the robot at time t; Reward o The expression is: Among them, Δw is the change of the robot's angular velocity at adjacent moments.

5. The coal mine underground robot path planning method based on deep reinforcement learning according to claim 1 is characterized in that: The improved DDPG two-layer network model includes: The first Actor network module is used to obtain the current action to be performed based on the current state information; A first Critic network module is used to obtain the evaluation value of the current action to be performed based on the current state information and the current action to be performed; The second Actor network module is used to obtain the best action to be performed at the next moment according to the next moment state information obtained after the action is executed; The second critic network module is used to obtain the evaluation value of the best action based on the next moment state information and the best action.

6. The coal mine underground robot path planning method based on deep reinforcement learning according to claim 1 is characterized in that: Improvements to the dueling network include: On the basis of the duel network, the average value of the action advantage function is added as a regulation factor: Among them, Q(s,a) is the evaluation value calculated by the duel network, V(s) is the state value, and A(s,z a ) is to take z in state s a The advantage of the action interval, n is the number of action intervals, and z is the set of action intervals.

7. The coal mine robot path planning method based on deep reinforcement learning according to claim 1 is characterized in that: The improved DDPG two-layer network model trained using the training set includes: The sample data is sorted in real time in descending order according to the reward value, the data before the target ranking is divided into a priority sampling pool, and the remaining sample data is divided into a uniform sampling pool, and in each iteration, the new data sample reward value is judged with the preset priority sampling pool threshold, and when the new data sample reward value is greater than the priority sampling pool threshold, the new data sample is stored in the priority sampling pool; A sampling weight is set between the priority sampling pool and the uniform sampling pool, sample data is collected from the priority sampling pool and the uniform sampling pool according to the sampling weight as the training set, and the improved DDPG two-layer network model is trained using the training set.

Images