Mobile robot complex environment path planning method based on DQN

Abstract

A mobile robot complex environment path planning method based on DQN relates to the field of path planning, and comprises the following steps: data acquisition, data preprocessing, deep Q network design, reward function design and path planning. According to the method, the deep Q network is trained by using the state information of the robot, so that the deep Q network learns the ability of selecting the action strategy through the environment information, and the method has stronger adaptability and superior generalization ability; comprehensive heuristic rewards are designed, and the performance in the three aspects of path length, energy consumption and safety is balanced; when a danger penalty function is designed, various factors such as the distance between a moving path and an obstacle, the terrain gradient, the orientation of the gradient and the direction of the obstacle are comprehensively considered, and the task safety of the mobile robot is improved; by introducing a 2.5 D layered grid map into a state space, the dimensionality of the state space is reduced, obstacle information and terrain information are dispersed, and the convergence speed, learning efficiency and path planning performance of the algorithm are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of path planning, and in particular to a DQN-based mobile robot complex environment path planning method. Background Art

[0002] Path planning refers to the decision-making process of a mobile robot starting from a starting point and reaching a target point while avoiding obstacles. However, existing general robot path planning methods require environmental modeling in advance, which makes it difficult to apply to unknown environments. In addition, general robot path planning methods have problems with the improper balance between energy consumption, efficiency, and safety. There are also problems with the lax design of safety rewards, low safety, and the inability to accurately capture the complex relationship between states and actions, resulting in low success rate and poor results of path planning tasks. Summary of the invention

[0003] In view of the above situation, in order to solve the technical problems existing in the existing general robot path planning methods, such as the need to perform environmental modeling in advance and thus being difficult to apply to unknown environments, improper balance between energy consumption, efficiency and safety, imprecise safety reward design and low safety, inability to accurately capture the complex relationship between state and action, and low success rate and poor effect of path planning tasks, the present invention provides a complex environment path planning method for a mobile robot based on DQN.

[0004] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0005] The present invention provides a DQN-based mobile robot complex environment path planning method, which specifically includes the following steps:

[0006] Step S1: Collect robot motion data;

[0007] Step S2: cleaning, normalizing and labeling the robot motion data;

[0008] Step S3: Design the action value function and construct the deep Q network structure;

[0009] Step S4: designing a comprehensive heuristic reward function including a path reward and penalty function, an extra energy consumption penalty function, and a dangerousness penalty function;

[0010] Step S5: Set the state space according to the 2.5D layered grid map, and set n groups of robots to perform path planning in the environment. Update the network parameters every 100 rounds. When the success rate reaches the threshold, increase the difficulty until the maximum number of training times.

[0011] Furthermore, the robot motion data includes state, action, environmental information, obstacle location, target location, surrounding terrain information, robot energy consumption, actual path and rewards obtained.

[0012] Furthermore, data cleaning is used to remove noise and erroneous data in the robot motion data, and missing values ​​are processed; after the robot motion data is normalized, data annotation is used to ensure that each data point contains the state, action taken and result.

[0013] Furthermore, the deep Q network is a five-layer static forward network, including four hidden layers and an output layer, and each hidden layer consists of a fully connected layer and an activation function.

[0014] Furthermore, the mathematical expression of the hidden layer is:

[0015] output i =PReLU(W i ×input i +b i )

[0016] Among them, W i represents the weight matrix of the i-th hidden layer; b i represents the bias vector of the i-th hidden layer; input i represents the input of the i-th hidden layer; for the first hidden layer, input1 represents the input data x of the deep Q network, whose shape is (n, input_dim), n represents the batch size, and input_dim represents the state space dimension; for subsequent hidden layers, input i Represents the output of the previous hidden layer; PReLU represents the parameterized ReLU activation function, and its specific calculation formula is as follows:

[0017] PReLU(x)=max(0,x)+αmin(0,x)

[0018] Among them, x represents the input data; α represents a learnable parameter; when the value of the input data x is positive, the behavior of PReLU is the same as ReLU, that is, it directly outputs x; when the value of the input data x is negative, PReLU outputs a scaled negative value, and the scaling factor is the parameter α.

[0019] Furthermore, the mathematical expression of the output layer is:

[0020] Q value =W final ×ouput4+b final

[0021] Among them, W final represents the weight matrix of the output layer; b finalrepresents the bias vector of the output layer; ouput4 represents the output of the last hidden layer; the deep Q network calculates the Q value of each action according to the input state.

[0022] Furthermore, the mathematical expression of the comprehensive heuristic reward function is:

[0023] r=r d +r e +r h +r s +r t +r o +r ob

[0024] Among them, r d represents the distance reward and penalty function, r e represents the additional energy consumption penalty function, y h represents the dangerousness penalty function, r s Indicates the penalty for standing still, r t Represents the reward for reaching the target point, r o represents the penalty for exceeding the worst path length and exhausting energy, r ob Indicates the penalty for colliding with an obstacle;

[0025] The mathematical expression of the distance reward and penalty function is:

[0026]

[0027] Among them, λ represents the distance reward and penalty factor, d or Indicates the distance from the starting point to the target point, d n-1 Indicates the distance between the robot's previous position and the target point, d n Indicates the distance between the current robot position and the target point; when (d n-1 -d n ) is positive, it means that the robot is approaching the target point, and this distance reward and penalty function will give a reward; when (d n-1 -d n ) is negative, it means that the robot is moving away from the target point, and the distance reward and penalty function will give a penalty;

[0028] The mathematical expression of the additional energy consumption penalty function is:

[0029]

[0030] Among them, e b Indicates the basic energy consumption of the robot in one operation, E(P n ,P n+1 ) represents the energy consumption between two points, P n+1 Indicates the next position, P nrepresents the current position, k represents the additional energy penalty factor. When the robot moves uphill, it consumes more energy. (E(P n ,P n+1 )-e b ) is positive, this extra energy consumption penalty function will give a penalty; when the robot moves downhill, it consumes less energy and no penalty is given;

[0031] The mathematical expression of the dangerousness penalty function is:

[0032] r h =η·H(P n ,P n+1 )

[0033] Among them, η represents the risk penalty factor, H(P n ,P n+1 ) indicates the danger from the current location to the next location.

[0034] Furthermore, the energy consumption E(P n ,P n+1 ) is calculated as follows:

[0035] E(P n ,P n+1 )=e b +c·g(z n+1 -z n )

[0036] Where c represents the mass of the robot, g represents the acceleration due to gravity, and z n+1 Indicates the height of the next position, z n Indicates the current location altitude.

[0037] Furthermore, the risk H(P n ,P n+1 ) is calculated as follows:

[0038]

[0039] Where b represents the risk parameter, d ob represents the distance between the nearest obstacle and the mobile robot, m represents the slope at which the robot can operate safely, and β(P n ,P n+1 ) represents the terrain slope between two points, θ represents the angle between the direction of the nearest obstacle relative to the robot and the terrain orientation;

[0040] The terrain slope β(P n ,P n+1 ) is calculated as follows:

[0041]

[0042] Among them, x n+1 Indicates the horizontal coordinate of the next position, x n Indicates the horizontal coordinate of the current position, y n+1 Indicates the ordinate of the next position, y n Indicates the current position vertical coordinate;

[0043] The specific calculation formula for the angle θ between the direction of the nearest obstacle relative to the robot and the terrain orientation is:

[0044] θ=|O t -O ob |

[0045] Among them, O t Indicates the terrain orientation, and its specific calculation formula is:

[0046]

[0047] Among them, Z represents the height, x represents the horizontal coordinate, and y represents the vertical coordinate;

[0048] Among them, O ob Indicates the direction of the nearest obstacle relative to the robot. The specific calculation formula is:

[0049]

[0050] Among them, x ob Indicates the horizontal coordinate of the obstacle relative to the robot, y ob The vertical coordinate representing the direction of the obstacle relative to the robot.

[0051] Furthermore, the mathematical expression of the state space is:

[0052] S 2.5d =[x,y,dx,dy,x t ,y t ,d or ,step,d,H,e cs ,e,ob r ,l r ]

[0053] Among them, x and y represent the position of the robot, dx and dy represent the relative position of the robot and the target point, and x t ,y t represents the location of the target point, d or represents the distance from the starting point to the target point, step represents the number of steps the robot moves, d represents the distance from the robot to the target point, e cs represents the energy consumption of the robot in its current state, e represents the total energy consumption of the robot, and obr Indicates the obstacles around the robot, l r Represents the elevation information of the terrain around the robot;

[0054] The specific updating formula of the network parameters is:

[0055] L=Ε[(Q local (s,a;θ local )-Q target ) 2 ]

[0056] Where L is the loss function, which represents the mean square error between the predicted Q value and the target Q value; Q local (s,a;θ local ) represents the Q value prediction of the current deep Q network for the state and action, θ local Represents the parameters of the current deep Q network; Q target Represents the target Q value, and its specific calculation formula is:

[0057] Q target = r + γ max a′ Q target (s′,a′;θ target )·(1-ov)

[0058] Where r represents the comprehensive heuristic reward function, γ represents the discount factor, s represents the current state, s′ represents the next state, ov represents the end flag, a represents the current action, and a′ represents the next action.

[0059] The beneficial effects achieved by the present invention using the above scheme are as follows:

[0060] 1. In view of the problem that the existing general robot path planning methods need to perform environmental modeling in advance and are difficult to apply to unknown environments, the present invention uses the robot's state information to train the deep Q network, so that it can learn the ability to select action strategies through environmental information. Because the object of its learning is the relationship between environmental information and action rather than the relationship between position information and action, the present invention can be applied to environment B after training in environment A, and has stronger adaptability and superior generalization ability.

[0061] 2. In view of the problem of improper balance between energy consumption, efficiency and safety in existing general robot path planning methods, the present invention designs a comprehensive heuristic reward function including path reward and punishment function, additional energy consumption penalty function and dangerousness penalty function, which balances the three aspects of robot path length, energy consumption and safety.

[0062] 3. In view of the problem that the existing general robot path planning methods have imprecise safety reward design and low safety, the present invention comprehensively considers multiple factors such as the distance between the moving path and the obstacle, the terrain slope, the direction of the slope, the direction of the obstacle, etc. in the risk penalty function, and pays special attention to the danger of the robot's tendency to tip over and slide to the moving path, thereby improving the safety of the mobile robot's mission.

[0063] 4. In view of the problems that the existing general robot path planning methods cannot accurately capture the complex relationship between state and action, and the low success rate and poor effect of path planning tasks, the present invention introduces a 2.5D layered grid map into the state space, reduces the dimension of the state space, and disperses obstacle information and terrain information, thereby improving the convergence speed, learning efficiency and path planning performance of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A flow chart of a DQN-based mobile robot complex environment path planning method provided by the present invention.

[0065] Figure 2 This is a schematic diagram of the specific implementation process of step S3.

[0066] Figure 3 This is a schematic diagram of the specific implementation process of step S4. DETAILED DESCRIPTION

[0067] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0068] See also Figure 1 As shown, the present invention provides a DQN-based mobile robot complex environment path planning method, and its specific implementation process is as follows:

[0069] Step S1: data collection;

[0070] The robot motion data is collected by the data collection operation. The robot motion data collected by the present invention is the robot motion data recorded in a complex environment, including but not limited to the state, action, environmental information, obstacle position, target position, surrounding terrain information, robot energy consumption, actual path and rewards obtained.

[0071] Step S2: data preprocessing;

[0072] Data preprocessing is used to complete data cleaning, normalization, and data labeling of the collected robot motion data. Data cleaning can remove noise and erroneous data in the robot motion data and process missing values. After normalizing the robot motion data, data labeling can ensure that each data point contains information such as status, actions taken, and results.

[0073] Step S3: Design the action value function and construct the deep Q network structure;

[0074] In view of the problem that existing general robot path planning methods require environmental modeling in advance and are difficult to apply to unknown environments, the present invention uses the robot's state information to train the deep Q network so that it can learn the ability to select action strategies based on environmental information.

[0075] See also Figure 2 As shown, the specific implementation process is as follows:

[0076] Step S3.1: Deep Q network structure design;

[0077] The deep Q network established in the present invention is a five-layer static forward network, which mainly includes four hidden layers and one output layer, and each hidden layer is composed of a fully connected layer and an activation function.

[0078] Step S3.2: hidden layer design;

[0079] The hidden layer designed in the present invention specifically uses the PReLU activation function, which is specifically expressed as follows:

[0080] output i =PReLU(W i ×input i +b i )

[0081] Among them, W i represents the weight matrix of the i-th hidden layer; b i represents the bias vector of the i-th hidden layer; input i represents the input of the i-th hidden layer; for the first hidden layer, input1 represents the input data w of the deep Q network, whose shape is (n, input_dim), where n represents the batch size and input_dim represents the state space dimension; for subsequent hidden layers, input i Represents the output of the previous hidden layer; PReLU represents the parameterized ReLU activation function, and its specific calculation formula is as follows:

[0082] PReLU(w)=max(0,w)+αmin(0,w)

[0083] Among them, w represents the input data; α represents a learnable parameter; the intuitive explanation of PReLU is that when the value of the input data x is positive, PReLU behaves the same as ReLU, that is, it directly outputs x; when the value of the input data x is negative, PReLU will output a scaled negative value, and the scaling factor is the parameter α.

[0084] This design of the present invention aims to solve the problem that the ReLU activation function is completely "dead" in the negative input region, that is, the output is always 0, allowing the deep Q network to have gradient flow in the negative input region, so that those weight initializations can be trained to cause some neurons to output negative values ​​in the early stage of training.

[0085] Step S3.3: output layer design;

[0086] For the final output layer, since no activation function is used, the specific calculation formula is as follows:

[0087] Q value =W final ×ouput4+b final

[0088] Among them, W final represents the weight matrix of the output layer; b final represents the bias vector of the output layer; ouput4 represents the output of the last hidden layer. In this way, the deep Q network can calculate the Q value of each action based on the input state.

[0089] Step S4: design reward function;

[0090] In view of the problem of improper balance between energy consumption, efficiency and safety in existing general robot path planning methods, the present invention designs a comprehensive heuristic reward function including a path reward and punishment function, an extra energy consumption penalty function and a danger penalty function.

[0091] A distance reward and penalty function, an extra energy consumption penalty function, and a danger penalty function are designed. The danger penalty function comprehensively considers multiple factors such as the distance between the moving path and the obstacle, the terrain slope, the direction of the slope, the direction of the obstacle, etc., with special attention paid to the danger of the robot's tendency to tip over and slide to the moving path.

[0092] See also Figure 3 As shown, the specific implementation process is as follows:

[0093] Step S4.1: comprehensive inspiration reward function design;

[0094] In order to guide the robot to reach the target point, penalties and rewards should be given based on the distance from the target point. The closer the distance to the target point, the greater the reward for moving closer to the target, and the greater the penalty for moving away from the target point; considering the energy consumption in path planning, penalties should be given based on the additional energy consumption when elevation changes occur; considering the safety in path planning, penalties should be given to the robot for dangerousness; in order to prevent the robot from cheating by stopping, a stronger penalty should be given for this situation; a larger reward should be given when the robot reaches the target point to encourage the robot to remember the experience when reaching the target; set the longest path length and the upper limit of the robot's energy, determine that the robot's task exceeds the worst path length and the energy is exhausted, and give a stronger penalty; determine that the robot's task collides with an obstacle fails, and give a stronger penalty; therefore, the comprehensive heuristic reward function designed by the present invention is specifically expressed as follows:

[0095] r=r d +r e +r h +r s +r t +r o +r ob

[0096] Among them, r d represents the distance reward and penalty function, r e represents the additional energy consumption penalty function, r h represents the dangerousness penalty function, r s Indicates the penalty for standing still, r t Represents the reward for reaching the target point, r o represents the penalty for exceeding the worst path length and exhausting energy, r ob represents the penalty for colliding with an obstacle. Among them, the penalty for stopping is r s , the reward r for reaching the target point t , the penalty r for exceeding the worst path length and exhausting energy o , Penalty for collision with obstacles ob are all large constants. The difference is that the reward r for reaching the target point t Much greater than the penalty for standing still s , the penalty r for exceeding the worst path length and exhausting energy o , Penalty for collision with obstacles ob , so that the robot can remember the experience of reaching the target point.

[0097] Step S4.2: Design a distance reward and penalty function, the specific mathematical expression of which is as follows:

[0098]

[0099] Among them, λ represents the distance reward and penalty factor, d or Indicates the distance from the starting point to the target point, d n-1 Indicates the distance between the robot's previous position and the target point, d n Indicates the distance between the current robot position and the target point; thus, when (d n-1 -d n ) is positive, it means that the robot is approaching the target point, and this distance reward and penalty function will give a reward; when (d n-1 -d n ) is negative, it means that the robot is moving away from the target point, and this distance reward and penalty function will give a penalty. And as the robot continues to approach the target point, the distance d between the current robot position and the target point n The value will continue to decrease, and the intensity of rewards and punishments will continue to increase to encourage the robot to approach the target point.

[0100] Step S4.3: designing an extra energy consumption penalty function;

[0101] The specific implementation process is as follows:

[0102] The energy consumption between two points is modeled as:

[0103] E(P n ,P n+1 )=e b +c·g(z n+1 -z n )

[0104] Among them, e b represents the basic energy consumption of the robot in one step, c represents the mass of the robot, g represents the acceleration of gravity, P n+1 Indicates the next position, P n Indicates the current position, z n+1 Indicates the height of the next position, z n Indicates the current location altitude.

[0105] The extra energy penalty function is designed to reduce the extra energy consumed by the robot when crossing the terrain. When the robot encounters wavy terrain, it will repeatedly move up and downhill. If the robot is rewarded for moving downhill, it will not be able to effectively guide the robot to learn the strategy for wavy terrain. Therefore, there is no penalty or reward for downhill movement that consumes less energy. The mathematical expression of the extra energy penalty function is as follows:

[0106]

[0107] Among them, E(P n ,P n+1 ) represents the energy consumption between the current position and the next position, and k represents the additional energy consumption penalty factor. When the robot moves uphill, it consumes more energy. (E(Pn ,P n+1 )-e b ) is positive, this extra energy consumption penalty function will give a penalty; when the robot moves downhill, it consumes less energy and no penalty is given.

[0108] Step S4.4: designing a dangerousness penalty function;

[0109] The danger of the robot's moving path is affected by many factors, such as the distance between the moving path and the obstacle, the terrain slope, the direction of the slope, and the direction of the obstacle. Therefore, the danger is modeled as follows:

[0110]

[0111] Where b represents the risk parameter, d ob represents the distance between the nearest obstacle and the mobile robot, m represents the slope at which the robot can operate safely, and β(P n ,P n+1 ) represents the terrain slope between two points, and θ represents the angle between the direction of the nearest obstacle relative to the robot and the terrain orientation.

[0112] Terrain slope β(P n ,P n+1 ) is calculated as follows:

[0113]

[0114] Among them, x n+1 Indicates the horizontal coordinate of the next position, x n Indicates the horizontal coordinate of the current position, y n+1 Indicates the current position ordinate, y n Indicates the vertical coordinate of the next position.

[0115] The specific calculation formula for the angle θ between the direction of the nearest obstacle relative to the robot and the terrain orientation is as follows:

[0116] θ=|O t -O ob |

[0117] Among them, O t Indicates the terrain orientation. The specific calculation formula is as follows:

[0118]

[0119] Among them, Z represents the height, x represents the horizontal coordinate, and y represents the vertical coordinate.

[0120] O ob Indicates the direction of the nearest obstacle relative to the robot. The specific calculation formula is as follows:

[0121]

[0122] Among them, x ob Indicates the horizontal coordinate of the obstacle relative to the robot, y ob Represents the vertical coordinate of the obstacle relative to the robot's direction.

[0123] In summary, the specific mathematical expression of the dangerousness penalty function is as follows:

[0124] r h =η·H(P n ,P n+1 )

[0125] Among them, η represents the risk penalty factor, H(P n ,P n+1 ) indicates the danger from the current location to the next location.

[0126] In view of the problems of imprecise safety reward design and low safety in existing general robot path planning methods, the present invention comprehensively considers multiple factors such as the distance between the moving path and the obstacle, the terrain slope, the direction of the slope and the direction of the obstacle when designing the danger penalty function, and pays special attention to the danger of the robot's tendency to tip over and slide to the moving path.

[0127] Step S5: path planning;

[0128] By setting the state space, n groups of robots are set to perform path planning in the environment, and the network parameters are updated every 100 rounds. After the success rate reaches the threshold, the difficulty is increased until the maximum number of training times.

[0129] The specific implementation process is as follows:

[0130] Step S5.1: Set the state space according to the 2.5D layered grid map, and its mathematical expression is as follows:

[0131] S 2.5d =[x,y,dx,dy,x t ,y t ,d or ,step,d,H,e cs ,e,ob r ,l r ]

[0132] Among them, S 2.5d represents the state space, x and y represent the position of the robot, dx and dy represent the relative position of the robot and the target point, and x t ,y t represents the location of the target point, d orrepresents the distance from the starting point to the target point, step represents the number of steps the robot moves, d represents the distance from the robot to the target point, e cs represents the energy consumption of the robot in its current state, e represents the total energy consumption of the robot, and ob r Indicates the obstacles around the robot, l r Represents the elevation information of the terrain around the robot.

[0133] Step S5.2: path planning, with an accurate threshold set in advance;

[0134] First, initialize the parameters and set n groups of robots at the same time; then use Q-learning to select the optimal action according to the state to achieve path planning; when the accuracy of the n groups of robots completing the path planning task is higher than the accuracy threshold, increase the number of obstacles in the environment; update the network parameters every 100 rounds of training; when the maximum number of training times is reached, the training is completed; the specific formula for updating the network parameters is as follows:

[0135] L=Ε[(Q local (s,a;θ local )-Q target ) 2 ]

[0136] Where L is the loss function, which represents the mean square error between the predicted Q value and the target Q value; Q local (s,a;θ local ) represents the Q value prediction of the current deep Q network for the state and action, θ local Represents the parameters of the current deep Q network; Q target Represents the target Q value, and its specific calculation formula is as follows:

[0137] Q target = r + γ max a′ Q target (s′,a′;θ target )·(1-ov)

[0138] Where r represents the comprehensive heuristic reward function, γ represents the discount factor, s represents the current state, s′ represents the next state, ov represents the end flag, a represents the current action, and a′ represents the next action.

[0139] In view of the problems that the existing general robot path planning methods cannot accurately capture the complex relationship between states and actions, and the low success rate and poor effect of path planning tasks, the present invention introduces a 2.5D layered grid map into the state space, reduces the dimension of the state space, and disperses obstacle information and terrain information, thereby improving the convergence speed, learning efficiency and path planning performance.

[0140] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A DQN-based mobile robot complex environment path planning method, characterized in that: The following steps are involved: Step S1: Collect robot motion data; Step S2: cleaning, normalizing and labeling the robot motion data; Step S3: Design the action value function and construct the deep Q network structure; Step S4: designing a comprehensive heuristic reward function including a path reward and penalty function, an extra energy consumption penalty function, and a dangerousness penalty function; Step S5: Set the state space according to the 2.5D layered grid map, and set n groups of robots to perform path planning in the environment. Update the network parameters every 100 rounds. When the success rate reaches the threshold, increase the difficulty until the maximum number of training times.

2. According to claim 1, a DQN-based mobile robot complex environment path planning method is characterized in that: The robot motion data includes status, action, environmental information, obstacle location, target location, surrounding terrain information, robot energy consumption, actual path and rewards obtained.

3. According to claim 1, a DQN-based mobile robot complex environment path planning method is characterized in that: Data cleaning is used to remove noise and erroneous data from the robot motion data, and to process missing values. After the robot motion data is normalized, data annotation is used to ensure that each data point contains the state, action taken, and result.

4. According to claim 1, a DQN-based mobile robot complex environment path planning method is characterized in that: The deep Q network is a five-layer static forward network, including four hidden layers and one output layer, each hidden layer consists of a fully connected layer and an activation function.

5. A DQN-based mobile robot complex environment path planning method according to claim 4, characterized in that: The mathematical expression of the hidden layer is: output i =PReLU(W i ×input i +b i ) Among them, W i represents the weight matrix of the i-th hidden layer; b i represents the bias vector of the i-th hidden layer; input i represents the input of the i-th hidden layer; for the first hidden layer, input1 represents the input data x of the deep Q network, whose shape is (n, input_dim), n represents the batch size, and input_dim represents the state space dimension; for subsequent hidden layers, input i Represents the output of the previous hidden layer; PReLU represents the parameterized ReLU activation function, and its specific calculation formula is as follows: PReLU(x)=max(0,x)+αmin(0,x) Among them, x represents the input data; α represents a learnable parameter; when the value of the input data x is positive, the behavior of PReLU is the same as ReLU, that is, it directly outputs x; when the value of the input data x is negative, PReLU outputs a scaled negative value, and the scaling factor is the parameter α.

6. A DQN-based mobile robot complex environment path planning method according to claim 4, characterized in that: The mathematical expression of the output layer is: Q value =W final ×ouput4+b final Among them, W final represents the weight matrix of the output layer; b final represents the bias vector of the output layer; ouput4 represents the output of the last hidden layer; the deep Q network calculates the Q value of each action according to the input state.

7. The DQN-based mobile robot complex environment path planning method according to claim 1, characterized in that: The mathematical expression of the comprehensive heuristic reward function is: r＝r d +r e +r h +r s +r t +r o +r ob Among them, r d represents the distance reward and penalty function, r e represents the additional energy consumption penalty function, r h represents the dangerousness penalty function, r s Indicates the penalty for standing still, r t Represents the reward for reaching the target point, r o represents the penalty for exceeding the worst path length and exhausting energy, r ob Indicates the penalty for colliding with an obstacle; The mathematical expression of the distance reward and penalty function is: Among them, λ represents the distance reward and penalty factor, d or Indicates the distance from the starting point to the target point, d n-1 Indicates the distance between the robot's previous position and the target point, d n Indicates the distance between the current robot position and the target point; when (d n-1 -d n ) is positive, it means that the robot is approaching the target point, and this distance reward and penalty function will give a reward; when (d n-1 -d n ) is negative, it means that the robot is moving away from the target point, and the distance reward and penalty function will give a penalty; The mathematical expression of the additional energy consumption penalty function is: Among them, e b Indicates the basic energy consumption of the robot in one operation, E(P n ,P n+1 ) represents the energy consumption between two points, P n+1 Indicates the current position, P n represents the next position, k represents the additional energy penalty factor. When the robot moves uphill, it consumes more energy. (E(P n ,P n+1 )-e b ) is positive, this extra energy consumption penalty function will give a penalty; when the robot moves downhill, it consumes less energy and no penalty is given; The mathematical expression of the dangerousness penalty function is: r h η·H(P n ,P n+1 ) Among them, η represents the risk penalty factor, H(P n ,P n+1 ) indicates the danger from the current location to the next location.

8. The DQN-based mobile robot complex environment path planning method according to claim 7, characterized in that: The energy consumption E(P n ,P n+1 ) is calculated as follows: E(P n ,P n+1 )=e b +c·g(z n+1 -z n ) Where c represents the mass of the robot, g represents the acceleration due to gravity, and z n+1 Indicates the height of the next position, z n Indicates the current location altitude.

9. The DQN-based mobile robot complex environment path planning method according to claim 7, characterized in that: The risk H(P n ,P n+1 ) is calculated as follows: Where b represents the risk parameter, d ob represents the distance between the nearest obstacle and the mobile robot, m represents the slope at which the robot can operate safely, and β(P n ,P n+1 ) represents the terrain slope between two points, θ represents the angle between the direction of the nearest obstacle relative to the robot and the terrain orientation; The terrain slope β(P n ,P n+1 ) is calculated as follows: Among them, x n+1 Indicates the horizontal coordinate of the next position, x n Indicates the horizontal coordinate of the current position, y n+1 Indicates the ordinate of the next position, y n Indicates the current position vertical coordinate; The specific calculation formula for the angle θ between the direction of the nearest obstacle relative to the robot and the terrain orientation is: θ=|O t -O ob | Among them, O t Indicates the terrain orientation, and its specific calculation formula is: Among them, Z represents the height, x represents the horizontal coordinate, and y represents the vertical coordinate; Among them, O ob Indicates the direction of the nearest obstacle relative to the robot. The specific calculation formula is: Among them, x ob Indicates the horizontal coordinate of the obstacle relative to the robot, y ob The vertical coordinate representing the direction of the obstacle relative to the robot.

10. The DQN-based mobile robot complex environment path planning method according to claim 1, characterized in that: The mathematical expression of the state space is: S 2.5d =[x,y,dx,dy,x t ,the t ,d or ,step,d,H,e cs ,e,ob r ,l r ] Among them, x and y represent the position of the robot, dx and dy represent the relative position of the robot and the target point, and x t ,y t represents the location of the target point, d or represents the distance from the starting point to the target point, step represents the number of steps the robot moves, d represents the distance from the robot to the target point, e cs represents the energy consumption of the robot in its current state, e represents the total energy consumption of the robot, and ob r Indicates the obstacles around the robot, l r Represents the elevation information of the terrain around the robot; The specific updating formula of the network parameters is: L=E[(Q local (s,a;θ local )-Q target ) 2 ] Where L is the loss function, which represents the mean square error between the predicted Q value and the target Q value; Q local (s,a;θ local ) represents the Q value prediction of the current deep Q network for the state and action, θ local Represents the parameters of the current deep Q network; Q target Represents the target Q value, and its specific calculation formula is: Q target =r+γ·max a′ Q target (s′,a′;θ target )·(1-ov) Where r represents the comprehensive heuristic reward function, γ represents the discount factor, s represents the current state, s′ represents the next state, ov represents the end flag, a represents the current action, and a′ represents the next action.

Images