Mountain path planning method for quadruped robot based on deep reinforcement learning

Abstract

The invention relates to a quadruped robot mountain path planning method based on deep reinforcement learning, which comprises the following steps: S1, constructing a mountain digital terrain model, dividing the mountain digital terrain model according to a gradient grading principle, and defining a safe area and a dangerous area; s2, introducing a black hole principle to optimize and improve an artificial potential field method, constructing a global slope potential field, and simplifying a high-dimensional complex environment; s3, performing weighting processing on potential energy in the potential field, inputting the potential energy into a deep reinforcement learning network, guiding the quadruped robot to avoid the abrupt slope area and move towards a target point, and improving the training speed of the algorithm; s4, introducing a gradient security domain and a gradient reward function, and re-planning a real-time path; according to the path planning method combining the artificial potential field method and the deep reinforcement learning, the robot can plan an optimal path with a proper gradient and a short distance only according to interaction data of the current and historical environments, and the problem that path planning is difficult due to the fact that the terrain gradient is too large in the complex mountain environment is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot decision-making and planning, and relates to a quadruped robot mountain path planning method based on deep reinforcement learning, and in particular to a quadruped robot mountain path planning method based on potential energy guided reinforcement learning and a quadruped robot mountain path planning system. Background Art

[0002] As a bionic robot, the quadruped robot can easily move quickly on rugged and undulating surfaces with four flexible legs. It is suitable for inspection, exploration, rescue and other tasks in complex terrain environments such as mountains, steep slopes, and caves. In mountainous environments, the terrain is undulating and there are steep slopes that rise sharply. When walking in such dangerous terrain, the robot's own stability and safety are difficult to guarantee. Moving and navigating in mountainous environments places high demands on the path planning method of quadruped robots.

[0003] At present, the path planning algorithms used in quadruped robots are mainly based on traditional methods such as graph search, heuristics and sensors, and mainly focus on avoiding known obstacles and solving the shortest smooth path. The robot perceives the environment, plans paths, and avoids obstacles in real time to overcome the difficulties brought by various terrains and obstacles, ensuring that the robot completes the task safely and efficiently. At the same time, deep reinforcement learning achieves the optimal decision from raw input to output in an end-to-end manner through interactive learning between the intelligent agent and the environment, which can meet the robot's path planning in mountainous environments.

[0004] However, traditional path planning methods are suitable for low-dimensional plane spaces or relatively simple structured environments. In complex terrain environments, the increase in spatial dimensions causes the algorithm to have long planning times and is prone to falling into local optimality. At the same time, the path planning method based on deep reinforcement learning relies only on visual or radar information, and cannot fully observe the steep terrain in mountainous environments, nor does it consider the slope as a factor in the path. There are problems such as slow algorithm convergence, random training actions, lack of early guidance, and high hardware requirements.

[0005] Specifically, traditional methods cannot effectively solve the problem of path planning in steep slopes. One of the important reasons is that the robot cannot accurately define steep slopes with excessive slopes, quickly identify and effectively avoid the dangerous area. Therefore, by combining traditional algorithms with deep reinforcement learning, the interaction efficiency of the terrain environment can be improved and the algorithm convergence speed can be accelerated. The reward function design in deep reinforcement learning can accurately guide the robot to avoid steep slopes and effectively escape from the local optimal point. Summary of the invention

[0006] In view of this, in order to solve the problem that the traditional quadruped robot path planning algorithm in mountainous environment does not take the slope factor into consideration, resulting in difficulty in walking and poor adaptability, the present invention provides a quadruped robot mountain path planning method based on deep reinforcement learning.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A mountain path planning method for a quadruped robot based on deep reinforcement learning, comprising the following steps:

[0009] S1: Construct a mountain digital ground model and divide it according to the slope classification principle to define safe areas and dangerous areas, thereby improving the interaction efficiency between the algorithm and the environment;

[0010] S2: Based on S1, the slope is used as a parameter and the slope potential energy is used to characterize the terrain undulation in combination with the improved artificial potential field (APF) algorithm. The black hole principle is introduced to further optimize and improve the artificial potential field method, and a slope potential field that can comprehensively reflect the overall slope of the terrain is constructed to simplify the high-dimensional complex environment.

[0011] S3: By weighting the potential energy probability of the slope potential field in S2 and inputting it into the deep reinforcement learning network, the quadruped robot is guided to avoid the steep slope area and move to the target point, thereby improving the training speed of the algorithm;

[0012] S4: Based on the unique walking characteristics of the quadruped robot, a slope safety domain is introduced and the walking slope of the quadruped robot is estimated and optimized in real time. The quadruped robot is trained through a reward function and the real-time path is replanned.

[0013] Furthermore, the construction of the mountain digital terrain model in step S1 is specifically as follows: a series of ground point coordinates and surface attributes (elevation, slope, slope morphology, etc.) are used to form a data array, and a digital elevation map is generated by constructing plane data and elevation data; terrain analysis and slope aspect calculation are performed, and after optimizing and adjusting the initially established mountain model, the complete mountain digital terrain model is finally imported into the simulation environment in the URDF model file format.

[0014] Furthermore, the slope classification principle in step S1 is specifically as follows: based on the density of contour lines, the denser the contour lines, the greater the terrain slope; the terrain with a slope of 25°≤i≤50° is defined as steep slope terrain, and the terrain with a slope i>50° is defined as steep slope terrain; the mountainous environment is divided into dangerous areas with steep slopes and safe areas with gentle slopes suitable for walking, and the distribution of steep slope terrain is represented by a contour map.

[0015] Further, in step S2, the black hole principle is specifically as follows:

[0016] A black hole domain is established within the radius R1 of the target point, and the potential energy in the black hole domain is much lower than that of other points; based on the distribution of the steep slope terrain, the radius r0 of the circle is regarded as a no-go zone with the point with the maximum slope as the center, and a repulsive potential field is established within the no-go zone, and the potential energy in the potential field is much higher than that in other areas;

[0017] The repulsive potential function of the forbidden area is:

[0018]

[0019] In the formula, k gra is the forbidden zone coefficient; (q-s0) is the distance between the robot's current position q and the point with the maximum slope s0 in the steep slope (m); R0 is the radius of the forbidden zone (m);

[0020] The quadruped robot is in the black hole domain and will be guided to the target point by the rapidly decreasing potential energy. The potential function formula of the black hole domain is:

[0021]

[0022] In the formula, k str is the black hole coefficient; |qq g | is the distance between the current position and the target point; R1 is the radius of the black hole domain;

[0023] Specifically, in order to ensure that the robot quickly enters the black hole potential field and avoids the steep slope prohibited area, the black hole coefficient k str The value should be much higher than the gravitational coefficient k att , and is less than the no-drive zone coefficient k gra ; Finally, the mountain environment is simplified into a two-dimensional terrain distribution map with a starting point, an end point and 8 steep slopes; the slope potential energy distribution map includes the slope repulsive potential field and the black hole domain gravitational potential field; the quadruped robot avoids the steep slope restricted area according to the decreasing trend of potential energy and moves quickly towards the target lowest potential energy point.

[0024] Further, in step S3, the action of the quadruped robot is guided and predicted according to the global potential energy information, and the slope potential field potential energy is weighted and used as the state S g Input to the Actor-Critic training network to finally make action decisions and convert the slope potential energy of each point into a probability series p i , in the probability series, the probability of the lowest potential energy point is the largest, while the probability of the highest potential energy point is the smallest;

[0025]

[0026] Where U(i) is the potential energy of each point; i is the path point in the target direction, i = (1, 2, ..., k).

[0027] Weight the slope potential energy probability series, the potential energy weighting function σ(p) i , after weighting, the point with the lowest slope potential energy accounts for the largest proportion among all weighted probabilities; the continuous path with the highest accumulation probability is obtained by calculating the slope potential energy weighting function, and this direction is the fastest descending gradient of the slope potential energy;

[0028]

[0029] In the formula, p i is the slope potential energy probability sequence; k is the capacity of the probability sequence.

[0030] Furthermore, in step S3, the deep reinforcement learning networks are respectively a policy network and an evaluation network, wherein the policy network processes a large number of continuous high-dimensional states and weighted slope potential energy, and selects actions according to the states; the evaluation network is updated by the reward function and the experience pool, and is responsible for judging the current strategy and then updating the policy network.

[0031] Furthermore, in step S3, both the policy network and the evaluation network are hidden layers composed of fully connected networks, the network input layer is a state space sequence, the first layer has 256 nodes, and the activation function is Relu; the second layer has 128 nodes, the activation function is tanh, and the policy network output layer is an action space; considering the deviation between the simulation training environment and the real environment, Ornstein-Uhlenbeck noise is added to generate inertial time series when outputting the action strategy. Beneficial effects: Improve the exploration efficiency of the quadruped robot in the training environment, and further learn a better path planning strategy.

[0032] Further, in step S4, based on the three-dimensional ground contact positions of the four legs of the robot in the world coordinate system, a standing plane is formed by four points, and the plane estimation formula is:

[0033]

[0034] Where b0, b1, b2 are the plane formula coefficients, and matrices A and Z are:

[0035]

[0036] According to the above terrain plane formula, the estimated ground slope θ can be calculated.

[0037]

[0038] Furthermore, in step S4, the slope safety region θ=(21°, 25°). Beneficial effect: The robot is prevented from tipping over due to steep slopes, while the robot's climbing ability under a certain slope is brought into play.

[0039] Furthermore, the reward function in step S4 includes potential energy reward, slope reward and other sparse rewards. Specifically, in the slope potential field, the quadruped robot moves quickly from a high potential energy starting point to a target point with zero potential energy according to the decreasing potential energy gradient. The potential energy reward function r1 is designed as follows:

[0040] r1=-β(U t -U t-1 ) (8)

[0041] Where U t , U t-1 is the potential energy of the robot at time t and t-1; β is the potential energy reward coefficient.

[0042] Furthermore, after each action of the strategy, the slope reward function evaluates the estimated slope size of the quadruped robot's location, punishes the behavior that exceeds the safe slope range, and prevents the robot from further falling into the steep slope terrain; at the same time, rewards the behavior within the slope safety domain, thereby saving the total path length. The slope reward function r2 is designed as follows:

[0043]

[0044] Where: θ is the estimated slope of the robot at the current position; θ min is the minimum value of the slope safety region of the quadruped robot; θ max is the maximum value of the slope safety domain of the quadruped robot; W1 and W2 are fixed reward values ​​and fixed penalty values.

[0045] Furthermore, a series of sparse rewards are included: when the robot reaches the target point, a fixed reward r is given g ; When the robot capsizes or moves out of the boundary, a fixed penalty W3 is given; Set the time reward r time , enabling the robot to reach the target point in a shorter time.

[0046] Furthermore, the mountain path planning process of the quadruped robot based on deep reinforcement learning consists of four parts: environment generation, slope potential energy and current position slope calculation, action execution and network update; when the training termination condition is reached or the number of training rounds n=n max , the accumulated rewards of this round are counted and the status is updated, and the training environment and robot position information are reinitialized.

[0047] The beneficial effects of the present invention are:

[0048] 1. The mountain path planning method of a quadruped robot based on deep reinforcement learning disclosed in the present invention has the following advantages:

[0049] (1) The mountain digital terrain model is divided according to the slope classification principle. This classification uses the slope classification principle to effectively divide the terrain into manageable parts. By introducing the "black hole" principle into the artificial potential field (APF) algorithm, it is improved to enhance its function. The introduction of the black hole principle helps to optimize the APF algorithm and reduce the local minimum problem, thereby traversing complex terrain more smoothly.

[0050] (2) When constructing the global slope potential field, the potential field comprehensively reflects the overall slope characteristics of the terrain. This method aims to reduce the computational complexity of navigation in a multidimensional environment. As a comprehensive representation of the terrain slope conditions, the global slope potential field is more effective for path planning and steep slope avoidance.

[0051] (3) The potential energy in the constructed potential field is processed by probability weighting and then input into the deep reinforcement learning network. This integration guides the reinforcement learning process by using the preprocessed terrain data, accelerates its convergence speed, and thus improves the overall efficiency of the algorithm.

[0052] (4) Considering the unique walking characteristics of quadruped robots, a slope optimization method is proposed. The method involves defining a safe slope threshold range and using a slope reward function to adjust and optimize the robot's state. By setting a safety threshold based on the optimized slope range, the algorithm ensures that the quadruped robot remains stable and efficient in challenging terrain.

[0053] 2. The mountain path planning method for quadruped robots based on deep reinforcement learning disclosed in the present invention constructs a slope potential field that characterizes the undulation of the mountain, and weights the potential energy and inputs it into the reinforcement learning network to guide the early training. A slope safety domain is set in the training network, and a slope reward function is proposed to reward and punish the state, so as to achieve effective optimization of the path slope in the middle and late stages of training. The improved slope potential field method can effectively process high-dimensional terrain information and simplify the complexity of path planning problems in undulating terrain. The slope optimization method can effectively improve and re-plan local paths, guide quadruped robots to avoid steep slopes, and find the optimal path within a safe slope range.

[0054] 3. The mountain path planning method for a quadruped robot based on deep reinforcement learning disclosed in the present invention can process slope information in complex terrain, integrate the artificial potential field method and guide the forward movement of the quadruped robot through a deep reinforcement learning network, and at the same time propose a slope safety domain to prevent the quadruped robot from entering terrain with a larger slope.

[0055] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0057] Figure 1 It is an overall flow chart of the mountain path planning method of a quadruped robot based on deep reinforcement learning of the present invention;

[0058] Figure 2 The overall framework diagram of the algorithm of the mountain path planning method of the quadruped robot based on deep reinforcement learning of the present invention;

[0059] Figure 3 This is a distribution map of steep slope terrain potential in the mountain path planning method of a quadruped robot based on deep reinforcement learning of the present invention;

[0060] Figure 4 Schematic diagram of steep slope terrain division in the mountain path planning method of a quadruped robot based on deep reinforcement learning of the present invention. DETAILED DESCRIPTION

[0061] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0062] Mountain path planning based on potential energy guided reinforcement learning mainly consists of two parts: slope potential energy module and Actor-Critic training network. The slope potential energy module establishes a global slope potential field based on the divided terrain distribution map and weights the slope potential energy probability; the Actor-Critic training network mainly processes the input weighted potential energy, perceived slope, relative position and other states, outputs action strategies and updates the network, and finally the robot motion control system executes the action instructions to complete the path planning.

[0063] Refer to Figure 1-Figure 4 A mountain path planning method for a quadruped robot based on deep reinforcement learning is shown, comprising the following steps:

[0064] S1: The mountain model is divided according to the slope classification principle, which specifically includes constructing a mountain digital ground model and using the slope as a parameter, combined with the improved APF algorithm to use the slope potential to characterize the terrain undulation.

[0065] To build a mountain digital ground model, a series of ground point coordinates and surface attributes (elevation, slope, slope morphology, etc.) are used to form a data array, and a digital elevation map is generated by constructing plane data and elevation data. Terrain analysis and slope aspect calculation are performed, and after optimizing and adjusting the initially established mountain model, the complete mountain digital ground model is finally imported into the simulation environment in the URDF model file format.

[0066] Reference Figure 4 The steep slope terrain division map in the mountain environment shown is a mountain environment established based on the DTM method. The digital ground model has a length of L = 25m, a width of D = 20m, and a maximum altitude of H max =13.5m. The contour map can more accurately describe the terrain features. The digital ground model is a mountain environment consisting of three mountain tops and two valleys composed of multiple undulating slopes.

[0067] Figure 3 The energy distribution map of steep slope terrain in the mountain path planning method shown in the figure; according to the slope classification principle: the more dense the contour lines are, the greater the terrain slope is, and the terrain with a slope of 25°≤i≤50° is defined as steep slope terrain, and the terrain with a slope i>50° is defined as steep slope terrain. The following formula is the terrain slope calculation formula, which can obtain the terrain slope size of the area with dense contour lines in the figure.

[0068]

[0069] Where i is the slope; ΔH is the elevation difference of the terrain; and ΔL is the horizontal distance difference of the terrain.

[0070] According to the slope calculation formula, the areas shown in the circle in the figure are all steep slopes with a slope of 25°≤i≤50°, and the slopes diverge from the center to the outside and gradually become gentler. Therefore, the mountain environment is divided into dangerous areas with steep slopes and safe areas with gentle slopes suitable for walking, and the distribution of steep slopes is represented by contour maps.

[0071] S2: Based on S1, the black hole principle is introduced to construct a slope potential field that can comprehensively reflect the overall slope of the terrain.

[0072] The black hole principle is as follows: a black hole domain is established within the radius R1 of the target point, and the potential energy in the black hole domain is much lower than that of other points; based on the distribution of the steep slope terrain, the radius r0 of the circle is regarded as a no-go zone with the point with the maximum slope as the center, and a repulsive potential field is established within the no-go zone, and the potential energy in the potential field is much higher than that in other areas;

[0073] The repulsive potential function of the forbidden area is:

[0074]

[0075] In the formula, k gra is the forbidden zone coefficient; (q-s0) is the distance between the robot's current position q and the point with the maximum slope s0 in the steep slope (m); R0 is the radius of the forbidden zone (m);

[0076] The quadruped robot is in the black hole domain and will be guided to the target point by the rapidly decreasing potential energy. The potential function formula of the black hole domain is:

[0077]

[0078] In the formula, k str is the black hole coefficient; |qq g | is the distance between the current position and the target point; R1 is the radius of the black hole domain;

[0079] Specifically, in order to ensure that the robot quickly enters the black hole potential field and avoids the steep slope prohibited area, the black hole coefficient k str The value should be much higher than the gravitational coefficient k att , and is less than the no-drive zone coefficient k gra ; Finally, the mountain environment is simplified into a two-dimensional terrain distribution map with a starting point, an end point and 8 steep slopes; the slope potential energy distribution map includes the slope repulsive potential field and the black hole domain gravitational potential field; the quadruped robot avoids the steep slope restricted area according to the decreasing trend of potential energy and moves quickly towards the target lowest potential energy point.

[0080] S3: Through the potential energy probability weighted processing of the slope potential field in S2, it is input into the deep reinforcement learning network.

[0081] According to the global potential energy information, the action of the quadruped robot is guided and predicted, and the slope potential field potential energy is weighted and used as the state S g Input to the Actor-Critic training network to finally realize action decision. First, the slope potential energy of each point is converted into the slope potential energy probability series p i , in the slope potential energy probability series, the probability of the lowest potential energy point is the largest, while the probability of the highest potential energy point is the smallest;

[0082]

[0083] Where U(i) is the potential energy of each point; i is the path point in the target direction, i = (1, 2, ..., k).

[0084] Weight the slope potential energy probability series, the potential energy weighting function σ(p) i , after weighting, the point with the lowest slope potential energy accounts for the largest proportion among all weighted probabilities; the continuous path with the highest accumulation probability is obtained by calculating the slope potential energy weighting function, and this direction is the fastest descending gradient of the slope potential energy;

[0085]

[0086] In the formula, p i is the slope potential energy probability sequence; k is the capacity of the probability sequence.

[0087] The deep reinforcement learning networks are divided into a policy network and an evaluation network. The policy network processes a large number of continuous high-dimensional states and weighted slope potential energy, and selects actions according to the state; the evaluation network is updated by the reward function and the experience pool, and is responsible for judging the current strategy and then updating the policy network. The network input layer is a state space sequence. Both the policy network and the evaluation network are hidden layers composed of fully connected networks. The first layer has 256 nodes and the activation function is Relu; the second layer has 128 nodes and the activation function is tanh. The output layer of the policy network is the action space. Considering the deviation between the simulation training environment and the real environment, Ornstein-Uhlenbeck noise is added to generate inertial time series when outputting the action strategy, which improves the exploration efficiency of the quadruped robot in the training environment, so as to further learn a better path planning strategy.

[0088] S4: Based on the unique walking characteristics of the quadruped robot, the walking slope of the quadruped robot is estimated and optimized in real time, and the quadruped robot is trained through the reward function.

[0089] In undulating terrain, the attitude angle of the quadruped robot is inconsistent with the slope angle, and the pitch angle obtained by the IMU cannot be directly converted to It is approximately equivalent to the slope of the slope. Therefore, based on the three-dimensional ground contact positions of the robot's four legs in the world coordinate system, the standing plane is composed of four points, and the plane estimation formula is:

[0090]

[0091] Where b0, b1, b2 are the plane formula coefficients, and matrices A and Z are:

[0092]

[0093] According to the above terrain plane formula, the estimated ground slope θ can be calculated.

[0094]

[0095] In the related research of the model predictive control (MPC) method of quadruped robots, the robot can move safely on a slope with a slope of more than 21° using a trotting gait while maintaining a low energy consumption level. Therefore, this chapter sets the slope safety domain θ = (21°, 25°) to prevent the robot from overturning due to steep slopes, while giving full play to the robot's climbing ability under a certain slope.

[0096] The reward function includes potential energy reward, slope reward and other sparse rewards. Specifically, in the slope potential field, the quadruped robot moves quickly from the high potential energy starting point to the target point with zero potential energy according to the decreasing potential energy gradient. The potential energy reward function r1 is designed as:

[0097] r1=-β(U t -U t-1 ) (8)

[0098] Where U t , U t-1 is the potential energy of the robot at time t and t-1; β is the potential energy reward coefficient.

[0099] After each action of the strategy, the slope reward function evaluates the estimated slope size of the quadruped robot's location, punishes the behavior that exceeds the safe slope range, and prevents the robot from further falling into the steep terrain; at the same time, rewards the behavior within the slope safety domain, thereby saving the total path length. The slope reward function r2 is designed as follows:

[0100]

[0101] Where: θ is the estimated slope of the robot at the current position; θ min is the minimum value of the slope safety region of the quadruped robot; θ max is the maximum value of the slope safety domain of the quadruped robot; W1 and W2 are fixed reward values ​​and fixed penalty values.

[0102] In addition to the above two main rewards, a series of sparse rewards are also set: when the robot reaches the target point, a fixed reward r is given g ; When the robot capsizes or moves out of the boundary, a fixed penalty W3 is given; Set the time reward r time , enabling the robot to reach the target point in a shorter time.

[0103] The mountain path planning process of a quadruped robot based on deep reinforcement learning consists of four parts: environment generation, slope potential energy and current position slope calculation, action execution, and network update. When the training termination condition is reached or the number of training rounds n=n max , the accumulated rewards of this round are counted and the status is updated, and the training environment and robot position information are reinitialized.

[0104] This quadruped robot mountain path planning method based on deep reinforcement learning can process slope information in complex terrain, integrate the artificial potential field method and guide the quadruped robot forward movement through a deep reinforcement learning network. At the same time, it proposes a slope safety domain to prevent the quadruped robot from entering terrain with a larger slope.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A mountain path planning method for a quadruped robot based on deep reinforcement learning, characterized in that: The following steps are involved: S1: Construct a mountain digital ground model and divide it according to the slope classification principle to define safe areas and dangerous areas, thereby improving the interaction efficiency between the algorithm and the environment; S2: Based on S1, the slope is used as a parameter and the slope potential is used to characterize the terrain undulation in combination with the improved artificial potential field (APF) algorithm; The black hole principle is introduced to further optimize and improve the artificial potential field method, construct a slope potential field that can comprehensively reflect the overall slope of the terrain, and simplify the high-dimensional complex environment. S3: By weighting the potential energy probability of the slope potential field in S2 and inputting it into the deep reinforcement learning network, the quadruped robot is guided to avoid the steep slope area and move to the target point, thereby improving the training speed of the algorithm; S4: Based on the unique walking characteristics of the quadruped robot, a slope safety domain is introduced and the walking slope of the quadruped robot is estimated and optimized in real time. The quadruped robot is trained through a reward function and the real-time path is replanned.

2. The mountain path planning method for a quadruped robot based on deep reinforcement learning as claimed in claim 1, characterized in that: The step S1 of constructing the mountain digital terrain model specifically comprises: using a series of ground point coordinates and surface attributes to form a data array, and generating a digital elevation map by constructing plane data and elevation data; performing terrain analysis and slope aspect calculation, and finally importing the complete mountain digital terrain model into the simulation environment in the URDF model file format after optimizing and adjusting the initially established mountain model; the slope classification principle is specifically as follows: based on the density of contour lines, the terrain with a slope of 25°≤i≤50° is defined as a steep slope terrain, and the terrain with a slope i>50° is defined as a steep steep slope terrain; the mountain environment is divided into a dangerous area with steep slopes and a safe area with a gentle slope suitable for walking, and the distribution of the steep slope terrain is represented by a contour map.

3. The mountain path planning method for a quadruped robot based on deep reinforcement learning as claimed in claim 1, characterized in that: In step S2, the black hole principle is specifically: A black hole domain is established within the radius R1 of the target point, and the potential energy in the black hole domain is much lower than that of other points; based on the distribution of the steep slope terrain, the radius r0 of the circle is regarded as a no-go zone with the point with the maximum slope as the center, and a repulsive potential field is established within the no-go zone, and the potential energy in the potential field is much higher than that in other areas; The repulsive potential function of the forbidden area is: In the formula, k gra is the forbidden zone coefficient; (q-s0) is the distance between the robot's current position q and the point with the maximum slope s0 in the steep slope (m); R0 is the radius of the forbidden zone (m); The quadruped robot is in the black hole domain and will be guided to the target point by the rapidly decreasing potential energy. The potential function formula of the black hole domain is: In the formula, k str is the black hole coefficient; |qq g | is the distance between the current position and the target point; R1 is the radius of the black hole domain; In order to ensure that the robot quickly enters the black hole potential field and avoids the steep slope prohibited area, the black hole coefficient k str The value should be much higher than the gravitational coefficient k att , and is less than the no-drive zone coefficient k gra ; The mountain environment is simplified into a two-dimensional terrain distribution map with a starting point, an end point and steep slope terrain; the slope potential energy distribution map includes the slope repulsive potential field and the black hole domain gravitational potential field; the quadruped robot avoids the steep slope restricted area according to the decreasing trend of potential energy and moves quickly towards the target potential energy lowest point.

4. The mountain path planning method for a quadruped robot based on deep reinforcement learning as claimed in claim 1, characterized in that: In step S3, the action of the quadruped robot is guided and predicted according to the global potential energy information, and the weighted processing of the slope potential field potential energy is performed and used as the state S g Input to the Actor-Critic training network to finally make action decisions and convert the slope potential energy of each point into a probability series p i , in the probability series, the probability of the lowest potential energy point is the largest, while the probability of the highest potential energy point is the smallest; Where U(i) is the potential energy of each point; i is the path point in the target direction, i = (1, 2, ..., k); Weight the slope potential energy probability series, the potential energy weighting function σ(p) i , after weighting, the point with the lowest slope potential energy accounts for the largest proportion among all weighted probabilities; the continuous path with the highest accumulation probability is obtained by calculating the slope potential energy weighting function, and this direction is the fastest descending gradient of the slope potential energy; In the formula, p i is the slope potential energy probability sequence; k is the capacity of the probability sequence.

5. The mountain path planning method for a quadruped robot based on deep reinforcement learning as claimed in claim 4, characterized in that: The deep reinforcement learning networks are respectively a policy network and an evaluation network, wherein the policy network processes a large number of continuous high-dimensional states and weighted slope potential energy, and selects actions according to the states; the evaluation network is updated by the reward function and the experience pool, and is responsible for judging the current strategy and then updating the policy network.

6. The mountain path planning method for a quadruped robot based on deep reinforcement learning as claimed in claim 1, characterized in that: In step S4, based on the three-dimensional ground contact positions of the four legs of the robot in the world coordinate system, a standing plane is formed by four points, and the plane estimation formula is: Where b0, b1, b2 are the plane formula coefficients, and matrices A and Z are: According to the solved terrain plane formula, the estimated ground slope θ can be calculated.

7. The mountain path planning method for a quadruped robot based on deep reinforcement learning as claimed in claim 6, characterized in that: The slope safety zone θ=(21°, 25°).

8. The method for mountain path planning of a quadruped robot based on deep reinforcement learning as claimed in claim 7, characterized in that: The reward function in step S4 includes potential energy reward, slope reward and sparse reward. Specifically, in the slope potential field, the quadruped robot moves quickly from a high potential energy starting point to a target point with zero potential energy according to the decreasing potential energy gradient. The potential energy reward function r1 is designed as follows: r1=-β(U t -IN t-1 ) (8) Where U t , U t-1 is the potential energy of the robot at time t and t-1; β is the potential energy reward coefficient; After each action of the strategy, the slope reward function evaluates the estimated slope size of the quadruped robot's location, punishes the behavior that exceeds the safe slope range, and prevents the robot from further falling into the steep terrain; at the same time, rewards the behavior within the slope safety domain, thereby saving the total path length. The slope reward function r2 is designed as follows: Where: θ is the estimated slope of the robot at the current position; θ min is the minimum value of the slope safety region of the quadruped robot; θ max is the maximum value of the slope safety domain of the quadruped robot; W1 and W2 are fixed reward values ​​and fixed penalty values.

9. The method for mountain path planning of a quadruped robot based on deep reinforcement learning as claimed in claim 8, characterized in that: The mountain path planning process of the quadruped robot based on deep reinforcement learning consists of four parts: environment generation, slope potential energy and current position slope calculation, action execution and network update. When the training termination condition is reached or the number of training rounds n=n max , the accumulated rewards of this round are counted and the status is updated, and the training environment and robot position information are reinitialized.

Images