A foot-ground interaction trajectory planning method and system

Abstract

The application discloses a foot-ground interaction trajectory planning method and system, relates to the technical field of foot robot trajectory planning, and comprises the following steps: constructing a foot-ground interaction expert trajectory library based on a DMPs imitation learning method; adopting a residual reinforcement learning method to perform trajectory self-adaptive correction on foot-ground interaction expert trajectories in the foot-ground interaction expert trajectory library; utilizing an adaptive trajectory iteration selection mechanism based on interaction feedback to dynamically select an optimal reference trajectory from the foot-ground interaction expert trajectory library; constructing a multi-terrain contact foot robot simulation environment, and performing simulation test verification on the trajectory self-adaptive correction process and the process of dynamically selecting the optimal reference trajectory to obtain simulation test verification results. The application can improve the accuracy of foot-ground interaction trajectory planning and improve the stability of foot-ground interaction of a foot robot in complex terrain.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of foot-type robot trajectory planning, in particular to a foot-ground interaction trajectory planning method and system. BACKGROUND

[0002] With the development of mobile foot-type robot technology, foot-type robots have gradually become an important platform for work in complex terrain environments due to their excellent obstacle crossing ability and adaptability to unstructured terrain. In practical applications, the tasks faced by foot-type robots often require them to move stably in terrain with mixed physical properties (such as alternating soft and hard, uneven, and variable friction). Therefore, how to plan a feasible, stable, and efficient foot-ground interaction trajectory while ensuring dynamic stability and obstacle avoidance ability has become an important direction of current research.

[0003] Traditional foot-type robot path planning methods include graph search algorithms, sampling methods, and reinforcement learning-based strategy search methods. These methods mainly address path generation problems for continuous rigid body models, and their unified goal is to find a collision-free path. However, for foot-type robots, which rely on discrete foot contact for movement, traditional path planning techniques have obvious shortcomings when directly applied.

[0004] On the one hand, if the entire foot-type robot is considered as a rigid body for path planning, although collision detection can be performed on the entire structure, overly conservative planning results may occur, and many paths that can be navigated by adjusting leg movements to avoid obstacles may be incorrectly judged as unreachable. On the other hand, if only the collision-free path of the main body is considered without considering the interaction between the legs and the terrain, the legs may not be able to complete the foot placement under physical constraints, ultimately making the path infeasible. Therefore, foot-type robots must consider multiple dimensional constraint conditions such as body posture, gait sequence, foot placement position, and environmental geometry and physical properties during path planning.

[0005] Existing technologies attempt to compensate for the shortcomings of traditional methods in foot-type robot planning from different angles. For example, Tonneau et al. considered the scenario where the body does not collide but the legs may come into contact during path planning, proposed that the leg contact space be a necessary condition for path feasibility, and embedded this criterion into the RRT planning framework. Although this method has enhanced the physical reality of planning to some extent, it has not solved the problem of whether all points on the path can be supported by a stable contact sequence, and therefore cannot guarantee the executability of the overall path.

[0006] In the aspect of trajectory planning, the research work is usually divided into two sub-problems of body trajectory planning and foot trajectory planning. The main goal of body trajectory planning is to generate a center of mass trajectory from a given contact sequence, satisfying the mechanical equilibrium and contact constraints. To simplify the complex whole-body dynamics solving, existing methods generally introduce approximate models such as inverted pendulum model (LIPM), spring-mass model (SLIP), center of mass dynamic model or zero moment point model (ZMP) to realize the rapid solving of trajectory. However, such models have limited modeling capability for terrain features, and it is difficult to handle unstructured terrain with high and low relief and mixed materials.

[0007] On the other hand, foot trajectory planning usually adopts geometric obstacle avoidance methods, such as regarding foot motion as a point path planning problem and applying general path planning algorithms such as A* algorithm, RRT (rapidly-exploring random tree) and the like. These methods can avoid collision between the landing point and obstacles to a certain extent, but generally do not consider dynamic constraints and landing stability. Some research introduces gradient-based optimization methods to model discrete foot trajectory key points as a unified optimization problem, and realizes trajectory smoothing and obstacle avoidance optimization by minimizing the cost function, but its adaptability to terrain changes is limited, and the effect is not good when dealing with mixed mechanical conditions.

[0008] In summary, although the existing technology attempts to consider the dynamic stability and environmental constraints of the legged robot from the path layer and the trajectory layer respectively, when facing complex terrain with mixed mechanical characteristics (such as soft and hard, high and low changes, and uncertain contact), there is still a lack of systematic, unified, and generalizable foot-ground interaction trajectory planning scheme. A general trajectory planning framework that considers the discrete contact mechanism of the legged robot, the change of the physical properties of the terrain, and the dynamic execution feasibility, ultimately resulting in low accuracy of foot-ground interaction trajectory planning and poor stability of foot-ground interaction of the legged robot in complex terrain. SUMMARY

[0009] The purpose of the present application is to provide a foot-ground interaction trajectory planning method and system, which can improve the accuracy of foot-ground interaction trajectory planning and improve the stability of foot-ground interaction of the legged robot in complex terrain.

[0010] To achieve the above-mentioned purpose, the present application provides the following scheme.

[0011] In a first aspect, the present application provides a foot-ground interaction trajectory planning method, which comprises the following steps.

[0012] S1: Based on the DMPs imitation learning method, a foot-ground interaction expert trajectory library is constructed; the foot-ground interaction expert trajectory library includes foot-ground interaction expert trajectories adapted to multiple different terrains.

[0013] S2: adopting a residual reinforcement learning method to perform trajectory adaptive correction on the foot-ground interaction expert trajectory in the foot-ground interaction expert trajectory library, to obtain a corrected foot-ground interaction expert trajectory.

[0014] S3: according to the corrected foot-ground interaction expert trajectory, utilizing an adaptive trajectory iterative selection mechanism based on interaction feedback to dynamically select an optimal reference trajectory from the foot-ground interaction expert trajectory library, to obtain an updated reference trajectory.

[0015] S4: constructing a multi-terrain contact foot-type robot simulation environment.

[0016] S5: based on the multi-terrain contact foot-type robot simulation environment, the updated reference trajectory and the corrected foot-ground interaction expert trajectory, simulating and testing the trajectory adaptive correction process of step S2 and the process of dynamically selecting the optimal reference trajectory of step S3 to obtain a simulation test verification result; the simulation test verification result includes foot-ground interaction task success rate, trajectory compliance degree and training efficiency verification result.

[0017] In a second aspect, the present application provides a foot-ground interaction trajectory planning system, which is used to implement the foot-ground interaction trajectory planning method of the first aspect, and includes the following modules.

[0018] A trajectory library construction module is configured to construct a foot-ground interaction expert trajectory library based on a DMPs imitation learning method; the foot-ground interaction expert trajectory library includes foot-ground interaction expert trajectories adapted to multiple different terrains.

[0019] A trajectory adaptive correction module is configured to adopt a residual reinforcement learning method to perform trajectory adaptive correction on the foot-ground interaction expert trajectory in the foot-ground interaction expert trajectory library, to obtain a corrected foot-ground interaction expert trajectory.

[0020] A trajectory iterative selection module is configured to, according to the corrected foot-ground interaction expert trajectory, utilize an adaptive trajectory iterative selection mechanism based on interaction feedback to dynamically select an optimal reference trajectory from the foot-ground interaction expert trajectory library, to obtain an updated reference trajectory.

[0021] A simulation environment construction module is configured to construct a multi-terrain contact foot-type robot simulation environment.

[0022] The simulation test verification module is configured to perform simulation test verification on the trajectory self-adaptive correction process and the process of dynamically selecting an optimal reference trajectory based on the multi-terrain contact foot-type robot simulation environment, the updated reference trajectory, and the corrected foot-ground interaction expert trajectory, and obtain a simulation test verification result. The simulation test verification result includes a foot-ground interaction task success rate, a trajectory compliance degree, and a training efficiency verification result.

[0023] According to the specific embodiments provided in the present application, the present application has the following technical effects.

[0024] The present application provides a foot-ground interaction trajectory planning method and system. The advantages of the method are as follows: (1) improving multi-terrain adaptation capability: by constructing an expert library containing multiple terrain adaptation trajectories in step S1 and combining step S2 residual reinforcement learning correction, the foot-ground interaction trajectory can flexibly adapt to different terrain characteristics, and the problem of a single trajectory being difficult to cope with complex terrain can be solved. (2) enhancing trajectory self-adaptive precision: by using the residual reinforcement learning method in step S2 to make targeted corrections based on the expert trajectory, and by using the interactive feedback dynamic optimization in step S3, the trajectory can be accurately adjusted according to the real-time interaction state, thereby improving the matching degree of the trajectory and the actual terrain. (3) ensuring interaction safety and stability: the corrected trajectory and the dynamically selected optimal reference trajectory can reflect their motion smoothness through the verified trajectory compliance degree, reducing the foot-end impact or body instability caused by trajectory mutation, thereby improving the safety of the interaction process. (4) improving task execution reliability: the foot-ground interaction task success rate verified in step S5 directly reflects the trajectory planning effect. By combining the expert library and the dynamic correction and optimization mechanism, the success rate of the foot-ground interaction task of the foot-type robot in the multi-terrain scene is improved, and the stability of the foot-ground interaction of the foot-type robot in the complex terrain is improved. (5) optimizing algorithm training efficiency: step S5 explicitly takes the training efficiency as a verification result, indicating that by combining residual reinforcement learning with an expert trajectory library, the efficiency of the algorithm training process can be improved while ensuring the performance of the trajectory, and the training resource consumption can be reduced. (6) supporting algorithm effectiveness verification: the multi-terrain simulation environment constructed in step S4 can provide a high-fidelity verification platform for the test in step S5, thereby ensuring the reliability of the evaluation results of the trajectory correction and optimization process, and providing a basis for the practical application of the algorithm. The accuracy of the foot-ground interaction trajectory planning is improved, and the stability of the foot-ground interaction of the foot-type robot in the complex terrain is improved. BRIEF DESCRIPTION OF DRAWINGS

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart illustrating a foot-ground interaction trajectory planning method provided in one embodiment of this application.

[0027] Figure 2 This is a schematic diagram illustrating the motion planning and control principle of a legged robot under the influence of foot-ground interaction, as provided in an embodiment of this application.

[0028] Figure 3 This is a schematic diagram of a dynamic motion primitive system provided in an embodiment of this application.

[0029] Figure 4 This is a schematic diagram of different foot-ground interaction trajectories set by an expert, provided as an embodiment of this application. Figure 4 (a) in the diagram is a schematic diagram of the foot-ground interaction trajectory of trajectory type 1. Figure 4 (b) in the diagram is a schematic diagram of the foot-ground interaction trajectory of trajectory type 2. Figure 4 (c) in the figure is a schematic diagram of the foot-ground interaction trajectory of trajectory type 3.

[0030] Figure 5 This is a schematic diagram of DMPs trajectory modification provided in one embodiment of this application. Figure 5 (a) in the diagram is a schematic diagram of the random interference trajectory of DMPs. Figure 5 (b) in the diagram is a schematic diagram of another DMP random interference trajectory.

[0031] Figure 6 This is a schematic diagram illustrating the foot-ground interaction after selecting different trajectories, provided in one embodiment of this application. Figure 6 (a) in the figure is a schematic diagram of the iterative trajectory under mixed soft and hard terrain. Figure 6 (b) in the figure is a schematic diagram of the iterative trajectory under soft terrain.

[0032] Figure 7 This is a schematic diagram of the foot-ground interaction trajectory planning algorithm provided in an embodiment of this application.

[0033] Figure 8 This is a schematic diagram of the normal foot-ground contact process provided in an embodiment of this application.

[0034] Figure 9 This is a schematic diagram of the tangential foot-ground contact process provided in an embodiment of this application.

[0035] Figure 10A schematic diagram of a random foot-ground contact force field construction process provided by an embodiment of the present application.

[0036] Figure 11 A schematic diagram of a foot-ground contact simulation process provided by an embodiment of the present application.

[0037] Figure 12 A schematic diagram of an expert trajectory and a training reward curve provided by an embodiment of the present application, Figure 12 (a) in the above is a schematic diagram of an expert trajectory, Figure 12 (b) in the above is a schematic diagram of a training reward curve.

[0038] Figure 13 A sample diagram of successfully finding a feasible support region provided by an embodiment of the present application, Figure 13 (a) in the above is a schematic diagram of a successful case 1 of successfully finding a feasible support region, Figure 13 (b) in the above is a schematic diagram of a successful case 2 of successfully finding a feasible support region, Figure 13 (c) in the above is a schematic diagram of a successful case 3 of successfully finding a feasible support region, Figure 13 (d) in the above is a schematic diagram of a successful case 4 of successfully finding a feasible support region.

[0039] Figure 14 A schematic diagram of failing to find a feasible support region provided by an embodiment of the present application, Figure 14 (a) in the above is a schematic diagram of a failed case 1 of failing to find a feasible support region, Figure 14 (b) in the above is a schematic diagram of a failed case 2 of failing to find a feasible support region, Figure 14 (c) in the above is a schematic diagram of a failed case 3 of failing to find a feasible support region.

[0040] Figure 15 A schematic diagram of a training reward curve of different cost functions provided by an embodiment of the present application, Figure 15 (a) in the above is a schematic diagram of a training reward curve of a discrete reward function, Figure 15 (b) in the above is a schematic diagram of a training reward curve of a continuous reward function.

[0041] Figure 16 A schematic diagram of a foot-ground interaction control process provided by an embodiment of the present application.

[0042] Figure 17 A sample diagram of successfully finding a feasible contact region in an Isaac Gym platform simulation environment provided by an embodiment of the present application, Figure 17 (a) in the above is a schematic diagram of a successful case 1 of successfully finding a feasible contact region in an Isaac Gym platform simulation environment, Figure 17(b) in FIG. 1 is a schematic diagram of a successful case 2 of successfully finding a feasible contact area in an Isaac Gym platform simulation environment, Figure 17 (c) in FIG. 1 is a schematic diagram of a successful case 3 of successfully finding a feasible contact area in an Isaac Gym platform simulation environment, Figure 17 (d) in FIG. 1 is a schematic diagram of a successful case 4 of successfully finding a feasible contact area in an Isaac Gym platform simulation environment.

[0043] Figure 18 FIG. 2 is a schematic diagram of an iterative trajectory interaction experiment provided by an embodiment of the present application, Figure 18 (a) in FIG. 2 is a schematic diagram of a trajectory iteration successful case 1, Figure 18 (b) in FIG. 2 is a schematic diagram of a trajectory iteration successful case 2, Figure 18 (c) in FIG. 2 is a schematic diagram of a trajectory iteration successful case 3.

[0044] Figure 19 FIG. 3 is a schematic diagram of trajectory comparison of different foot-ground interaction methods provided by an embodiment of the present application, Figure 19 (a) in FIG. 3 is a schematic diagram of trajectory results obtained by an imitation learning method, Figure 19 (b) in FIG. 3 is a schematic diagram of trajectory results obtained by a method of Egli et al., Figure 19 (c) in FIG. 3 is a schematic diagram of trajectory results obtained by a method proposed by the present application.

[0045] Figure 20 FIG. 4 is a structural schematic diagram of a foot-ground interaction trajectory planning system provided by an embodiment of the present application. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0047] The purpose of the present application is to provide a foot-ground interaction trajectory planning method and system, which can be widely used in mine underground inspection, open-pit mine equipment inspection, mountain power transmission line inspection, post-disaster rescue and debris operation, etc., especially suitable for high-low undulating rock ground, soft slag pile, mixed gravel terrain, irregular obstacles, soft collapse, etc., mainly used for trajectory planning in the foot-ground interaction process of a foot-type robot, which can improve the accuracy of foot-ground interaction trajectory planning and improve the stability of foot-ground interaction of the foot-type robot in complex terrain.

[0048] In order to make the above objectives, characteristics and advantages of the present application more apparent, more comprehensible, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0049] As shown in Figure 1 The embodiment proposes a foot-ground interaction trajectory planning method, which specifically comprises the following steps.

[0050] S1: Based on the DMPs (Dynamic Movement Primitives) imitation learning method, a foot-ground interaction expert trajectory library is constructed. The foot-ground interaction expert trajectory library includes foot-ground interaction expert trajectories suitable for multiple different terrains.

[0051] S2: A residual reinforcement learning method is used to adaptively correct the foot-ground interaction expert trajectories in the foot-ground interaction expert trajectory library to obtain corrected foot-ground interaction expert trajectories.

[0052] S3: According to the corrected foot-ground interaction expert trajectories, an adaptive trajectory iteration selection mechanism based on interaction feedback is used to dynamically select the optimal reference trajectory from the foot-ground interaction expert trajectory library to obtain an updated reference trajectory.

[0053] S4: A multi-terrain contact foot robot simulation environment is constructed.

[0054] S5: Based on the multi-terrain contact foot robot simulation environment, the updated reference trajectory and the corrected foot-ground interaction expert trajectory, the trajectory adaptive correction process of step S2 and the process of dynamically selecting the optimal reference trajectory of step S3 are simulated and tested to obtain simulation test verification results; the simulation test verification results include foot-ground interaction task success rate, trajectory compliance degree and training efficiency verification results.

[0055] As an optional implementation, step S1 specifically comprises the following steps.

[0056] S11: Based on the dynamic primitive algorithm model, the DMPs imitation learning algorithm and the DMPs equation are determined.

[0057] S12: Using the teaching imitation learning method, the weights of the forcing function in the DMPs equation are learned using teaching data, a foot-ground interaction expert trajectory set including vertical downward and horizontal movement, vertical downward and oblique downward movement, and multiple changes of downward position types is constructed, and a foot-ground interaction expert trajectory library is formed.

[0058] As an optional implementation, step S2 specifically comprises the following steps.

[0059] S21: learning a residual strategy based on the foot-ground interaction expert trajectory in the foot-ground interaction expert trajectory library as an initial strategy, and establishing a residual strategy model; the residual strategy model adopts a POMDP (Partially Observable Markov Decision Process) model.

[0060] S22: training the residual strategy model by using a PPO (Proximal Policy Optimization) algorithm, and encoding historical interaction data by using an LSTM (Long Short-Term Memory Network) to obtain environment-related hidden features and input the residual strategy model, correct the forcing function in the DMPs equation, and obtain a corrected foot-ground interaction expert trajectory.

[0061] As an optional implementation, step S3 specifically includes the following steps.

[0062] S31: setting observation, action, episode, and reward function parameters of a residual network and an iterative network.

[0063] S32: training a trajectory iterative strategy by reinforcement learning based on the corrected foot-ground interaction expert trajectory and the observation, action, episode, and reward function parameters of the residual network and the iterative network, and dynamically adjusting a reference trajectory according to interaction feedback, repeatedly executing a residual network controller until a contact force threshold condition is met or an exploration episode number limit is exceeded, and obtaining an updated reference trajectory.

[0064] As an optional implementation, the observation of the residual network includes single-leg joint position, single-leg joint speed, single-leg joint position and speed at the last time, and foot end contact force information of the legged robot. The observation of the iterative network includes a current trajectory sequence number, cumulative reward, and exploration area annotation map, and the action of the iterative network is defined as selecting the foot-ground interaction expert trajectory in the foot-ground interaction expert trajectory library.

[0065] As an optional implementation, step S4 specifically includes the following steps.

[0066] S41: establishing a foot-ground contact mechanics model; the foot-ground contact mechanics model includes a normal dynamic contact model and a tangential dynamic contact model.

[0067] S42: constructing a random contact force field based on a Gaussian mixture model.

[0068] S43: Based on the foot-ground contact mechanics model and the random contact force field, a simulation process is built using the Isaac Gym platform to form a simulation environment for a multi-terrain contact footed robot.

[0069] To make the technical solution of this application clearer, the following examples will be used to illustrate the overall implementation process and detailed sub-steps of the technical solution of this application. Figure 2 This paper illustrates the motion planning and control principles of a legged robot under the influence of foot-ground interaction. The main steps include:

[0070] Step 1: Construct a foot-ground interaction expert trajectory library based on the DMPs imitation learning method.

[0071] Step one's main task is to build an expert foot-to-ground interaction trajectory library suitable for various terrain environments, providing a foundation for trajectory control of legged robots in complex terrains. Building the foot-to-ground interaction expert trajectory library includes the following steps.

[0072] 1) Establish a DMPs imitation learning algorithm based on the dynamic primitive algorithm model.

[0073] Dynamic Motion Primitives (DMPs) are a classic imitation learning method used to plan the motion trajectories of legged robots. A key challenge in DMPs is formulating nonlinear dynamic equations that can be flexibly adjusted to represent arbitrarily complex motion behaviors without manually adjusting equation parameters and ensuring system and trajectory stability. DMP algorithms are based on nonlinear dynamic systems, generating complex trajectories by combining point attractor models and nonlinear functions. The point attractor model ensures that the trajectory converges to the target position, while the nonlinear function allows for arbitrary changes to the trajectory during motion. By simply adjusting the target position and orientation, DMPs can easily generalize the learned trajectory to new target points. Furthermore, DMPs maintain stylistic similarity to the taught trajectory when generating new trajectories.

[0074] The mathematical expression for a nonlinear dynamic system is as follows.

[0075] (1).

[0076] In the formula, The position of the trajectory; The velocity of the trajectory; The acceleration of the trajectory; The target location of the trajectory; and This is the proportionality coefficient; This is a forced function used to adjust the motion trajectory.

[0077] The forced function is defined as follows.

[0078] (2).

[0079] (3).

[0080] where, is the variance of the th Gaussian basis function; is the center of the th Gaussian basis function; denotes the output value of the th Gaussian basis function; is the weight of the th Gaussian basis function; is the number of basis functions; is the initial position of the motion, is the phase variable.

[0081] By setting different start and end positions, the trajectory can be automatically corrected without changing the basic shape. The function does not directly depend on time, but rather on a phase variable which monotonically varies from 1 to 0 during the motion and is obtained from the following equation.

[0082] (4).

[0083] where, denotes the rate of change of the phase variable , and is the decay coefficient.

[0084] This differential equation is called a regular system, whose system equation is independent of time and whose state update only depends on the current state value. This feature makes the regular system stable and consistent in time, while ensuring the invariance of the curve represented by the equation in space and time, i.e., the motion is self-similar to the target, start point, and time scaling changes without changing the weight 。 The state will gradually tend to zero, and the decay rate is controlled by the decay coefficient , ensuring the stability and convergence of the system.

[0085] Figure 3 The structural composition of the dynamic motion primitive system is shown. For applications, the focus is on the forcing function that controls the trajectory shape. According to equation (2), the forcing function is obtained by weighting different Gaussian basis functions with weights . Each Gaussian basis function corresponds to a weight , and the forcing function The output is the sum of the activation values ​​of the Gaussian function multiplied by their weights. This is achieved by adjusting the weight coefficients. By obtaining the forced function curve of arbitrary shape, it is possible to control the trajectory of legged robots of different shapes.

[0086] To quickly obtain motion trajectory parameters for legged robots, a common method is learning through teaching and imitation. This problem sets out to determine how to learn the forcing function in the DMP equations using data obtained through teaching. weight Assume that sample trajectories are obtained through teaching. , , The duration is Based on the teaching information, this can be modeled as a supervised learning problem to estimate the objective forcing function controlling the shape of the DMP. .

[0087] (5).

[0088] In the formula, For the objective function, Indicates the position of the teaching trajectory. Indicates the speed of the taught trajectory. This represents the acceleration of the teaching trajectory.

[0089] In calculation During this process, it is necessary to ensure that the regularization system is integrable. At this point, the target position... Set as teaching data The position of time, the starting point The initial position of the teaching data will be set, and the learning objective is to minimize the following equation.

[0090] (6).

[0091] In the formula, This represents the cumulative cost of error. The phase variable represents the state.

[0092] At this point, by minimizing the cumulative cost of error The weights in formula (2) can be obtained. Typically, the simplest and easiest way is to use the least squares method for nonlinear fitting. The solution is given by the following equation.

[0093] (7).

[0094] (8).

[0095] wherein, represents the th Gaussian basis diagonal matrix; represents the phase variable at time t; represents the th Gaussian basis function at time t; represents the state transition function.

[0096] Through the above method, the foot-terrain interaction expert skill of the foot-type robot can be quickly imitated to make the foot-type robot obtain the foot-terrain interaction expert skill conveniently.

[0097] 2) Constructing a foot-terrain interaction expert trajectory set.

[0098] In order to adapt to different terrains, the foot-type robot will perform foot-terrain interaction at the end of the swing leg swing, mainly including three links: (1) the foot-type robot adopts an arbitrary foot end trajectory to move from the starting point to the target foot landing point. The selection of the target foot landing point needs to be as much as possible in the center position of the single leg working space to meet the subsequent foot end interaction space. (2) When the planned foot end trajectory is executed, a vertical downward trajectory is executed, and when the foot end force reaches the target threshold, the next trajectory mode is switched. (3) The third trajectory is the last part of the swing leg execution environment, which will execute a foot-terrain exploration trajectory to obtain a foot landing point that can provide greater support force and driving force.

[0099] According to human walking experience, the foot-terrain interaction trajectory usually executed includes the following steps: first, a vertical downward action is performed to judge the softness of the terrain; then, a tangential sliding in the opposite direction of the forward direction is performed to evaluate the friction characteristics of the ground. On the basis of this rule, various foot-terrain interaction trajectories are designed, as shown in FIG. 3, to adapt to the needs of different terrain conditions. The designed trajectories mainly include three types: (1) vertical downward, followed by horizontal movement. This type of trajectory is suitable for the case where the contact position is located in a relatively hard area, such as a rigid ground environment, or a soft soil or snow covered on a rigid ground; (2) vertical downward, followed by oblique downward movement. This type of trajectory can better cope with soft ground environment, and obtain greater support force and tangential driving force through soil accumulation effect; (3) the third type of trajectory is to change the downward position multiple times to cope with complex mixed terrain environment, and search for a foot landing point. Through the remote control handle, the foot-type robot collects the interaction trajectory data of the above three types, and then a preliminary foot-terrain interaction trajectory set of the foot-type robot is established by using the imitation learning algorithm. Figure 4 Step two: trajectory adaptive correction based on residual reinforcement learning.

[0100]

[0101] ​The embodiment adopts an imitation learning method to construct a trajectory library of interaction between a single leg of a legged robot and a terrain environment. However, the real environment is extremely complex, and simply relying on expert interaction trajectories cannot effectively cope with complex and variable terrain environments. At this time, the success of deep reinforcement learning provides a new idea for solving this problem. Unlike imitation learning, reinforcement learning learns the optimal strategy through the continuous interaction between the agent and the environment, and has stronger generalization ability and adaptability. During the training process, the agent continuously interacts with the simulation environment, and optimizes the policy network through trial and error learning.

[0102] Therefore, the embodiment adopts a method of improving the strategy based on deep reinforcement learning, that is, residual policy learning, to solve this problem. The main idea is to enhance the initial strategy by learning the residual term of any strategy to improve the generalization ability of the initial strategy. The specific process is as follows: given an initial strategy : → , state ∈ , action ∈ , learn a residual policy : → , get a final strategy : → .

[0103] (9).

[0104] In the formula, indicates the final strategy, indicates the initial strategy, indicates the residual strategy.

[0105] In the embodiment, the initial strategy refers to the expert trajectory obtained by imitation learning, and the residual strategy is obtained by reinforcement learning training.

[0106] The residual strategy model is established as a standard partially observable POMDP model, and the model feature is that the future state only depends on the current state, and is irrelevant to the past state history. A single Markov decision process is a tuple.

[0107] (10).

[0108] In the formula, indicates the POMDP model; indicates the state of the agent; indicates the action of the agent; indicates the observation set, = {joint angles and velocities, IMU states, foot end forces / torques, local terrain information, contact signals, etc.}; represents the immediate reward function; is the state transition function, ( , , ’ )=Pr( ’ | , ), represents the probability that the state and action afterwards transitions to state ’ ; is the observation probability function, ( | ’ ) represents the probability of observing an observation ’ in state ; is the time discount factor, 0≤ ≤1.

[0109] At time , the environment is in some state ∈ , the agent produces an action ∈ , the environment produces a new state ( +1| , ) according to the dynamics +1∈ , the agent gets a reward ( , , +1). At each time period, an agent in some state 𝑠 takes an action in , which means that all trajectories or episodes involving a finite number of actions are sampled based on the assumption of a Markov decision process. In all experiments described in this embodiment, the states and actions are real-valued vectors. A policy : → maps states to actions. Given an initial state , the reinforcement learning problem is to find a policy such that starting from any initial state, following this policy leads to the maximum expected cumulative reward J= .

[0110] Definition is the action-value function that defines the expected future discounted reward that can be obtained by performing action starting from state following policy . The Bellman equation is often used as the action-value function, denoted as follows.

[0111] (11).

[0112] where is the action-value function under policy ; is the state transition function, denotes the probability of transitioning to another state after state and action 𝑎; denotes the expectation based on state transition; is the immediate reward function for state and action 𝑎; denotes the next state, denotes the policy function; denotes the action output of the policy in state .

[0113] Actor-Critic is used to estimate the action-value function , Actor-Critic is an important class of reinforcement learning algorithms that combines the advantages of policy gradient and value function approximation. In this method, Actor (policy network) is responsible for selecting actions according to the current state, and Critic (value function network) is used to evaluate the quality of the action selected by the policy network. During training, the value function network is updated using the loss function derived from the Bellman equation, while the policy network is trained to produce actions that can maximize the Critic's evaluation. Since the value function network provides a reliable baseline to evaluate the goodness of actions, this double structure is usually more stable than training the policy network alone.

[0114] As the final residual strategy learning, PPO algorithm is used for learning. PPO improves the stability and efficiency of training by limiting the magnitude of each policy update. It uses a clipped objective function to avoid excessive policy updates, thereby solving the instability problem in traditional policy gradient methods. PPO combines the advantages of policy optimization and importance sampling, and is widely used in various complex reinforcement learning tasks. It is favored because of its easy implementation and excellent performance. At the same time, in the foot-ground interaction problem, the legged robot needs a large amount of parallel training. Compared with TD3 (double-delay deep deterministic policy gradient) and other offline algorithms, PPO is an online learning method that does not need to store too much historical interaction data, which makes it a classic algorithm for legged robot reinforcement learning. The objective function of PPO is as follows.

[0115] (12).

[0116] In the formula, The objective function of PPO algorithm is represented by The advantage function is The probability ratio of new and old policies is The hyperparameter is used to limit the magnitude of policy update The expectation is represented by The clipping function is

[0117] Through this method, the residual model of the legged robot interacting with the ground can be trained.

[0118] The foot-ground interaction task is a typical partially observable Markov decision control problem. In POMDP, the agent needs to make decisions without fully understanding the system state. The agent selects actions based on the current belief state, and updates its belief state according to the selected action and observation. The ultimate goal is to maximize the expected cumulative reward by selecting the optimal policy. Similarly, in the foot-ground interaction process, the physical information of the terrain cannot be known in advance. The legged robot needs to estimate the current state of the foot in the terrain environment according to the real-time interaction data, such as by executing an arbitrary controller , generating the observable trajectory of the legged robot estimating the current state of the foot in the terrain environment, and then making control according to the observable trajectory. In order to reduce the input of the network, the historical data is encoded through an LSTM network to obtain the hidden features , which is represented as follows.

[0119] (13).

[0120] Finally, the environment-related hidden features are obtained, which are used as the input of the policy network At this time, the policy network can make better decisions to meet different terrain environments and execution conditions according to historical information and the current state.

[0121] Back to the DMPs model at the beginning , the trajectory is controlled by the forcing function. The output command of the initial policy is the control force of the trajectory, which is calculated by the forcing function according to the current phase, and the residual policy outputs a residual force , which is used to modify the initial DMPs model. This method indirectly controls the foot trajectory by changing the forcing term and then passing through a spring damping model, so the method is more flexible and avoids the sudden change of the legged robot. The final DMPs model is modified as follows.

[0122] (14).

[0123] As Figure 5 shown, the untrained residual learning algorithm is tested in the simulation environment, and the solid line curve in the figure represents the DMPs trajectory drawn by the expert, and the red dashed line is the trajectory after adding the random residual term. It can be seen that by adding the residual force , the expert trajectory can be easily fine-tuned to achieve adaptive adjustment in different environments.

[0124] Step three: adaptive trajectory iterative selection mechanism based on interactive feedback.

[0125] The foregoing introduces a method based on DMPs combined with residual reinforcement learning to solve the problem of finding reliable foot landing points under different terrain interactions. The residual reinforcement learning method perceives the terrain environment and estimates its own state through LSTM on historical interaction data, and then obtains a more reliable control amount. However, short-time trajectory interaction cannot fully explore the environment to obtain reliable environmental information. For example, as shown in Figure 6 (a), where the gray area represents a relatively solid terrain area, and the remaining areas represent soft terrain, the execution of the left trajectory still cannot estimate that there is a better exploratory area in the lower left corner. Therefore, in order to improve the success rate of algorithm exploration, it is necessary for the legged robot to quickly dynamically adjust the next interaction behavior after executing a segment of trajectory, as shown in Figure 6 (a) on the right side of the trajectory, the legged robot can find more valuable support areas after iteration. This is particularly important for soft terrain, Figure 6(b) in the above equation shows the change of the terrain pressure area in the interaction process. After the foot robot interacts with the soft terrain, the shape and mechanical properties of the terrain will change, so the interaction trajectory needs to be dynamically adjusted with the change of the terrain. The embodiment solves the problem of iterative selection of expert trajectory by reinforcement learning. The foot robot needs to train and learn in a random force field environment to obtain efficient trajectory iteration skills. The trajectory iteration strategy to be optimized is defined as .

[0126] (15).

[0127] wherein, represents the optimal trajectory iteration strategy; represents the trajectory iteration strategy to be optimized; argmax represents the maximum value of the argument point set function.

[0128] 1) Residual network parameter setting.

[0129] In this embodiment, the parameter setting of the residual strategy network mainly includes the following contents.

[0130] (1) Observation: The observation is the input of the strategy network, which can be obtained directly or indirectly through the foot robot sensor. The selection of the observation directly affects the effect of the system. In this task, the observation obs includes the single leg joint position, the single leg joint velocity, the single leg joint position and velocity at the last time, and if the foot end is equipped with a foot end force sensor, the contact force information perceived by the foot end also needs to be included.

[0131] In order to successfully realize the "simulation to reality" deployment, it is necessary to add noise to the observation and action value during the training process. Therefore, the embodiment adds uniform sampling white noise to the observation, wherein the maximum noise amplitude added to the joint position obtained by observation is 0.02 rad, and the maximum noise amplitude added to the joint velocity is 0.04 rad / s. In addition, in order to consider the actuator model error, the action output of the controller is randomly scaled, and the factor remains unchanged during the process of an episode. This randomization is hidden from the agent, because it only receives unmodified past behavior data as part of the observation.

[0132] (2) Action: The action is the output of the strategy network. The output quantity of the residual strategy network is set to be the correction of the imitated trajectory DMP, that is, the external disturbance quantity of formula (14). In order to speed up the training speed, the embodiment uses an approximate mean and standard deviation to normalize all actions and observations.

[0133] (3) Round: In the foot-ground interaction task, a round is defined from the beginning of the foot-ground contact to the end condition being met. There are multiple end conditions, including: finding a contact force threshold (i.e., finding a foot landing point that meets the support and friction conditions); the number of executions / time reaching the set limit; or the foot of the biped robot exceeding its movement space range.

[0134] (4) Reward function: A certain reward value needs to be obtained after each round ends for the learning of the value function network. The reward function of the foot-ground interaction task is as follows.

[0135] (16).

[0136] In the formula, is the total reward function; is the contact force reward function; is the reward function in the working space; is the speed reward function; is the time step number; represents taking the maximum value of the sum of the item rewards (contact force reward function, reward function in working space, speed reward function) and 0.0, i.e., if the sum of the item rewards is greater than or equal to 0.0, the value of the sum of the item rewards is taken as the final value; if the sum of the item rewards is less than 0.0, 0.0 is taken.

[0137] Contact force reward function The purpose of the contact force reward function is to reward the behavior of the biped robot finding a terrain area with greater bearing capacity and friction. The specific reward function is as follows.

[0138] (17).

[0139] In the formula, is the normal force, i.e., the normal contact force; is the normal force threshold; is the tangential force, i.e., the tangential contact force; is the tangential force threshold; is the reward value, > 0.

[0140] This is a typical sparse reward. When the biped robot detects that the normal force of the foot end is greater than and the tangential force is greater than during the foot-ground contact, a positive reward is given. Conversely, if the threshold requirement is not met, the reward value is 0.

[0141] At the same time, in order to prevent the foot end of the biped robot from exceeding its working space during exploration, the reward function is set.

[0142] (18).

[0143] wherein, is the spatial constraint penalty coefficient. This reward function is also a kind of penalty function, giving a negative value reward when the legged robot joint exceeds the limit, wherein > 0.

[0144] In addition, in order to prevent the sudden change of the speed of the legged robot, the compliance of the legged robot in the contact process should also be ensured, and a speed reward function is added, which is expressed as follows.

[0145] (19).

[0146] wherein, is the reward coefficient, and > 0; represents the single-leg joint speed at the moment; represents the single-leg joint speed at the moment; represents the L1 norm.

[0147] 2) Iterative network parameter setting.

[0148] (1) Observation: The observation includes the serial number of the current execution trajectory in the trajectory library , the cumulative reward in the execution trajectory process , and the label map of the explored area in the execution process. In addition, the iterative network also trains an LSTM encoder to consider some environmental characteristic quantities that cannot be artificially designed.

[0149] (2) Action: The action of the iterative strategy is defined as selecting the foot-ground interaction expert trajectory in the foot-ground interaction expert trajectory library.

[0150] (3) Round: The round here is basically the same as the residual strategy, and when the reference trajectory ends, it means that the round ends.

[0151] (4) Reward function: The reward function of the iterative network only contains the reward of the contact force, and the setting of the reward function is consistent with the residual strategy.

[0152] Table 1 shows the flow of the foot-ground interaction algorithm, which consists of two loops, the inner loop is to execute the controller based on the residual learning network, and the reference trajectory is . If during the execution of the interaction controller, once the contact point position provides a contact force less than the target threshold and If the foot robot finds a footfall that meets the requirements, the algorithm ends. If the contact force requirement is not met in a complete round of inner loop, the trajectory iteration strategy is executed to update the reference trajectory Then the residual network controller in the inner loop is repeated until the contact force meets the threshold condition or exceeds the exploration round limit.

[0153] Table 1 Flowchart of foot-ground interaction algorithm

[0154]

[0155] As shown in Figure 7 , the autonomous learning algorithm of foot-ground contact under hybrid mechanical properties terrain is divided into four parts. The first part is the expert trajectory learning module. In order to help the foot robot quickly obtain the skill of foot-ground interaction, the dynamic motion primitive method is adopted as the trajectory model of the foot robot in this embodiment; the basic operation trajectory of the foot robot under different terrains is generated by expert pre-programming or teleoperation method, and the interaction skill trajectory library of the foot robot is constructed by imitation learning method after data collection, which is called by the subsequent algorithm. The second part is the trajectory iteration selection module. The purpose of this module is that the foot robot can gradually learn to select better interaction trajectory through multiple interactions. The information obtained in each interaction is used as the input information of the trajectory iterator, and the trajectory iterator is trained by reinforcement learning method. The third part is the residual network module based on reinforcement learning. In order to improve the generalization performance of the pre-constructed expert interaction trajectory, the residual network learns a residual external force item of the control trajectory on the basis of the original dynamic primitive model , which modifies the interaction trajectory of the foot robot in real time according to the terrain information (obtained by the long short-term memory neural network according to the historical interaction data) to meet the dynamic changing terrain conditions. The last interaction control module takes the modified trajectory data as input and adopts PD controller for tracking.

[0156] Step four: construction of foot-ground contact simulation environment of multi-terrain.

[0157] The foot-ground interaction algorithm proposed in this embodiment hopes to improve the generalization performance of the algorithm through reinforcement learning. However, although reinforcement learning can easily obtain satisfactory results in the simulation environment, the simulation to hardware migration has a large gap due to the difference in dynamics between simulation and real system. Therefore, one of the bottleneck problems restricting the application of the proposed algorithm on real foot robots is to establish a foot-ground contact environment with higher fidelity, and the random terrain makes the adaptability of the trained strategy stronger.

[0158] 1) Foot-ground contact mechanics model.

[0159] Unlike the modeling of different foot-ground, the main parameters of a terrain (model parameters that have the greatest influence on contact force) are predicted by vision, but here a high-precision simulation system needs to be established, so this embodiment adopts a higher-precision foot-ground contact mechanics model, including a normal dynamic contact model and a tangential dynamic contact model.

[0160] a) Normal dynamic contact model.

[0161] As shown in Figure 8 , the complete normal contact process is divided into four parts, including non-contact, contact, far contact and re-contact parts, and the main links can be summarized as two parts: the downward contact process and the upward far-off process. For downward contact, the Hunt-Crossley model is usually used to construct contact simulation.

[0162] (20).

[0163] In the formula, is the normal force, i.e. the normal contact force; is the normal equivalent stiffness coefficient; is the normal equivalent damping coefficient; is the settlement amount; is the normal settlement velocity, is the maximum settlement amount; and are both settlement amount exponential coefficients; is the settlement velocity exponential coefficient.

[0164] Since the foot end or part of the ground still has elastic deformation, there is a process of gradually decreasing normal force at the beginning of the foot end leaving the ground. For a rigid ground and a rigid foot, this process is very fast or even sudden. The Hunt-Crossley model can still be used during the foot end lifting process.

[0165] (21).

[0166] In the formula, is the normal equivalent stiffness coefficient; is the normal equivalent damping coefficient; is the ground plastic deformation amount, which is related to the ground material and the total amount of deformation.

[0167] b) Tangential dynamic contact model.

[0168] The tangential contact process can also be divided into Figure 9The tangential force increases with the increase of tangential displacement, and when the foot end is in the process of switching from the swing leg to the support leg, the tangential force will gradually reverse. In the foot-ground interaction process of the embodiment, the tangential motion direction will also change naturally. The contact process in this embodiment uses an exponential function model to simulate it.

[0169] (22).

[0170] (23).

[0171] In the formula, is the tangential force, i.e. the tangential contact force; is the shear displacement; is the tangential velocity; is the shear displacement modulus; is the friction coefficient between the foot and the ground; is the tangential force when the motion is reversed; is the tangential damping coefficient; is a sign function, is a hyperbolic tangent function.

[0172] In order to deal with the soil terrain with obvious bulldozing effect, the embodiment adds a bulldozing force term to the above model, and with the increase of the settlement amount , the bulldozing force also gradually increases.

[0173] (24).

[0174] In the formula, is the bulldozing effect coefficient.

[0175] So far, according to the above contact model, the foot-ground contact process under different terrain conditions can be simulated.

[0176] 2) Random contact force field construction method.

[0177] The embodiment sets different model parameters to simulate terrains of different materials. According to the common terrain parameter distribution obtained by literature review, the value range of the model parameters in Table 2 is obtained. In order to generate force field environments under different conditions, the embodiment proposes a random terrain representation method based on a mixture Gaussian model. The mixture Gaussian model is widely used to describe complex probability distributions due to its good fitting ability and flexibility. As Figure 10 shown, the algorithm uses a mixture Gaussian model to generate a parameter related to the terrain position, and the parameter value is proportional to the probability density at the contact position , where the maximum and minimum values correspond to the maximum and minimum probability densities, respectively.

[0178] Table 2 Distribution of Random Terrain Parameters

[0179]

[0180] First, a Gaussian mixture model is defined to describe the independent variables. probability density function The set of parameters It includes the mixing coefficient. mean Covariance Matrix The set of probabilities. The probability density function of a Gaussian mixture model. It is expressed as the following formula.

[0181] (25).

[0182] in, The number of Gaussian components; Indicates the first The probability density function of a single Gaussian distribution.

[0183] Based on this, this embodiment defines a parameter. Its value is probability density at Proportional. Specifically, It is expressed as the following formula.

[0184] (26).

[0185] In the formula, Indicates terrain location The target parameter value at the location; This is the adjustment coefficient; This is the adjustment coefficient.

[0186] Adjustment coefficient and This is used to control the range and position of the parameters. To ensure that the maximum and minimum values ​​of the parameters correspond to the maximum and minimum probability densities, appropriate values ​​are selected here. and The specific calculation method is as follows.

[0187] (27).

[0188] (28).

[0189] in, For parameters The maximum value; For parameters The minimum value.

[0190] Through the above method, the parameters can be dynamically generated according to the probability density of the Gaussian mixture model, producing a rich enough variety of terrain environments.

[0191] 3) Multi-terrain contact simulation process based on Isaac Gym platform.

[0192] Isaac Gym platform is a high-performance physics simulation platform developed by NVIDIA, aiming to accelerate the research of legged robot control and reinforcement learning algorithms. It utilizes the powerful computing capability of NVIDIA GPU to achieve efficient and realistic physical simulation through high-fidelity PhysX engine and parallel computing technology. The contact model of Isaac Gym is based on NVIDIA's PhysX physics engine, which provides advanced physical simulation functions, including rigid body dynamics, collision detection and contact force calculation. Although PhysX provides a powerful contact model, it mainly focuses on the simulation of rigid body contact, and the built-in functions of PhysX may have certain limitations for soft or elastic ground simulation, such as soft soil or sponge. Isaac Gym mainly handles rigid body dynamics in standard configuration and does not directly support soft ground contact model. Therefore, based on the introduced foot-ground contact model, the solving module as shown in Figure 11 is constructed. To avoid interference of the built-in contact solver to the simulator, a new contact plane is established 1 meter above the original ground to simulate the contact of soft ground. The specific process is that the foot-ground contact solving module obtains the position and velocity of the foot end rigid body in the world coordinate system, and then solves the corresponding foot-ground contact force. The program applies the calculated contact force to the corresponding rigid body of the foot end. In addition, for the convenience of debugging, a force field visualization module is also added. In order to introduce the solution of foot-ground contact module in more detail, Table 3 details the calculation process of foot-ground contact force simulation algorithm. First, the solving module will read the position and velocity information of the foot end and the contact terrain in the world coordinate system, and then judge whether a collision occurs according to their relative position. If a collision occurs, first determine whether it is consistent with the previous motion direction, then set the value of the flag bit RECORD according to the judgment information. To record the initial contact position, then calculate the settlement and shear displacement according to the initial contact position and the current position, and finally calculate the foot-ground contact force to complete the entire algorithm process.

[0193] Table 3 Foot-ground contact force simulation algorithm

[0194]

[0195] Step five: simulation verification of foot-ground contact learning algorithm under mixed mechanical property terrain.

[0196] 1) Simulation verification of foot-ground interaction learning in a two-dimensional scene.

[0197] To quickly analyze the above problems, a two-dimensional environment was first constructed. The constructed environment includes a two-bar linkage robotic arm, with the upper and lower arms of the robotic arm being 100mm in length and the rotation range of both joints being 0-360 degrees. It also includes a ground mechanical force field environment with randomly set parameters. In order to better determine whether the end effector of the robotic arm has found a better position, the visualization module of the environment does not strictly follow the model settings (Equations (20), (21), and (24)). This module ignores the damping term and sets a fixed slip amount. The contact length is 2cm, and the contact force at the corresponding contact position is calculated to create a visual force field. In the actual simulation system, the solution is still obtained using the foot-ground contact module constructed according to the algorithm in Table 3. It is worth noting that this simulation system does not exhibit dynamic behavior, so once the DMPs trajectory is generated, it is assumed that the foot will definitely be able to reach the target position. Nevertheless, the trajectory planning is still consistent with the actual system, and the results obtained still accurately reflect the algorithm's performance.

[0198] To verify whether the algorithm can help legged robots find areas with greater load-bearing capacity and friction in random terrain, this embodiment sets the following... Figure 12 The expert trajectory shown in (a) is used to train the residual strategy. In this simulation experiment, the PPO algorithm was used to train the residual network. During training, a new force field environment was randomly generated in each round. Training began again after collecting 20 sets of data, and the network was evaluated every 10 training iterations. Figure 13 As shown in (b) of the figure, the purple curve represents the actual value, the red curve represents the average value, and the orange area represents the error band. It can be seen that as the number of training rounds increases, the total reward value obtained by the legged robot becomes higher and higher. After about 200 rounds of training, the algorithm's reward begins to converge. The interactive experiment set up in this embodiment includes terrain environments with three difficulty levels. Table 4 records the distribution range of terrain parameters under the three difficulty levels. According to the analysis, the dominant parameter controlling the terrain mechanical properties is mainly the normal equivalent stiffness coefficient. coefficient of friction (In this embodiment, the coefficient of friction between the foot and the ground is used.) and shear displacement modulus Three core parameters. Difficult terrain corresponds to very soft and slippery terrain, while medium and easy terrains correspond to terrains that gradually become harder and have greater friction.

[0199] Table 4. Contact parameter values ​​for terrains of different difficulty.

[0200]

[0201] Subsequently, training and testing experiments were conducted under different terrain conditions. For each terrain difficulty, 100 different force field environments were generated for testing. The specific test results are shown in Table 5.

[0202] Table 5 Success rate of foot-ground interaction tasks under different terrain conditions

[0203]

[0204] As the terrain difficulty level increases, the average reward value gradually decreases, and the success rate also decreases. To better analyze the situation, some successful and unsuccessful cases were randomly selected for analysis. Figure 13 Four success stories were presented. Figure 13 The terrain shown in (a) is a relatively ideal environment, where the foot can find an area that meets the contact mechanics conditions as soon as it makes contact; Figure 13 (b) The terrain is more complex, and the support and driving force provided by the area where the foot just makes contact cannot meet the requirements. Therefore, the foot actively moves forward to find a contact area that meets the conditions. Figure 13 Compared to the initial contact positions of the foot in (c) and (d) Figure 13 (b) of them are further away from the feasible contact area; however, they successfully found locations that met the conditions through different methods. Specifically, Figure 13 In (c), the legged robot slid forward to first satisfy the tangential contact force threshold requirement, and then, maintaining its tangential position vertically downward, successfully explored a contact position that satisfied the normal force. Conversely, Figure 13 In (d), the legged robot first satisfies the normal contact force threshold condition during its forward sliding, and then continues to slide laterally until it reaches the tangential force threshold condition. It can be seen that through the proposed method, the legged robot can actively adapt to different terrain conditions and successfully explore and find an area with greater support force and friction.

[0205] Figure 14 This demonstrates some failure scenarios, for Figure 14 The terrain represented by (a) is too complex, and there is no terrain area that meets the conditions in the workspace of the legged robot, so this situation is difficult to avoid. Figure 14 The terrain represented by (b) is a feasible contact area that exists behind the default trajectory of single-leg exploration. Even if LSTM is used to represent the terrain environment, the limited exploration still cannot clearly represent the overall situation of the environment. Figure 14In the diagram (c), the terrain represents a relatively solid area found before the legged robot can explore a more solid area, at which point the robot stops exploring. It can be seen that single exploration can only obtain local environmental information and lacks the ability to observe the entire environment, leading to the failure of the interaction task. This embodiment verifies the algorithm's performance while also setting different forms of cost functions to examine the impact of different cost functions on the foot-ground interaction effect. A sparse reward function setting method has been introduced above (formulas (16), (17), (18)). However, continuous rewards are often more conducive to the rapid training of the algorithm. This embodiment also compares and tests continuous reward function forms, where the function reward provides greater support force (normal force) upon touch. ) and friction (tangential force) To modify the behavior of the position, the formula (17) is modified as follows.

[0206] (29).

[0207] In the formula, This is the normal force reward coefficient; This is the tangential force reward coefficient.

[0208] like Figure 15 As shown, the reward value using a continuous reward function converges faster, while that using a sparse reward function is relatively slower. However, in practice, if a continuous reward function is used, the legged robot wiggles back and forth at different positions to obtain more rewards, failing to quickly reach the contact area that meets the conditions. In contrast, with the sparse reward function scheme, the legged robot can directly and quickly move to the area that meets the conditions. Therefore, this embodiment ultimately uses a sparse reward function as the objective function of the agent.

[0209] 2) Simulation verification of foot-ground interaction trajectory planning based on the Isaac Gym platform.

[0210] The aforementioned simulations in a two-dimensional environment demonstrate that the proposed algorithm performs well in foot-ground contact tasks, and also verify that the sparse reward function form described above can find terrain regions that meet the contact conditions more quickly and directly compared to the continuous form. Next, to further verify the algorithm's performance in a dynamic simulation environment and its advantages over other methods, simulations based on the Isaac Gym simulator were conducted. Compared to the two-dimensional simulation environment, the Isaac Gym simulator has dynamic simulation capabilities; therefore, the DMPs trajectory adjusted through the residual strategy still requires the design of a compliant controller for tracking control, such as... Figure 16The actual contact force is introduced to modify the DMPs model after the residual strategy. When the actual contact force is greater than the maximum tangential force, the movement in the tangential direction will be limited; similarly, when the normal force is greater than the maximum threshold, it cannot continue to move downward. When both conditions are met, the foot robot successfully finds the required landing point. While the DMPs external force term is constrained by the feedback force, the contact force is also directly fed back to the single leg controller, allowing it to quickly adjust the foot position to prevent excessive contact force.

[0211] After integrating the controller into the Isaac Gym simulator, the effectiveness of the proposed foot-ground interaction algorithm in a three-dimensional environment was verified. Similarly, as shown in Figure 17 , the foot robot can gradually find the terrain area that meets the conditions in most cases under a random force field environment. Some extreme harsh terrain (no feasible contact area in the workspace) and terrain that cannot perceive force field changes before contact are still unsolvable. To further improve the success rate of the algorithm, the next iteration learning strategy was tested to see if it could improve the success rate of the contact task in complex environments. First, the switching strategy of the trajectory was trained separately . Then a residual strategy that is not related or weakly related to the reference trajectory was trained separately for subsequent external force adjustment on the DMPs trajectory. During the training process, only the external force term obtained by multiplying a small coefficient or ignoring the forcing function of the default reference DMPs can complete the training. Finally, combining the two is the final foot-ground interaction algorithm. After training, all algorithms were tested on 100 different random terrains (medium difficulty terrain) to obtain the task success rate indicator, as shown in Table 6. On the basis of the switching strategy and the switching strategy plus the residual strategy , the experiment added two other schemes, the random DMPs strategy and the "random DMPs strategy ", to compare the effectiveness of the proposed method. From the experimental results (Table 6), it can be seen that the random DMPs scheme is equivalent to using a purely random expert scheme, with a task success rate of only 62%. In contrast, by iterating the expert trajectory (selected strategy ), the task success rate increased by 12% to 74%. This shows that iterative adjustment of the trajectory can effectively help the foot robot select a more suitable reference trajectory. When the combination scheme of the random DMPs strategy and is used, the task success rate increases to 83%, and it can be seen that the residual controller can still effectively improve the foot-ground interaction ability of the foot robot under a lower quality reference trajectory, which directly illustrates the effectiveness of the residual strategy. When a better reference trajectory is used, i.e. The success rate of the task reached 90%, which is much higher than other schemes. Further, the "Residual Scheme" was tested on some failed cases in the "Iterative Scheme", Figure 18 showed new results. After adjusting the trajectory iteratively, the foot robot successfully explored the area that the previous scheme could not explore, and thus successfully completed the interaction task. In summary, the above experiments show that the iterative trajectory algorithm combined with the residual strategy can effectively help the foot robot improve the success rate of the foot-ground interaction task.

[0212] Table 6 Task success rate of different foot-ground interaction strategies

[0213]

[0214] 3) Foot-ground interaction comparison experiment.

[0215] Finally, the proposed scheme and the latest method were compared. For the task of finding a feasible contact area for foot-ground contact, there is no relevant research by scholars at present, but there is a similar task of excavator earthwork. The difference between the task of this embodiment and the excavator task is that the excavator task is to avoid hard and large tangential force areas as much as possible, and it is more inclined to shovel out soft sand; while the task of this embodiment is to find hard and large tangential force areas. Although so, the method used can be adjusted by adjusting the control target as a comparison scheme of this embodiment. The first scheme compared in this embodiment is from the method of Egli et al. This method uses reinforcement learning to directly control the joint position of the foot robot to achieve the shovel work. Referring to this scheme, the reward equation is consistent with the setting of the proposed method, and 500 foot robots are trained in parallel for 800 rounds to obtain a control strategy. The second comparison method is a mimic learning method, which sets a general contact trajectory by human, and then uses impedance control to track. The experimental terrain environment still adopts medium difficulty, and the comparison indexes include task success rate, flexibility of operation trajectory, and training time of controller for comparison. The experimental results are shown in Table 7. In terms of task success rate, the success rate of the algorithm proposed in this embodiment is the highest, which is basically the same as the method of Egli et al., and the pure mimic learning result is the worst. However, in terms of flexibility, the reinforcement learning scheme designed by Egli et al. has a great disadvantage. Because the behavior generated by the control strategy occasionally mutates, it leads to the shaking of the execution trajectory, such as Figure 19 ​As shown, the mutation caused by the reference trajectory may lead to unstable robot body and even hardware damage; in contrast, the trajectory of the imitation learning scheme is smooth, and the proposed scheme is also dynamically filtered by DMPs, and can also achieve the effect of softness. Finally, in terms of the number of training rounds, the proposed method can also greatly reduce the training time under the guidance of the reference trajectory. Therefore, compared with other methods, the embodiment method has obvious advantages in success rate, trajectory softness and training time.

[0216] Table 7 Comparison of experimental data of different foot-ground interaction methods

[0217]

[0218] Based on the same inventive concept, the embodiment of the present application also provides a foot-ground interaction trajectory planning system for implementing the foot-ground interaction trajectory planning method described above. The implementation scheme of the problem solving provided by the system is similar to the implementation scheme described in the above method, so the specific limitations in the following embodiment of the foot-ground interaction trajectory planning system can be referred to the limitations of the foot-ground interaction trajectory planning method in the above text, which will not be repeated here.

[0219] In one exemplary embodiment, as Figure 20 shown, a foot-ground interaction trajectory planning system is provided, comprising the following modules.

[0220] A trajectory library construction module is configured to construct a foot-ground interaction expert trajectory library based on a DMPs imitation learning method; the foot-ground interaction expert trajectory library comprises foot-ground interaction expert trajectories adapted to multiple different terrains.

[0221] A trajectory adaptive correction module is configured to perform trajectory adaptive correction on the foot-ground interaction expert trajectories in the foot-ground interaction expert trajectory library using a residual reinforcement learning method, to obtain corrected foot-ground interaction expert trajectories.

[0222] A trajectory iterative selection module is configured to dynamically select an optimal reference trajectory from the foot-ground interaction expert trajectory library according to the corrected foot-ground interaction expert trajectories, using an adaptive trajectory iterative selection mechanism based on interaction feedback, to obtain an updated reference trajectory.

[0223] A simulation environment construction module is configured to construct a multi-terrain contact foot robot simulation environment.

[0224] The simulation test verification module is configured to perform simulation test verification on the trajectory self-adaptive correction process and the process of dynamically selecting an optimal reference trajectory based on the multi-terrain contact legged robot simulation environment, the updated reference trajectory, and the corrected leg-ground interaction expert trajectory, to obtain a simulation test verification result. The simulation test verification result includes a leg-ground interaction task success rate, a trajectory compliance degree, and a training efficiency verification result.

[0225] The application adopts a hybrid trajectory planning framework based on DMPs and residual reinforcement learning, takes dynamic motion primitives as a basic trajectory generation model, and combines residual reinforcement learning to realize dynamic correction of trajectories. Through a residual DMPs model, the residual force correction DMPs base model output by reinforcement learning is corrected, the original trajectory generation capability of DMPs is maintained, and compliant trajectory adjustment is realized, thereby effectively solving the problem of insufficient adaptability of traditional DMPs in a complex environment. In addition, the application also designs a leg-ground interaction strategy architecture. First, an expert trajectory library containing typical modes such as vertical-horizontal, vertical-oblique downward, and multi-position search is constructed through DMPs imitation learning. Then, a PPO reinforcement learning algorithm is used to train a residual strategy network, the historical interaction data is encoded through LSTM, and the dynamic correction amount of the DMPs forcing function is output, to realize online self-adaptive adjustment of the trajectory. Finally, through a trajectory iterative selection strategy, an optimal trajectory is dynamically selected from the expert library based on real-time interaction feedback, to form a closed-loop control system of “basic trajectory generation-online fine adjustment-global strategy optimization”. The architecture effectively combines the stability of imitation learning and the adaptability of reinforcement learning, and significantly improves the interaction success rate of the legged robot in a complex terrain. In addition, based on a high-precision ground mechanics model, a high-fidelity randomization simulation environment construction method is adopted. The method dynamically generates diversified random contact force fields through a mixture Gaussian model, and realizes realistic interaction simulation in combination with a refined leg-ground contact mechanics model. The mechanics model includes four-stage modeling of normal dynamic contact and an exponential function model of tangential dynamic contact, and can accurately simulate the mechanical properties under different terrains. In the Isaac Gym platform, a virtual contact plane construction method is innovatively adopted, the soft ground contact force is accurately calculated by establishing a virtual contact layer, the simulation efficiency is maintained, and the physical accuracy is ensured, thereby providing a high-fidelity simulation environment for reinforcement learning training of the legged robot.

[0226] The application provides a foot-ground interaction trajectory planning method and system. For the foot-ground interaction part, the motion interaction provides a more reliable (large support force and sufficient friction) foot-falling point and its contact constraint condition for the rear motion planning part. The foot robot can plan the optimal gait, body posture and contact state of the foot-falling point according to the foot-falling point position, normal vector and contact constraint information. Finally, the motion control part can use the classical model predictive control or the popular and effective reinforcement learning control method to perform stable control, and find a reliable area for the foot robot in a complex terrain environment in real time, so as to provide more relaxed contact constraint conditions for subsequent planning and whole body control.

[0227] The technical features of the above embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0228] The principles and implementation manners of the application are described by using specific examples in the embodiments. The above embodiments are only used to help understand the method and core idea of the application. For those skilled in the art, the specific implementation manners and application ranges can be changed according to the idea of the application. In conclusion, the content of the specification should not be understood as a limitation of the application.

Claims

1. A foot-ground interaction trajectory planning method, characterized in that, The foot-ground interaction trajectory planning method includes: S1: Based on the DMPs imitation learning method, a foot-to-ground interaction expert trajectory library is constructed; the foot-to-ground interaction expert trajectory library includes foot-to-ground interaction expert trajectories adapted to various terrains; S2: Using residual reinforcement learning, the foot-ground interaction expert trajectories in the foot-ground interaction expert trajectory library are adaptively corrected to obtain the corrected foot-ground interaction expert trajectories. S3: Based on the corrected foot-ground interaction expert trajectory, the optimal reference trajectory is dynamically selected from the foot-ground interaction expert trajectory library using an adaptive trajectory iteration selection mechanism based on interactive feedback, and the updated reference trajectory is obtained. S4: Construct a simulation environment for multi-terrain contact legged robots; S5: Based on the multi-terrain contact legged robot simulation environment, the updated reference trajectory, and the corrected foot-ground interaction expert trajectory, perform simulation tests to verify the trajectory adaptive correction process in step S2 and the process of dynamically selecting the optimal reference trajectory in step S3, and obtain simulation test verification results; the simulation test verification results include the success rate of the foot-ground interaction task, the smoothness of the trajectory, and the training efficiency verification results.

2. The foot-ground interaction trajectory planning method according to claim 1, characterized in that, Step S1 specifically includes: Based on the dynamic primitive algorithm model, the DMPs imitation learning algorithm and DMPs equations are determined. The teaching and imitation learning method is adopted to learn the weight of the forcing function in the DMPs equation using teaching data, and to construct a set of foot-ground interaction expert trajectories including vertical downward movement followed by horizontal movement, vertical downward movement followed by diagonal downward movement, and multiple changes in the downward position type, thus forming a foot-ground interaction expert trajectory library.

3. The foot-ground interaction trajectory planning method according to claim 2, characterized in that, The mathematical expression for the nonlinear dynamic system of the DMPs imitation learning method is as follows: ； In the formula, The position of the trajectory. The velocity of the trajectory. Let be the acceleration of the trajectory. The target location of the trajectory. and This is the proportionality coefficient. This is a forced function used to adjust the motion trajectory; The expression for the forcing function is: ； ； In the formula, For the first The variance of the Gaussian function, For the first The center of the Gaussian function, Indicates the first The output value of the Gaussian function, For the first The weights of the Gaussian function, The number of basis functions. The initial position of the motion, For phase variables; The weights are solved using the least squares method. To minimize the cumulative error cost, the expression for the cumulative error cost is: ； In the formula, This represents the cumulative cost of error. For the objective function, The phase variable represents the state.

4. The foot-ground interaction trajectory planning method according to claim 2, characterized in that, Step S2 specifically includes: Using the foot-to-ground interaction expert trajectories in the foot-to-ground interaction expert trajectory library as the initial strategy, a residual strategy is learned, and a residual strategy model is established; the residual strategy model adopts the POMDP model. The residual policy model is trained using the PPO algorithm, and the historical interaction data is encoded using an LSTM network to obtain environment-related latent features, which are then input into the residual policy model to correct the forcing function in the DMPs equation, resulting in the corrected foot-ground interaction expert trajectory.

5. The foot-ground interaction trajectory planning method according to claim 4, characterized in that, The residual strategy satisfies: ； In the formula, Indicates the final strategy. Indicates the initial policy. Represent the residual strategy; The expression for the POMDP model is: ； In the formula, Represents the POMDP model. Indicates the state of the agent. Represents the actions of the intelligent agent. This represents the instant reward function. Represents the observation set, This is the state transition function. To observe the probability function, The time discount factor; The objective function of the PPO algorithm is: ； In the formula, This represents the objective function of the PPO algorithm. For the dominant function, This represents the probability ratio between the old and new strategies. For hyperparameters, Expressing expectations, This is the clipping function.

6. The foot-ground interaction trajectory planning method according to claim 1, characterized in that, Step S3 specifically includes: Set the observation, action, round, and reward function parameters for the residual network and the iterative network; Based on the corrected foot-to-ground interaction expert trajectory and the observations, actions, rounds, and reward function parameters of the residual network and iterative network, a trajectory iteration strategy is trained through reinforcement learning, and the reference trajectory is dynamically adjusted according to the interaction feedback. The residual network controller is repeatedly executed until the contact force threshold condition is met or the exploration round limit is exceeded, thus obtaining the updated reference trajectory.

7. The foot-ground interaction trajectory planning method according to claim 6, characterized in that, The observations of the residual network include the single-leg joint position, single-leg joint velocity, single-leg joint position and velocity at the previous moment, and foot contact force information of the legged robot. The expression for the reward function is: ； In the formula, For the total reward function, For contact force reward function, For the reward function in the workspace, For speed reward function; The observations of the iterative network include the current trajectory number, cumulative reward, and exploration area annotation map. The action of the iterative network is defined as selecting the foot-to-ground interaction expert trajectory from the foot-to-ground interaction expert trajectory library.

8. The foot-ground interaction trajectory planning method according to claim 1, characterized in that, Step S4 specifically includes: A foot-ground contact mechanics model is established; the foot-ground contact mechanics model includes a normal dynamic contact model and a tangential dynamic contact model. A stochastic contact force field is constructed based on a Gaussian mixture model. Based on the foot-ground contact mechanics model and the random contact force field, a simulation process was built using the Isaac Gym platform to form a simulation environment for a multi-terrain contact footed robot.

9. The foot-ground interaction trajectory planning method according to claim 8, characterized in that, The expression for the normal dynamic contact model is: ； ； In the formula, For normal contact force, The normal equivalent stiffness coefficient is... The normal equivalent damping coefficient is... For the amount of subsidence, Normal subsidence velocity, For the maximum settlement, and All are subsidence index coefficients. The subsidence velocity exponent coefficient, The normal equivalent stiffness coefficient; The normal equivalent damping coefficient; For ground plastic deformation; The expression for the tangential dynamic contact model is: ； In the formula, For tangential force, This is the shear displacement. For tangential velocity, The coefficient of friction between the foot and the ground. The tangential force during reverse motion. The tangential damping coefficient is... This is the soil accumulation effect coefficient. For symbolic functions, It is the hyperbolic tangent function.

10. A foot-ground interaction trajectory planning system, characterized in that, The foot-to-ground interaction trajectory planning system is used to implement the foot-to-ground interaction trajectory planning method according to any one of claims 1-9, and the foot-to-ground interaction trajectory planning system includes: The trajectory library construction module is used to construct a foot-ground interaction expert trajectory library based on the DMPs imitation learning method; the foot-ground interaction expert trajectory library includes foot-ground interaction expert trajectories adapted to various terrains. The trajectory adaptive correction module is used to perform trajectory adaptive correction on the foot-ground interaction expert trajectories in the foot-ground interaction expert trajectory library using a residual reinforcement learning method, so as to obtain the corrected foot-ground interaction expert trajectories. The trajectory iteration selection module is used to dynamically select the optimal reference trajectory from the foot-ground interaction expert trajectory library based on the corrected foot-ground interaction expert trajectory and an adaptive trajectory iteration selection mechanism based on interactive feedback, so as to obtain the updated reference trajectory. The simulation environment construction module is used to build a simulation environment for multi-terrain contact legged robots. The simulation test and verification module is used to perform simulation tests and verifications on the trajectory adaptive correction process and the process of dynamically selecting the optimal reference trajectory based on the multi-terrain contact legged robot simulation environment, the updated reference trajectory, and the corrected leg-ground interaction expert trajectory, and to obtain simulation test and verification results. The simulation test and verification results include the success rate of the leg-ground interaction task, the smoothness of the trajectory, and the training efficiency verification results.

Images