Imitation learning-based robot motion control strategy network training method, robot motion control method, device, equipment, robot and storage medium

Abstract

This disclosure provides a robot motion control policy network training method based on imitation learning, a robot motion control method, device, equipment, robot, and storage medium, relating to the field of robotics technology. While maintaining anthropomorphic characteristics of actions through a style reward layer, this disclosure innovatively employs a terrain-independent gait reward layer based on contact sequences and biomechanical constraint functions. This incorporates the robot's interaction with the ground into the optimization objective of the policy network, so that the training process no longer simply pursues visual motion reproduction but explicitly constrains the biomechanical characteristics in the motion generation process. Physical robustness is embedded in the generated motion trajectory, significantly reducing the dynamic deviation between the simulation environment and the real physical environment, effectively eliminating foot slippage and gait instability caused by non-physical actions, and improving the stability and safety of the control strategy when deployed on a real machine.

Description

Technical Field

[0001] This disclosure relates to the field of robotics technology, and to a method for training a robot motion control strategy network based on imitation learning, a robot motion control method, a device, an equipment, a robot, and a storage medium. Background Technology

[0002] Robot imitation learning technology enables robots to master complex motor skills by observing and imitating teaching data from experts (such as human demonstrators or expert policies). Compared to traditional model-based control methods, imitation learning can guide policy networks to generate visually more human-like motion trajectories, avoiding the stiffness and mechanical feel common in traditional methods.

[0003] However, in existing imitation learning frameworks, the optimization objective typically relies on the geometric pose error between the reference action and the generated action, aiming to reproduce human data as accurately as possible in terms of the robot's joint angles and limb positions. This kinematic error-oriented training mechanism may cause the control strategy to overfit the visual appearance, neglecting the physical constraints of the robot's interaction with the environment.

[0004] For example, the paper “Experience-Learning Inspired Two-Step Reward Method for Efficient Legged Locomotion Learning Towards Natural and Robust Gaits” (Yinghui Li et al., June 20, 2024) employs a two-stage strategy: the first stage trains the robot on flat ground using explicit gait rewards to acquire basic experience; the second stage removes explicit gait rewards in complex terrain and instead uses a discriminator to generate style rewards based on the experience from the first stage, thereby guiding the robot to adapt to the environment.

[0005] This paper and similar related technologies exhibit a significant technical bias: the belief that introducing explicit gait constraints during training in complex environments will over-constrain the robot's exploration capabilities, leading to training failure or motion stagnation. For example, the paper explicitly states that "adding gait rewards leads to failure because it over-constrains the robot." This bias stems from the fact that gait rewards used in related technologies are typically based on strict geometric kinematic constraints. For instance, the Raibert heuristic used in the first stage of the aforementioned paper calculates ideal foot placement on flat ground, and the foot swing height constraint forces the foot to reach a specific geometric height. While these geometric constraints are effective on flat terrain, they conflict with the actual undulations of the terrain when facing unstructured terrain such as stairs or boulders, requiring the robot to raise its feet to a fixed height relative to the base or to land on preset geometric coordinates. Furthermore, the related technologies conclude that gait constraints and style learning must be decoupled in the final policy learning stage, relying solely on the discriminator's probabilistic assessment of the action distribution to evaluate whether the action "looks like" it is completed, thereby implicitly maintaining the rationality of the movement.

[0006] However, the inventors discovered in practice that this style learning without physical constraints has flaws. While removing gait constraints gives the robot the freedom to explore space, it also deprives it of oversight of contact dynamics. Because the discriminator only focuses on the statistical similarity between the generated action and the reference action (i.e., visual anthropomorphism), it cannot perceive the microscopic physical contact state. This lack of physical constraints may only manifest as subtle differences in movement in a simulation environment, but in a real physical environment, it is amplified through the Sim2Real Gap. Consequently, when the trained policy network is deployed to the robot, it is prone to slippage, gait instability, uncontrollable airtime, and even hardware damage due to excessive ground impact. Summary of the Invention

[0007] This disclosure provides a robot motion control strategy network training method based on imitation learning, a robot motion control method, device, equipment, robot, and storage medium, which can effectively eliminate foot slippage and step frequency instability caused by non-physical movements, and prevent hardware damage caused by excessive ground reaction force, thereby improving the stability and safety of the control strategy when deployed on a real machine.

[0008] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure.

[0009] According to a first aspect of this disclosure, a method for training a robot motion control policy network based on imitation learning is provided, comprising:

[0010] The style reward layer receives reference human motion data and motion data output from the motion control strategy network, and outputs a first reward item based on the feature differences between the two.

[0011] The gait reward layer receives motion data output from the motion control policy network and corresponding contact sequence data, and calculates the second reward item through a constraint function based on biomechanical design.

[0012] A comprehensive reward is generated based on the first and second reward items, and the parameters of the motion control policy network are updated using the comprehensive reward item to train the motion control policy network.

[0013] Optionally, the contact sequence data mentioned above includes:

[0014] Contact force sensing data and contact phase of each foot end aligned with the motion data in time steps;

[0015] The contact phase includes at least two of the following: the single-leg swing phase, the double-leg support phase, and the airborne phase.

[0016] Optionally, the constraint function based on biomechanical design described above is configured as follows:

[0017] The current state of each foot is determined based on the contact phase, including the swinging state and the supporting state.

[0018] Kinematic trajectory tracking constraints are applied to the foot in a swinging state, and ground contact dynamic constraints are applied to the foot in a supporting state.

[0019] Optionally, the above-mentioned kinematic trajectory tracking constraints for the foot in a swinging state include:

[0020] Generate the desired motion trajectory of the foot during the swinging process. The desired motion trajectory includes the desired height curve and the desired vertical velocity curve.

[0021] The system acquires the real-time motion state of the foot and calculates the height tracking error between the real-time motion state and the desired height curve, as well as the speed tracking error between the real-time motion state and the desired vertical speed curve.

[0022] The real-time contact force of the foot is obtained from the contact sequence data, and a contact force deviation value is generated when the real-time contact force is not zero.

[0023] Based on the height tracking error, velocity tracking error, and contact force deviation, the reward value of the kinematic trajectory tracking constraint is calculated as a component of the second reward item.

[0024] Optionally, generating the desired motion trajectory of the foot during the swinging process includes:

[0025] Obtain the current gait phase value, which changes periodically over time within a preset range;

[0026] The gait phase values ​​are mapped to foot height values ​​that vary with the phase using a preset spline curve function, in order to form the desired height curve;

[0027] Calculate the derivative of the desired height curve with respect to time to generate the desired vertical velocity curve.

[0028] Optionally, the reward value for calculating the kinematic trajectory tracking constraint mentioned above includes:

[0029] The altitude tracking reward component is obtained by processing the altitude tracking error using the Gaussian kernel function; the velocity tracking reward component is obtained by processing the velocity tracking error using the Gaussian kernel function; and the contact force compliance reward component is obtained by processing the contact force deviation value using the Gaussian kernel function.

[0030] The reward value is obtained by weighting and summing the height tracking reward component, the speed tracking reward component, and the contact force compliance reward component.

[0031] Optionally, the above-mentioned ground contact dynamic constraints applied to the foot in a supported state include:

[0032] Construct a foot-locked target, where the desired height off the ground, desired vertical velocity, and desired horizontal velocity of the foot are all set to zero.

[0033] The real-time motion state of the foot is acquired, and the steady-state deviations of the height, vertical and horizontal relative to the locked target are calculated.

[0034] The reward value for ground contact dynamic constraints, calculated based on height steady-state deviation, vertical steady-state deviation, and horizontal steady-state deviation, is used as a component of the second reward item.

[0035] Optionally, the reward value for calculating the ground contact dynamic constraints based on the above-mentioned height steady-state deviation, vertical steady-state deviation, and horizontal steady-state deviation includes:

[0036] Substituting the height steady-state deviation into the Gaussian radial basis function generates the foot-touching reward; substituting the vertical steady-state deviation into the Gaussian radial basis function generates the vertical stability reward; and substituting the horizontal steady-state deviation into the Gaussian radial basis function generates the anti-slip reward.

[0037] The reward values ​​are summed for the foot contact reward, vertical stability reward, and anti-slip reward.

[0038] Optionally, the constraint function based on biomechanical design described above is further configured as follows:

[0039] Perform limb coordination rhythm constraints based on gait frequency for the robot's arm joints.

[0040] Optionally, the above-mentioned limb coordination rhythm constraints based on gait frequency applied to the robot's arm joints include:

[0041] The linear velocity command in the input instructions of the motion control strategy network is analyzed, and the basic amplitude value of the arm swing is determined based on the linear velocity command.

[0042] The base amplitude value is periodically modulated to generate a desired arm state sequence containing the desired position and desired velocity;

[0043] Acquire the real-time state of the robot arm and calculate the cooperative tracking error between the real-time state of the robot arm and the desired state sequence of the robot arm;

[0044] The reward value generated based on the collaborative tracking error to create limb coordination rhythm constraints is used as a component of the second reward item.

[0045] Optionally, the above-mentioned periodic modulation of the base amplitude value to generate a desired arm state sequence containing the desired position and desired velocity includes:

[0046] Extract the sine and cosine eigenvalues ​​of the current gait phase;

[0047] The base amplitude value is multiplied by the sine and cosine eigenvalues ​​respectively to obtain the desired position;

[0048] The desired velocity is obtained by multiplying the base amplitude value by the time derivatives of the sine and cosine eigenvalues, respectively.

[0049] Based on the desired position and desired velocity, construct a desired arm state sequence containing the desired position and desired velocity.

[0050] Optionally, the above method also includes: merging the components of each second reward item to obtain the second reward item.

[0051] Optionally, before receiving reference human motion data and motion data output by the motion control policy network using the style reward layer, the method further includes:

[0052] Construct a discriminator network for an adversarial motion prior framework;

[0053] The discriminator network is trained based on reference human motion data and motion data output by the motion control strategy network, and the parameters of the discriminator network are updated to obtain a trained discriminator network.

[0054] The trained discriminator network was designated as the style reward layer.

[0055] Optionally, the discriminator network is trained based on reference human motion data and motion data output by the motion control policy network, and the parameters of the discriminator network are updated to obtain a trained discriminator network, including:

[0056] The first action segment is extracted from reference human action data as a reference sample;

[0057] The second motion segment is extracted from the motion data output by the motion control policy network and used as the generated sample.

[0058] The parameters of the discriminator network are updated based on the reference samples and generated samples to obtain a trained discriminator network.

[0059] Optionally, the above-mentioned updating of the discriminator network parameters based on reference samples and generated samples to obtain a trained discriminator network includes:

[0060] The reference sample is input into the discriminator network to obtain the first discriminant output;

[0061] The generated sample is input into the discriminator network to obtain a second discriminant output;

[0062] The discriminant loss is calculated based on the first and second discriminant outputs, and backpropagation is performed on the discriminant network based on the discriminant loss to update the parameters of the discriminant network, thus obtaining the trained discriminant network.

[0063] Optionally, the style reward layer includes a first encoding subnetwork, a second encoding subnetwork, and a style scoring subnetwork; the style reward layer receives reference human motion data and motion data output from the motion control policy network, and outputs a first reward item based on the feature differences between the two, including:

[0064] The first encoding subnetwork with shared weights is used to generate a first state action feature vector by inputting reference human action data, and the action data output by the motion control strategy network is input into the first encoding subnetwork with shared weights to generate a second state action feature vector.

[0065] The first state action feature vector and the second state action feature vector are respectively input into the second encoding sub-network with shared weights to obtain the first embedding vector and the second embedding vector;

[0066] The first and second embedding vectors are respectively input into the style scoring subnetwork to obtain the discrimination score;

[0067] The identification score is mapped to the first reward item.

[0068] Optionally, the above-mentioned comprehensive reward item generated based on the first reward item and the second reward item includes:

[0069] Calculate the batch average for the first reward item and the second reward item respectively to obtain the first average and the second average;

[0070] The first weight corresponding to the first reward item is updated based on the first mean, and the second weight corresponding to the second reward item is updated based on the second mean;

[0071] The first reward item is weighted based on the updated first weight, and the second reward item is weighted based on the updated second weight.

[0072] The weighted first and second reward items are summed to obtain the comprehensive reward item.

[0073] Optionally, the above-mentioned updating the first weight corresponding to the first reward item based on the first mean and updating the second weight corresponding to the second reward item based on the second mean includes:

[0074] Obtain the historical sliding window sequence of the first mean and the second mean;

[0075] A moving average is calculated for the historical sliding window sequence of the first mean to obtain the first moving average, and a moving average is calculated for the historical sliding window sequence of the second mean to obtain the second moving average;

[0076] The normalization ratio is calculated based on the first moving average and the second moving average to obtain the first weight and the second weight.

[0077] Optionally, the parameters of the motion control policy network updated using the comprehensive reward term include:

[0078] Trajectory data is collected based on a motion control strategy network in a simulation environment, and the comprehensive reward corresponding to the trajectory data is recorded.

[0079] The strategy gradient update amount is calculated based on trajectory data and comprehensive reward items;

[0080] The parameters of the motion control policy network are updated based on the policy gradient update amount to obtain the updated parameters of the motion control policy network.

[0081] Optionally, the above calculation of the strategy gradient update amount based on trajectory data and comprehensive reward term includes:

[0082] The trajectory data is accumulated with a comprehensive reward item at each time step to generate a reward sequence, and a reward vector is formed based on the reward sequence;

[0083] The baseline estimation vector is calculated based on the return vector, and the advantage estimation sequence is calculated based on the return vector and the baseline estimation vector.

[0084] The advantage estimation sequence and trajectory data are input together into a preset optimizer to generate policy gradient updates.

[0085] According to a second aspect of this disclosure, a robot motion control method based on imitation learning is provided, comprising:

[0086] Obtain the robot's current observation state;

[0087] The current observation state is input into the pre-trained motion control policy network, and the motion control policy network is used to output the action policy to control the robot's movement.

[0088] The motion control policy network is obtained according to the robot motion control policy network training method based on imitation learning described in the first aspect above.

[0089] According to a third aspect of this disclosure, a robot motion control policy network training device based on imitation learning is provided, comprising:

[0090] The style reward layer construction module is used to receive reference human motion data and motion data output by the motion control strategy network using the style reward layer, and output the first reward item based on the feature differences between the two.

[0091] The gait reward layer construction module is used to receive motion data output by the motion control strategy network and the corresponding contact sequence data of the motion data using the gait reward layer, and calculate the second reward item through a constraint function based on biomechanical design.

[0092] The policy network update module is used to generate a comprehensive reward based on the first reward and the second reward, and to update the parameters of the motion control policy network using the comprehensive reward to train the motion control policy network.

[0093] According to a fourth aspect of this disclosure, a robot motion control device based on imitation learning is provided, comprising:

[0094] The data acquisition module is used to acquire the robot's current observation status;

[0095] The control strategy module is used to input the current observation state into the pre-trained motion control strategy network, and output the action strategy to control the robot's movement based on the motion control strategy network.

[0096] The motion control policy network is obtained according to the robot motion control policy network training method based on imitation learning described in the first aspect above.

[0097] According to a fifth aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory storing computer-readable instructions that, when executed by the processor, implement the methods of the first or second aspect described above.

[0098] According to a sixth aspect of this disclosure, a robot is provided, comprising: a processor; and a memory storing computer-readable instructions that, when executed by the processor, implement the method of the second aspect described above.

[0099] Optionally, the aforementioned robots include any one of legged robots, quadruped robots, bipedal robots, wheeled robots, wheel-legged robots, four-wheeled legged robots, humanoid robots, cleaning robots, transport robots, mobile robots, and robotic arms.

[0100] According to a seventh aspect of this disclosure, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, the method of the first or second aspect described above is implemented.

[0101] As can be seen from the above technical solution, this disclosure possesses at least one of the following advantages and positive effects:

[0102] The robot motion control policy network training method based on imitation learning in the exemplary embodiments of this disclosure overcomes the technical bias in related technologies that "explicit gait constraints must be removed in training on complex terrain to avoid over-restriction." By constructing a collaborative mechanism between the style reward layer and the gait reward layer, it significantly improves the physical robustness and environmental adaptability of robot motion control. Specifically, unlike the geometric gait constraints (such as forcibly constraining the absolute lift height of the foot or the preset foot landing point based on flat ground) that easily lead to training failure in related technologies, the exemplary embodiments of this disclosure maintain the anthropomorphic characteristics of the action through the style reward layer, while innovatively adopting a gait reward layer based on contact sequence and biomechanical constraint functions that is independent of terrain. This incorporates the robot's interaction state with the ground (e.g., single-leg swing, bipedal support, airborne phase, etc.) into the optimization objective of the policy network, so that the training process no longer simply pursues visual action reproduction, but explicitly constrains the biomechanical characteristics in the action generation process. Therefore, the motion control strategy network in the exemplary embodiments of this disclosure, since it no longer forcibly specifies the geometric spatial trajectory of the feet, allows the robot to not only freely adjust its foot placement to adapt to the undulations of unstructured terrain such as stairs and rocks, retaining sufficient spatial exploration capabilities, but also to actively learn and adapt to biomechanical physical constraints during the training phase. This embeds physical robustness into the generated motion trajectory, significantly reducing the dynamic deviation between the simulation environment and the real physical environment, effectively eliminating foot slippage and gait instability caused by non-physical actions, and preventing hardware damage caused by excessive ground reaction forces, thereby improving the stability and safety of the control strategy when deployed on a real machine. Attached Figure Description

[0103] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0104] Figure 1 This is a system architecture diagram provided in an embodiment of the present disclosure;

[0105] Figure 2 This is a flowchart of a robot motion control strategy network training method based on imitation learning provided in an embodiment of this disclosure;

[0106] Figure 3 This is a flowchart of a robot motion control strategy network training method based on imitation learning provided in an embodiment of this disclosure;

[0107] Figure 4 This is a flowchart of a method for constructing a style reward layer provided in an embodiment of this disclosure;

[0108] Figure 5 This is a flowchart of a robot motion control strategy network training method based on imitation learning provided in an embodiment of this disclosure;

[0109] Figure 6 This is a flowchart of a robot motion control strategy network training method based on imitation learning provided in an embodiment of this disclosure;

[0110] Figure 7 This is a flowchart of a robot motion control strategy network training method based on imitation learning provided in an embodiment of this disclosure;

[0111] Figure 8 This is a flowchart of a robot motion control strategy network training method based on imitation learning provided in an embodiment of this disclosure;

[0112] Figure 9 This is a flowchart of a robot motion control strategy network training method based on imitation learning provided in an embodiment of this disclosure;

[0113] Figure 10 This is a flowchart of a robot motion control method based on imitation learning provided in an embodiment of this disclosure;

[0114] Figure 11 This is a schematic diagram of the structure of a robot motion control strategy network training device based on imitation learning provided in an embodiment of this disclosure;

[0115] Figure 12This is a schematic diagram of the structure of a robot motion control device based on imitation learning provided in an embodiment of this disclosure;

[0116] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure;

[0117] Figure 14 This is a schematic diagram of a robot provided in an embodiment of this disclosure. Detailed Implementation

[0118] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0119] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0120] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0121] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0122] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0123] To better understand the inventive concept of the embodiments of this disclosure, the technical solutions of the embodiments of this disclosure will be described below in conjunction with exemplary embodiments.

[0124] First, the relevant terms involved in the embodiments of this disclosure will be explained.

[0125] The style reward layer can be understood as a network for generating style reward values.

[0126] Reference human motion data can be understood as human temporal motion data from human demonstrations, motion capture systems, or high-quality expert trajectory libraries.

[0127] Motion data can be understood as the robot's motion data output by the motion control policy network.

[0128] The contact phase can be understood as a discrete or continuous representation of the contact state interval of a robot's foot within a gait cycle.

[0129] Contact sequence data can be understood as a time-series data set used to describe the contact state and contact force changes between the robot's feet and the ground or support surface in the time dimension. It is a sequence arranged by time steps.

[0130] The Gaussian kernel function can be used to map continuous error quantities to continuous, differentiable, and upper-bounded reward components of 1.

[0131] Ground contact dynamics constraints can be used to encourage the foot to remain highly stable in a supported state, without vertical shaking or horizontal slippage.

[0132] Limb coordination rhythm can be understood as the coordinated rhythmic pattern of different limbs in the time dimension during periodic movements, where the frequency, phase relationship and amplitude changes of their movements are consistent.

[0133] Gait frequency-based rhythm constraints can be understood as using the robot's current gait frequency as a time reference to construct a desired periodic motion model for the arm joints, so that the movement rhythm of the arm joints is synchronized with or proportionally related to the gait rhythm of the lower limbs.

[0134] Periodic modulation can be understood as using gait frequency or gait phase as a time reference to periodically change the fundamental amplitude value, thereby generating a rhythmic desired motion signal. Periodic modulation is typically implemented using sine functions, cosine functions, or other periodic functions.

[0135] A historical sliding window sequence can be understood as a sequence composed of the most recent fixed number of first or second means in chronological order for multiple consecutive training batches in the time dimension.

[0136] A simulation environment can be understood as a computational environment used to simulate the interaction between robot dynamics and the environment.

[0137] Figure 1The diagram illustrates a system architecture for a robot motion control strategy network training method and a robot motion control method based on imitation learning, which can be applied to embodiments of this disclosure.

[0138] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a robot 102, a network 103, and a server 104. The terminal device 101 includes, but is not limited to, desktop computers, laptops, smartphones, and tablets. The terminal device 101 is equipped with a graphical user interface, which can provide visualization functions to display the robot 102's operating status, motion trajectory, etc. It can also be used to configure training parameters, control the training process, monitor the model training status, and support sending motion control commands to the robot 102.

[0139] Robot 102 is equipped with a variety of sensors, such as inertial measurement units, joint position sensors, speed sensors, and angle sensors, to collect its own state data in real time during movement.

[0140] Server 104 is equipped with a training module for implementing a robot motion control policy network training method based on imitation learning, which can train the motion control policy network. After training, server 104 can deploy the trained motion control policy network to robot 102, so that robot 102 can output action policies to control robot movement based on the motion control policy network.

[0141] Network 103 serves as the medium for providing a communication link between terminal device 101, robot 102, and server 104. Network 103 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc. It should be understood that... Figure 1 The number and type of terminal devices, robots, networks, and servers shown are merely illustrative. Depending on implementation needs, any number and type of terminal devices, robots, networks, and servers can be included.

[0142] Through the coordinated operation of various parts of the system architecture 100, the robot's motion control strategy network can be trained efficiently. This network enables the robot to not only freely adjust its footing to adapt to the undulations of unstructured terrain such as stairs and rocks, thus preserving its spatial exploration capabilities, but also to actively learn and adapt to biomechanical physical constraints during the training phase. This embeds physical robustness into the generated motion trajectory, significantly reducing the dynamic deviation between the simulation environment and the real physical environment. It effectively eliminates foot slippage and gait instability caused by non-physical actions, and prevents hardware damage caused by excessive ground reaction forces, thereby improving the stability and safety of the control strategy when deployed on a real machine.

[0143] This disclosure provides a method for training a robot motion control policy network based on imitation learning, such as... Figure 2 The flowchart illustrates a method for training a robot motion control policy network based on imitation learning. This method may include the following steps 210 to 230:

[0144] Step 210: Utilize the style reward layer to receive reference human motion data and motion data output by the motion control strategy network, and output the first reward item based on the feature differences between the two.

[0145] Step 220: Receive motion data and corresponding contact sequence data output by the motion control strategy network using the gait reward layer, and calculate the second reward item using a constraint function based on biomechanical design.

[0146] Step 230: Generate a comprehensive reward based on the first reward and the second reward, and use the comprehensive reward to update the parameters of the motion control policy network in order to train the motion control policy network.

[0147] This embodiment of the disclosure overcomes the technical bias in related technologies that "explicit gait constraints must be removed in training on complex terrain to avoid over-restriction" by using a robot motion control policy network training method based on imitation learning. By constructing a collaborative mechanism between a style reward layer and a gait reward layer, it significantly improves the physical robustness and environmental adaptability of robot motion control. Specifically, unlike geometric gait constraints (such as forcibly constraining the absolute lift height of the foot or a preset foot landing point based on a flat ground) that easily lead to training failure in related technologies, the exemplary embodiment of this disclosure maintains the anthropomorphic characteristics of the action through a style reward layer while innovatively adopting a terrain-independent gait reward layer based on contact sequences and biomechanical constraint functions. This incorporates the robot's interaction state with the ground (e.g., single-leg swing, bipedal support, airborne phase, etc.) into the optimization objective of the policy network, so that the training process no longer simply pursues visual action reproduction, but explicitly constrains the biomechanical characteristics in the action generation process. Therefore, the motion control strategy network in the exemplary embodiments of this disclosure, since it no longer forcibly specifies the geometric spatial trajectory of the feet, allows the robot to not only freely adjust its foot placement to adapt to the undulations of unstructured terrain such as stairs and rocks, retaining sufficient spatial exploration capabilities, but also to actively learn and adapt to biomechanical physical constraints during the training phase. This embeds physical robustness into the generated motion trajectory, significantly reducing the dynamic deviation between the simulation environment and the real physical environment, effectively eliminating foot slippage and gait instability caused by non-physical actions, and preventing hardware damage caused by excessive ground reaction forces, thereby improving the stability and safety of the control strategy when deployed on a real machine.

[0148] The following will provide a detailed description of the robot motion control strategy network training method based on imitation learning in the embodiments of this disclosure.

[0149] In step 210, the style reward layer receives reference human motion data and motion data output by the motion control policy network, and outputs a first reward item based on the feature differences between the two.

[0150] The style reward layer can be understood as a style reward value generation network.

[0151] Reference human motion data can be understood as human temporal motion data from human demonstrations, motion capture systems, or high-quality expert trajectory libraries.

[0152] Motion data can be understood as the robot's motion data output by the motion control policy network.

[0153] The first reward item can be understood as an indicator used to characterize the difference in features between the action data output by the motion control strategy network and the reference human action data, that is, to characterize the similarity between the action data output by the motion control strategy network and the reference human action data.

[0154] Specifically, reference human motion data and motion data output by the motion control strategy network can be input into the style reward layer. The style reward layer receives the reference human motion data and the motion data output by the motion control strategy network, and based on the feature differences between the motion data output by the motion control strategy network and the reference human motion data, using the reference human motion data as a benchmark, the style consistency between the motion data output by the motion control strategy network and the reference human motion data is evaluated by comparing the feature differences between them, and a first reward item is generated based on the feature differences.

[0155] The first reward term can be the result of a monotonic mapping of feature differences.

[0156] In some embodiments, the style reward layer may include a first encoding subnetwork, a second encoding subnetwork, and a style scoring subnetwork. The style reward layer receives reference human motion data and motion data output from a motion control policy network, and outputs a first reward item based on the feature differences between the two. Figure 3 The flowchart shown is a method for training a robot motion control policy network based on imitation learning. Figure 3 As shown, the method may include the following steps:

[0157] Step 310: Input the reference human motion data into the first encoding sub-network with shared weights to generate the first state motion feature vector, and input the motion data output by the motion control strategy network into the first encoding sub-network with shared weights to generate the second state motion feature vector.

[0158] In this embodiment of the present disclosure, reference human motion data can be input into a first encoding sub-network with shared weights, and the reference human motion data can be encoded based on the first encoding sub-network to obtain a first state motion feature vector; motion data output from the motion control strategy network can be input into the first encoding sub-network with shared weights, and the motion data can be encoded based on the first encoding sub-network to obtain a second state motion feature vector.

[0159] Step 320: Input the first state action feature vector and the second state action feature vector into the second encoding sub-network with shared weights to obtain the first embedding vector and the second embedding vector.

[0160] The encoding subnetwork can be understood as a network used to abstract features from high-dimensional state-action feature vectors.

[0161] Embedding vectors can be understood as low-dimensional feature representations used to characterize the latent features of action segments at the motion style level.

[0162] The first state action feature vector can be input into the second encoding sub-network with shared weights, and the first state action feature vector can be encoded based on the second encoding sub-network to obtain the first embedding vector; the second state action feature vector can be input into the second encoding sub-network with shared weights, and the second state action feature vector can be encoded based on the second encoding sub-network to obtain the second embedding vector.

[0163] Shared weights can be understood as allowing action data from different sources to be compared under the same feature extraction rules and the same embedding space, thereby ensuring the fairness and effectiveness of style identification.

[0164] Step 330: Input the first embedding vector and the second embedding vector into the style scoring subnetwork to obtain the corresponding discrimination scores.

[0165] Identification scores can be used to characterize the degree of similarity between corresponding action segments and human movements in terms of movement style.

[0166] The first embedding vector can be input into the style scoring subnetwork to obtain the first discrimination score; the second embedding vector can be input into the style scoring subnetwork to obtain the second discrimination score.

[0167] For example, the identification score can range from 0 to 1, with a higher value indicating that the movement style is closer to human movement.

[0168] Therefore, by constructing a shared-weight encoding subnetwork and a style scoring subnetwork, human action samples and policy-generated action samples are mapped to a unified embedding space and corresponding discrimination scores are output, thus achieving a refined evaluation of the similarity between robot action styles and human action styles.

[0169] Step 340: Map the identification score to the first reward item.

[0170] Thus, by using an adversarial motion prior discriminator and a discriminant rating reward mapping, human movement styles are introduced into the policy learning process in the form of endogenous rewards, rather than being externally forced to be corrected. This deeply couples movement style constraints with the policy optimization process, improving learning efficiency and stability, and effectively avoiding unnatural movement problems such as stiffness and rhythm disorder.

[0171] In some embodiments, before the style reward layer receives the reference human motion data and the motion data output by the motion control policy network, the following can be performed: Figure 4 A flowchart illustrating a method for constructing a style reward layer is shown, as follows: Figure 4 As shown, the method may include the following steps:

[0172] Step 410: Construct a discriminator network for the adversarial motion prior framework.

[0173] In this embodiment of the disclosure, a discriminator network of the Adversarial Motion Priors (AMP) framework can be constructed.

[0174] The adversarial motion prior framework can be understood as a method of learning motion priors from human motion data using adversarial learning, and then using these priors as reward signals for training imitation learning control strategies.

[0175] Step 420: Train the discriminator network based on the reference human motion data and the motion data output by the motion control strategy network, and update the parameters of the discriminator network to obtain the trained discriminator network.

[0176] In this embodiment of the present disclosure, reference human motion data and motion data output by the motion control strategy network can be input into the discriminator network. The discriminator network is trained based on the reference human motion data and motion data output by the motion control strategy network, and the parameters of the discriminator network are updated to obtain a trained discriminator network.

[0177] In some embodiments, training the discriminator network based on reference human motion data and motion data output by the motion control strategy network, and updating the parameters of the discriminator network to obtain a trained discriminator network, may include steps 4201-4203:

[0178] Step 4201: Extract the first action segment from the reference human action data as a reference sample.

[0179] Extracting the first action segment from reference human action data as a reference sample can be understood as selecting a sequence of actions over a continuous period of time from the temporal data of human demonstration actions as a human style sample.

[0180] Step 4202: Extract the second motion segment from the motion data output by the motion control strategy network as a generated sample.

[0181] Extracting a second motion segment from the motion data output by the motion control policy network as a generated sample can be understood as selecting a continuous motion sequence from the robot motion trajectory generated by the current policy network that corresponds to the reference sample in terms of time length and feature dimension.

[0182] Step 4203: Update the parameters of the discriminator network based on the reference samples and generated samples to obtain the trained discriminator network.

[0183] In this embodiment of the disclosure, updating the parameters of the discriminator network based on reference samples and generated samples can be understood as updating the parameters of the discriminator network by comparing the feature distributions of the two types of action segments, so that it learns to distinguish the style differences between human actions and robot-generated actions, and thus obtains a trained discriminator network.

[0184] In some embodiments, the above-mentioned updating of the discriminator network parameters based on reference samples and generated samples to obtain a trained discriminator network may include S11-S13:

[0185] S11. Input the reference sample into the discriminator network to obtain the first discriminant output.

[0186] The first discriminant output can be used to characterize the confidence level that the reference sample belongs to human action.

[0187] S12. The generated sample is input into the discriminator network to obtain the second discriminant output.

[0188] The second discriminant output can be used to characterize the confidence level that the generated sample belongs to human action.

[0189] S13. Calculate the discrimination loss based on the first discrimination output and the second discrimination output, and perform backpropagation on the discriminator network based on the discrimination loss to update the parameters of the discriminator network, thereby obtaining the trained discriminator network.

[0190] In this embodiment, a reference sample is input into a discriminator network to obtain a first discriminant output representing that the reference sample belongs to a human action; a generated sample is input into the same discriminator network to obtain a second discriminant output representing that the generated sample belongs to a human action; a discriminant loss is constructed based on the difference between the first and second discriminant outputs, and the discriminant loss is used to perform backpropagation updates on the discriminator network, so that the discriminator network gradually enhances its ability to distinguish between human action samples and policy-generated action samples.

[0191] Step 430: Determine the trained discriminator network as the style reward layer.

[0192] Therefore, the discriminator network can be trained based on reference human action data and action data output by the motion control policy network to obtain a style reward layer. The style reward layer can then receive reference human action data and action data output by the motion control policy network, and output a first reward item based on the feature differences between the two.

[0193] In step 220, the gait reward layer receives motion data output from the motion control strategy network and contact sequence data corresponding to the motion data, and calculates the second reward term through a constraint function based on biomechanical design.

[0194] The gait reward layer can be understood as a gait reward value generation network.

[0195] The contact sequence data may include contact force sensing data and contact phases that are time-step aligned between the robot's foot segments and the motion control strategy network output motion data. The contact phases may include at least two of the following: a single-leg swing phase, a two-leg support phase, and a flight phase.

[0196] The single-leg swing phase can be understood as a period during which the foot does not contact the ground.

[0197] The bipedal support phase can be understood as the period when both feet are in contact with the ground simultaneously.

[0198] The airborne phase can be understood as the period when neither foot is in contact with the ground.

[0199] Contact force sensing data can be understood as data characterizing the contact state and force conditions between the robot's foot (or end effector) and the supporting surface, used to reflect "whether it is stepping on the ground, how stable it is, and whether there is a tendency to slip," etc. For example, contact force sensing data can include vertical contact force (contact normal force), tangential force, friction, and other data.

[0200] The contact phase can be understood as a discrete or continuous representation of the contact state interval of a robot's foot within a gait cycle.

[0201] The constraint function of biomechanical design can be understood as a reward calculation function that evaluates the quality of the robot's motion data and the corresponding contact sequence data output by the motion control strategy network based on the physical motion laws in human or humanoid gait.

[0202] The second reward can be understood as a reward value obtained by evaluating the robot's motion data and the corresponding contact sequence data output by the motion control strategy network based on the constraint function designed by the gait reward layer using biomechanical design. It is used to characterize the degree of conformity of the robot's motion data output by the motion control strategy network at the physical or gait level, such as gait stability, foot contact rationality, and kinematic continuity.

[0203] Specifically, the motion data output by the motion control strategy network and the corresponding contact sequence data can be input into the gait reward layer, and the second reward can be calculated using a constraint function based on biomechanical design.

[0204] In some embodiments of this disclosure, the above-described biomechanical design-based constraint function can be configured as follows:

[0205] The current state of each foot is determined based on the contact phase, including the swinging state and the supporting state.

[0206] Kinematic trajectory tracking constraints are applied to the foot in a swinging state, and ground contact dynamic constraints are applied to the foot in a supporting state.

[0207] In some embodiments, the above-described kinematic trajectory tracking constraint applied to the foot in a swinging state can be executed. Figure 5 The flowchart shown is a method for training a robot motion control policy network based on imitation learning. Figure 5 As shown, the method may include the following steps:

[0208] Step 510: Generate the desired motion trajectory of the foot during the swinging process. The desired motion trajectory includes the desired height curve and the desired vertical velocity curve.

[0209] The expected trajectory can be understood as a reference trajectory used to evaluate whether the current foot movement is reasonable.

[0210] The desired height curve can be understood as a function of the height of the foot relative to the ground or the body coordinate system as a function of time during the swing phase.

[0211] The desired vertical velocity curve can be understood as the derivative of the desired height curve with respect to time.

[0212] In this embodiment of the disclosure, generating the desired motion trajectory of the foot during the swing process means that when the foot is in the swing phase, a reference motion model for the vertical direction is constructed based on the gait phase, a height curve of the desired height of the foot as a function of the phase is generated by a preset function, and the desired vertical velocity curve is generated by the time derivative of the height curve, so as to serve as a reference benchmark for evaluating the continuity, smoothness and non-contact nature of the foot swing behavior in the gait reward layer.

[0213] In some embodiments, generating the desired motion trajectory of the foot during the swinging process may include steps 5101-5103:

[0214] Step 5101: Obtain the current gait phase value. The gait phase value changes periodically over time within a preset range.

[0215] In this embodiment of the disclosure, the current gait phase value of the robot can be obtained. The gait phase value changes periodically over time within a preset interval.

[0216] Gait phase values ​​can be understood as continuous variables used to characterize the relative temporal position of a robot within a gait cycle. The gait phase values ​​change monotonically with time within a preset phase interval and are periodically updated after completing a gait cycle, serving as the phase independent variable for generating the desired foot trajectory.

[0217] Gait phase values ​​are not geometric constraints on the spatial position of the foot, but rather a continuous phase parameter used to uniformly describe the temporal relationships within the gait cycle. Their role is to provide a time reference for biomechanical constraints, rather than limiting the specific foot landing position.

[0218] Step 5102: Use a preset spline curve function to map the gait phase value to the foot height value that changes with the phase, so as to form the desired height curve.

[0219] In this embodiment of the disclosure, the gait phase value is mapped to the foot height value that varies with the phase using a preset spline curve function. This means that the gait phase value is used as the independent variable, and the expected height value of the foot at the corresponding phase is calculated by using a pre-constructed continuous differentiable spline curve function within the swing phase interval. This forms an expected height reference curve that varies continuously with the gait phase, which is used to smooth the lifting and landing behavior of the foot during the swing process, without limiting the specific horizontal landing position of the foot.

[0220] Step 5103: Calculate the derivative of the desired height curve with respect to time to generate the desired vertical velocity curve.

[0221] Therefore, the desired height curve and desired vertical velocity curve can be accurately generated, and the desired motion trajectory of the foot during the swinging process can be obtained.

[0222] Step 520: Obtain the real-time motion state of the foot and calculate the height tracking error between the real-time motion state and the desired height curve, as well as the speed tracking error between the real-time motion state and the desired vertical speed curve.

[0223] Height tracking error can be understood as the difference between the actual vertical height of the foot at the current moment and the reference vertical height calculated from the expected height curve when the foot is in the swing phase. It is used to quantify whether the foot swing height conforms to the expected foot lift and landing trajectory.

[0224] This height tracking error essentially measures whether the robot's feet lift to the expected height when swinging.

[0225] Speed ​​tracking error can be understood as the difference between the actual vertical velocity of the foot at the current moment when the foot is in the swing phase and the expected vertical velocity calculated based on the expected height curve. It is used to quantify the smoothness and dynamic rationality of the foot lifting and landing process.

[0226] Speed ​​tracking error essentially measures how smoothly a robot's feet lift when swinging.

[0227] Step 530: Obtain the real-time contact force of the foot from the contact sequence data, and generate the contact force deviation value when the real-time contact force is not zero.

[0228] Contact sequence data can be understood as a time-series data set used to describe the contact state and contact force changes between the robot's feet and the ground or support surface in the time dimension. It is a sequence arranged by time steps.

[0229] Contact sequence data may include contact force sensing data and contact phase information. Contact force sensing data may include the normal contact force or equivalent contact reaction force between the foot and the ground. Contact phase information is used to characterize the phase label or determination result of whether the foot belongs to the swing phase, support phase or take-off phase in the current time step.

[0230] Contact force can be understood as the mechanical quantity of the reaction force exerted by the ground on the foot when the foot comes into contact with the ground, in the foot coordinate system or the world coordinate system.

[0231] The contact force deviation value can be understood as a deviation index generated when a non-zero contact force is detected at the foot during the swinging phase, which is used to characterize the degree of undesired contact during the swinging phase. The contact force deviation value is used to quantify the degree of undesired contact that occurs at the foot during the swinging phase.

[0232] This embodiment introduces a contact force deviation value generation mechanism based on contact sequence data within the foot swing phase, enabling the robot to explicitly avoid undesirable foot contact behaviors during training, thereby ensuring the non-contact nature of the swing phase and the physical feasibility of the gait.

[0233] Step 540: Based on the height tracking error, velocity tracking error, and contact force deviation, calculate the reward value of the kinematic trajectory tracking constraint as a component of the second reward item.

[0234] In this embodiment of the disclosure, the height tracking error, speed tracking error, and contact force deviation value can be weighted and summed to obtain the reward value of the kinematic trajectory tracking constraint, and the reward value of the kinematic trajectory tracking constraint can be used as a component of the second reward item.

[0235] In some embodiments, the calculation of the reward value for the kinematic trajectory tracking constraint described above may include S21-S22:

[0236] S21. Use the Gaussian kernel function to process the height tracking error to obtain the height tracking reward component, use the Gaussian kernel function to process the speed tracking error to obtain the speed tracking reward component, and use the Gaussian kernel function to process the contact force deviation value to obtain the contact force compliance reward component.

[0237] The Gaussian kernel function can be used to map continuous error quantities to a continuous, differentiable, upper-bounded reward component. When the error value is zero, the Gaussian kernel function outputs its maximum value, and the output value continuously decreases as the error value increases.

[0238] For example, let the altitude tracking error be e. h Then the highly tracked reward component r h for:

[0239] ;

[0240] Where, σ h This indicates the tolerance scale for high error.

[0241] For example, let the speed tracking error be e. v Then the speed tracking reward component r v for:

[0242] ;

[0243] Where, σ v This indicates the tolerance scale for speed error.

[0244] For example, let the contact force deviation value be e. F Then the contact force compliance reward component r F for:

[0245] ;

[0246] Where, σ F This indicates the tolerance scale for contact force error.

[0247] Therefore, by using the Gaussian kernel function to process the height tracking error and obtain the height tracking reward component, the foot can be continuously and smoothly guided to form reasonable changes in lift and land height during the swing phase without forcibly limiting the geometric trajectory of the foot, thereby reducing the risk of tripping and improving gait stability.

[0248] By using the Gaussian kernel function to process the velocity tracking error and obtain the velocity tracking reward component, the dynamic smoothness of the foot swing process can be explicitly constrained, effectively suppressing the problems of sudden velocity changes and excessive impact during the foot lifting and falling process, thereby reducing the peak landing impact force and improving the hardware safety of the real robot during operation.

[0249] By using the Gaussian kernel function to process the contact force deviation value, the contact force compliance reward component can be obtained. This can explicitly constrain the non-contact nature of the foot during the swing phase, preventing the strategy from obtaining rewards by reducing the height of the foot lift or touching the ground prematurely, thereby effectively eliminating non-physical movement behaviors such as dragging the foot and wiping the ground.

[0250] S22. The reward value is obtained by weighted summation of the height tracking reward component, the speed tracking reward component, and the contact force compliance reward component.

[0251] This results in the foot swinging behavior being jointly constrained by spatial height, temporal rate of change, and contact state; the constraints are balanced through weight parameters to avoid a single constraint dominating the training process.

[0252] The reward function remains continuous and differentiable, which is beneficial for stable training and convergence. Without introducing strong geometric foot placement constraints, multi-dimensional, soft-constraint biomechanical restrictions are imposed on the foot swing phase movement behavior, so that the generated motion trajectory achieves a balance between visual anthropomorphism and physical feasibility.

[0253] In some embodiments, the above-described application of ground contact dynamic constraints to the foot in a supported state can be performed. Figure 6 The flowchart shown is a method for training a robot motion control policy network based on imitation learning. Figure 6 As shown, the method may include the following steps:

[0254] Step 610: Construct a foot-locking target. Set the desired height off the ground, desired vertical velocity, and desired horizontal velocity of the foot to zero.

[0255] In this embodiment of the disclosure, the robot has at least one foot, and each foot can be in different contact states during walking. The supporting state can be understood as the state in which the foot remains in contact with the ground and is used to support the weight of the robot body.

[0256] The support state can be determined based on any one or a combination of the following conditions: the normal contact force detected at the foot is greater than the preset contact force threshold, the gait phase value is within the preset support phase range, and the foot contact indicator signal is "contact".

[0257] In a supported state, the desired motion characteristics of the foot are significantly different from those in a swinging state, with the core objective being to maintain stability relative to the ground.

[0258] Foot-locking targets can be understood as a set of desired motion constraints set to maintain stable contact between the foot and the ground when the foot is in a supporting position. Foot-locking targets can be understood as reference targets used to evaluate whether foot movement meets the requirements of support stability.

[0259] The target is locked at the foot, and the desired height off the ground, desired vertical velocity, and desired horizontal velocity of the foot are all set to zero.

[0260] A desired ground clearance of zero means that the foot should remain in contact with the ground and not lift off when in a supported position. Ground clearance refers to the vertical distance of the foot relative to a ground reference plane.

[0261] The expectation that the vertical velocity is zero means that the foot should not move up and down in the vertical direction. Vertical velocity refers to the instantaneous velocity component of the foot in the vertical direction.

[0262] A desired horizontal velocity of zero indicates that the foot should not slip in the horizontal direction. Horizontal velocity refers to the velocity component of the foot within the plane of the ground.

[0263] Step 620: Obtain the real-time motion state of the foot and calculate the height steady-state deviation, vertical steady-state deviation, and horizontal steady-state deviation of the real-time motion state relative to the foot-locked target.

[0264] Real-time motion states can include at least real-time ground clearance, real-time vertical velocity, and real-time horizontal velocity. These real-time motion states can be obtained through simulation environments, kinematic calculations, or sensor data.

[0265] In this embodiment of the disclosure, steady-state deviation can be used to describe the degree of deviation of the foot from the locked target in a supported state.

[0266] Steady-state height deviation can be understood as the difference between the real-time ground clearance of the foot in a supported state and the desired ground clearance. Steady-state height deviation can be used to measure whether undesirable foot lifting or sinking occurs.

[0267] Vertical steady-state deviation can be understood as the difference between the real-time vertical velocity of the foot in a supported state and the desired vertical velocity. Vertical steady-state deviation can be used to measure whether the foot is shaking or oscillating in the vertical direction.

[0268] Horizontal steady-state deviation can be understood as the deviation between the real-time horizontal velocity of the foot in a supported state and the desired horizontal velocity. Horizontal steady-state deviation can be used to measure whether the foot slips on the ground.

[0269] Step 630: Calculate the reward value of the ground contact dynamic constraint based on the height steady-state deviation, vertical steady-state deviation and horizontal steady-state deviation, and use it as a component of the second reward item.

[0270] In this embodiment, the reward value of the ground contact dynamics constraint can be calculated based on the height steady-state deviation, vertical steady-state deviation, and horizontal steady-state deviation, and the reward value of the ground contact dynamics constraint can be used as a component of the second reward item. The component of the second reward item can be used together with other reward components to participate in the parameter update of the motion control strategy network.

[0271] Ground contact dynamics constraints can be used to encourage the foot to remain highly stable in a supported state, without vertical shaking or horizontal slippage.

[0272] In some embodiments, the calculation of the reward value for ground contact dynamic constraints based on the height steady-state deviation, vertical steady-state deviation, and horizontal steady-state deviation may include steps 6301-6302:

[0273] Step 6301: Substitute the height steady-state deviation into the Gaussian radial basis function to generate the foot-touching reward item; substitute the vertical steady-state deviation into the Gaussian radial basis function to generate the vertical stability reward item; substitute the horizontal steady-state deviation into the Gaussian radial basis function to generate the anti-slip reward item.

[0274] In this embodiment of the disclosure, the Gaussian Radial Basis Function (RBF) can be understood as a nonlinear function with the magnitude of the error as the independent variable and the output value continuously decaying as the error increases.

[0275] Foot-to-ground reward can be used to encourage the feet to remain in contact with the ground while in a supporting position.

[0276] Vertical stability bonus terms can be used to constrain the dynamic stability of the foot in the vertical direction under support.

[0277] Anti-slip bonuses can be used to suppress horizontal slippage of the foot while in a supporting position.

[0278] Substituting the steady-state height deviation into the Gaussian radial basis function generates a foot-to-ground reward term. When the foot height is close to the ground (steady-state height deviation ≈ 0), the foot-to-ground reward term takes a larger value; when the foot lifts up or floats, the foot-to-ground reward term decays rapidly with increasing deviation.

[0279] Substituting the vertical steady-state deviation into the Gaussian radial basis function generates a vertical stability reward term. This vertical stability reward term takes a larger value when the vertical velocity at the foot tip is close to zero, thereby suppressing the up-and-down oscillations during the support phase.

[0280] Substituting the horizontal steady-state deviation into the Gaussian radial basis function generates an anti-slip reward term. When the foot does not move significantly within the ground plane, the anti-slip reward term approaches its maximum value; when the foot slides or drags, the reward term drops rapidly.

[0281] Step 6302: Sum the foot contact bonus, vertical stability bonus, and anti-slip bonus to output the bonus value.

[0282] Therefore, by constructing a foot-locked target and generating ground contact dynamics constraint rewards based on steady-state deviations of height, vertical velocity, and horizontal velocity, the relative stability between the foot and the ground is explicitly constrained in the support state; this effectively suppresses foot lifting, vibration, or slippage during the support period; reduces abrupt changes in ground reaction force and improves overall gait stability; and enhances the robot's walking reliability and safety in real physical environments. It achieves continuous constraint on foot stability in the support state, thereby significantly improving the physical feasibility and operational stability of the gait.

[0283] In some embodiments of this disclosure, the above-mentioned biomechanical design-based constraint function is configured as follows:

[0284] The current state of each foot is determined based on the contact phase, including the swinging state and the supporting state.

[0285] Kinematic trajectory tracking constraints are applied to the foot in a swinging state, and ground contact dynamic constraints are applied to the foot in a supporting state.

[0286] Perform limb coordination rhythm constraints based on gait frequency for the robot's arm joints.

[0287] The robot's arm joints can be understood as the rotational joints that make up the robot's upper limb kinematic chain, typically including but not limited to: shoulder joint, elbow joint, and wrist joint.

[0288] Gait frequency can be understood as the number of times a robot completes a full gait cycle per unit of time during walking.

[0289] Limb coordination rhythm can be understood as the coordinated rhythmic pattern of different limbs in the time dimension during periodic movements, where the frequency, phase relationship and amplitude changes of their movements are consistent.

[0290] Gait frequency-based rhythm constraints can be understood as using the robot's current gait frequency as a time reference to construct a desired periodic motion model for the arm joints, so that the movement rhythm of the arm joints is synchronized with or proportionally related to the gait rhythm of the lower limbs.

[0291] This disclosure embodiment constructs a desired rhythm model of the arm joint based on gait frequency and compares it with the real-time motion state of the arm joint to generate a coordinated rhythm reward, thereby realizing the coordinated constraint of the robot's upper and lower limbs at the temporal rhythm level, and thus significantly improving the motion coordination and anthropomorphism during walking.

[0292] In some embodiments, the above-described application of limb coordination rhythm constraints based on gait frequency to the robot's arm joints can perform... Figure 7 The flowchart shown is a method for training a robot motion control policy network based on imitation learning. Figure 7 As shown, the method may include the following steps:

[0293] Step 710: Analyze the linear velocity command in the input command of the motion control strategy network, and determine the basic amplitude value of the arm swing based on the linear velocity command.

[0294] The input instructions to the motion control strategy network can be understood as the control target information input to the motion control strategy network by the high-level planning module or human-computer interaction module, which describes the overall motion intention that the robot is expected to perform.

[0295] Linear velocity commands can be understood as control quantities describing the magnitude and direction of a robot's overall walking or moving speed. Linear velocity commands reflect the robot's current walking intensity and are an important basis for determining gait amplitude and limb swing amplitude.

[0296] The basic amplitude value of arm swing can be understood as the reference swing amplitude set for the arm joint based on the overall motion intensity of the robot.

[0297] The baseline amplitude of the arm swing can be determined based on the linear velocity command. For example, the baseline amplitude is smaller when walking at low speed, and larger when walking at high speed to maintain natural and coordinated movements.

[0298] Step 720: Periodically modulate the base amplitude value to generate a desired arm state sequence containing the desired position and desired velocity.

[0299] Periodic modulation can be understood as using gait frequency or gait phase as a time reference to periodically change the fundamental amplitude value, thereby generating a rhythmic desired motion signal. Periodic modulation is typically implemented using sine functions, cosine functions, or other periodic functions.

[0300] The desired arm state sequence can be understood as a set of desired motion states generated for the arm joints within a continuous time step, used to describe the ideal motion behavior that the arm should achieve under rhythmic constraints.

[0301] The desired arm state sequence can include at least the desired position and the desired velocity.

[0302] In some embodiments, the periodic modulation of the base amplitude value to generate a desired arm state sequence containing the desired position and desired velocity may include steps 7201-7204:

[0303] Step 7201: Extract the sine and cosine eigenvalues ​​of the current gait phase.

[0304] Step 7202: Multiply the basic amplitude value by the sine eigenvalue and the cosine eigenvalue respectively to obtain the desired position.

[0305] Step 7203: Multiply the basic amplitude value by the time derivatives of the sine and cosine eigenvalues ​​respectively to obtain the desired velocity.

[0306] Step 7204: Based on the desired position and desired velocity, construct a desired arm state sequence containing the desired position and desired velocity.

[0307] Step 730: Obtain the real-time state of the robot arm and calculate the cooperative tracking error between the real-time state of the robot arm and the desired arm state sequence.

[0308] The real-time state of a robotic arm can be understood as the current actual motion state of the arm joints, obtained by sensors or a simulation environment within each control cycle. For example, this may include real-time joint angles and real-time joint angular velocities.

[0309] Cooperative tracking error can be understood as the deviation between the real-time motion state of the robot arm and the expected sequence of arm states. It is used to measure whether the arm motion meets the cooperative rhythm requirements based on gait frequency.

[0310] Step 740: Generate a reward value for limb coordination rhythm constraint based on the collaborative tracking error as a component of the second reward item.

[0311] The reward value of limb coordination rhythm constraint can be understood as a reward signal constructed based on the coordination tracking error, which is used to guide the motor control policy network to learn the coordinated movement pattern of the upper and lower limbs at the temporal rhythm level during training.

[0312] The embodiments of this disclosure adaptively determine the arm swing amplitude by parsing the linear velocity command, generate an arm rhythm state sequence containing the desired position and desired velocity based on the gait frequency, and construct a reward signal based on the cooperative tracking error. This realizes the cooperative constraint of the robot arm and lower limb gait at the temporal rhythm level, thereby significantly improving the overall motion coordination and anthropomorphism.

[0313] In some embodiments, after obtaining the components of the second reward item, the components of each second reward item can be combined to obtain the second reward item.

[0314] This embodiment of the present disclosure achieves multi-constraint collaborative expression by merging the components of each second reward item after calculating the components of each second reward item separately to form a unified second reward item. It integrates reward components from different motion constraint dimensions (including but not limited to foot swing constraint, support contact constraint, and limb coordination rhythm constraint) into a single reward signal, enabling the motion control strategy network to simultaneously perceive multiple physical and kinematic constraint requirements in a single parameter update.

[0315] By merging multiple reward components, the policy network is prevented from over-optimizing for a single constraint, and the training process is avoided from being dominated by a single reward. The components of each second reward term are continuous functions, and the overall reward signal remains continuous and differentiable after merging, ensuring gradient stability and convergence during the imitation learning training process.

[0316] In step 230, a comprehensive reward is generated based on the first reward and the second reward, and the parameters of the motion control policy network are updated using the comprehensive reward to train the motion control policy network.

[0317] In this embodiment, a comprehensive reward can be generated based on a first reward item corresponding to the action style and a second reward item corresponding to the gait. For example, the first reward item and the second reward item can be weighted and summed to obtain the comprehensive reward item. The parameters of the motion control strategy network can be updated using the comprehensive reward item to train the motion control strategy network. In other words, the comprehensive reward item can be used as a training feedback signal to optimize the parameters of the motion control strategy network, thereby adjusting the parameter distribution of the motion control strategy network so that the motion control strategy network outputs a motion control strategy that meets the requirements of style consistency and gait stability during the interaction with the environment.

[0318] In some embodiments of this disclosure, the above-described generation of a comprehensive reward based on the first reward item and the second reward item can be executed. Figure 8 The flowchart shown is a method for training a robot motion control policy network based on imitation learning. Figure 8 As shown, the method may include the following steps:

[0319] Step 810: Calculate the batch average for the first reward item and the second reward item respectively to obtain the first average and the second average.

[0320] A batch can be understood as a set of time steps or trajectory samples continuously collected by the motion control policy network in a simulation or real environment before a single imitation learning parameter update.

[0321] The batch mean can be understood as a statistic obtained by summing the values ​​of a certain reward item across all samples within the same batch and dividing by the sample size. It is used to characterize the overall level of that reward item in the current batch.

[0322] The mean of the first reward item within the current batch is the first mean. The first mean can be used to reflect the overall level of style rewards (such as adversarial motion prior identification scores) in the current batch.

[0323] The average of the second reward item within the current batch is the second mean. The second mean can be used to reflect the overall level of physical kinematic constraint rewards (such as swing phase, support phase, synergistic rhythm, etc.) in the current batch.

[0324] Step 820: Update the first weight corresponding to the first reward item based on the first mean, and update the second weight corresponding to the second reward item based on the second mean.

[0325] In some embodiments, the above-mentioned updating the first weight corresponding to the first reward item based on the first mean and updating the second weight corresponding to the second reward item based on the second mean may include steps 8201-8203:

[0326] Step 8201: Obtain the historical sliding window sequence of the first mean and the second mean.

[0327] A historical sliding window sequence can be understood as a sequence composed of the most recent fixed number of first or second means in chronological order for multiple consecutive training batches in the time dimension.

[0328] Step 8202: Calculate the moving average of the historical sliding window sequence of the first mean to obtain the first moving average, and calculate the moving average of the historical sliding window sequence of the second mean to obtain the second moving average.

[0329] Moving average can be understood as a statistical method that smooths the values ​​within a sliding window to reduce short-term fluctuations and highlight long-term trends.

[0330] In this embodiment, a moving average is used to stabilize the reward scale estimation and avoid outliers in a single batch having an excessive impact on the weight calculation.

[0331] Step 8203: Calculate the normalization ratio based on the first moving average and the second moving average to obtain the first weight and the second weight.

[0332] The normalization ratio can be understood as a set of ratio coefficients calculated based on the first moving average and the second moving average, used to map rewards from different sources to the same weight scale.

[0333] The first weight can be used to adjust the relative contribution of the first reward item in the joint reward.

[0334] The second weight can be used to adjust the relative contribution of the second reward item in the joint reward.

[0335] Therefore, by constructing a historical sliding window and calculating a moving average based on the first and second means, and then generating normalized weights based on the moving average, an adaptive balance of multiple reward items in the time dimension is achieved, thereby significantly improving the stability and robustness of the multi-constraint joint training process.

[0336] Step 830: Perform weighted processing on the first reward item based on the updated first weight, and perform weighted processing on the second reward item based on the updated second weight.

[0337] Step 840: Sum the weighted first reward item and the second reward item to obtain the comprehensive reward item.

[0338] This embodiment of the disclosure constructs a comprehensive reward item that adaptively balances as the training process by performing batch-level statistics on different reward items and dynamically updating the weights based on the statistical results, and then performing weighted fusion on the reward items. This significantly improves the coordination and anthropomorphism of the robot's movements while ensuring physical stability.

[0339] In some embodiments of this disclosure, the above-described method of updating the parameters of the motion control policy network using a comprehensive reward term can be implemented. Figure 9 The flowchart shown is a method for training a robot motion control policy network based on imitation learning. Figure 9 As shown, the method may include the following steps:

[0340] Step 910: Collect trajectory data based on the motion control strategy network in the simulation environment, and record the comprehensive reward items corresponding to the trajectory data.

[0341] A simulation environment can be understood as a computational environment used to simulate the interaction between robot dynamics and the environment, capable of generating robot state transitions, contact feedback, and reward signals under offline conditions.

[0342] In the simulation environment, the robot moves in the simulation environment based on the motion control strategy network. The motion control strategy network collects the trajectory data of the robot in the simulation environment and records the comprehensive reward items corresponding to the trajectory data.

[0343] The trajectory data is time series data.

[0344] Step 920: Calculate the strategy gradient update amount based on trajectory data and comprehensive reward items.

[0345] The policy gradient update can be understood as the direction and magnitude of network parameter updates calculated based on the policy gradient algorithm, which is used to indicate how to adjust the policy parameters to improve the expected cumulative reward.

[0346] In some embodiments, the above calculation of the strategy gradient update based on trajectory data and comprehensive reward term may include steps 9201-9203:

[0347] Step 9201: Accumulate the comprehensive reward item for the trajectory data by time step to generate a reward sequence, and form a reward vector based on the reward sequence.

[0348] The reward can be understood as the weighted sum of the comprehensive rewards obtained from a certain time step and accumulated along the trajectory, used to measure the long-term contribution of the action at that time step to the overall task objective.

[0349] The return sequence can be understood as a sequence of return values ​​calculated along the time dimension for each time step.

[0350] The reward vector can be understood as organizing the reward sequence in vector form, which facilitates subsequent batch calculations and matrix operations.

[0351] Step 9202: Calculate the baseline estimation vector based on the return vector, and calculate the advantage estimation sequence based on the return vector and the baseline estimation vector.

[0352] The baseline estimation vector can be understood as a sequence of reference values ​​obtained by estimating the reward vector, used to reduce the variance of the policy gradient estimation without changing the gradient expectation.

[0353] The advantage estimation sequence can be understood as the sequence of differences between the reward vector and the corresponding baseline estimation vector, used to measure the superiority or inferiority of a certain action relative to the average level.

[0354] Step 9203: Input the advantage estimation sequence and trajectory data into the preset optimizer to generate policy gradient update.

[0355] A pre-defined optimizer can be understood as an optimization algorithm module used to calculate policy parameter updates based on advantage estimation results and trajectory data, such as a policy gradient-based optimizer or its variants.

[0356] Step 930: Update the parameters of the motion control policy network based on the policy gradient update amount to obtain the updated parameters of the motion control policy network.

[0357] In this embodiment, trajectory data is collected in a simulation environment and the policy gradient update amount is calculated based on the comprehensive reward term. The execution parameters of the motion control policy network are updated, and a closed-loop optimization mechanism with comprehensive reward as the core is constructed. This significantly improves the coordination, anthropomorphism and executability of robot motion in real environment while ensuring physical stability.

[0358] This disclosure provides a robot motion control method based on imitation learning, such as... Figure 10 A flowchart of a robot motion control method based on imitation learning is shown, such as... Figure 10 As shown, the method may include the following steps:

[0359] Step 1010: Obtain the robot's current observation status.

[0360] Step 1020: Input the current observation state into the pre-trained motion control policy network, and output the action policy for controlling the robot's movement based on the motion control policy network;

[0361] The motion control policy network is obtained according to the robot motion control policy network training method based on imitation learning in any of the above embodiments.

[0362] In this exemplary embodiment, while maintaining the anthropomorphic characteristics of the movements through a style reward layer, an innovative gait reward layer based on contact sequences and biomechanical constraint functions, independent of terrain, is adopted. This incorporates the robot's interaction with the ground (e.g., single-leg swing, bipedal support, airborne phase, etc.) into the optimization objective of the policy network. This means the training process no longer simply pursues visual motion reproduction but explicitly constrains the biomechanical characteristics during motion generation. Therefore, the motion control policy network in this exemplary embodiment, by no longer forcibly defining the geometric spatial trajectory of the feet, allows the robot to freely adjust its foot placement to adapt to the undulations of unstructured terrain such as stairs and rocks, retaining ample spatial exploration capabilities. Furthermore, it can actively learn and adapt to biomechanical physical constraints during the training phase, thus embedding physical robustness into the generated motion trajectory. This significantly reduces the dynamic deviation between the simulation environment and the real physical environment, effectively eliminating foot slippage and gait instability caused by non-physical movements, and preventing hardware damage due to excessive ground reaction forces, thereby improving the stability and safety of the control strategy when deployed on a real machine.

[0363] This disclosure provides a robot motion control strategy network training device based on imitation learning, such as... Figure 11 The diagram shows a structural schematic of a robot motion control strategy network training device based on imitation learning. Figure 11 As shown, the robot motion control policy network training device 1100 based on imitation learning includes:

[0364] The style reward layer construction module 1110 is used to receive reference human motion data and motion data output by the motion control strategy network using the style reward layer, and output a first reward item based on the feature difference between the two.

[0365] Gait reward layer construction module 1120 is used to receive motion data output by the motion control strategy network and contact sequence data corresponding to the motion data using the gait reward layer, and calculate the second reward item through a constraint function based on biomechanical design.

[0366] The policy network update module 1130 is used to generate a comprehensive reward based on the first reward and the second reward, and to update the parameters of the motion control policy network using the comprehensive reward to train the motion control policy network.

[0367] The robot motion control strategy network training device based on imitation learning provided in this disclosure can implement the robot motion control strategy network training method based on imitation learning in any of the above embodiments. Its execution method and beneficial effects are similar, and will not be described again here.

[0368] This disclosure provides a robot motion control device based on imitation learning, such as... Figure 12 The diagram shows a structural schematic of a robot motion control device based on imitation learning. Figure 12 As shown, the robot motion control device 1200 based on imitation learning includes:

[0369] The data acquisition module 1210 is used to acquire the robot's current observation status;

[0370] The control strategy module 1220 is used to input the current observation state into the pre-trained motion control strategy network and output the action strategy for controlling the robot's motion based on the motion control strategy network.

[0371] The motion control policy network is obtained according to the robot motion control policy network training method based on imitation learning in any of the above embodiments.

[0372] The robot motion control device based on imitation learning provided in this disclosure can implement the robot motion control method based on imitation learning in any of the above embodiments. Its execution method and beneficial effects are similar, and will not be described again here.

[0373] This disclosure also provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program. When the computer program is executed by the processor, it can implement the methods of any of the above embodiments. The execution method and beneficial effects are similar, and will not be described again here.

[0374] The electronic device in this disclosure can be understood as any device with processing and computing capabilities. This device may include, but is not limited to, mobile terminals such as smartphones, laptops, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), in-vehicle terminals, and wearable devices, as well as fixed electronic devices such as digital TVs, desktop computers, and smart home devices.

[0375] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure, such as... Figure 13 As shown, the electronic device 1300 may include a processor 1310 and a memory 1320. The memory 1320 stores a computer program 1321. When the computer program 1321 is executed by the processor 1310, it can implement the method provided in any of the above embodiments. The execution mode and beneficial effects are similar and will not be described again here.

[0376] Of course, for the sake of simplicity, Figure 13 Only some of the components of the electronic device 1300 relevant to the present invention are shown in this illustration; components such as buses, input / output interfaces, input devices, and output devices are omitted. In addition, the electronic device 1300 may include any other suitable components depending on the specific application.

[0377] This disclosure also provides a robot, which includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, it can implement the above-mentioned robot motion control method based on imitation learning. Its execution method and beneficial effects are similar and will not be described again here.

[0378] The robot can include any one of the following: legged robot, quadruped robot, bipedal robot, wheeled robot, wheel-legged robot, quadrupedal robot, humanoid robot, cleaning robot, transport robot, mobile robot, and robotic arm. (Reference) Figure 14 As shown, Figure 14 A schematic diagram of a robot is shown.

[0379] This disclosure provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the methods of any of the above embodiments. The execution method and beneficial effects are similar, and will not be described again here.

[0380] The aforementioned computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0381] The computer program described above can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's electronic device, partially on the user's device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server.

[0382] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0383] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0384] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for training a robot motion control policy network based on imitation learning, characterized in that, The method comprises: receiving reference human motion data and motion data output by a motion control policy network through a style reward layer, and outputting a first reward item based on the feature difference between the two; receiving the motion data output by the motion control policy network and contact sequence data corresponding to the motion data through a gait reward layer, and calculating a second reward item through a constraint function designed based on biomechanics; generating a comprehensive reward item based on the first reward item and the second reward item, and updating the parameters of the motion control policy network using the comprehensive reward item to train the motion control policy network. 2.The method of claim 1, wherein, The contact sequence data comprises: contact force sensing data and contact phase of each foot end aligned with the motion data at time steps; wherein the contact phase comprises at least two of single foot swing period, double foot support period and flight period. 3.The method of claim 2, wherein, The constraint function designed based on biomechanics is configured to: determine the current state of each foot end based on the contact phase, the current state comprising swing state and support state; perform kinematic trajectory tracking constraint for the foot end in the swing state, and perform ground contact dynamics constraint for the foot end in the support state.

4. The robot motion control policy network training method based on imitation learning according to claim 3, characterized in that, The kinematic trajectory tracking constraint for the foot end in the swing state comprises: generating an expected motion trajectory of the foot end during swing, the expected motion trajectory comprising expected height curve and expected vertical velocity curve; obtaining real-time motion state of the foot end, and calculating height tracking error between the real-time motion state and the expected height curve, and velocity tracking error between the real-time motion state and the expected vertical velocity curve; obtaining real-time contact force of the foot end from the contact sequence data, and generating contact force deviation value when the real-time contact force is not zero; based on the height tracking error, the velocity tracking error and the contact force deviation value, calculating the reward value of the kinematic trajectory tracking constraint as a component of the second reward item. 5.The method of claim 4, wherein, The generation of the expected motion trajectory of the foot end during swing comprises: obtaining a current gait phase value, the gait phase value periodically changes in a preset interval over time; mapping the gait phase value to foot end height value varying with phase using a preset spline curve function to form the expected height curve; calculating the derivative of the expected height curve with respect to time to generate the expected vertical velocity curve. 6.The method of claim 4, wherein, The calculation of the reward value of the kinematic trajectory tracking constraint comprises: processing the height tracking error using a Gaussian kernel function to obtain a height tracking reward component, processing the velocity tracking error using a Gaussian kernel function to obtain a velocity tracking reward component, and processing the contact force deviation value using a Gaussian kernel function to obtain a contact force compliance reward component; weighting and summing the height tracking reward component, the velocity tracking reward component and the contact force compliance reward component to obtain the reward value. 7.The method of claim 3, wherein, The ground contact dynamics constraint for the foot end in the support state comprises: constructing a foot end locking target, the foot end locking target setting the expected take-off height, the expected vertical velocity and the expected horizontal velocity of the foot end to be zero; acquire real-time motion state of the foot end, and calculate height steady-state deviation, vertical steady-state deviation and horizontal steady-state deviation of the real-time motion state relative to the foot end locking target respectively; calculate reward value of the ground contact dynamics constraint based on the height steady-state deviation, the vertical steady-state deviation and the horizontal steady-state deviation as a component of the second reward item. 8.The method of claim 7, wherein, The calculating the reward value of the ground contact dynamics constraint based on the height steady-state deviation, the vertical steady-state deviation and the horizontal steady-state deviation comprises: substituting the height steady-state deviation into a Gaussian radial basis function to generate a foot end ground adhesion reward item, substituting the vertical steady-state deviation into a Gaussian radial basis function to generate a vertical stability reward item, and substituting the horizontal steady-state deviation into a Gaussian radial basis function to generate a skid prevention reward item; summing the foot end ground adhesion reward item, the vertical stability reward item and the skid prevention reward item to output the reward value. 9.The method of claim 3, wherein, The constraint function based on biomechanical design is further configured to: perform a gait frequency based limb coordination rhythm constraint on the arm joints of the robot. 10.The method of claim 9, wherein, The performing the gait frequency based limb coordination rhythm constraint on the arm joints of the robot comprises: analyzing linear velocity instructions in input instructions of the motion control strategy network, and determining a basic amplitude value of arm swing according to the linear velocity instructions; periodically modulating the basic amplitude value to generate an expected arm state sequence containing expected position and expected velocity; acquiring real-time state of the robot arm, and calculating a coordination tracking error between the real-time state of the robot arm and the expected arm state sequence; generating a reward value of the limb coordination rhythm constraint based on the coordination tracking error as a component of the second reward item. 11.The method of claim 10, wherein, The periodically modulating the basic amplitude value to generate an expected arm state sequence containing expected position and expected velocity comprises: extracting sine characteristic value and cosine characteristic value of a current gait phase; multiplying the basic amplitude value by the sine characteristic value and the cosine characteristic value respectively to obtain the expected position; multiplying the basic amplitude value by time derivatives of the sine characteristic value and the cosine characteristic value respectively to obtain the expected velocity; constructing an expected arm state sequence containing the expected position and the expected velocity based on the expected position and the expected velocity.

12. The method of training an imitative learning based robot motion control policy network according to any one of claims 4, 7 or 10, wherein, The method further comprises: merging components of each of the second reward items to obtain the second reward item. 13.The method of claim 1, wherein, Before the receiving, by the style reward layer, the reference human motion data and the motion data output by the motion control strategy network, the method further comprises: constructing a discriminator network of an adversarial motion prior framework; training the discriminator network based on the reference human motion data and the motion data output by the motion control strategy network, and updating parameters of the discriminator network to obtain a trained discriminator network; determining the trained discriminator network as the style reward layer.

14. The robot motion control policy network training method based on imitation learning according to claim 13, characterized in that, The training the discriminator network based on the reference human motion data and the motion data output by the motion control strategy network, and updating parameters of the discriminator network to obtain a trained discriminator network comprises: cutting a first action segment from the reference human action data as a reference sample; cutting a second action segment from the action data output by the motion control policy network as a generated sample; updating parameters of the discriminator network based on the reference sample and the generated sample to obtain a trained discriminator network.

15. The robot motion control policy network training method based on imitation learning according to claim 14, characterized in that, The updating parameters of the discriminator network based on the reference sample and the generated sample to obtain a trained discriminator network comprises: inputting the reference sample into the discriminator network to obtain a first discrimination output; inputting the generated sample into the discriminator network to obtain a second discrimination output; calculating a discrimination loss based on the first discrimination output and the second discrimination output, and performing back propagation on the discriminator network based on the discrimination loss to update parameters of the discriminator network to obtain a trained discriminator network. 16.The method of claim 1, wherein, The style reward layer comprises a first encoding sub-network, a second encoding sub-network, and a style score sub-network; and the use of the style reward layer to receive reference human action data and action data output by the motion control policy network and output a first reward item based on feature differences between the two comprises: inputting the reference human action data into the first encoding sub-network with shared weights to generate a first state-action feature vector, and inputting the action data output by the motion control policy network into the first encoding sub-network with shared weights to generate a second state-action feature vector; inputting the first state-action feature vector and the second state-action feature vector into the second encoding sub-network with shared weights respectively to obtain a first embedding vector and a second embedding vector; inputting the first embedding vector and the second embedding vector into the style score sub-network respectively to obtain a discrimination score; mapping the discrimination score into the first reward item. 17.The method of claim 1, wherein, The generation of a comprehensive reward item based on the first reward item and a second reward item comprises: calculating batch means of the first reward item and the second reward item respectively to obtain a first mean and a second mean; updating a first weight corresponding to the first reward item based on the first mean, and updating a second weight corresponding to the second reward item based on the second mean; performing weighted processing on the first reward item based on the updated first weight, and performing weighted processing on the second reward item based on the updated second weight; performing summation on the weighted first reward item and the weighted second reward item to obtain the comprehensive reward item.

18. The robot motion control policy network training method based on imitation learning according to claim 17, characterized in that, The updating of the first weight corresponding to the first reward item based on the first mean, and the updating of the second weight corresponding to the second reward item based on the second mean comprises: obtaining a historical sliding window sequence of the first mean and the second mean; calculating a moving average of the historical sliding window sequence of the first mean to obtain a first moving average, and calculating a moving average of the historical sliding window sequence of the second mean to obtain a second moving average; calculating a normalization ratio based on the first moving average and the second moving average to obtain the first weight and the second weight. 19.The method of claim 1, wherein, The parameter updating of the motion control policy network by using the comprehensive reward item comprises: Trajectory data is collected based on the motion control policy network in a simulation environment, and the trajectory data corresponding to the comprehensive reward item is recorded; A policy gradient update quantity is calculated based on the trajectory data and the comprehensive reward item; A parameter update is performed on the motion control policy network based on the policy gradient update quantity to obtain the parameter of the updated motion control policy network.

20. The robot motion control policy network training method based on imitation learning according to claim 19, wherein, The calculation of the policy gradient update quantity based on the trajectory data and the comprehensive reward item comprises: The trajectory data is accumulated by time step according to the comprehensive reward item to generate a reward sequence, and a reward vector is formed based on the reward sequence; A baseline estimation vector is calculated based on the reward vector, and an advantage estimation sequence is calculated based on the reward vector and the baseline estimation vector; The advantage estimation sequence and the trajectory data are jointly input into a preset optimizer to generate the policy gradient update quantity.

21. A method of robot motion control based on imitation learning, the method comprising: It comprises: Obtaining the current observation state of the robot; The current observation state is input into the pre-trained motion control policy network, and the action policy for controlling the motion of the robot is output based on the motion control policy network; The motion control policy network is obtained according to the robot motion control policy network training method based on imitation learning in any one of claims 1-20. 22.A robot motion control policy network training device based on imitation learning, characterized by It comprises: A style reward layer construction module is configured to receive reference human motion data and action data output by the motion control policy network using a style reward layer, and output a first reward item based on the feature difference between the two; A gait reward layer construction module is configured to receive the action data output by the motion control policy network and the contact sequence data corresponding to the action data using a gait reward layer, and calculate a second reward item based on a constraint function designed based on biomechanics; A policy network updating module is configured to generate a comprehensive reward item based on the first reward item and the second reward item, and update the parameters of the motion control policy network using the comprehensive reward item to train the motion control policy network.

23. A robot motion control apparatus based on imitation learning, characterized by, It comprises: A data acquisition module is configured to obtain the current observation state of the robot; A control policy module is configured to input the current observation state into the pre-trained motion control policy network, and output the action policy for controlling the motion of the robot based on the motion control policy network; The motion control policy network is obtained according to the robot motion control policy network training method based on imitation learning in any one of claims 1-20.

24. An electronic device, comprising: It comprises: A processor; and A memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement the method of any one of claims 1-21.

25. A robot characterized by It comprises: A processor; and A memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement the method of claim 21.

26. The robot of claim 25, wherein, The robot comprises a mobile robot.

27. The robot of claim 25, wherein, The robot comprises a humanoid robot.

28. The robot of claim 25, wherein, The robot comprises any one of a legged robot and a wheeled robot.

29. The robot of claim 25, wherein, The robot comprises a wheeled-legged robot.

30. The robot of claim 25, wherein, The robot includes any one of a quadruped robot, a biped robot, a quadruped robot.

31. The robot of claim 25, wherein, The robot includes any one of a cleaning robot and a transport robot.

32. A computer-readable storage medium, comprising: The storage medium has stored therein a computer program which, when executed by a processor, implements the method of any one of claims 1-21.

Images