A method and apparatus for training a policy network, a robot control method, and a robot.

Abstract

This application relates to a policy network training method and apparatus, a robot control method, and a robot. The policy network training method includes: dividing the robot's target action into multiple sub-stages according to the execution time sequence; constructing a basic reward function for the target action and a corresponding stage reward function for each sub-stage; determining the comprehensive reward value for each time step based on the basic reward function and the stage reward function; and determining the empirical data for that time step based on the comprehensive reward value; and training the model to be trained based on the empirical data for each time step to obtain the trained policy network. This application solves the technical problems of sparse reward signals, difficult policy convergence, and strong dependence on expert demonstration data in deep reinforcement learning policy training for complex action learning.

Description

Technical Field

[0001] This application relates to the field of robot motion control, and in particular to a policy network training method and apparatus, a robot control method, and a robot. Background Technology

[0002] With the rapid development of robotics technology, robots have gained widespread attention in fields such as logistics, disaster relief, and entertainment due to their stable motion, high payload capacity, and continuous operation. Existing robot motion control methods primarily employ kinematic planning, model-based trajectory optimization, and heuristic motion sequence design to achieve simple gait walking. These methods have significant limitations when dealing with complex target actions. Recent research has proposed using deep reinforcement learning-based motion control methods to achieve robot motion interaction and complex actions. Deep reinforcement learning, through the interaction between the agent and the environment, autonomously learns control strategies, achieving end-to-end policy mapping from perception to action without the need for explicit solution of the dynamic model.

[0003] In existing deep reinforcement learning-based robot motion control methods, special actions are generally achieved by designing reward functions based on heuristic experience or by tracking expert actions. While these methods can provide reference targets or incentive signals for the target action, they suffer from the following problems: First, the single reward function designed based on heuristic experience suffers from sparse incentive signals, conflicting incentive signals, and high exploration difficulty, leading to policy convergence difficulties. Second, obtaining expert references for complex actions is difficult, costly, and suffers from transfer gaps. In addition, some studies have attempted to design event-triggered multi-stage rewards, but such designs rely on successful event triggering, are complex, and produce large-amplitude policies with high requirements for motor hardware, making transfer and deployment difficult.

[0004] Currently, no effective solutions have been proposed to address the problems of sparse rewards, difficulty in policy convergence, and low training efficiency in the robot action learning process in related technologies. Summary of the Invention

[0005] This application provides a policy network training method and apparatus, a robot control method, and a robot, to at least solve the problems of reward sparsity, policy convergence difficulty, and low training efficiency in the process of learning complex robot actions in related technologies.

[0006] In a first aspect, embodiments of this application provide a policy network training method, the method comprising:

[0007] The robot's target action is divided into multiple sub-stages according to the order of execution time. A basic reward function is constructed for the target action, and a stage reward function is constructed for each sub-stage. The basic reward function is used to constrain the robot's action to meet physical constraints and safety requirements. The stage reward function corresponding to each sub-stage is used to guide the model to be trained to learn the action features corresponding to that stage. The model to be trained includes a policy network and a commentator network.

[0008] During the experience collection process, specific data values ​​are collected at each time step. The specific data values ​​include the current state parameters at each time step, the action instructions at each time step, the next state parameters after the action instructions are executed at each time step, and the termination flag at each time step.

[0009] The basic reward value for this time step is determined based on the basic reward function, and the stage reward value for this time step is determined based on the stage reward function corresponding to the target action stage to which this time step belongs.

[0010] Based on the stage reward value and the base reward value, determine the comprehensive reward value for that time step, and based on the reward value and the specified data value for that time step, determine the experience data for that time step.

[0011] The model to be trained is trained based on the empirical data at each time step to obtain the trained policy network.

[0012] Secondly, embodiments of this application provide a policy network training apparatus, the apparatus comprising:

[0013] The action segmentation module is used to divide the robot's target action into multiple sub-stages according to the execution time sequence, and to construct a basic reward function for the target action and a corresponding stage reward function for each sub-stage. The basic reward function is used to constrain the robot's actions to meet physical constraints and safety requirements. The stage reward function for each sub-stage is used to guide the model to be trained to learn the action features corresponding to that stage. The model to be trained includes a policy network and a commentator network.

[0014] The experience acquisition module is used to acquire specified data values ​​at a given time step during the experience acquisition process. The specified data values ​​include the current state parameters at the given time step, the action instructions at the given time step, the next state parameters after the action instructions are executed at the given time step, and the termination flag at the given time step.

[0015] The reward calculation module is used to determine the basic reward value for the time step based on the basic reward function, determine the stage reward value for the time step based on the stage reward function corresponding to the target action stage to which the time step belongs, and determine the comprehensive reward value for the time step based on the stage reward value and the basic reward value.

[0016] The experience acquisition module is also used to determine the experience data at the time step based on the reward value at the time step and the specified data value at the time step.

[0017] The policy training module is used to train the model to be trained based on the empirical data at each time step, so as to obtain the trained policy network.

[0018] Thirdly, embodiments of this application provide a robot control method, including:

[0019] Collect the robot's current state parameters;

[0020] The current state parameters are input into the policy network so that the policy network outputs an action command based on the current state parameters; wherein the policy network is trained according to the policy network training method as described in any of the first aspects;

[0021] Control the robot to execute the action corresponding to the action command.

[0022] Fourthly, embodiments of this application provide a robot, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the robot control method as described in the third aspect above.

[0023] Compared to related technologies, the policy network training method and apparatus, robot training method, and robot provided in this application divide the robot's target action into multiple sub-stages according to the execution sequence, construct a basic reward function that is effective throughout the entire time for the target action, and construct a targeted stage reward function for each sub-stage. At each time step, the reward value is determined jointly based on the stage reward function corresponding to the sub-stage and the basic reward function to generate experience data and train the policy network. This solves the technical problems of sparse reward signals, difficult policy convergence, and strong dependence on expert demonstration data in the learning of complex actions in existing deep reinforcement learning methods. It realizes the effective guidance of the policy to autonomously master the key action features of each stage according to the execution sequence and in stages without relying on high-cost expert trajectory data, and significantly improves the policy training efficiency and convergence stability.

[0024] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0025] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0026] Figure 1 A flowchart of the strategy network training method provided in this application;

[0027] Figure 2 This is a schematic diagram illustrating the division of the upright heart-shaped action phases as an exemplary embodiment of this application;

[0028] Figure 3 A schematic diagram illustrating the implementation principle of training a model to be trained, as shown in an exemplary embodiment of this application;

[0029] Figure 4 This is a schematic diagram illustrating the training and deployment process of a policy network as an exemplary embodiment of this application;

[0030] Figure 5 A structural block diagram of the policy network training device provided in this application;

[0031] Figure 6 A flowchart of the robot control method provided in this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application. Furthermore, it is understood that although the efforts made in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, modifications to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0033] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0034] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application means two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The terms “first,” “second,” “third,” etc., used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0035] This application provides a method for training a policy network. Figure 1 The flowchart of the policy network training method provided in this application is as follows: Figure 1 As shown, the process includes the following steps:

[0036] Step S110: Divide the robot's target action into multiple sub-stages according to the order of execution time, and construct a basic reward function for the target action and a corresponding stage reward function for each sub-stage; wherein, the basic reward function is used to constrain the robot's action to meet physical constraints and safety requirements; the stage reward function corresponding to each sub-stage is used to guide the model to be trained to learn the action features corresponding to that stage; the model to be trained includes a policy network and a commentator network.

[0037] In this context, a target action refers to a specific complex action that the robot expects to learn and execute, such as a quadruped robot's high-dynamic stunts like "standing and making a heart shape," "backflipping," and "jumping." Furthermore, a complex action can be understood as an action with a long execution cycle, containing multiple consecutive stages with different dynamic characteristics, and with strong coupling relationships between these stages.

[0038] Specifically, the temporal characteristics of the target action to be performed by the robot can be obtained by analyzing the motion capture data or prior knowledge from expert demonstration videos. Further, the target action can be divided into stages according to its temporal characteristics. For example, the "standing and making a heart shape" action can be broken down into several continuous stages with clear dynamic characteristics, such as "squatting to store energy," "front leg exertion to stand upright," "front foot positioning to the trajectory start point," "execution of the heart-shaped trajectory," and "leg retraction and landing." It is understandable that after dividing the target action into stages, each sub-stage has a fixed start and end time, and all sub-stages together completely cover the entire execution process of the target action.

[0039] Furthermore, a basic reward function is constructed for the entire target action execution process. This basic reward function is used to constrain the robot's actions to meet basic physical constraints and safety requirements, such as limiting joint torque, penalizing energy loss, and preventing joint over-limit. Simultaneously, a stage reward function is constructed for each sub-stage to guide the model to be trained in learning the action features corresponding to that sub-stage. For example, the "squatting and storing energy" stage rewards a lower center of gravity, while the "foreleg exertion and uprighting" stage rewards for foreleg take-off and changes in body posture.

[0040] This requires introducing the basic concepts of reinforcement learning. The model to be trained in this application is a deep reinforcement learning model, whose core components include a policy network (Actor network) and a critic network. The policy network is responsible for reasoning and outputting the action instruction to be executed next based on the robot's currently observed state parameters; the critic network is used to evaluate the merits of performing a certain action in the current state, providing gradient direction for the policy network's updates. The training process of reinforcement learning is essentially a process in which the robot repeatedly interacts with the environment through trial and error, learning the optimal policy by maximizing cumulative rewards. This application guides the policy network to efficiently learn complex target actions by carefully designing reward functions at different stages.

[0041] By taking the steps described above, and by decomposing complex target actions into time-sequential stages and designing highly specific reward functions for each stage, the problems of sparse reward signals and target conflicts that exist in traditional single reward functions when facing long-term complex tasks are solved. This achieves refined guidance of the policy learning process and lays the foundation for subsequent efficient training.

[0042] Step S120: For each time step, collect the specified data value under that time step; the specified data value includes the current state parameter under that time step, the action command under that time step, the next state parameter after the action command is executed under that time step, and the termination flag under that time step.

[0043] Specifically, the specific implementation of this step depends on the training environment.

[0044] On the one hand, when training is conducted in a virtual simulation environment, the agent object is a robot model set in the simulation environment. The robot model needs to be deployed on a pre-built simulation platform (such as IsaacSim). At each discrete time step, the current state parameters of the robot model are obtained from the simulation environment. These parameters include, but are not limited to, body posture angles, angular velocities, joint angles, joint angular velocities, and foot contact states. The policy network performs forward inference based on the state parameters and outputs an action command. For example, in one embodiment, the action command includes the target angles of each joint. This action command is converted into torque commands by the underlying joint controller, driving the robot model to move for one time step, thereby changing the environmental state and obtaining the next state parameters. Simultaneously, the simulation platform automatically determines whether the current round should end based on preset termination conditions and returns a termination flag for that time step. These termination conditions may include at least one of the following: the robot successfully completes the target action and lands stably (task successful termination); the robot rolls over, its joints exceed limits, or it experiences an unexpected and violent collision with the ground (task failed termination); and the accumulated time or number of steps in the current round reaches a preset limit (timeout termination). The termination flag is a Boolean value; for example, a value of 1 indicates the end of the current round, while a value of 0 indicates the continuation of the current round.

[0045] Furthermore, the conversion from the aforementioned motion commands to joint torque commands can be achieved through a position-controlled motion-torque mapping model, which can be represented as:

[0046] ;

[0047] in, Indicates joint torque command; This indicates the action instructions output by the policy network; Indicates the default angle of the joint; This indicates the current actual angle of the joint; This indicates the current angular velocity of the joint; Indicates the stiffness coefficient; This represents the damping coefficient. The model uses the motion commands output by the policy network as the position control target, and calculates the desired joint torque based on the current joint state, thereby achieving drive control of each joint of the robot model.

[0048] On the other hand, when training takes place in a real physical environment, sensors deployed on the physical robot (such as inertial measurement units, joint encoders, foot contact sensors, etc.) collect current state parameters in real time. The policy network outputs action commands based on these state parameters and converts them into torque commands through an underlying torque mapping model, sending them to the joint actuators of the physical robot to drive its actual movement. After one control cycle, the sensors collect new state parameters after the movement. Simultaneously, a corresponding state monitoring module can detect preset termination conditions in real time and generate a termination flag for that time step when the condition is triggered. In a real environment, the termination conditions are similar to those in the simulation environment and may include: the robot autonomously recovering a stable standing posture after completing the target action (task success); the robot's body posture angle exceeding a safety threshold or any joint angle exceeding physical limits (safe termination); and the operator manually issuing a stop command (human intervention termination). The introduction of termination flags helps to promptly interrupt abnormal actions during training or deployment in a real environment, ensuring the safety of equipment and personnel.

[0049] Therefore, regardless of the training environment used, the above interaction process can collect the current state parameters, action instructions, next state parameters, and termination flag. These four types of data together constitute the specified data value at this time step, and also provide the original data foundation for generating complete empirical data at this time step.

[0050] Through the above steps, by discretizing the continuous interaction process between the robot and the environment and obtaining specified data values ​​at a series of time steps, a structured and standardized data foundation is provided for subsequent reward calculation and experience data encapsulation, ensuring that the reinforcement learning algorithm can learn stably and orderly from the interaction data.

[0051] Step S130: Determine the basic reward value for the time step based on the basic reward function, and determine the stage reward value for the time step based on the stage reward function corresponding to the target action stage to which the time step belongs.

[0052] It should be noted that, at each time step, the data generated by the robot's interaction with the environment, in addition to the specified data values ​​mentioned above (i.e., current state parameters, action commands, next state parameters, and termination flag), also includes the comprehensive reward value returned by the environment at that time step. This comprehensive reward value is used to characterize the immediate evaluation feedback obtained by executing a specific action command in the current state. By accumulating the comprehensive reward value, the reinforcement learning algorithm can quantitatively evaluate the performance of the current policy network in long-term tasks, and iteratively update the policy network parameters with the goal of maximizing the accumulated comprehensive reward value, ultimately enabling the robot to learn the optimal behavioral policy to complete the target action.

[0053] In one embodiment, the comprehensive reward value can be calculated using the predefined basic reward function and stage reward function in step S110.

[0054] Specifically, firstly, a general reward component, i.e., the basic reward value, is calculated based on a fundamental reward function (such as energy consumption penalty, torque penalty, etc.) that is independent of the specific action stage. This reward component remains effective throughout the entire process of the target action to ensure that the strategy never violates basic physical laws and safety constraints.

[0055] Secondly, based on the cumulative duration corresponding to the current time step, it is determined which sub-stage (i.e., the target action stage) the robot is currently in. It can be understood that after the target action is divided into time sequences, each sub-stage corresponds to a fixed time interval, which constitutes the time window of that sub-stage. Therefore, based on the cumulative duration from the start of training to the current time step, it can be determined which time window the current time step falls into, and thus determine which sub-stage (i.e., the target action stage) the current time step is in. Further, the corresponding stage reward function is used to calculate the corresponding reward component, while the reward functions of other sub-stages are not included in the calculation, thus obtaining the stage reward value for the current time step. This mechanism of phased reward switching based on time windows allows the reward function to dynamically adjust the incentive target as the action progresses.

[0056] By adaptively switching the reward function according to the time progress of the action execution through the above steps, the problem of mutual interference between reward objectives at different stages is avoided, and the policy network is accurately and conflict-free incentivized and guided within a specific time interval.

[0057] Step S140: Determine the comprehensive reward value for this time step based on the stage reward value and the basic reward value, and determine the experience data for this time step based on the comprehensive reward value and the specified data value for this time step.

[0058] Specifically, the weighted sum of the base reward value and the stage reward value calculated in step S130 yields the comprehensive reward value for that time step, which can be expressed as follows:

[0059] ;

[0060] in, This represents the total reward value at that time step; Indicates the base reward value; This represents the total number of sub-stages into which the target action is divided; Indicates the first The stage reward weight of each sub-stage; This is a time window indicator function, at the current time. In the first Time interval of each sub-stage The value is 1 if the time condition is met; otherwise, it is 0. Indicates the first The stage reward function corresponding to each sub-stage is in the current state parameter and action commands The next stage reward value.

[0061] Subsequently, the comprehensive reward value is combined with the specified data values ​​(current state parameter, action instruction, next state parameter, termination flag) collected in step S120 to form a complete set of experience data for that time step, namely (current state parameter, action instruction, comprehensive reward value, next state parameter, termination flag). This experience data is stored in an experience playback buffer for use in subsequent policy update steps.

[0062] Through the above steps, by integrating the comprehensive reward value calculated in stages at each time step with the specified data value at each corresponding time step, complete empirical data containing clear guiding signals at each time step is constructed, providing key data input for subsequent policy evaluation and policy improvement using reinforcement learning algorithms.

[0063] Step S150: Train the model to be trained based on the empirical data at each time step to obtain the trained policy network.

[0064] Specifically, in this step, reinforcement learning algorithms (such as Proximal Policy Optimization, PPO) periodically sample a batch of empirical data from the experience replay buffer. The critic network uses this data to update its network parameters through temporal difference learning to more accurately estimate state values. The policy network (actor network) then uses the advantage function provided by the critic network as guidance to update its network parameters through policy gradient methods, increasing the probability of actions that yield higher cumulative rewards being selected. By iteratively executing steps S120 to S150—that is, the loop of collecting specified data values, calculating rewards, constructing the empirical data required for training, and updating network parameters—the policy network gradually learns the behavior that maximizes cumulative rewards, eventually converging to an optimal policy network (i.e., the trained policy network) that can stably and smoothly complete the target action.

[0065] Through the above steps, by utilizing the rich experience data generated by the phased reward mechanism and combining iterative optimization of the policy network and the critic network with deep reinforcement learning algorithms, the problems of unstable policy convergence and low training efficiency in the process of complex action learning by traditional methods are solved, and efficient and stable learning of quadruped robot action policies is achieved.

[0066] Through steps S110 to S150, the target action is divided into multiple sub-stages with defined time windows according to the execution time sequence. A stage reward function matching the action characteristics of each sub-stage is constructed. This allows the policy network to obtain clear and progressive stage guidance signals based on the objective time progress during training. This solves the problem of reward sparsity and unclear policy exploration direction caused by conflicting incentive targets in existing technologies, achieving refined decoupling of the complex action learning process and significantly improving the stability and convergence efficiency of policy training. Furthermore, this solution replaces the event-triggered stage switching mechanism with a time window-based reward gating mechanism. This makes the switching of action stages independent of whether the policy successfully completes a specific action event. This solves the problems of low exploration efficiency and reliance on manual experience in the early stages of training for event-triggered stage rewards in existing technologies. It enables effective guidance for the policy from the early stages of training, significantly reducing exploration difficulty and improving training efficiency. Furthermore, this solution organically combines the basic reward function with the stage reward function of each sub-stage. While constraining actions to meet physical feasibility and safety requirements globally, it also specifically strengthens the key action features of each stage in different time intervals. This solves the problem of excessively large action amplitudes learned by the policy in existing technologies and deployment gaps when transferred to real hardware. It achieves a balance between compliance and robustness in motion control and improves the transferability of the policy from simulation to real environment.

[0067] In some embodiments, the robot is a quadruped robot, and the target action is a standing heart-shaped gesture. The robot's target action is divided into multiple sub-stages according to the order of execution time, including:

[0068] The standing heart-shaped gesture is divided into the initial adjustment stage, the upright stage, the heart-shaped trajectory preparation stage, the heart-shaped trajectory execution stage, and the closing stage.

[0069] Specifically, this solution will be described in detail using the example of a quadruped robot performing a highly dynamic stunt like "standing and making a heart shape with its hands." The diagram below illustrates the division of the standing heart-making action stages as an exemplary embodiment of this application. Figure 2 As shown, based on motion capture data or expert video demonstrations, the "standing heart gesture" can be divided into the following five sub-stages according to time sequence:

[0070] (1) The initial adjustment phase has a time window of 0s to 0.5s. During this phase, the robot squats down, lowers its center of gravity, and adjusts the corresponding joints to the designated positions to store energy in the joints.

[0071] (2) The time window for the uprighting stage is 0.5s to 1.5s. During this stage, the robot’s two front legs act as execution legs and the two hind legs act as support legs. The front legs exert force to make the robot upright, and the hind legs gradually extend until they reach the support state.

[0072] (3) The preparation phase of the heart-shaped trajectory has a time window of 1.5s to 3.0s. In this phase, the robot’s left and right front feet are first retracted to the preparation position, and then smoothly reach the starting point of the preset heart-shaped trajectory.

[0073] (4) The heart-shaped trajectory execution phase has a time window of 3.0s to 5.0s. During this phase, the robot controls its left and right forelegs to draw a standard heart-shaped trajectory in three-dimensional space.

[0074] (5) The closing phase has a time window of 5.0s and beyond. In this phase, after the robot completes the heart-shaped trajectory, it controls the left and right front feet to retract, the posture to return to normal, and then controls the left and right front feet to touch the ground to achieve a stable landing. In actual training, since the robot's joints basically return to the default standing posture after completing the posture retraction, and the movement no longer changes significantly, setting too long a round time will lead to ineffective data calculation and waste of training resources. Therefore, the termination time of this phase can be set to 10s, that is, the time window of the closing phase is specifically 5.0s to 10.0s.

[0075] Through the steps described above, the abstract "standing heart gesture" action is concretized into five sub-stages with clearly defined start and end times and dynamic objectives. This solves the problem of directly modeling and optimizing complex actions, enabling a structured and hierarchical understanding and expression of acrobatic movements. Furthermore, the detailed phase division of the target action according to time sequence lays the foundation for subsequently constructing targeted phase reward functions.

[0076] In some embodiments, determining the base reward value at that time step based on the base reward function includes:

[0077] Step S201: Determine the first penalty term based on the torque and angular velocity of each joint of the robot.

[0078] Specifically, the first penalty term can be expressed as:

[0079] ;

[0080] Among them, For joints torque, For joints angular velocity, The robot's joints are numbered. This first penalty, also known as the energy penalty, is used to suppress unnecessary energy consumption by the robot.

[0081] Step S202: Determine the second penalty term based on the absolute value of the torque of each joint.

[0082] Specifically, the second penalty term can be expressed as:

[0083] ;

[0084] Among them, For joints torque, The joints of the robot are numbered. The second penalty term, also known as the torque penalty term, is used to limit the magnitude of the joint output torque to prevent excessive torque from causing motor overload or energy loss.

[0085] Step S203: Determine the third penalty item based on the angle of each joint, the preset safety upper limit, and the preset safety lower limit.

[0086] Furthermore, step S203 can be further broken down into the following three steps:

[0087] Step 1: Based on the angles of each joint and the preset safe upper limit angle, determine the first excess deviation of the angle of each joint relative to the preset safe upper limit angle;

[0088] Step 2: Based on the angles of each joint and the preset lower safety limit angle, determine the second excess deviation of the angle of each joint relative to the preset lower safety limit angle;

[0089] Step 3: Based on the first and second over-limit deviations, determine the overall deviation value of the angle of each joint, and based on the overall deviation value of the angle of each joint, determine the joint state over-penalty item.

[0090] Specifically, the third penalty can be expressed as:

[0091] ;

[0092] in, This is the first deviation exceeding the limit; This is the second excessive deviation; For the first The current angle of each joint; For the first The preset safety limit angle of each joint; For the first The preset safety lower limit angle of each joint; The joints of the robot are numbered. This third penalty, also known as the joint state over-limit penalty, is used to prevent the angles of each joint from exceeding the safe range of motion.

[0093] Step S204: Determine the fourth penalty item based on the deviation between the action command of each joint in the current time step and the action command of each joint in the previous time step.

[0094] Specifically, the fourth penalty item can be expressed as:

[0095] ;

[0096] in, For the first The action instructions for each joint at the current time step; For the first The motion command of each joint in the previous time step; The robot's joints are numbered. This fourth penalty, also known as the motion change rate penalty, is used to suppress drastic changes in motion commands and improve the smoothness of movement.

[0097] Step S205: Determine the base reward value based on the first penalty item, the second penalty item, the third penalty item, and the fourth penalty item.

[0098] Specifically, at each time step, the base reward value can be expressed as:

[0099] ;

[0100] in, The first penalty item mentioned above; The second penalty item mentioned above; The third penalty item mentioned above; This is the fourth penalty item mentioned above.

[0101] Through the above steps, by designing a basic reward function that covers energy consumption, torque, joint state over-limit, and rate of change of motion, the problem of excessive motion, excessive energy consumption, or exceeding the physical limits of hardware that may occur when the strategy learns complex motions is solved. This achieves basic constraints on the behavior of the strategy, ensuring the safety of the training process and the physical realizability of the final strategy.

[0102] In some embodiments, corresponding stage reward functions are activated for different target action stages. The following section continues with "standing and making a heart shape with your hands" as the target action, detailing how the stage reward value corresponding to the current time step is determined in each sub-stage of the target action.

[0103] It is understandable that the target action stage of the current time step is determined based on the cumulative time corresponding to the current time step. During policy training in a simulation environment, each time step corresponds to a fixed simulation step size; therefore, the cumulative time corresponding to the current time step can be determined using the following method:

[0104] ;

[0105] in, This indicates the cumulative time corresponding to the current time step; This indicates the number of time steps elapsed from the start of the round to the current time step; This represents the simulation step size corresponding to each time step. (This is used when obtaining the cumulative time.) Then, through judgment By determining which sub-stage of the target action the current time step falls within the preset time window, the target action stage to which it belongs can be identified. For example, when... At that time, the target action phase of the current time step is determined to be the fuselage uprighting phase.

[0106] When strategy training is conducted in a real environment based on a physical robot, the cumulative time of the current round can be directly obtained through a high-precision timer or system clock in the robot control system, and the target action stage to which the current time step belongs can be determined based on the cumulative time. The determination method is consistent with that in the simulation environment.

[0107] Through the above methods, regardless of whether the training is conducted in a simulation environment or a real environment, the current action stage can be accurately and consistently determined according to the time process, providing a reliable timing reference for the call of the reward function in subsequent stages.

[0108] The process of determining the stage reward value will be introduced below.

[0109] (I) Initial Adjustment Phase: The process of determining the phase reward value:

[0110] In some embodiments, when the target action phase is the initial adjustment phase, the stage reward value for that time step is determined according to the stage reward function corresponding to the target action phase to which that time step belongs, including:

[0111] Step S301: Based on the cumulative duration corresponding to the current time step, the preset starting target height of the robot's body base in the initial adjustment phase, and the preset ending target height of the body base in the initial adjustment phase, determine the target height of the body base in the current time step, and determine the first reward item based on the actual height of the body base in the current time step and the target height of the body base in the current time step.

[0112] Specifically, the first reward item can be determined through the following two steps:

[0113] Step 1: Determine the target altitude of the fuselage base at the current time step.

[0114] The target height can be calculated by interpolation and can be expressed as:

[0115] ;

[0116] in, This indicates the cumulative duration corresponding to the current time step; This indicates the preset target height of the fuselage base at the start of the initial adjustment phase (i.e., t=0.0s); This indicates the preset target height of the fuselage base at the end of the initial adjustment phase (i.e., t=0.5s); This indicates the target altitude of the fuselage base at the current time step.

[0117] Step 2: Determine the first reward item based on the actual height of the fuselage base at the current time step and the target height determined above.

[0118] The first reward item can be represented as:

[0119] ;

[0120] in, For time window indication function, when The value is 1 if the condition is met, and 0 otherwise. This indicates the actual height of the fuselage base at the current time step; This indicates the target height of the robot's base at the current time step. This reward, by penalizing the squared error between the actual height and the target height, guides the robot to smoothly lower its center of gravity to the preset target height during the initial adjustment phase.

[0121] The first reward, namely the base center of gravity adjustment reward, is used to guide the robot to complete the center of gravity lowering action during this stage.

[0122] Step S302: Determine the target angle of each joint in the current time step based on the accumulated time, the preset starting reference angle of each joint in the initial adjustment phase, and the preset ending reference angle of each joint in the initial adjustment phase. Then, determine the second reward item based on the actual angle of each joint in the current time step and the target angle of each joint in the current time step.

[0123] Specifically, the second reward item can be determined through the following two steps:

[0124] Step 1: Determine the target angle of each joint at the current time step.

[0125] For the The target angle of each joint can be calculated by interpolation and can be expressed as:

[0126] ;

[0127] in, This indicates the cumulative duration corresponding to the current time step; These are the interpolation coefficients, which can be obtained through... Calculations show that this coefficient takes a value of 0 at t=0.0s and a value of 1 at t=0.5s, with a linear transition in between time points; Indicates the first The preset starting reference angle of each joint at the start of the initial adjustment phase (i.e., t=0.0s); Indicates the first The preset termination reference angle of each joint at the end of the initial adjustment phase (i.e., t=0.5s); Indicates the first The target angle of each joint at the current time step.

[0128] Step 2: Determine the second reward item based on the actual angles of each joint at the current time step and the target angles determined above.

[0129] This second reward item can be represented as:

[0130] ;

[0131] in, For time window indication function, when The value is 1 if the condition is met, and 0 otherwise. Indicates the first The actual angle of each joint at the current time step; Indicates the first The target angle of each joint at the current time step.

[0132] The second reward, namely the joint state adjustment reward, guides the robot joints to move smoothly to the preset ready posture according to the preset time process by penalizing the deviation between the actual angle and the target angle of each joint, thus preparing for the subsequent uprighting of the robot body.

[0133] Step S303: Determine the third reward item based on the actual angle of each front leg hip joint of the robot at this time step and the preset angle of each front leg hip joint at this time step.

[0134] Specifically, during this target movement phase, for the first hip joint set belonging to the foreleg... Each joint has a preset angle denoted as . This preset angle typically corresponds to the nominal angle of the front leg hip joint in a standing posture. Subsequently, based on the deviation between the actual angle of each front leg hip joint and the preset angle at the current time step, the third reward item can be determined, which can be expressed as:

[0135] ;

[0136] in, For time window indication function, when The value is 1 if the condition is met, and 0 otherwise. This indicates a set of joint numbers belonging to the hip joints of the foreleg; Indicates the first The actual angle of the foreleg hip joint at the current time step; Indicates the first The preset angle of the front leg hip joint.

[0137] The third reward, namely the foreleg abduction joint reward, punishes the deviation between the actual angle of the foreleg hip joint and the preset angle, inhibiting excessive abduction or adduction of the foreleg in this stage, which helps to stabilize the subsequent upright movement. Step S304: Determine the fourth reward based on the projection of the robot's gravity vector onto the horizontal plane.

[0138] Specifically, the projection of the gravity vector in the fuselage coordinate system can be expressed as: ,in and These represent the gravity vector in the fuselage coordinate system. shaft and The projection component along the axial direction reflects the degree of tilt of the fuselage relative to the horizontal plane. When the fuselage is perfectly horizontal, the gravity vector is perpendicular to the fuselage. When the axes are parallel, and All are 0. Subsequently, the fourth reward item can be determined based on the projection components of the gravity vector onto the horizontal plane of the fuselage coordinate system, which can be expressed as:

[0139] ;

[0140] in, For time window indication function, when The value is 1 if the condition is met, and 0 otherwise. and These represent the gravity vector in the fuselage coordinate system. shaft and Projection component along the axial direction.

[0141] The fourth reward, namely the horizontal posture retention reward, is achieved through a penalty. and The sum of these factors guides the robot to maintain a horizontal position during the initial adjustment phase, preventing tilting during the lowering of the center of gravity from affecting subsequent movements.

[0142] Step S305: Determine the stage reward value for this time step based on the first reward item, the second reward item, the third reward item, and the fourth reward item.

[0143] Specifically, in the initial adjustment phase, the phase reward value corresponding to the current time step can be expressed as:

[0144] ;

[0145] in, The first reward item mentioned above; The second reward item mentioned above; The third reward item mentioned above; This is the fourth reward item mentioned above.

[0146] Through the above steps, by designing specific reward items for center of gravity height, joint configuration, foreleg posture, and body level in the initial adjustment phase, the policy network can obtain clear and conflict-free guidance signals in this phase. This solves the problems of unclear exploration direction and uncoordinated joint movements in the initial stage of action, enabling the robot to complete center of gravity lowering and posture preparation stably and quickly, laying a good foundation for the subsequent body uprighting phase.

[0147] (II) The process of determining the stage reward value during the fuselage uprighting stage:

[0148] In some embodiments, when the target action phase is the fuselage uprighting phase, the stage reward value for that time step is determined according to the stage reward function corresponding to the target action phase to which that time step belongs, including:

[0149] S401. Determine the fifth reward item based on the airborne contact state of the robot's left and right front feet.

[0150] Specifically, in this phase, the ground contact states of the left and right forelegs are recorded as follows: and This can be determined using foot force sensors or contact detection algorithms. When the tips of the left and right forefeet contact the ground, and The value of is 1; when the left and right forelegs are in the air, and The value is 0. Therefore, this fifth reward item can be represented as:

[0151] ;

[0152] in, This is a time window indicator function, indicating the current cumulative time. The value is 1 if the aircraft is upright, otherwise it is 0. This reward is valid during the upright phase of the aircraft and in multiple subsequent phases. This is an indicator function that takes the value 1 when the condition inside the parentheses is true, and 0 otherwise.

[0153] The fifth reward, namely the foreleg lift-off reward, is given only when both forelegs are in the air at the same time, and is used to guide the strategy to learn the behavior of lifting the forelegs off the ground.

[0154] S402. The sixth reward item is determined based on the duration of the left and right forelegs remaining airborne.

[0155] Specifically, for each foreleg Define its levitation duration , used to represent the cumulative time since the most recent takeoff; its first touchdown indication function is defined. , among which, when the first When each foot changes from an airborne state to a contact state at the current time step The value is 1 for all cases and 0 for all others. Therefore, the sixth reward item can be represented as:

[0156] ;

[0157] in, This is a time window indicator function, indicating the current cumulative time. The value is 1 if the condition is met, and 0 otherwise. Indicates the first The current airtime of each foot; 0.5 is the preset minimum expected airtime threshold, in seconds; This is the first ground contact indicator function.

[0158] The sixth reward, namely the foreleg airtime reward, awards a bonus based on how long the foreleg's airtime exceeds a threshold of 0.5 seconds upon its first touchdown. The longer the airtime, the higher the reward, thus encouraging the strategy to maintain the foreleg's airtime. It is understood that the preset minimum expected airtime threshold can be set according to actual needs; the 0.5-second value here is merely an example.

[0159] S403. Determine the seventh reward item based on the actual ground clearance of the left and right forelegs and the preset ground clearance of the left and right forelegs.

[0160] Specifically, the actual height of the left and right forelegs from the ground is recorded as follows: and Both can be obtained through forward kinematics and terrain perception. Let the preset desired ground clearance be 0.5 meters. Then, the seventh reward item can be expressed as:

[0161] ;

[0162] in, For time window indication function, when The value is 1 if the condition is met, and 0 otherwise. and These represent the actual ground clearance heights of the left and right forelegs, respectively; 0.5 is the preset target ground clearance height in meters; and 0.25 is the standard deviation parameter, used to control the rate at which the reward value decays as the deviation increases. This reward uses a Gaussian function, with the reward value being maximum when the actual height is close to the target ground clearance height of 0.5 meters. The larger the deviation, the smaller the reward, thus guiding the forelegs to smoothly rise to the preset target ground clearance height. It is understood that the preset target ground clearance height can be set according to actual needs; the value of 0.5 meters here is merely an example.

[0163] The seventh bonus, namely the foreleg height bonus, is used to guide the forelegs to be raised to a specified height to achieve full fuselage uprightness.

[0164] Step S404: Determine the eighth reward item based on the robot's actual head height above the ground and the preset head height above the ground.

[0165] Specifically, at this stage, the position of the robot's head in the body coordinate system is fixed, and the actual height of the head above the ground can be calculated using kinematics. The preset desired head height can be set to 0.7 meters, then the eighth reward item can be represented as:

[0166] ;

[0167] in, For time window indication function, when The value is 1 if the condition is met, and 0 otherwise. This indicates the actual height of the head at the current time step; 0.7 represents the preset target head height, in meters. This reward, by penalizing the squared error between the actual head height and the target height, guides the robot to maintain a suitable working height during the uprighting process. It is understood that the preset expected head height here can be set according to actual needs; the value of 0.7 meters here is merely an example.

[0168] The eighth reward, namely the head height constraint reward, is used to ensure that the fuselage height is within a reasonable range during the uprighting process, so as to avoid leg interference due to being too low or instability due to being too high.

[0169] Step S405: Determine the ninth reward item based on the cumulative duration corresponding to the current time step, the preset starting pitch angle of the robot's body during the upright phase, the preset ending pitch angle of the robot's body during the upright phase, and the actual gravity vector of the robot at this time step.

[0170] Specifically, during the uprighting phase of the aircraft, the target pitch angle at the initial moment (t=0.5s) is set to 0 (horizontal state), and the target pitch angle at the final moment (t=1.5s) is set to... (Upright state), then the current time step Corresponding target pitch angle It can be calculated using linear interpolation and can be expressed as:

[0171] ;

[0172] Based on the target pitch angle Target attitude quaternions can be constructed This will further change the gravity vector in the world coordinate system. Transform to the fuselage coordinate system to obtain the target gravity vector. Among them, the actual gravity vector This can be obtained through an airborne inertial measurement unit (IMU). Therefore, the ninth reward item can be represented as:

[0173] ;

[0174] in, For time window indication function, when The value is 1 if the condition is met, and 0 otherwise. The first is the actual gravity vector in the fuselage coordinate system. One component; The first is the target gravity vector in the fuselage coordinate system. Each component.

[0175] The ninth reward, namely the fuselage attitude upright tracking reward, guides the fuselage attitude to smoothly transition from horizontal to vertical according to a preset time process by penalizing the absolute deviation between the actual gravity vector and the target gravity vector in each component.

[0176] Step S406: Determine the stage reward value for this time step based on the fifth, sixth, seventh, eighth, and ninth reward items.

[0177] Specifically, when the target action phase is the aircraft uprighting phase, the phase reward value corresponding to the current time step can be expressed as:

[0178] ;

[0179] in, This is the fifth reward item; This is the sixth reward item; This is the seventh reward item; This is the eighth reward item; This is the ninth reward item.

[0180] Through the above steps, by designing specific reward items for the airborne contact state of the left and right forelegs, the duration of continuous airborne movement of the left and right forelegs, the ground clearance of the left and right forelegs, the ground clearance of the head, and the tracking of the body pitch attitude during the uprighting phase, the policy network can obtain guidance signals from easy to difficult and progressively more complex during this phase. This solves the problem that a single reward function is difficult to stimulate explosive uprighting movements and realizes a continuous movement of the robot's front legs powerfully pushing off the ground, the hind legs providing stable support, and the body smoothly flipping to an upright position.

[0181] (III) The process of determining the stage reward value in the preparation stage of the heart-shaped trajectory:

[0182] Optionally, a course switch function can be set before the cardioid trajectory preparation stage to limit the start of the actual operation of the cardioid trajectory preparation stage.

[0183] Specifically, in the training process of the standing heart-shaped gesture, the policy network needs to first master the basic action of upright posture. Introducing reward items related to cardioid trajectory tracking in the early stages of training will interfere with the learning of the upright posture due to goal conflict. Therefore, a cardioid trajectory course switch function is set up to solve this problem. Cardioid trajectory course switch function It can be represented as:

[0184] ;

[0185] in, This is an indicator function that returns 1 if the condition within the parentheses is met, and 0 otherwise. This represents the global environment step count, starting from the beginning of training. This represents the number of environmental steps in one round of data acquisition. The physical meaning of this course switching function is that the reward items related to the cardioid trajectory (such as the subsequent eleventh, twelfth, thirteenth, and fifteenth reward items) are only activated when the global training steps reach the preset activation threshold (i.e., the policy network's learning of the upright posture has basically converged). Before this, the cardioid trajectory-related reward items are not effective, and the policy network focuses on learning basic movements such as aircraft uprightness. Through this mechanism, a course-based learning approach from easy to difficult is achieved, effectively avoiding the training instability problem caused by multi-objective conflicts.

[0186] In some embodiments, the cardioid trajectory preparation phase includes a first sub-phase and a second sub-phase.

[0187] Specifically, when the target action phase is the heart-shaped trajectory preparation phase, this phase is further subdivided into a first sub-phase and a second sub-phase. The phase reward value corresponding to the current time step is determined according to the sub-phase it belongs to. The time window corresponding to the first sub-phase is 1.5s to 2.0s, and the time window corresponding to the second sub-phase is 2.0s to 3.0s.

[0188] When the target sub-stage is the first sub-stage, the stage reward value corresponding to the current time step can be determined through the following steps:

[0189] S501, Determine the fifth, sixth, and eighth reward items.

[0190] Specifically, the fifth reward item (forelegs take-off reward item), the sixth reward item (forelegs take-off time reward item), and the eighth reward item (head height constraint reward item) continue to be effective in this first sub-stage. Their definitions are the same as the corresponding reward items in the fuselage upright stage, and will not be repeated here.

[0191] S502. Determine the tenth reward item based on the components of the robot's gravity vector and the components of the target gravity vector preset for the cardioid trajectory preparation stage.

[0192] Specifically, when the fuselage is in a completely upright position, the target gravity vector has definite component values ​​in the fuselage coordinate system, which are denoted as... Actual gravity vector It can be obtained through the IMU. Therefore, the tenth reward item can be represented as:

[0193] ;

[0194] in, For time window indication function, when The value is 1 if the condition is met, and 0 otherwise. The first is the actual gravity vector in the fuselage coordinate system. One component; The first is the gravity vector of the target in an upright state. Each component.

[0195] The tenth reward, namely the upright posture tracking reward, guides the robot to maintain an upright posture throughout the cardioid trajectory preparation phase by penalizing the deviation between the actual gravity vector and the gravity vector of the upright target.

[0196] Step S503: Determine the eleventh reward item based on the first actual position of the robot's left and right front feet at the current time step and the preset first target position of the left and right front feet.

[0197] Specifically, the actual positions of the ends of the left and right forelegs in the fuselage coordinate system are denoted as follows: and This is referred to as the first actual position, which can be calculated from the joint angles using forward kinematics. The preset left and right forefoot preparatory points are denoted as... and The first target position, whose value can be set according to actual needs, for example, can be set to... , Therefore, the eleventh reward item can be represented as:

[0198] ;

[0199] in, Switch on / off for the heart-shaped trajectory course; For time window indication function, when The value is 1 if the condition is met, and 0 otherwise. This represents the L1 norm, used to calculate the sum of the absolute deviations of the components of the position vector.

[0200] The eleventh reward, namely the first sub-stage forefoot trajectory tracking reward, guides the forefoot to move smoothly to the preset preparation point position by penalizing the deviation between the actual position of the left and right forefoot ends and the preparation point position.

[0201] Step S504: Determine the stage reward value for the current time step based on the fifth, sixth, eighth, tenth, and eleventh reward items.

[0202] Specifically, in the first sub-stage of the heart-shaped trajectory preparation phase, the stage reward value corresponding to the current time step can be expressed as:

[0203] ;

[0204] in, This is the fifth reward item; This is the sixth reward item; This is the eighth reward item; This is the tenth award item; This is the eleventh award item.

[0205] When the current target sub-stage is the second sub-stage of the heart-shaped trajectory preparation stage, determine the stage reward value corresponding to the current time step, including:

[0206] S601, Determine the fifth, sixth, and eighth reward items.

[0207] Specifically, the fifth reward item (forelegs take-off reward item), the sixth reward item (forelegs take-off time reward item), and the eighth reward item (head height constraint reward item) continue to be effective in this second sub-stage. Their definitions are the same as the corresponding reward items in the fuselage upright stage, and will not be repeated here.

[0208] Step S602: Based on the preset left and right front foot positions and the cumulative duration corresponding to the current time step, determine the second target position of the robot's left and right front feet in the current time step, and determine the twelfth reward item based on the second actual position of the left and right front feet in the current time step and the second target position of the left and right front feet in the current time step.

[0209] Specifically, to facilitate understanding and maintain consistency in terminology, in this stage, the actual positions of the left and right foreleg ends in the fuselage coordinate system will be described. and This is referred to as the second actual position, and its calculation method can be the same as that of the first actual position described above; the target position at the current time step is obtained through interpolation. and This is called the second target location.

[0210] In detail, the starting positions of the heart-shaped trajectories are denoted as follows: and Current time step The corresponding interpolation coefficients can be calculated using normalized time schedules:

[0211] ;

[0212] in, The function restricts the input values ​​to the interval [0,1]. Based on the interpolation coefficients, the target positions of the left and right forelegs at the current time step, i.e., the second target positions, can be expressed as follows:

[0213] ;

[0214] .

[0215] Therefore, the twelfth reward item can be represented as:

[0216] ;

[0217] in, Switch on / off for the heart-shaped trajectory course; For time window indication function, when The value is 1 if the condition is met, and 0 otherwise. and This refers to the second actual position of the distal ends of the left and right forelegs; and This is the second target position obtained by the above interpolation.

[0218] The twelfth reward, namely the second sub-stage forefoot trajectory tracking reward, guides the forefoot tip to move smoothly from the preparation point to the starting position of the heart-shaped trajectory according to a preset time process by penalizing the deviation between the second actual position and the second target position of the left and right forefoot tips.

[0219] Step S603: Determine the stage reward value for the current time step based on the fifth, sixth, eighth, tenth, and twelfth reward items.

[0220] Specifically, in the second sub-stage of the heart-shaped preparation phase, the stage reward value corresponding to the current time step can be expressed as:

[0221] ;

[0222] in, This is the fifth reward item; This is the sixth reward item; This is the eighth reward item; This is the tenth award item; This is the twelfth reward item.

[0223] Through the above steps, by subdividing the cardioid trajectory preparation stage into two sub-stages and designing corresponding stage reward functions for each sub-stage, the policy network can first learn the action of retracting the left and right forelegs to the preset preparation point position, and then learn the action of smoothly transitioning from the preset preparation point position to the starting position of the cardioid trajectory. This solves the problem of unsmooth transition and large error accumulation between the dynamic upright posture and the precise trajectory tracking posture, and achieves stable and accurate positioning of the left and right foreleg ends, laying the foundation for high-precision trajectory tracking in the next stage.

[0224] (iv) Method for determining the stage reward value during the execution phase of the heart-shaped trajectory:

[0225] In some embodiments, when the target action phase is the cardioid trajectory execution phase, the stage reward value for that time step is determined according to the stage reward function corresponding to the target action phase to which that time step belongs, including:

[0226] Step S701: Determine the fifth, sixth, and eighth reward items.

[0227] Specifically, the fifth reward item (forelegs take-off reward item), the sixth reward item (forelegs take-off time reward item), and the eighth reward item (head height constraint reward item) continue to be effective in this stage, and their definitions are the same as the corresponding reward items in the aforementioned fuselage upright stage, so they will not be repeated here.

[0228] Step S702: Based on the cumulative duration corresponding to the current time step, the start time of the cardioid trajectory execution phase, and the end time of the cardioid trajectory execution phase, determine the trajectory reference angles of the robot's left and right front feet, and based on the trajectory reference angles and the pre-constructed cardioid trajectory equation, determine the third target position of the left and right front feet in the current time step.

[0229] The thirteenth reward is determined based on the third actual position of the left and right forelegs at the current time step and the third target position of the robot at the current time step.

[0230] Specifically, to facilitate understanding and maintain consistency in terminology, in this stage, the actual positions of the left and right foreleg ends in the fuselage coordinate system will be described. and This is referred to as the third actual position, and its calculation method is the same as that of the first actual position. The target position at the current time step, calculated from the cardioid trajectory parametric equation, is then used. and This is referred to as the third target location.

[0231] Specifically, the time window for the execution phase of the heart-shaped trajectory is 3.0s to 5.0s. Therefore, the current time step... The corresponding normalized phase progress can be calculated as follows: Thus, the trajectory reference angle is obtained. :

[0232] .

[0233] Based on trajectory reference angle Define the trajectory reference angles for the left forefoot and right forefoot respectively. , This achieves a symmetrical cardioid trajectory movement. Substituting the aforementioned trajectory reference angles into the preset cardioid trajectory parameter equations, the third target position of the left and right forelegs at the current time step can be obtained. and .

[0234] Furthermore, the parametric equation of this cardioid trajectory can be the parametric equation of a Cartesian cardioid trajectory. Taking the extension of the Cartesian cardioid into three-dimensional space as an example, the parametric equation for the third target position can be expressed as:

[0235] For the right forefoot:

[0236] ;

[0237] ;

[0238] ;

[0239] For the left forefoot:

[0240] ;

[0241] ;

[0242] ;

[0243] Therefore, the thirteenth reward item can be represented as:

[0244] .

[0245] in, Switch on / off for the heart-shaped trajectory course; For time window indication function, when The value is 1 if the condition is met, and 0 otherwise. and This refers to the actual position of the ends of the left and right forelegs; and The target position is determined by the cardioid trajectory equation.

[0246] The thirteenth reward, namely the cardioid trajectory tracking reward, guides the forefoot end to accurately reproduce the spatial shape of the cardioid trajectory by penalizing the L1 norm deviation between the third actual position and the third target position of the left and right forefoot ends.

[0247] Step S703: Determine the fourteenth reward based on the actual angles of each hind leg joint of the robot and the preset angles of each hind leg joint during the execution phase of the heart-shaped trajectory.

[0248] Specifically, the set of hind leg joints is denoted as... The preset angles of each hind leg joint in the upright supporting posture are denoted as follows: Therefore, the fourteenth reward item can be represented as:

[0249] ;

[0250] in, For time window indication function, when The value is 1 if the condition is met, and 0 otherwise. For the first The actual angle of the hind leg joint at the current time step; For the first The preset angle of the hind leg joint.

[0251] The fourteenth bonus, namely the rear leg upright posture joint maintenance bonus, guides the rear leg to maintain stable support during trajectory execution by penalizing deviations between the rear leg joint angle and the preset angle, thus avoiding imbalance of the aircraft caused by forefoot movement disturbances.

[0252] Step S704: Determine the stage reward value for this time step based on the fifth, sixth, eighth, thirteenth, and fourteenth reward items.

[0253] Specifically, during the execution phase of the heart-shaped trajectory, the stage reward value corresponding to the current time step can be expressed as:

[0254] ;

[0255] in, This is the fifth reward item; This is the sixth reward item; This is the eighth reward item; To achieve the reward item; This is the fourteenth reward item.

[0256] Through the above steps, by generating dynamic target positions using the cardioid trajectory parametric equation and designing corresponding cardioid trajectory tracking rewards, and combining them with hind leg upright posture joint maintenance rewards, the policy network can accurately control the foreleg end to move along a predetermined cardioid trajectory while maintaining the upright stability of the robot body. This solves the problem that complex spatial trajectories are difficult to learn through sparse rewards, and enables the quadruped robot to perform the iconic "heart-making" action smoothly and accurately.

[0257] (v) Final stage, method for determining stage reward value:

[0258] In some embodiments, the closing phase includes a first closing phase and a second closing phase.

[0259] Specifically, the closing phase can be further subdivided into a first closing phase and a second closing phase, with the phase reward value corresponding to the current time step determined according to the closing phase. The time window for the first closing phase is 5.0s to 6.0s, and the time window for the second closing phase is 6.0s to 10.0s.

[0260] During the first closing phase, the stage reward value corresponding to the current time step can be determined through the following steps:

[0261] S801. Determine the fifth, sixth, eighth, and fourteenth reward items.

[0262] Specifically, the fifth reward item (forefoot airborne reward item), the sixth reward item (forefoot airborne time reward item), the eighth reward item (head height constraint reward item), and the fourteenth reward item (hind leg upright posture maintenance reward item) continue to be effective in this first closing stage, and their definitions are the same as the corresponding reward items in the aforementioned stages, so they will not be repeated here.

[0263] S802. Based on the robot's endpoint position in the heart-shaped trajectory execution phase, the robot's preset end position in the closing phase, and the cumulative duration corresponding to the current time step, determine the fourth target position of the robot's left and right front feet in the current time step, and determine the fifteenth reward item based on the fourth actual position of the left and right front feet in the current time step and the fourth target position of the left and right front feet in the current time step.

[0264] Specifically, to facilitate understanding and maintain consistency in terminology, in this stage, the actual positions of the left and right foreleg ends in the fuselage coordinate system will be described. and This is referred to as the fourth actual location, and its calculation method is the same as that of the first actual location. The target recovery location is then calculated through interpolation. and This is referred to as the fourth target location.

[0265] Specifically, at the end of the cardioid trajectory execution phase (t=5.0s), the positions of the ends of the left and right forelegs are recorded as follows: and The preset closing preparation point can be the same as the preparation point in the cardioid trajectory preparation stage, denoted as... and Current time step The corresponding recovery progress coefficient can be calculated using normalized time schedule:

[0266] ;

[0267] in, The function restricts the input values ​​to the range [0,1]. Based on the recovery progress coefficient, the fourth target position of the left and right forelegs at the current time step can be represented as follows:

[0268] ;

[0269] ;

[0270] Therefore, the fifteenth reward item can be represented as:

[0271] .

[0272] in, Switch on / off for the heart-shaped trajectory course; For time window indication function, when The value is 1 if the condition is met, and 0 otherwise. and This is the fourth actual position at the tip of the left and right forelegs; and This is the fourth target position obtained by the above interpolation.

[0273] The fifteenth reward, namely the foreleg recovery reward, guides the forelegs to smoothly recover from the end of the heart-shaped trajectory to the preset preparation point by penalizing the deviation between the fourth actual position and the fourth target position of the left and right foreleg ends.

[0274] Step S803: Determine the stage reward value for the current time step based on the fifth, sixth, eighth, fourteenth, and fifteenth reward items.

[0275] Specifically, in the first closing phase, the phase reward value corresponding to the current time step can be expressed as:

[0276] ;

[0277] in, This is the fifth reward item; This is the sixth reward item; This is the eighth reward item; This is the fourteenth award item; This is the fifteenth reward item.

[0278] During the first closing phase, the stage reward value corresponding to the current time step can be determined through the following steps:

[0279] Step S901: Determine the sixteenth reward item based on the actual angles of each joint of the robot and the preset joint angles of each joint in the final stage.

[0280] Specifically, the preset joint angles of each joint in the second closing stage are denoted as follows: The preset joint angle corresponds to the joint configuration of the robot in its standard standing posture. Therefore, the sixteenth reward item can be represented as:

[0281] ;

[0282] in, For time window indication function, when The value is 1 if the condition is met, and 0 otherwise. For the first The actual angle of each joint at the current time step; For the first The preset joint angles of each joint in the second closing phase.

[0283] The sixteenth reward, namely the deviation from the default joint posture reward, guides all joints of the robot to smoothly return to the standard standing configuration by penalizing the deviation between the actual angle of each joint and the preset joint angle in the second closing stage.

[0284] Step S902: Determine the seventeenth reward item based on the actual height of the robot's body and the preset height.

[0285] Specifically, the height of the quadruped robot's base in its default standing posture is set to 0.42 meters. Therefore, the seventeenth reward item can be represented as:

[0286] ;

[0287] in, This is a time window indicator function; The actual height of the fuselage base at the current time step; 0.42 is the preset standing height in meters, which can be set according to actual needs.

[0288] The seventeenth reward item, namely the base restoration default height reward item, guides the robot to restore its center of gravity to the normal standing height by penalizing the square error between the actual height of the base and the preset height.

[0289] Step S903: Determine the eighteenth reward item based on the projection of the aircraft's gravity vector onto the horizontal plane.

[0290] Specifically, the projection component of the gravity vector in the fuselage coordinate system is denoted as... and When the fuselage is completely horizontal, both values ​​are 0. Therefore, the eighteenth reward item can be represented as:

[0291] ;

[0292] in, This is a time window indicator function; and These represent the gravity vector in the fuselage coordinate system. shaft and Projection component along the axial direction.

[0293] The eighteenth reward, namely the posture-maintaining reward, is achieved through a penalty. and The sum of these factors guides the robot to smoothly return from an upright position to a horizontal standing position.

[0294] Step S904: Determine the stage reward value for this time step based on the sixteenth, seventeenth, and eighteenth reward items.

[0295] Specifically, in the second closing phase, the stage reward value corresponding to the current time step can be expressed as:

[0296] ;

[0297] in, This is the sixteenth award item; This is the seventeenth award item; This is the eighteenth award.

[0298] Through the above steps, by subdividing the closing phase into a first closing phase and a second closing phase, and designing rewards for forefoot retraction, default posture deviation, default height base recovery, and horizontal posture maintenance respectively, the policy network can smoothly and orderly return to a stable standing state after completing the stunt, solving the problem of how to land smoothly after the stunt, avoid impact and loss of control, and achieving a smooth closing of the entire "standing heart" action and safe recovery of the robot.

[0299] In some embodiments, collecting specified data values ​​at that time step includes:

[0300] Place the robot model corresponding to the robot in a pre-built simulation environment;

[0301] Specified data values ​​are collected through the interaction between the robot model and the simulation environment.

[0302] Specifically, when conducting policy training in a simulation environment, it is first necessary to build a virtual interactive environment for the quadruped robot. This simulation environment can be built based on mainstream robot simulation platforms such as NVIDIA Isaac Sim, PyBullet, and MuJoCo. Within the simulation environment, a corresponding URDF (Unified Robot Description Format) model description file is created based on the design, manufacturing, and assembly parameters of the real robot. This model description file includes the physical information of each link of the robot (such as mass, center of mass position, inertia tensor, etc.) and the parameter constraints of each joint (such as angle range, maximum torque, maximum angular velocity, etc.). Taking a quadruped robot as an example, its model mainly includes the body trunk, hip links, thigh links, lower leg links, and foot components, totaling 12 joints in a serial tree structure. The four legs each contain hip joints, thigh joints, and lower leg joints. The physical parameters of each link are kept the same as those of the real robot to ensure that the simulation training results can be accurately transferred to the real hardware.

[0303] In the simulation environment, the interaction between the robot model and the environment is as follows: At each discrete time step, the simulation platform performs dynamic calculations based on the current physical state, updating the state parameters of the robot model and the environment. The policy network receives the current state parameters (including body posture, joint angles, joint angular velocities, foot contact state, etc.) obtained from the simulation environment and outputs motion commands. These motion commands are converted into joint torque commands by the underlying joint controller, driving the robot model to move. After one simulation step, the environment state changes, and the system collects the next state parameters again. The current state parameters, motion commands, next state parameters, and termination flag generated during the above interaction process constitute the specified data value for that time step.

[0304] To more realistically verify and train the robot's movement capabilities in acrobatic scenarios, the simulation environment needs to be equipped with various typical terrain environments based on specific task requirements. These include flat ground for basic movement testing, ramps for assessing balance and power distribution, and stairs for testing gait planning and coordinated control. Simultaneously, physical parameters need to be precisely set for different terrains, including the coefficient of friction to simulate non-slip or slippery surfaces, the elastic coefficient to reflect energy feedback during collisions and landings, and gravity parameters to approximate real-world and even extreme conditions.

[0305] Through the above embodiments, by constructing high-fidelity robot models and diverse training terrains in the simulation platform, and collecting experience data based on the interaction between the robot model and the simulation environment, the problems of high training costs, long cycles, and safety risks of real robots are solved. This achieves rapid, safe, and low-cost collection of the massive interactive data required for reinforcement learning algorithms, while ensuring the feasibility and accuracy of transferring simulation training results to real robots.

[0306] In some embodiments, the above method further includes:

[0307] Introducing domain randomization into the simulation environment; wherein, introducing domain randomization includes:

[0308] At the start of the simulation, the physical properties of the robot model are randomized, including the link center of gravity position, link mass, link size, and joint friction coefficient.

[0309] When resetting the simulation environment, the initial state of the robot model is randomized, including the birth position, initial velocity, and joint angles.

[0310] External disturbances are applied to the robot model at preset time intervals; the external disturbances include external force, impulse, and propulsion speed.

[0311] Specifically, the policy network trained in the simulation environment ultimately needs to be deployed to run on a real robot. However, dynamic differences inevitably exist between the simulation and real environments. These differences arise from several factors: deviations between the physical parameters of the real robot (such as center of mass position, link mass, and joint friction coefficient) and the simulation model; various external disturbances in the real environment that are difficult to model accurately (such as uneven ground, wind effects, and sensor noise); and response delays and measurement errors in the actuators and sensors of the real robot. If training is only performed in an idealized simulation environment, the policy network often overfits to the specific dynamic characteristics of the simulation environment, leading to a significant performance degradation or even complete failure after migration to a real robot.

[0312] To overcome the above problems, this application introduces a domain randomization mechanism during simulation training. The core idea of ​​domain randomization is to randomize the dynamic parameters of the simulation environment within a certain range during training, allowing the policy network to encounter various dynamic conditions during the training phase, thereby forcing it to learn a robust control policy that is insensitive to changes in dynamic parameters. Specifically, the domain randomization methods used in this application include the following three categories:

[0313] The first type involves randomization at the start of the simulation. At the beginning of each training round, the physical properties of the robot model are randomized. These physical properties include, but are not limited to: the center of gravity position of each link (randomly shifted within ±5% of the nominal value), the mass of each link (randomly scaled within ±10% of the nominal value), the size of each link (randomly varied within ±2% of the nominal value), and the coefficient of friction of each joint (randomly sampled within a preset range). This type of randomization remains constant within a single round, but varies between rounds, used to simulate individual differences in robots caused by manufacturing tolerances, assembly errors, etc., in the real world.

[0314] The second type involves randomization during environment reset. At the reset of each training round, the initial state of the robot model is randomized. These initial states include, but are not limited to: the robot's birth position (randomly offset within a preset area), initial velocity (randomly assigned within a preset range), and initial angles of each joint (randomly perturbed within ±5 degrees of the default standing posture). This type of randomization ensures that the policy network can cope with different starting conditions and avoids dependence on a specific initial state.

[0315] The third type involves randomization at time intervals. During training, random external perturbations are applied to the robot model at preset time intervals (e.g., every 1 second). These external perturbations include, but are not limited to, applying external forces or impulses of random direction and magnitude to the robot body, and applying random push speeds to specific links. This type of randomization is used to simulate sudden disturbances that the robot may encounter during real-world operation, such as bumps caused by uneven ground or accidental collisions with obstacles, thereby training the policy network's ability to quickly recover stability after being disturbed.

[0316] Furthermore, to further narrow the gap between simulation and reality, a Gaussian noise model can be introduced into the observed input. (and time delay) to simulate the measurement noise and communication delay problems present in real sensors.

[0317] Through the above embodiments, by introducing three domain randomization strategies during the simulation training process, the policy network is exposed to rich dynamic changes and external disturbances during the training phase. This solves the problems of overfitting of the policy to specific dynamic conditions and performance degradation after transfer to real robots caused by idealized simulation training. As a result, the trained policy network has stronger robustness, generalization ability and disturbance resistance when deployed on real robots, and the success rate of simulation to real transfer is significantly improved.

[0318] To facilitate a more comprehensive understanding of the technical solution presented in this application, supplementary explanations are provided regarding the network architecture design, policy optimization algorithms, and the migration and deployment process from simulation to the real environment related to deep reinforcement learning.

[0319] First, an explanation of the network architecture design. Figure 3 This is a schematic diagram illustrating the implementation principle of training a model to be trained, as shown in an exemplary embodiment of this application. Figure 3 As described above, the embodiments of this application may adopt an asymmetric actor-critic architecture to balance the computational resource limitations of real robot deployment with the sample efficiency and stability during the training phase.

[0320] The policy network (Actor network) only receives sensor observations available to the robot in the actual system, including but not limited to: body angular velocity, body gravity vector, relative angles and joint angular velocities of each joint, motion commands from the previous time step, and body linear and angular velocity commands. The Actor network performs forward inference based on the current observations and outputs motion commands. These motion commands are then converted into torque commands for each joint through a position-controlled motion-torque mapping model, driving the robot's movement.

[0321] The Critic network differs from the Policy network in its structure and input design, receiving richer "privileged information" during training to more accurately evaluate the value of the current state and action. Specifically, the input to the Critic network consists of four concatenated parts:

[0322] (1) Ontological observation information: the same sensor observations as the actor network.

[0323] (2) Privileged Encoder Output: The privileged encoder receives privileged information that cannot be directly observed by the actual machine, such as the true value of external disturbances, the true value of the robot's random mass and centroid position, the true value of the random physical parameters of the links, and the true height of the foot relative to the ground. It compresses and maps these high-dimensional information into low-dimensional privileged space features through a multi-layer perceptron (MLP).

[0324] (3) Height Scan Encoder Output: The height scan encoder uses the ray scan points aligned with the robot's yaw angle as input to perceive the height distribution of the surrounding terrain and extract low-dimensional terrain feature representations.

[0325] (4) Fuselage linear velocity: as a supplement to the observation of the main body.

[0326] The comment network concatenates the above information and then uses a multilayer perceptron to infer and output a state value function, which is used to calculate the advantage function and guide the update of the policy network.

[0327] The aforementioned policy network, comment network, privileged encoder, and height scan encoder can all adopt a Multi-Layer Perceptron (MLP) as the basic network architecture. A MLP consists of an input layer, several hidden layers, and an output layer. Through linear transformations and nonlinear activation functions between layers, it extracts and compresses features from high-dimensional input information layer by layer, effectively learning the nonlinear mapping relationship between input observations and control policies. The core advantages of this design are its simple structure, controllable parameter scale, and low inference computational overhead, making it particularly suitable for the real-time and large-scale parallel training requirements in reinforcement learning scenarios.

[0328] Next, an explanation of policy optimization based on the PPO algorithm.

[0329] This application embodiment can use the Proximal Policy Optimization (PPO) algorithm to jointly update the policy network and the critic network. After collecting a batch of empirical data, data can be sampled from the empirical replay buffer, the objective function of the PPO algorithm can be calculated, and gradient descent updates can be performed. The objective function (i.e., the loss function) of the PPO algorithm can be expressed as:

[0330] ;

[0331] in, The probability ratio represents the difference between the new policy and the old policy in terms of action instructions. The ratio of the probabilities, i.e. ,in Representation strategy, Indicates the current state parameter Below, the current policy network (with parameters) Output action commands The probability of; This indicates that under the same state and action, the old policy network before the update (with parameters) will be used instead. The probability of the output; The dominant function represents the current action instruction. The relative performance of strategies compared to their average performance, i.e. ; The pruning factor is used to constrain the policy update magnitude within a certain range. Within a certain range, to ensure training stability; The function is for finding the minimum value; This is the clipping function, used to limit the probability ratio within a preset range; Let be the expectation operator, representing the calculation of the average value over a batch of empirical data. By minimizing the above loss function, the policy network parameters... The strategy will be updated in the direction of increasing the expected value of the advantage function, thereby gradually converging to the optimal strategy.

[0332] Finally, a note on Sim-to-Real migration deployment.

[0333] In practical deployment, due to engineering constraints such as size, weight, and power consumption, robots typically use embedded platforms or industrial PCs as the main control computing unit. This type of hardware usually has strong CPU computing power but often lacks a GPU, making it difficult to directly run Python inference environments that rely on PyTorch and GPU acceleration. To meet the requirements of high real-time performance and high stability inference, this embodiment exports the converged policy model to ONNX (Open Neural Network Exchange) format and performs efficient cross-platform and cross-language deployment based on the ONNX Runtime inference engine, thereby fully utilizing the CPU instruction set or hardware acceleration unit to improve inference performance.

[0334] During deployment, to ensure consistency between the strategy behavior and the training phase, a complete and accurate observation input chain aligned with the simulation environment needs to be constructed. Specifically, the current time progress is calculated according to the control cycle and the number of loop steps; the robot acquires the angle information of each joint through joint encoders and calculates the joint angular velocity through numerical difference; the inertial measurement unit installed on the body provides the body's attitude angle and angular velocity information, and further calculates the gravity vector in the body coordinate system based on the attitude angles to supplement attitude and balance-related observations. To achieve time-discrete characteristics consistent with simulation training, the system uses a high-precision timer to control the inference cycle, clearly defining the time interval for each strategy inference and the update frequency of various observation signals, and ensuring that the actual update frequency of all sensors is not lower than the observation frequency required by the strategy, thereby avoiding performance degradation due to observation lag or asynchrony.

[0335] Furthermore, to ensure the correctness and stability of the inference input, the actual machine observation space needs to be strictly aligned, including the observation dimension, units, normalization method, and numerical range. Appropriate filtering (such as low-pass filtering or moving average) should be introduced into some sensor outputs to make their statistical characteristics as close as possible to the observation distribution during the training phase, reducing simulation-to-real-machine transfer errors. From an engineering safety perspective, the action commands output by the strategy need to be properly pruned, limiting the action amplitude within a safe range. Simultaneously, joint torque commands need to be saturated to prevent the output from exceeding the physical limits of robot joints, actuators, or structural components, thereby improving the overall safety and reliability of the system while ensuring strategy performance.

[0336] Figure 4 This is a schematic diagram illustrating the training and deployment process of a policy network as shown in an exemplary embodiment of this application, such as... Figure 4 As shown, the implementation process includes creating a model and designing a training environment. Optionally, domain randomization can be introduced when designing the training environment, and course learning and corresponding termination conditions can be set. Then, the target actions are divided, basic reward functions are constructed, stage reward functions are constructed, the policy network is trained, and the trained policy network is deployed. Throughout the implementation process, the performance of the policy network can be evaluated according to the actual training situation. Evaluation indicators may include cumulative reward value, action smoothness, task completion rate, and action reproduction accuracy after simulation to real-world transfer. Based on the evaluation results, the weights of each reward function, the threshold for starting course learning, and the parameter range of domain randomization can be further adjusted, and the training and optimization can be repeated until the policy network reaches the preset performance indicator requirements. Through the above complete training and deployment process, the embodiment of this application realizes efficient training and stable and reliable deployment of complex motion policy networks for quadruped robots in real environments.

[0337] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0338] This embodiment also provides a policy network training device for implementing the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0339] Figure 5 The structural block diagram of the policy network training device provided in this application is as follows: Figure 5 As shown, the device includes:

[0340] The action segmentation module 10 is used to divide the robot's target action into multiple sub-stages according to the order of execution time, and to construct a basic reward function for the target action and a corresponding stage reward function for each sub-stage. The basic reward function is used to constrain the robot's action to meet physical constraints and safety requirements. The stage reward function corresponding to each sub-stage is used to guide the model to be trained to learn the action features corresponding to that stage. The model to be trained includes a policy network and a commentator network.

[0341] The experience acquisition module 20 is used to acquire specified data values ​​for each time step. The specified data values ​​include the current state parameters for the time step, the action instructions for the time step, the next state parameters after the action instructions are executed, and the termination flag.

[0342] The reward calculation module 30 is used to determine the basic reward value for the time step based on the basic reward function, determine the stage reward value for the time step based on the stage reward function corresponding to the target action stage to which the time step belongs, and determine the comprehensive reward value for the time step based on the stage reward value and the basic reward value.

[0343] The experience collection module 20 is also used to determine the experience data for the time step based on the comprehensive reward value and the specified data value for the time step.

[0344] The policy training module 40 is used to train the model to be trained based on the empirical data at each time step, so as to obtain the trained policy network.

[0345] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination. Specific examples in this embodiment can be found in the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0346] This embodiment also provides a robot control method. Figure 6 The flowchart of the robot control method provided in this application is as follows: Figure 6 As shown, the process includes the following steps:

[0347] Step S610: Collect the robot's current state parameters;

[0348] Step S620: Input the current state parameters into the policy network so that the policy network can output action instructions based on the current state parameters; wherein, the policy network is trained according to any of the above-mentioned policy network training methods.

[0349] Step S630: Control the robot to execute the actions corresponding to the above action instructions.

[0350] By employing the aforementioned robot control method, current state parameters collected from the actual environment are input into a pre-trained policy network. The policy network then infers and outputs action commands in real time to control the robot's execution. This solves the problem that traditional control methods rely on precise dynamic models or manually designed controllers when facing complex dynamic tasks. Since the policy network is trained using the aforementioned phased reward mechanism, it has fully learned the key dynamic characteristics of each stage of the target action during training. Therefore, complex online planning or parameter tuning is unnecessary during deployment; a single forward inference is sufficient to generate action commands that meet the task requirements. This achieves end-to-end rapid response from perception to action, significantly reducing the computational overhead and debugging difficulty of actual deployment. This embodiment also provides a robot, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to execute the aforementioned robot control method.

[0351] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0352] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0353] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0354] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0355] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for training a policy network, characterized in that, The method includes: The robot's target action is divided into multiple sub-stages according to the order of execution time. A basic reward function is constructed for the target action, and a corresponding stage reward function is constructed for each sub-stage. The basic reward function is used to constrain the robot's action to meet physical constraints and safety requirements. The stage reward function corresponding to each sub-stage is used to guide the model to be trained to learn the action features corresponding to that sub-stage. The model to be trained includes a policy network and a commentator network. For each time step, a specified data value is collected at that time step; the specified data value includes the current state parameter at that time step, the action instruction at that time step, the next state parameter after the action instruction is executed at that time step, and the termination flag at that time step. The basic reward value for this time step is determined based on the basic reward function, and the stage reward value for this time step is determined based on the stage reward function corresponding to the target action stage to which this time step belongs. Based on the stage reward value and the base reward value, determine the comprehensive reward value for that time step, and based on the comprehensive reward value and the specified data value for that time step, determine the experience data for that time step. The model to be trained is trained based on the empirical data at each time step to obtain the trained policy network; The robot is a quadruped robot, and the target action is a standing heart-shaped gesture; the target action of the robot is divided into multiple sub-stages according to the order of execution time, including: The standing heart-shaped action is divided into the initial adjustment stage, the upright stage, the heart-shaped trajectory preparation stage, the heart-shaped trajectory execution stage, and the closing stage. Determining the basic reward value at this time step based on the basic reward function includes: The first penalty term is determined based on the torque and angular velocity of each joint of the robot. The second penalty term is determined based on the absolute value of the torque at each joint; The third penalty is determined based on the angles of each joint, the preset upper safety limit angle, and the preset lower safety limit angle. The fourth penalty is determined based on the deviation between the action commands of each joint at the current time step and the action commands of each joint at the previous time step. The base reward value is determined based on the first penalty item, the second penalty item, the third penalty item, and the fourth penalty item.

2. The policy network training method according to claim 1, characterized in that, When the target action phase is the initial adjustment phase, determining the stage reward value for that time step based on the stage reward function corresponding to the target action phase to which that time step belongs includes: Based on the cumulative duration corresponding to the current time step, the preset starting target height of the robot's body base in the initial adjustment phase, and the preset ending target height of the body base in the initial adjustment phase, the target height of the body base in the current time step is determined, and the first reward item is determined based on the actual height of the body base in the current time step and the target height of the body base in the current time step. Based on the accumulated duration, the preset starting reference angle of each joint of the robot in the initial adjustment phase, and the preset ending reference angle of each joint in the initial adjustment phase, the target angle of each joint in the current time step is determined, and the second reward item is determined based on the actual angle of each joint in the current time step and the target angle of each joint in the current time step. The third reward item is determined based on the actual angle of each front leg hip joint of the robot at that time step and the preset angle of each front leg hip joint at that time step. The fourth reward item is determined based on the projection of the robot's gravity vector onto the horizontal plane; The stage reward value at this time step is determined based on the first reward item, the second reward item, the third reward item, and the fourth reward item.

3. The policy network training method according to claim 1, characterized in that, When the target action phase is the fuselage uprighting phase, determining the stage reward value for that time step according to the stage reward function corresponding to the target action phase to which that time step belongs includes: The fifth reward item is determined based on the airborne contact state of the robot's left and right front feet, and the sixth reward item is determined based on the duration of the continuous airborne state of the left and right front feet. The seventh reward item is determined based on the actual ground clearance of the left and right forelegs and the preset ground clearance of the left and right forelegs; The eighth reward item is determined based on the robot's actual head height above the ground and the preset head height above the ground. The ninth reward item is determined based on the cumulative duration corresponding to the current time step, the preset starting pitch angle of the robot's body during the upright phase, the preset ending pitch angle of the robot's body during the upright phase, and the actual gravity vector of the robot at that time step. The stage reward value at this time step is determined based on the fifth reward item, the sixth reward item, the seventh reward item, the eighth reward item, and the ninth reward item.

4. The policy network training method according to claim 3, characterized in that, The heart-shaped trajectory preparation phase includes a first sub-phase and a second sub-phase; when the target action phase is the heart-shaped trajectory preparation phase, determining the stage reward value for that time step according to the stage reward function corresponding to the target action phase to which that time step belongs includes: Determine the fifth reward item, the sixth reward item, and the eighth reward item; The tenth reward item is determined based on the components of the robot's gravity vector and the components of the target gravity vector preset for the preparation stage of the heart-shaped trajectory. Based on the cumulative duration corresponding to the current time step, determine the target sub-stage to which the current time step belongs; When the target sub-stage is the first sub-stage, the eleventh reward item is determined based on the first actual position of the robot's left and right front feet at the current time step and the preset first target position of the left and right front feet. The stage reward value for the current time step is determined based on the fifth, sixth, eighth, tenth, and eleventh reward items.

5. The policy network training method according to claim 4, characterized in that, When the target sub-stage is the second sub-stage, the method further includes: Based on the preset left and right front foot positions and the cumulative duration corresponding to the current time step, the second target position of the robot's left and right front feet in the current time step is determined, and based on the second actual position of the left and right front feet in the current time step and the second target position of the left and right front feet in the current time step, the twelfth reward item is determined. The stage reward value for the current time step is determined based on the fifth, sixth, eighth, tenth, and twelfth reward items.

6. The policy network training method according to claim 3, characterized in that, When the target action phase is the execution phase of the heart-shaped trajectory, determining the stage reward value for that time step according to the stage reward function corresponding to the target action phase to which that time step belongs includes: Determine the fifth reward item, the sixth reward item, and the eighth reward item; Based on the cumulative duration corresponding to the current time step, the start time of the heart-shaped trajectory execution phase, and the end time of the heart-shaped trajectory execution phase, the trajectory reference angles of the robot's left and right forelegs are determined, and based on the trajectory reference angles and the pre-constructed heart-shaped trajectory equation, the third target position of the left and right forelegs at the current time step is determined. The thirteenth reward item is determined based on the third actual position of the left and right forelegs at the current time step and the third target position of the robot at the current time step. The fourteenth reward item is determined based on the actual angles of each hind leg joint of the robot and the preset angles of each hind leg joint during the execution phase of the heart-shaped trajectory. The stage reward value at this time step is determined based on the fifth, sixth, eighth, thirteenth, and fourteenth reward items.

7. The policy network training method according to claim 6, characterized in that, The closing phase includes a first closing phase and a second closing phase; when the target action phase is the closing phase, determining the stage reward value for that time step according to the stage reward function corresponding to the target action phase to which that time step belongs includes: The fifth reward item, the sixth reward item, the eighth reward item, and the fourteenth reward item are determined; Based on the cumulative duration corresponding to the current time step, determine the target closing stage to which the current time step belongs; When the target closing stage is the first closing stage, the robot's left and right front feet are determined to be the fourth target position in the current time step based on the robot's end position in the heart-shaped trajectory execution stage, the robot's preset end position in the closing stage, and the cumulative duration corresponding to the current time step. The fifteenth reward item is determined based on the fourth actual position of the left and right front feet in the current time step and the fourth target position of the left and right front feet in the current time step. The stage reward value for the current time step is determined based on the fifth, sixth, eighth, fourteenth, and fifteenth reward items.

8. The policy network training method according to claim 7, characterized in that, When the target closing stage is the second closing stage, the method further includes: The sixteenth reward item is determined based on the actual angles of each joint of the robot and the preset joint angles of each joint in the closing stage; the seventeenth reward item is determined based on the actual height of the robot's body and the preset height; and the eighteenth reward item is determined based on the projection of the gravity vector of the robot's body onto the horizontal plane. The stage reward value at this time step is determined based on the sixteenth reward item, the seventeenth reward item, and the eighteenth reward item.

9. The policy network training method according to claim 1, characterized in that, The acquisition of specified data values ​​at this time step includes: Place the robot model corresponding to the robot in a pre-built simulation environment; The specified data values ​​are collected through the interaction between the robot model and the simulation environment.

10. The policy network training method according to claim 9, characterized in that, The method further includes: Introducing domain randomization into the simulation environment; wherein, the introduction of domain randomization includes: At the start of the simulation, the physical properties of the robot model are randomized, including the link center of gravity position, link mass, link size, and joint friction coefficient. When the simulation environment is reset, the initial state of the robot model is randomized, including the birth position, initial velocity, and joint angles. External disturbances are applied to the robot model at preset time intervals; wherein, the external disturbances include external force, impulse, and pushing speed.

11. The policy network training method according to claim 1, characterized in that, The determination of joint state over-limit penalty items based on the angle of each joint, the preset safety upper limit, and the preset safety lower limit includes: Based on the angles of each joint and the preset safety limit, determine the first excess deviation of each joint relative to the preset safety limit; Based on the angles of each joint and the preset safety lower limit, determine the second excessive deviation of each joint relative to the preset safety lower limit; Based on the first and second over-limit deviations, the overall deviation value of each joint is determined, and based on the overall deviation value of each joint, the joint state over-penalty term is determined.

12. A policy network training device, characterized in that, The device includes: The action segmentation module is used to divide the robot's target action into multiple sub-stages according to the execution time sequence, and to construct a basic reward function for the target action and a corresponding stage reward function for each sub-stage. The basic reward function is used to constrain the robot's actions to meet physical constraints and safety requirements. The stage reward function for each sub-stage is used to guide the model to be trained to learn the action features corresponding to that stage. The model to be trained includes a policy network and a commentator network. The experience acquisition module is used to collect specified data values ​​for each time step; the specified data values ​​include the current state parameters for the time step, the action instructions for the time step, the next state parameters after the action instructions are executed, and the termination flag for the time step. The reward calculation module is used to determine the basic reward value for the time step based on the basic reward function, determine the stage reward value for the time step based on the stage reward function corresponding to the target action stage to which the time step belongs, and determine the comprehensive reward value for the time step based on the stage reward value and the basic reward value. The experience acquisition module is also used to determine the experience data at the time step based on the comprehensive reward value at the time step and the specified data value at the time step. The policy training module is used to train the model to be trained based on the empirical data at each time step, so as to obtain the trained policy network. Determining the basic reward value at this time step based on the basic reward function includes: The first penalty term is determined based on the torque and angular velocity of each joint of the robot. The second penalty term is determined based on the absolute value of the torque at each joint; The third penalty is determined based on the angles of each joint, the preset upper safety limit angle, and the preset lower safety limit angle. The fourth penalty is determined based on the deviation between the action commands of each joint at the current time step and the action commands of each joint at the previous time step. The base reward value is determined based on the first penalty item, the second penalty item, the third penalty item, and the fourth penalty item.

13. A robot control method, characterized in that, include: Collect the robot's current state parameters; The current state parameters are input into the policy network, so that the policy network outputs action instructions based on the current state parameters; wherein the policy network is trained by the policy network training method according to any one of claims 1 to 11. Control the robot to execute the action corresponding to the action command.

14. A robot, comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the robot control method of claim 13.

Images