Robot motion control method, electronic device, storage medium, and program product

Abstract

The application relates to the technical field of robots, in particular to a robot motion control method, an electronic device, a storage medium and a program product. The method comprises the following steps: in a simulation environment, an initial control strategy network is iteratively trained according to a preset reward function by using a preset reinforcement learning algorithm, and a target control strategy network is obtained; the preset reward function comprises multiple reward items, and the multiple reward items are respectively determined based on speed deviation data, attitude data, chassis height offset data and an energy consumption model corresponding to each joint motor of a robot model in a motion process; and the target control strategy network is deployed on a target robot by using a simulation-to-real migration method. The method can make the control strategy network consider speed tracking, attitude stability, chassis stability and energy efficiency at the same time, realize minimization of average power consumption in a moving process through multi-objective optimization, and significantly reduce energy consumption of a heavy-load multi-legged robot on uneven terrain.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a robot motion control method, electronic device, storage medium, and program product. Background Technology

[0002] Currently, robotics technology is developing rapidly. Multi-legged robots, with their excellent terrain adaptability, are showing broad application prospects in complex operational scenarios such as mine exploration, building inspection, and disaster relief. As the load-bearing capacity requirements of multi-legged robots continue to increase, the proportion of their chassis weight in the robot's total mass is constantly increasing. This change directly affects the robot's energy consumption and endurance.

[0003] When multi-legged robots move on uneven terrain, the up-and-down movement of their chassis leads to significant energy consumption. Taking a hexapod robot with a chassis mass of 200kg as an example, when the chassis rises and falls by 5cm per step, the lifting work is approximately 98J. At a step frequency of 2 steps / second, the power consumption from chassis undulation alone reaches 196W. Therefore, reducing the rise and fall of the chassis's center of gravity during movement is a key way to reduce the energy consumption of heavy-duty multi-legged robots. Furthermore, the joint motors of multi-legged robots exhibit heterogeneous energy recovery characteristics in actual working conditions. Specifically, some motors have complete energy recovery mechanisms, some can only partially recover braking energy, and some have no energy recovery capability at all. Different joints of the same robot are usually equipped with joint motors with these different characteristics.

[0004] Currently, motion control for heavy-duty multi-legged robots primarily employs reinforcement learning strategies. However, during reinforcement learning training, the focus is typically solely on the robot's balance control and gait stability, neglecting the additional energy consumption caused by chassis center-of-gravity fluctuations. Furthermore, the energy models used in existing reinforcement learning strategies are often simplified, failing to fully consider the heterogeneous energy recovery characteristics of joint motors. This results in significant discrepancies between the energy consumption performance of the trained control strategies and the simulation results when deployed in practice. Summary of the Invention

[0005] This application provides a robot motion control method, electronic device, storage medium, and program product that can significantly reduce the energy consumption of heavy-duty multi-legged robots on uneven terrain.

[0006] Firstly, this application provides a robot motion control method, comprising: constructing a robot model corresponding to a target robot and various terrains in a simulation environment, including non-flat terrain; the robot model includes a chassis and multiple supporting legs supporting the chassis, with joint motors at the joints of each supporting leg; in the simulation environment, controlling the robot model to move on various terrains based on the control parameters of each joint motor output by an initial control strategy network, and iteratively training the initial control strategy network using a preset reinforcement learning algorithm according to a preset reward function to obtain a target control strategy network; wherein: the initial control strategy network is used to output the control parameters of each joint motor according to the environmental state information of the robot model in the simulation environment, the environmental state information including at least terrain perception information of various terrains; the preset reward function includes a first reward item, a second reward item, a third reward item, and a fourth reward item. The incentives are as follows: the first incentive is based on the speed deviation data of the robot model during movement; the second incentive is based on the posture data of the robot model during movement; the third incentive is based on the chassis height offset data of the robot model during movement; and the fourth incentive is based on the energy consumption model corresponding to each joint motor. The energy consumption model includes a fully recovered energy consumption model for joint motors in the fully recovered energy mode, a partially recovered energy consumption model for joint motors in the partially recovered energy mode, or a zero recovered energy consumption model for joint motors in the zero recovered energy mode. The target control strategy network is deployed on the target robot using the simulation-to-real-world transfer method. The target control strategy network is used to output the control parameters of each joint motor in the target robot based on the environmental state information of the target robot in the real environment when the target robot is running.

[0007] According to the above method, the preset reward function includes a first reward item, a second reward item, a third reward item, and a fourth reward item. The first reward item is determined based on the robot model's velocity deviation data during movement, which enables the control policy network to have accurate velocity tracking capabilities, ensuring that the robot maintains the expected movement speed in complex terrain. The second reward item is determined based on the robot model's posture data during movement, which can suppress the robot's body tilt and pitch during movement, enhancing the robot's walking stability on uneven terrain. The third reward item is determined based on the robot model's chassis height offset data during movement, realizing the integration of chassis center of gravity stability as an explicit optimization objective into the reinforcement learning training framework. This allows the control policy network to actively learn control strategies to suppress chassis undulations, avoiding unnecessary chassis lifting and lowering movements, thereby significantly reducing energy consumption caused by chassis center of gravity fluctuations. This is particularly suitable for heavy-duty multi-legged robots where chassis weight accounts for a major proportion. The fourth reward is determined based on the energy consumption model corresponding to each joint motor. This model includes energy consumption models for each joint motor with three energy recovery characteristics: complete energy recovery, partial energy recovery, and zero energy recovery. This improves the accuracy of energy consumption prediction and guides the control strategy network to prioritize energy-saving joint drive commands. Using this preset reward function, the control strategy network can simultaneously consider speed tracking, posture stability, chassis stability, and energy efficiency. Through multi-objective optimization, it minimizes the average power consumption during movement, significantly reducing energy consumption of heavy-duty multi-legged robots on complex, uneven terrains such as mines and construction sites while ensuring motion stability and terrain adaptability.

[0008] In one possible implementation of the first aspect, the control parameters of each joint motor are used to ensure that the robot model maintains the chassis in a horizontal orientation during movement and keeps the height change of the chassis less than a threshold.

[0009] In one possible implementation of the first aspect, the first reward term is calculated according to the following formula:

[0010] R_motion = w1×exp(-||v_xy - v_target||^2);

[0011] Where R_motion represents the first reward value, w1 represents the first weight parameter, v_xy represents the motion speed of the robot model, and v_target represents the target motion speed of the robot model;

[0012] Attitude data includes the chassis roll angle and chassis pitch angle. The second bonus item is calculated according to the following formula:

[0013] R_balance = -w2×(roll^2 + pitch^2);

[0014] Where R_balance represents the second reward value, w2 represents the second weight parameter, roll represents the chassis roll angle, and pitch represents the chassis pitch angle;

[0015] The chassis height offset data includes the chassis height offset amount and the speed of the chassis's center of gravity in the height direction. The third bonus item is calculated according to the following formula:

[0016] R_cog = -w3×|h(t)-h_ref|^2 - w4×|v_z(t)|^2;

[0017] Where R_cog represents the third reward value, w3 represents the third weight parameter, h(t) represents the center of gravity height of the chassis at the current moment, h_ref represents the reference height, h(t)-h_ref represents the chassis height offset, v_z(t) represents the speed of the center of gravity of the chassis in the height direction, and w4 represents the fourth weight data.

[0018] The fourth reward item is calculated using the following formula:

[0019] R_power = -w5×P_avg;

[0020] Where R_power represents the fourth reward value, w5 represents the fifth weight parameter, and P_avg represents the average power consumption of the robot model in a complete gait cycle. P_avg is determined based on the energy consumption model corresponding to each joint motor.

[0021] In one possible implementation of the first aspect, the joint motors in the multiple supporting legs include a first joint motor, a second joint motor, and a third joint motor. The energy consumption model corresponding to the first joint motor is a fully recovered energy consumption model, the energy consumption model corresponding to the second joint motor is a partially recovered energy consumption model, and the energy consumption model corresponding to the third joint motor is a zero-recovery energy consumption model. The expression for the fully recovered energy consumption model is:

[0022] P_j1(t)=τ_j1(t)×ω_j1(t);

[0023] Where j1 represents the first joint motor, P_j1(t) represents the power consumption of the first joint motor at time t, τ_j1(t) represents the torque of the first joint motor at time t, and ω_j1(t) represents the angular velocity of the first joint motor at time t; the expression for the partially recovered energy consumption model is:

[0024] ;

[0025] Where j2 represents the second joint motor, P_j2(t) represents the power consumption of the second joint motor at time t, τ_j2(t) represents the torque of the second joint motor at time t, ω_j2(t) represents the angular velocity of the second joint motor at time t, and x represents the percentage of energy recovery, which is less than or equal to 1; the expression for the zero-recovery energy consumption model is:

[0026] P_j3(t)= |τ_j3(t)×ω_j3(t)|;

[0027] Where j3 represents the third joint motor, P_j3(t) represents the power consumption of the third joint motor at time t, τ_j3(t) represents the torque of the third joint motor at time t, and ω_j1(t) represents the angular velocity of the third joint motor at time t.

[0028] In one possible implementation of the first aspect, the mass of the chassis accounts for more than 50% of the total mass of the robot model, and the terrain includes at least one of flat ground, slopes, undulating gravel ground, and steps.

[0029] In one possible implementation of the first aspect, the robot model includes six supporting legs, at least three of which are in contact with the ground when the robot model moves, and the contact points of the at least three supporting legs with the ground are not on the same straight line.

[0030] In one possible implementation of the first aspect, the supporting leg also includes a wheeled walking unit. The walking modes of the robot model on various terrains include wheeled walking mode, legged walking mode, and wheeled-legged hybrid walking mode. The preset reward function also includes a fifth reward item and a sixth reward item. The fifth reward item is determined based on the mode switching smoothness index, and the sixth reward item is determined based on the contact force between the wheeled walking unit and the ground. The mode switching smoothness index is used to characterize the smoothness of different walking mode switching.

[0031] In one possible implementation of the first aspect, the environmental state information of the robot model in the simulation environment includes the robot's ontological perception information, external disturbance information, and domain randomization information.

[0032] In one possible implementation of the first aspect, the ontological perception information includes at least one of the following: the roll angle of the chassis, the pitch angle of the chassis, the yaw angle of the chassis, the speed of the chassis in three-dimensional space, the angle and angular velocity of each joint motor, and terrain perception information; external disturbance information includes randomly applied external forces applied to the robot model; and domain randomization information includes randomization of at least one of the following parameters: ground friction coefficient, chassis mass, joint friction force, and sensor noise in the robot model.

[0033] In one possible implementation of the first aspect, the reinforcement learning algorithm is preset to be a proximal policy optimization algorithm, and the initial control policy network is a multilayer perceptron.

[0034] Secondly, this application provides an electronic device, which includes: a memory for storing one or more programs; and a processor for executing one or more programs to enable the electronic device to implement any of the robot motion control methods provided in the first aspect above.

[0035] Thirdly, this application provides a readable storage medium including one or more programs that, when executed on an electronic device, enable the electronic device to implement any of the robot motion control methods provided in the first aspect above.

[0036] Fourthly, this application provides a computer program product that, when run on an electronic device, enables the electronic device to implement any of the robot motion control methods provided in the first aspect above.

[0037] The technical effects that can be achieved by the second to fourth aspects mentioned above can be referred to the technical effects that can be achieved by the first aspect, and will not be elaborated here. Attached Figure Description

[0038] Figure 1 According to some embodiments of this application, a flowchart of a robot motion control method is shown;

[0039] Figure 2 According to some embodiments of this application, schematic diagrams of three energy recovery modes are shown;

[0040] Figure 3 According to some embodiments of this application, a structural diagram of a preset reward function is shown;

[0041] Figure 4 According to some embodiments of this application, a schematic diagram of a robot control strategy training and deployment framework based on Sim2Real transfer is shown;

[0042] Figure 5 According to some embodiments of this application, a gait diagram of a hexapod robot is shown;

[0043] Figure 6 According to some embodiments of this application, the chassis height change of a hexapod robot during movement is shown;

[0044] Figure 7 According to some embodiments of this application, a flowchart of another robot motion control method is shown;

[0045] Figure 8 According to some embodiments of this application, a block diagram of a robot motion control system is shown;

[0046] Figure 9 A block diagram of an electronic device is shown according to some embodiments of this application. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0048] The illustrative embodiments of this application include, but are not limited to, robot motion control methods, readable storage media, program products, and electronic devices.

[0049] As mentioned earlier, heavy-duty multi-legged robots traverse uneven terrain, and the significant mass of their chassis causes substantial energy consumption due to the up-and-down movement of their center of gravity. Existing reinforcement learning methods do not explicitly optimize chassis center of gravity stability, making it difficult for the trained control strategies to achieve energy reduction goals. Furthermore, the energy recovery characteristics of the joint motors within the robot are heterogeneous, and existing solutions, failing to consider this heterogeneity, ultimately lead to energy consumption prediction errors in practical applications.

[0050] To address the aforementioned problems, this application provides a robot motion control method. This method constructs a robot model corresponding to the target robot and various terrains in a simulation environment. The various terrains can include uneven terrain, and the robot model can include a chassis and multiple supporting legs supporting the chassis, with joint motors at the joints of each supporting leg. Then, in the simulation environment, the robot model is controlled to move on the various terrains based on the control parameters of each joint motor output by an initial control policy network. A preset reinforcement learning algorithm is used to iteratively train the initial control policy network according to a preset reward function to obtain the target control policy network. The initial control policy network is used to output the control parameters of each joint motor based on the environmental state information of the robot model in the simulation environment. The environmental state information includes at least terrain perception information for various terrains. The preset reward function includes a first reward item, a second reward item, a third reward item, and a fourth reward item. The first reward item is determined based on the robot model's velocity deviation data during movement; the second reward item is determined based on the robot model's posture data during movement; the third reward item is determined based on the robot model's chassis height offset data during movement; and the fourth reward item is determined based on the energy consumption model corresponding to each joint motor. The energy consumption model includes a fully recovered energy consumption model for joint motors in a fully recovered energy mode, a partially recovered energy consumption model for joint motors in a partially recovered energy mode, or a zero recovered energy consumption model for joint motors in a zero recovered energy mode. After obtaining the target control strategy network, it can be deployed to the target robot using a simulation-to-real-world transfer method. The target control strategy network is used to output the control parameters of each joint motor in the target robot based on the environmental state information of the target robot in the real environment during operation.

[0051] According to the above method, the preset reward function includes a first reward item, a second reward item, a third reward item, and a fourth reward item. The first reward item is determined based on the robot model's velocity deviation data during movement, which enables the control policy network to have accurate velocity tracking capabilities, ensuring that the robot maintains the expected movement speed in complex terrain. The second reward item is determined based on the robot model's posture data during movement, which can suppress the robot's body tilt and pitch during movement, enhancing the robot's walking stability on uneven terrain. The third reward item is determined based on the robot model's chassis height offset data during movement, realizing the integration of chassis center of gravity stability as an explicit optimization objective into the reinforcement learning training framework. This allows the control policy network to actively learn control strategies to suppress chassis undulations, avoiding unnecessary chassis lifting and lowering movements, thereby significantly reducing energy consumption caused by chassis center of gravity fluctuations. This is particularly suitable for heavy-duty multi-legged robots where chassis weight accounts for a major proportion. The fourth reward is determined based on the energy consumption model corresponding to each joint motor. This model includes energy consumption models for each joint motor with three energy recovery characteristics: complete energy recovery, partial energy recovery, and zero energy recovery. This improves the accuracy of energy consumption prediction and guides the control strategy network to prioritize energy-saving joint drive commands. Using this preset reward function, the control strategy network can simultaneously consider speed tracking, posture stability, chassis stability, and energy efficiency. Through multi-objective optimization, it minimizes the average power consumption during movement, significantly reducing energy consumption of heavy-duty multi-legged robots on complex, uneven terrains such as mines and construction sites while ensuring motion stability and terrain adaptability.

[0052] It is understood that the robot motion control method provided in this application can be applied to any electronic device, including but not limited to mobile stations (MS) and mobile terminals (MT). For example, electronic devices can be mobile phones, smart TVs, wearable devices, tablets, desktop computers, laptops, virtual reality (VR) devices, terminals in self-driving vehicles, terminals in remote medical surgery, terminals in smart grids, terminals in smart cities, terminals in smart homes, etc. This application does not limit the specific form of the electronic device.

[0053] The following is combined with Figure 1The flowchart shown illustrates the robot motion control method provided in this application. This method can be applied to electronic devices, such as the aforementioned computer and other electronic devices. Figure 1 As shown, specifically, the method is as follows:

[0054] S101: Construct the robot model corresponding to the target robot and various terrains in the simulation environment. The various terrains include uneven terrains. The robot model includes a chassis and multiple supporting legs supporting the chassis. Each supporting leg is equipped with a joint motor at its joint.

[0055] By constructing a robot model corresponding to the target robot and various terrains in a simulation environment, it is possible to train the control policy network based on reinforcement learning in the simulation environment. Then, the trained target control policy network can be deployed to the target robot through simulation-to-reality transfer methods (such as Sim2Real), which can significantly reduce the debugging cost of the target robot.

[0056] The simulation environment can be built using a pre-defined physics simulator (such as Saac Gym or MuJoCo). When building the simulation environment, a robot model corresponding to the target robot can be constructed within it. The target robot can be a heavy-duty multi-legged robot. Heavy-duty means that the weight of the chassis accounts for more than 50% of the total weight of the target robot, and multi-legged means that the target robot has two or more supporting legs.

[0057] A robot model is a simulation model of the target robot in a simulation environment. A robot model can include a chassis and multiple support legs that support the chassis. The chassis's mass accounts for more than 50% of the total mass of the robot model; for example, the chassis's weight accounts for 60% to 80% of the total mass of the robot model.

[0058] The robot model can have two or more supporting legs, such as four or six. Each supporting leg has a joint motor at its joint, resulting in multiple joint motors in the robot model. These joint motors can have the same or different energy recovery characteristics during braking; this application does not impose any limitations on this.

[0059] For example, the target robot can be a hexapod robot with a chassis mass of 150 kg and a total mass of 220 kg. The robot model corresponding to this target robot can include six supporting legs, each with three degrees of freedom: hip roll, hip pitch, and knee pitch. The robot model has a total of 18 joint motors. Of these, six hip roll motors have full energy recovery capability, six hip pitch motors have 50% partial energy recovery capability, and six knee motors have no energy recovery capability.

[0060] To train a control policy network with strong generalization capabilities, the simulation environment needs to include various terrains. These terrains can include one or more of the following: flat ground, slopes, undulating gravel surfaces, and steps. For example, flat ground, slopes of 5°–15°, randomly undulating gravel surfaces with a height of 0–10 cm, and steps with a height of 5–15 cm can be generated in the simulation environment. Furthermore, to improve the control policy network's resistance to interference, randomly placed or moving obstacles can be generated in the simulation environment as external disturbances, such as spherical or ellipsoidal stones, cylindrical logs, cuboid ditches, or protrusions.

[0061] S102: In the simulation environment, the robot model is controlled to move on various terrains based on the control parameters of each joint motor output by the initial control strategy network. The initial control strategy network is iteratively trained using a preset reinforcement learning algorithm according to a preset reward function to obtain the target control strategy network.

[0062] After the simulation environment is built, the robot model can be controlled to move on various terrains based on the control parameters of each joint motor output by the initial control strategy network in the simulation environment, so as to achieve reinforcement learning training of the initial control strategy network.

[0063] Before training the initial control policy network using reinforcement learning, it is necessary to design the reward function for the reinforcement learning process.

[0064] To reduce the energy consumption of the target robot, it is necessary to improve the accuracy of energy consumption calculation in the control policy network. To achieve this goal, when designing the reward function, an energy consumption model corresponding to each joint motor can be established first. Then, a reward function can be designed based on the energy consumption model of each joint motor. Subsequently, the initial control policy network can be iteratively trained according to the designed preset reward function.

[0065] Based on the energy recovery characteristics of joint motors during braking, the target robot may include three types of joint motors. These three types of joint motors can correspond to three energy recovery modes: complete energy recovery mode, partial energy recovery mode, and zero energy recovery mode. For example, the joint motors in multiple support legs may include a first joint motor, a second joint motor, and a third joint motor. The first joint motor operates in a complete energy recovery mode, the second joint motor operates in a partial energy recovery mode, and the third joint motor operates in a zero energy recovery mode. Figure 2 A schematic diagram of these three energy recovery modes is shown below. (Refer to...) Figure 2 The full energy recovery mode means that 100% of the braking energy generated by the joint motor during the braking phase is recovered and utilized. The partial energy recovery mode means that x% of the braking energy generated by the joint motor during the braking phase is recovered and utilized, and the remaining (1-x) braking energy is converted into heat energy and dissipated, where x is the percentage of energy recovered. The zero energy recovery mode means that all the braking energy generated by the joint motor during the braking phase is converted into heat energy and dissipated.

[0066] For the first joint motor, its corresponding energy consumption model is the fully recovered energy consumption model, and the expression for the fully recovered energy consumption model is:

[0067] P_j1(t) = τ_j1(t) × ω_j1(t), formula (1);

[0068] Where j1 represents the first joint motor, P_j1(t) represents the power consumption of the first joint motor at time t, τ_j1(t) represents the torque of the first joint motor at time t, and ω_j1(t) represents the angular velocity of the first joint motor at time t.

[0069] The braking energy generated by the first joint motor during the braking phase can be 100% recovered and returned to the battery of the power system. Specifically, when the output torque direction of the first joint motor is consistent with the angular velocity direction of the joint, the first joint motor draws energy from the battery, and the instantaneous power consumption P_j1(t) is positive. When the output torque direction of the first joint motor is opposite to the angular velocity direction of the joint, the first joint motor acts as a generator, converting all the braking energy into electrical energy and recovering it to the battery, and the instantaneous power consumption P_j1(t) is negative. That is, a negative power consumption value indicates that net energy is fed back to the battery.

[0070] For the second joint motor, its corresponding energy consumption model is a partially recovered energy consumption model, and the expression for the partially recovered energy consumption model is:

[0071] , formula (2);

[0072] Where j2 represents the second joint motor, P_j2(t) represents the power consumption of the second joint motor at time t, τ_j2(t) represents the torque of the second joint motor at time t, ω_j2(t) represents the angular velocity of the second joint motor at time t, and x represents the percentage of energy recovery, x is less than or equal to 1;

[0073] Part of the braking energy generated by the second joint motor during the braking phase can be recovered into the battery of the power system. Specifically, when the output torque direction of the second joint motor is consistent with the angular velocity direction of the joint, the second joint motor draws energy from the battery, and the instantaneous power consumption P_j2(t) is positive. When the output torque direction of the second joint motor is opposite to the angular velocity direction of the joint, When the value is negative, the second joint motor acts as a generator, converting some of the braking energy into electrical energy and recovering it to the battery. The remaining braking energy is converted into heat and dissipated. For example, assuming x is 70%, the power consumption of the second joint motor at time t is... 100W (i.e.) =-100W), then the net instantaneous power consumption P_j2(t) is 30W.

[0074] The energy consumption model corresponding to the third joint motor is the zero-recovery energy consumption model, and the expression for the zero-recovery energy consumption model is:

[0075] P_j3(t) = |τ_j3(t)×ω_j3(t)|, Formula (3);

[0076] Where j3 represents the third joint motor, P_j3(t) represents the power consumption of the third joint motor at time t, τ_j3(t) represents the torque of the third joint motor at time t, and ω_j1(t) represents the angular velocity of the third joint motor at time t.

[0077] The braking energy of the third joint motor is entirely dissipated as heat. During braking, braking energy is lost as heat.

[0078] It is understandable that the power consumption of the second and third joint motors is positive during braking. This means that the second and third joint motors will generate positive power consumption regardless of whether they are in driving or braking state. In this way, the control strategy network can avoid braking or frequent switching between positive and negative torque as much as possible to reduce power consumption.

[0079] It is understandable that a robot model can include multiple joint motors. Each joint motor can be one of three types: a first joint motor, a second joint motor, or a third joint motor. That is, each joint motor in the robot model can be independently configured with one of these three energy recovery modes, allowing different joint motors in the same robot model to have different energy recovery characteristics. During reinforcement learning training, parameters such as the energy recovery mode and percentage of energy recovery for each joint motor can be randomized. This ensures that the final trained control strategy network does not depend on any specific energy recovery mode of any particular joint motor. Therefore, even if a joint motor of the target robot malfunctions or its energy recovery efficiency decreases during actual application of the control strategy, the trained control strategy network can still achieve near-optimal energy efficiency by adaptively adjusting gait and joint control parameters.

[0080] In this embodiment of the application, after constructing the corresponding energy consumption model for each joint motor, a reward function can be designed based on the corresponding energy consumption model for each joint motor.

[0081] To ensure that the trained control policy network can simultaneously achieve speed tracking, attitude stabilization, chassis stability, and energy efficiency, the pre-defined reward function includes at least a first reward term, a second reward term, a third reward term, and a fourth reward term. For example, Figure 3 A structural diagram of a preset reward function is shown, for reference. Figure 3 The preset reward function can include a first reward item, a second reward item, a third reward item, a fourth reward item, and a fifth reward item. The first reward item is the motion target reward component, used to enable the robot model to track speed and guide its forward direction. The second reward item is the balance stability reward component, used to introduce a non-horizontal posture penalty to maintain the robot model's posture stability. The third reward item is the center of gravity stability penalty component, which explicitly penalizes the vertical offset of the chassis center of gravity relative to the reference height and the vertical velocity to suppress chassis vertical fluctuations during robot movement. The fourth reward item is the average power consumption optimization component, which calculates the instantaneous power consumption and / or average power consumption of each joint motor based on the heterogeneous energy consumption model established above, and incorporates it as a negative reward into the optimization objective. The fifth reward item is a regularization term, including joint acceleration penalties and joint torque smoothing penalties.

[0082] For example, the preset reward function can be calculated using the following formula:

[0083] R = R_motion + R_balance + R_cog + R_power + R_regularization, formula (4);

[0084] Where R is the preset reward function, R_motion is the first reward, R_balance is the second reward, R_cog is the third reward, R_power is the fourth reward, and R_regularization is the fifth reward.

[0085] In the preset reward function, the first reward item is determined based on the speed deviation data of the robot model during the movement process.

[0086] The first reward item can be calculated using the following formula:

[0087] R_motion = w1 × exp(-||v_xy - v_target||^2), formula (5);

[0088] Where R_motion represents the first reward value, w1 represents the first weight parameter, v_xy represents the robot model's motion speed (including speed magnitude and direction), and v_target represents the robot model's target motion speed (including speed magnitude and direction). Based on v_xy and v_target, the robot model's speed deviation data during motion can be calculated.

[0089] By setting a motion target reward component R_motion in the preset reward function, the robot model can be guided to move according to the target velocity magnitude and direction. When the robot's actual velocity perfectly matches the target velocity in both magnitude and direction, the reward approaches 1. Conversely, any deviation will cause the first reward term to decay exponentially, thus forcing the control policy network to learn accurate velocity tracking capabilities.

[0090] The second reward is determined based on the robot model's attitude data during motion. The attitude data can include the chassis's roll angle and pitch angle. The second reward is calculated using the following formula:

[0091] R_balance = -w2×(roll^2 + pitch^2), formula (6);

[0092] Where R_balance represents the second reward value, w2 represents the second weight parameter, roll represents the chassis roll angle, and pitch represents the chassis pitch angle.

[0093] By setting a balance stability reward component R_balance in the preset reward function, the robot model's posture stability can be maintained, preventing tipping over. In this way, the robot model's chassis can always remain level, and angular velocity is suppressed, thereby avoiding phenomena such as tipping over or hitting the chassis, and providing a stable platform for the mounted precision payloads (such as robotic arms and high-precision cameras).

[0094] The third reward item is determined based on the chassis height offset data of the robot model during movement. The chassis height offset data includes the chassis height offset amount and the velocity of the chassis's center of gravity in the height direction. The third reward item is calculated according to the following formula:

[0095] R_cog = -w3×|h(t)- h_ref|^2 - w4×|v_z(t)|^2,Formula (7);

[0096] In this context, R_cog represents the third reward value, w3 represents the third weight parameter, h(t) represents the current center of gravity height of the chassis, h_ref represents the reference height, which can be a fixed value (e.g., 0.6m) or a value dynamically adjusted according to terrain changes. h(t) and h_ref indicate that the reference height is in the same world coordinate system. The chassis height offset of the robot model can be calculated based on h(t) and h_ref. v_z(t) represents the velocity of the chassis's center of gravity in the height direction, v_z(t) is used to suppress the intensity of the chassis's vertical movement, and w4 represents the fourth weight data.

[0097] In traditional robot motion control methods, only posture balance (avoiding tipping over) is typically considered, allowing for changes in chassis height. However, for heavy-duty robots, every 1 cm vertical rise or fall of the chassis can mean an additional consumption of hundreds of watts of power to lift hundreds of kilograms of weight. Therefore, this embodiment uses chassis center of gravity height stability as an explicit penalty component. In formula (7), the vertical height deviation of the chassis center of gravity is considered... and vertical velocity As a penalty, this forces the control policy network to learn that when the terrain is uneven, it can adapt to the terrain by extending and retracting the support legs, while keeping the chassis height and horizontal attitude almost unchanged, thereby significantly reducing the work done in lifting and lowering heavy objects.

[0098] The fourth reward item is determined based on the energy consumption model corresponding to each joint motor, and is calculated according to the following formula:

[0099] R_power = -w5×P_avg, formula (8);

[0100] Where R_power represents the fourth reward value, w5 represents the fifth weight parameter, and P_avg represents the average power consumption of the robot model in a complete gait cycle. P_avg is determined based on the energy consumption model corresponding to each joint motor.

[0101] P_avg can be calculated using the following formula:

[0102] P_avg = (1 / T) × ∫P_total(t)dt, formula (9);

[0103] Where P_total(t) represents the total instantaneous power consumption of all joint motors in the robot model at time t, i.e., P_total(t) = ΣP_ji(t), and the value of i is 1, 2, 3.

[0104] According to formula (8), if P_avg is positive, it indicates net energy consumption, and a penalty is applied. If P_avg is negative, it indicates net energy recovery, and this component is negative, which is equivalent to providing a positive reward to encourage energy recovery behavior. In this way, the control strategy network tends to choose the gait with the highest energy efficiency. At the same time, the trained control strategy network can truly reflect the hardware energy consumption, narrowing the gap between simulation and actual energy consumption prediction, and making the energy-saving strategy more practical. For example, it enables the control strategy network to learn to actively adjust joint angles and current during downhill or deceleration, so that the motor enters the efficient power generation range, maximizing energy recovery, and enables the control strategy network to learn to reduce unnecessary braking losses.

[0105] In some embodiments, wheeled locomotion units are also installed at the ends of each supporting leg in the robot model. The wheeled locomotion unit may include wheels and hub motors that drive the wheels to rotate. This configuration of the robot model can exhibit various locomotion modes on different terrains, including wheeled locomotion, legged locomotion, and hybrid wheel-legged locomotion. For this type of robot model, during reinforcement learning, the initial control policy network outputs control parameters for each joint motor, as well as control parameters for each hub motor. On flat terrain, the control policy network tends to use wheeled locomotion to reduce energy consumption. On rough terrain, the control policy network switches to legged locomotion and utilizes a chassis center-of-gravity stabilization strategy to reduce energy loss. Furthermore, to ensure the coordination of motion between the hub motors of each wheeled locomotion unit and the joint motors in each supporting leg, a mode switching smoothness reward (e.g., maintaining relative stability in robot speed, motor output current, etc.) and wheel-ground contact force optimization terms (e.g., stable wheel-ground contact force, no impact, etc.) can be added to the preset reward function.

[0106] After completing the simulation environment construction, energy consumption model establishment, and reward function design, the initial control policy network can be trained in the simulation environment using a preset reinforcement learning algorithm.

[0107] In this embodiment, the initial control strategy network is used to output control parameters for each joint motor based on the environmental state information of the robot model in the simulation environment. The environmental state information includes at least terrain perception information for various terrain types. For example, the initial control strategy network can employ a multilayer perceptron (MLP), whose input is the body perception information of the robot model, and whose output is the target position of each joint. For example, when the robot model is a hexapod, the body perception information input to the initial control strategy network can include chassis posture, angular velocity, joint angles and angular velocities, and the previous moment's action, and the output is the target position commands for 18 joints. The control parameters for each joint motor can be determined based on the target position of each joint.

[0108] The preset reinforcement learning algorithm can be trained using proximal policy optimization (PPO). PPO is a deep reinforcement learning algorithm based on policy gradients, widely used in robot control due to its high training stability, strong hyperparameter robustness, and relatively good sampling efficiency. Its core mechanism uses a pruned surrogate objective function to limit the difference between the old and new policies at each policy update, thereby avoiding destructive parameter updates and ensuring monotonically convergent training.

[0109] During training, different terrain conditions and external disturbances can be applied to the robot model, enabling the control strategy to learn gait and joint control methods that maintain chassis center of gravity stability and minimize average power consumption under various operating conditions. To further improve the generalization ability of the strategy and accelerate training convergence, a curriculum learning strategy can be adopted during reinforcement learning. For example, basic motion capabilities and center of gravity stability can be trained first on flat terrain, and then complex terrain conditions such as slopes, gravel, and steps can be gradually introduced to improve the generalization performance of the strategy on diverse terrains.

[0110] For example, the training process can be divided into three stages, with a total training step count of approximately 100 million steps. The first stage trains basic gait and center of gravity stability on flat terrain, with approximately 20 million steps. In this stage, a flat, hard surface can be used in the simulation environment. The goal is for the initial control policy network to learn basic motor skills such as moving from a standstill, forward, backward, and turning in place, and to initially develop a stable gait. The second stage introduces moderately complex terrain (such as slopes), forcing the policy to learn the ability to maintain a level chassis on uneven terrain, for example, by adjusting the support length of different legs to compensate for the slope. This stage involves approximately 30 million steps. The third stage introduces fully complex terrain (such as gravel roads, discrete steps, random potholes, etc.), with approximately 50 million steps. In this stage, the control policy network not only needs to select safe foot placement points to avoid obstacles but also needs to plan smooth leg trajectories to reduce energy loss due to frequent acceleration / deceleration and collision impacts.

[0111] In some embodiments, the third stage may further incorporate external disturbances to enhance the robustness of the strategy. Specifically, while keeping the terrain unchanged, random external disturbances are applied to the robot's state. These disturbances can take the form of thrust disturbances or parameter perturbations. Thrust disturbances can involve applying a random pulse force of random direction and magnitude at the chassis center of gravity, with a force range of 50. N to 300 N, with a duration of 0.1 to 0.5 seconds. Parameter perturbations can be small, random changes to simulation parameters, such as ±30% change in joint friction, ±20% change in plantar friction, ±10% change in maximum motor torque, and ±15% change in load mass. These perturbations provide a robust foundational strategy for subsequent simulation-to-real (sim-to-real) transfer.

[0112] It is understandable that control policy networks trained in simulation environments often perform poorly or even fail completely when directly deployed on target robots due to inherent differences between simulation and the real world in terms of physical parameters, perceptual noise, and actuator latency (i.e., the reality gap). Therefore, to improve the generalization ability of control policy networks, domain randomization techniques can be introduced during reinforcement learning. This allows the policy to experience a wider range and more extreme dynamic changes in simulation than in the real world, thus exhibiting stronger robustness in the face of uncertainties in real-world environments.

[0113] The parameters involved in domain randomization can include at least one of the following: ground friction coefficient, chassis mass, joint friction, sensor noise in the robot model, etc. For example, the ground friction coefficient can be randomized within the range of 0.5 to 1.2, the chassis mass can be randomly varied by ±10%, and the joint friction can be randomly varied by ±20%. In this way, the control policy network continuously adapts to the changing system dynamics during training, thereby reducing overfitting to specific parameter values ​​and improving the transfer effect from simulation to real environment (Sim2Real).

[0114] S103: Deploy the target control strategy network onto the target robot using the simulation-to-real migration method. The target control strategy network is used to output the control parameters of each joint motor in the target robot based on the environmental state information of the target robot in the real environment when the target robot is running.

[0115] After obtaining the target control policy network, the finally trained target control policy network (including network weights, network structure, and normalized parameters) can be exported in a preset format, such as the Open Neural Network Exchange (ONNX) format. Subsequently, the exported target control policy network is deployed to the embedded controller of the target robot, and the runtime loop frequency can be controlled at 500Hz. After migration and deployment, the target robot can perform motion control in the real environment based on the target control policy network learned in the simulation environment.

[0116] It is understandable that, since simulators cannot fully simulate all the physical details of the real world (such as cable dragging, unstructured ground deformation, actual sensor noise distribution, and minor actuator delays), small but perceptible performance gaps may still exist after the target control strategy network is deployed. Therefore, after migration, a small amount of fine-tuning in the real environment can be performed to compensate for the residual differences between simulation and reality. Fine-tuning can employ low-cost adaptive methods, such as recalibrating the normalized parameters of the control strategy network based on a small number of real-world trajectories, or performing a small number of gradient updates on the network weights using real-world data collected online (keeping other frozen parameters unchanged). Through this type of fine-tuning, the target control strategy network can better adapt to the dynamic characteristics of the target robot, bringing motion stability and energy consumption efficiency closer to the optimal levels in the simulation.

[0117] Ultimately, through the Sim2Real migration process and fine-tuning steps, the target control strategy network trained in the simulation environment, which can maintain a high degree of stability of the chassis center of gravity and has high energy efficiency on complex terrain, can be transferred to the real target robot, enabling it to truly perform heavy-load tasks in varied and harsh environments such as gravel, sand, mud, and stairs, while maintaining low energy consumption.

[0118] Figure 4 This paper presents a framework for training and deploying robot control policies based on Sim2Real transfer learning, referencing... Figure 4 The Sim2Real migration method can be divided into two main stages: simulation training and real deployment.

[0119] refer to Figure 4 As shown, during the simulation training phase, physical simulators such as Isaac Gym and MuJoCo can be used to construct a simulation environment, providing high-fidelity and repeatable dynamics and contact interaction simulations for the robot model, serving as the basis for generating training data. Domain randomization is introduced during reinforcement learning to improve the generalization ability and robustness of the policy by perturbing parameters such as robot model quality, environmental contact friction coefficient, and sensor observation noise, thus covering uncertainties in the real world. A progressive learning mechanism is employed, gradually increasing terrain complexity in the order of flat terrain, moderately rugged terrain, and complex and varied terrain, guiding the policy to stable convergence and avoiding failure due to excessively difficult tasks in the early stages of training. The reinforcement learning algorithm uses proximal policy optimization and is trained using a pre-set reward function, which includes a center-of-gravity stability term and an energy consumption model based on three heterogeneous energy recovery modes (full recovery, partial recovery, and zero recovery). The initial control policy network can be implemented using a multilayer perceptron, which receives the observed state of the simulation environment (body perception information, etc.) and outputs the control parameters of each joint motor. After approximately 100 million optimization steps, training is completed, resulting in the target control policy network.

[0120] After obtaining the target control policy network, the Sim2Real method is used to migrate and deploy it to a real target robot. During migration, the converged MLP control policy network is first converted into the ONNX general format to decouple it from the training framework and ensure the model's portability. Then, the standardized model is deployed to the target embedded hardware through the migration process.

[0121] During the actual deployment phase, the target control policy network runs on an embedded controller (which can be based on ARM or x86 architecture). The loaded ONNX format target control policy network performs forward inference on the controller, generating control commands at a control frequency of 500Hz to ensure timely response. The target control policy network outputs control parameters (e.g., target position, torque) for each joint motor in the target robot, while sensors on the target robot feed their status information back to the controller, forming a control closed loop. After the control policy network is deployed, the policy can be fine-tuned using a small amount of real-world interaction data to further adapt to the dynamic characteristics of specific physical hardware, thereby improving the final performance.

[0122] The robot motion control method provided in this application is particularly suitable for hexapod robots. A hexapod robot has at least three legs in a supporting state at any given time. Since three points can define a plane, the position and orientation of the chassis remain unchanged during gait transitions. This characteristic makes it highly promising for hexapod robots to achieve center-of-gravity stability through optimized control strategies when the chassis weight constitutes a major proportion. Figure 5 A schematic diagram of the gait of a hexapod robot is shown. (Reference) Figure 5 The six legs of the hexapod robot employ an alternating tripedal support pattern during movement, meaning that at any given time, at least three legs are in a supporting state. Utilizing the geometric principle that three points define a plane, the chassis maintains a constant horizontal orientation and height during the support phase switching. Horizontal orientation means that the chassis is approximately parallel to the ground; for example, the angle between the chassis and the ground is less than a preset angle, such as less than 10°. For instance, support legs 1, 3, and 5, along with support legs 2, 4, and 6, can each form a supporting triangle, and the projection of the chassis's center of gravity always lies within this triangle.

[0123] Figure 6 The figure illustrates the change in chassis height of a hexapod robot during movement. The upper figure shows the change in chassis center of gravity when using the robot motion control method provided in this application, while the lower figure shows the change in chassis center of gravity without constrained chassis stability. (Reference) Figure 6 The robot motion control method provided in this application fully utilizes the structural advantages of a hexapod robot with alternating tripod support, achieving approximately constant chassis position and posture during movement. Using this method, the length of each supporting leg of the hexapod robot can be dynamically adjusted according to ground undulations, maintaining a stable chassis center of gravity. Without this center of gravity stability constraint, the chassis would fluctuate up and down with ground undulations. Compared to the chassis's movement following terrain undulations, this method can save an additional 30% to 70% of energy (the specific savings depend on the complexity of the terrain and the chassis's mass percentage).

[0124] Understandably, this method is also applicable to quadruped and bipedal robots. For both quadruped and bipedal robots, the chassis center-of-gravity stability penalty term can also guide the reinforcement learning strategy to learn a more stable gait. For example, the target robot could be a robot with a chassis mass of 80 kg and a total mass of 110 kg. A quadruped robot weighing kg was developed. By adding a chassis center-of-gravity height stability penalty to the reward function, the reinforcement learning policy was still able to learn a gait that minimizes chassis height fluctuations. Experiments show that in moderately complex terrain, adding this penalty can achieve an energy consumption reduction of approximately 15%–25% compared to the baseline policy without the center-of-gravity stability penalty.

[0125] Figure 7A flowchart of another robot motion control method provided in an embodiment of this application is shown. (Reference) Figure 7 This method may include processes such as building a physical simulation environment, establishing an energy consumption model, designing a reward function, reinforcement learning training, Sim2Real migration deployment, and actual operation.

[0126] When constructing a physical simulation environment, various complex terrains (such as gravel, ramps, steps, etc.) are created within the simulation environment to provide diverse challenging scenarios for robot training. The simulation environment enables safe and efficient large-scale trial-and-error training.

[0127] By establishing an energy consumption model, energy consumption models corresponding to three energy recovery modes are designed to simulate the energy consumption of motors at different joints in the target robot under different motion states, providing a foundation for energy-saving control. When designing the reward function, a center-of-gravity stability penalty term and a power consumption optimization term are added to guide the robot to maintain chassis center-of-gravity stability during movement and to adopt energy-saving motion methods.

[0128] In the reinforcement learning training process, a proximal policy optimization algorithm is used for training, and a curriculum learning strategy is introduced. Training begins on flat terrain, then transitions to moderately complex terrain, and finally completes training on fully complex terrain. This approach allows the robot to gradually improve its capabilities, avoiding facing overly difficult tasks from the outset. Simultaneously, domain randomization is introduced during reinforcement learning to increase the randomness of the simulation environment, thereby enhancing the generalization ability of the policy.

[0129] After training, the obtained policy is transferred and deployed to the target robot using Sim2Real. During the transfer process, the target control policy network is converted to a standard format (e.g., ONNX format). After deployment to the target robot, the target control policy network operates at a preset frequency (e.g., 500). (Hz) to control the movement of the target robot and achieve high-frequency control response.

[0130] Through the above methods, the target robot can achieve stable chassis center of gravity and energy-saving control during actual operation, enabling the target robot to maintain balance and effectively save energy in real complex environments.

[0131] Figure 8 A block diagram of a robot motion control system is shown, with reference to... Figure 8The system can include a simulation training module, an energy modeling module, a reward calculation module, a control strategy network module, a Sim2Real transfer module, and a target robot. These modules are coupled through data and command flows to form an integrated "perception-decision-execution" and "training-optimization-transfer" system, with the core data flow forming a complete closed loop from simulation perception to physical robot control. The core modules have clear divisions of labor and work collaboratively, achieving a closed-loop process from virtual strategy training and optimization to real-time deployment of the physical robot.

[0132] refer to Figure 8 As shown, the simulation training module is mainly used for training and data generation. It receives terrain parameters and robot dynamics models, constructs a high-fidelity multi-terrain simulation environment, simulates robot dynamics and sensor characteristics, outputs body perception information, and provides the interactive data required for training to the core decision-making unit. The control strategy network module, as the core decision-making unit of the robot model, is constructed using a multilayer perceptron. It receives body perception information from the simulation or real environment and outputs joint control commands (such as target position, target torque, or impedance parameters) in real time through forward propagation, achieving precise decision-making for robot motion. The energy modeling module establishes configurable energy recovery models for each driven joint, simulates energy consumption and recovery characteristics, and calculates real-time power consumption, providing data for reward calculation. The reward calculation module calculates multi-target reward signals by combining motion error, stability, energy consumption, and other states. A newly added center of gravity stability penalty term ensures the robot's dynamic balance, and finally, the reward value is fed back to the trainer to drive continuous iterative optimization of the control strategy network. The Sim2Real migration module acts as a crucial bridge between the virtual and physical environments. It enhances the generalization ability of policies to real-world environments through domain randomization technology, while converting the trained policy model into ONNX format for final deployment to an embedded controller supporting ARM or x86 architectures. The target robot, as the system's final execution terminal, uses a 500-series embedded controller. The ONNX model runs at a frequency of Hz and combines sensor feedback to form a complete perception-control closed loop, ensuring efficient implementation of the strategy.

[0133] Based on the above system, a robot motion control process of "offline training and online deployment" can be realized. In the offline phase, policy training is completed using a simulation environment and multi-objective optimization signals, and the robustness of the policy is improved through relevant technologies. In the migration phase, the matured policy is exported and deployed to the embedded controller, completing the transition from virtual to physical. In the online phase, the deployed policy receives real sensor information, issues control commands in real time, and drives the robot to efficiently complete the predetermined task.

[0134] In summary, this application provides a robot motion control method that can significantly reduce energy consumption of heavy-duty multi-legged robots in uneven terrain. By using chassis center-of-gravity stability as an explicit optimization objective, energy loss due to chassis lifting and lowering can be reduced by 30% to 70% in rugged terrain. By constructing a reward function based on heterogeneous differentiated energy consumption models, the trained target control strategy network can be adapted to the actual energy characteristics of heterogeneous motors in real hardware systems, reducing energy consumption prediction errors after Sim2Real migration. Furthermore, this method is applicable to both purely legged and wheel-legged hybrid locomotion modes, demonstrating broad applicability. Moreover, the framework of this method can be extended to quadrupedal and bipedal robots, exhibiting good versatility.

[0135] This application provides a computer program product that, when executed on an electronic device, enables the electronic device to implement the robot motion control methods provided in the foregoing embodiments.

[0136] This application also provides a computer-readable storage medium storing one or more programs / instructions, which, when executed by an electronic device, enable the electronic device to implement the robot motion control method provided in the foregoing embodiments.

[0137] This application also provides an electronic device, which includes at least one memory and one or more processors. The memory is used to store one or more programs / instructions, and the at least one processor is used to execute one or more programs / instructions stored in the memory to enable the electronic device to implement the robot motion control method provided in the foregoing embodiments.

[0138] The robot motion control method provided in this application can be applied to any electronic device, including but not limited to mobile stations (MS) and mobile terminals (MT). For example, the electronic device can be a mobile phone, smart TV, wearable device, tablet computer, desktop computer, laptop computer, virtual reality (VR) device, augmented reality (AR) device, terminal in industrial control, terminal in self-driving, terminal in remote medical surgery, terminal in smart grid, terminal in transportation safety, terminal in smart city, terminal in smart home, etc. This application does not limit the specific form of the electronic device.

[0139] Figure 9 A block diagram of an electronic device 10 is shown. (Reference) Figure 9 The electronic device 10 may include one or more processors 102, a system control logic unit 101 connected to at least one of the processors 102, a system memory 105 connected to the system control logic unit 101, a memory 103 connected to the system control logic unit 101, and a network interface 107 connected to the system control logic unit 101.

[0140] It is understood that the structures illustrated in the embodiments of this application do not constitute a limitation on the only possible implementation of the electronic device 10. In other embodiments of this application, the electronic device 10 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0141] Processor 102 may include one or more single-core or multi-core processors. In some embodiments, processor 102 may include any combination of general-purpose processors and special-purpose processors (e.g., application processors, baseband processors, etc.). It is understood that in this embodiment, processor 102 may be configured to execute executable instructions 104 stored in memory 103 to implement the interaction method of this embodiment. When at least one instruction is executed in processor 102, electronic device 10 implements the interaction method of this embodiment.

[0142] System control logic unit 101 may include any suitable interface controller to provide any suitable interface to at least one of the processors 102 and / or any suitable device or component communicating with system control logic unit 101. System control logic unit 101 may include one or more memory controllers to provide an interface to system memory 105. System memory 105 may be used to load and store data and / or instructions. In some embodiments, system memory 105 of electronic device 10 may include any suitable volatile memory, such as suitable dynamic random access memory.

[0143] Memory 103 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, memory 103 may include any suitable volatile memory and / or any suitable non-volatile storage device, such as memory 103 may include random access memory (RAM) and / or cache memory cells, and may further include read-only memory (ROM).

[0144] The memory 103 may include a portion of the storage resources on the device on which the electronic device 10 is installed, or it may be accessible by the device, but is not necessarily part of the device. For example, the memory 103 may be accessed over a network via the network interface 107.

[0145] Specifically, system memory 105 and memory 103 may each include a temporary copy and a permanent copy of instruction 104. Instruction 104 may include, when executed by at least one of processors 102, causing electronic device 10 to implement the interaction method of embodiments of this application. In some embodiments, instruction 104, hardware, firmware, and / or its software components may additionally / alternatively be located in system control logic unit 101, network interface 107, and / or processor 102.

[0146] Network interface 107 may include a transceiver for providing a radio interface to electronic device 10, thereby enabling communication with any other suitable device (such as a front-end module, antenna, etc.) via one or more networks. In some embodiments, network interface 107 may be integrated into other components of electronic device 10. For example, network interface 107 may be integrated into at least one of processor 102, system memory 105, memory 103, and firmware device (not shown) with instructions.

[0147] Network interface 107 may further include any suitable hardware and / or firmware to provide a multiple-input multiple-output radio interface. For example, network interface 107 may be a network adapter, a wireless network adapter, a telephone modem, and / or a wireless modem.

[0148] The electronic device 10 may further include an input / output (I / O) device 106. The input / output device 106 enables a user to interact with the electronic device 10.

[0149] In some embodiments, the electronic device 10 may further include, but is not limited to, a display (e.g., a liquid crystal display, a touch screen display, etc.), a speaker, a microphone, one or more cameras (e.g., a still image camera and / or a video camera), a flashlight (e.g., a light-emitting diode flash) and a keyboard.

[0150] In some embodiments, the electronic device 10 further includes a sensor for determining at least one of environmental conditions or location information associated with the electronic device 10.

[0151] In some embodiments, the sensor may include, but is not limited to, a gyroscope sensor, an accelerometer, a proximity sensor, an ambient light sensor, and a positioning unit. The positioning unit may also be part of or interact with the network interface 107 to communicate with components of the positioning network, such as global positioning system (GPS) satellites.

[0152] The embodiments disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0153] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digit gate processor, a microcontroller, an application-specific integrated circuit, or a microprocessor.

[0154] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0155] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, optical discs, magneto-optical disks, ROM, RAM, magnetic cards or optical cards, or tangible machine-readable memories for transmitting information using electrical, optical, acoustic, or other forms of propagation signals via the Internet (e.g., carrier waves, infrared signal digit gating, etc.). Therefore, machine-readable media include any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0156] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, including structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0157] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0158] It should be noted that in the examples and description of this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0159] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made thereto without departing from the scope of this application.

Claims

1. A robot motion control method, characterized in that, include: In a simulation environment, a robot model corresponding to the target robot and various terrains are constructed. The various terrains include non-flat terrains. The robot model includes a chassis and multiple support legs that support the chassis. Each support leg is equipped with a joint motor at its joint. In the simulation environment, the robot model moves on various terrains based on the control parameters of each joint motor output by the initial control policy network. A preset reinforcement learning algorithm is used to iteratively train the initial control policy network according to a preset reward function to obtain the target control policy network; wherein: The initial control strategy network is used to output control parameters for each joint motor based on the environmental state information of the robot model in the simulation environment. The environmental state information includes at least the terrain perception information of the various terrains. The preset reward function includes a first reward item, a second reward item, a third reward item, and a fourth reward item. The first reward item is determined based on the speed deviation data of the robot model during the movement process. The second reward item is determined based on the posture data of the robot model during the movement process. The third reward item is determined based on the chassis height offset data of the robot model during the movement process. The fourth reward item is determined based on the energy consumption model corresponding to each joint motor. The energy consumption model includes a fully recovered energy consumption model corresponding to a joint motor in a fully recovered energy mode, a partially recovered energy consumption model corresponding to a joint motor in a partially recovered energy mode, or a zero recovered energy consumption model corresponding to a joint motor in a zero recovered energy mode. The target control strategy network is deployed on the target robot using a simulation-to-real migration method. The target control strategy network is used to output the control parameters of each joint motor in the target robot based on the environmental state information of the target robot in the real environment when the target robot is running.

2. The method according to claim 1, characterized in that, The control parameters of each joint motor are used to ensure that the robot model maintains the chassis in a horizontal position during movement and keeps the height change of the chassis less than a threshold.

3. The method according to claim 1 or 2, characterized in that, The first reward item is calculated according to the following formula: R_motion = w1×exp(-||v_xy - v_target||^2); Wherein, R_motion represents the first reward value, w1 represents the first weight parameter, v_xy represents the motion speed of the robot model, and v_target represents the target motion speed of the robot model; The attitude data includes the chassis roll angle and the chassis pitch angle, and the second reward item is calculated according to the following formula: R_balance = -w2×(roll^2 + pitch^2); Where R_balance represents the second reward value, w2 represents the second weight parameter, roll represents the roll angle of the chassis, and pitch represents the pitch angle of the chassis; The chassis height offset data includes the chassis height offset amount and the speed of the chassis's center of gravity in the height direction. The third reward item is calculated according to the following formula: R_cog = -w3×|h(t)-h_ref|^2 - w4×|v_z(t)|^2; Where R_cog represents the third reward value, w3 represents the third weight parameter, h(t) represents the center of gravity height of the chassis at the current moment, h_ref represents the reference height, h(t)-h_ref represents the chassis height offset, v_z(t) represents the speed of the center of gravity of the chassis in the height direction, and w4 represents the fourth weight data. The fourth reward item is calculated according to the following formula: R_power = -w5×P_avg; Wherein, R_power represents the fourth reward value, w5 represents the fifth weight parameter, and P_avg represents the average power consumption of the robot model in a complete gait cycle. P_avg is determined based on the energy consumption model corresponding to each joint motor.

4. The method according to claim 3, characterized in that, The joint motors in the multiple support legs include a first joint motor, a second joint motor, and a third joint motor. The energy consumption model corresponding to the first joint motor is the fully recovered energy consumption model, the energy consumption model corresponding to the second joint motor is the partially recovered energy consumption model, and the energy consumption model corresponding to the third joint motor is the zero recovered energy consumption model. The expression for the fully recovered energy consumption model is: P_j1(t)=τ_j1(t)×ω_j1(t); Where j1 represents the first joint motor, P_j1(t) represents the power consumption of the first joint motor at time t, τ_j1(t) represents the torque of the first joint motor at time t, and ω_j1(t) represents the angular velocity of the first joint motor at time t. The expression for the partially recovered energy consumption model is as follows: ； Where j2 represents the second joint motor, P_j2(t) represents the power consumption of the second joint motor at time t, τ_j2(t) represents the torque of the second joint motor at time t, ω_j2(t) represents the angular velocity of the second joint motor at time t, and x represents the percentage of energy recovery, x is less than or equal to 1; The expression for the zero-recovery energy consumption model is: P_j3(t)= |τ_j3(t)×ω_j3(t)|; Where j3 represents the third joint motor, P_j3(t) represents the power consumption of the third joint motor at time t, τ_j3(t) represents the torque of the third joint motor at time t, and ω_j1(t) represents the angular velocity of the third joint motor at time t.

5. The method according to claim 1, characterized in that, The chassis mass accounts for more than 50% of the total mass of the robot model, and the various terrains include at least one of flat ground, slopes, undulating gravel ground, and steps.

6. The method according to claim 1 or 5, characterized in that, The robot model includes six supporting legs. When the robot model moves, at least three of the supporting legs are in contact with the ground, and the contact points of at least three of the supporting legs with the ground are not on the same straight line.

7. The method according to claim 1 or 5, characterized in that, The supporting leg also includes a wheeled walking unit, and the walking modes of the robot model on the various terrains include wheeled walking mode, legged walking mode and wheeled-legged hybrid walking mode; The preset reward function also includes a fifth reward item and a sixth reward item. The fifth reward item is determined based on the mode switching smoothness index, and the sixth reward item is determined based on the contact force between the wheeled walking unit and the ground. The mode switching smoothness index is used to characterize the smoothness of switching between different walking modes.

8. The method according to claim 1, characterized in that, The environmental state information of the robot model in the simulation environment includes the robot's own perception information, external disturbance information, and domain randomization information.

9. The method according to claim 8, characterized in that, The body perception information includes at least one of the following: the chassis roll angle, the chassis pitch angle, the chassis yaw angle, the chassis speed in three-dimensional space, the angle and angular velocity of each joint motor, and terrain perception information; The external interference information includes randomly applied external forces applied to the robot model; The domain randomization information includes randomization of at least one of the following parameters: ground friction coefficient, chassis mass, joint friction force, and sensor noise in the robot model.

10. The method according to claim 1, characterized in that, The preset reinforcement learning algorithm is a near-end policy optimization algorithm, and the initial control policy network is a multilayer perceptron.

11. An electronic device, characterized in that, include: A memory for storing instructions executed by one or more processors of the electronic device; A processor, when executing the instructions in the memory, causes the electronic device to perform the robot motion control method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the robot motion control method according to any one of claims 1 to 10.

13. A computer program product, characterized in that, When the computer program product is run on an electronic device, it enables the electronic device to implement the robot motion control method according to any one of claims 1 to 10.

Images