Robot training method and device, electronic equipment and storage medium

Abstract

The invention provides a robot training method and device, electronic equipment and a storage medium, and the method comprises the steps: constructing a training network which comprises a strategy output network and an evaluation network; a reward function of the training network is determined, and the reward function comprises a centroid speed tracking reward function, an upper limb trunk balance reward function and a gait guiding reward function; obtaining current posture information of the to-be-trained robot, and calculating a target reward according to the posture information; inputting the attitude information into a strategy output network to obtain an action strategy of the hydraulic artificial muscle; inputting the target reward into an evaluation network to obtain an evaluation result of the action strategy; and under the condition that the evaluation result is that the action strategy meets the preset driving requirement, the hydraulic artificial muscle is driven according to the action strategy. According to the invention, the training of the robot based on hydraulic artificial muscle driving can be realized, and the training difficulty is low.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and in particular to a robot training method, device, electronic device and storage medium. Background Technology

[0002] The reinforcement learning of existing humanoid robots is based on brushless motor drive. Since the joints only have one motor as the power input and there are no additional joint constraints, many different rewards are needed to guide the design of rewards, especially joint penalty rewards. These include joint torque penalties, joint velocity penalties, joint acceleration penalties, and motion smoothness penalties, as well as posture and motion guidance rewards, in order to enable the humanoid robot to walk.

[0003] However, the reward design and network design of traditional electric robots cannot be directly transferred to humanoid robots driven by hydraulic artificial muscles. Therefore, a new robot training method is urgently needed to solve the above problems. Summary of the Invention

[0004] In view of this, this application provides a robot training method, apparatus, electronic device, and computer-readable storage medium, which can realize the training of robots based on hydraulic artificial muscle drive, and the training difficulty is low.

[0005] A first aspect of this application provides a robot training method, wherein the robot is driven by a hydraulic artificial muscle. The method includes: constructing a training network, wherein the training network includes a policy output network and an evaluation network, the policy output network being used to output the motion policy of the hydraulic artificial muscle, and the evaluation network being used to detect whether the motion policy meets preset driving requirements; determining a reward function of the training network, wherein the reward function includes a center of mass velocity tracking reward function, an upper limb trunk balance reward function, and a gait guidance reward function; acquiring the current posture information of the robot to be trained, and calculating the center of mass velocity tracking reward corresponding to the center of mass velocity tracking reward function, the upper limb trunk balance reward corresponding to the upper limb trunk balance reward function, and the gait guidance reward corresponding to the gait guidance reward function based on the posture information; inputting the posture information into the policy output network to obtain the motion policy of the hydraulic artificial muscle; inputting the center of mass velocity tracking reward, the upper limb trunk balance reward, and the gait guidance reward into the evaluation network to obtain an evaluation result of the motion policy; and driving the hydraulic artificial muscle according to the motion policy if the evaluation result indicates that the motion policy meets the preset driving requirements.

[0006] In one possible implementation, the training network is built on a simulation platform that provides a Markov decision process environment, the Markov decision process environment including an action space; the robot includes a muscle actuator for driving the hydraulic artificial muscle; after building the training network, the method further includes: determining an activation layer of the training network, wherein the activation layer is used to limit the mean value of the action output in the action space to between (-1, 1) and to limit the control range of the muscle actuator to between (0, 1).

[0007] In one possible implementation, determining the activation layers of the training network includes: determining the activation layers according to the following formula: action_mean=(actuator_high+actuator_low) / 2; action_delta=(actuator_high-actuator_low) / 2; Where, action_mean is the mean of the action, actuator_high is the maximum control value of the muscle actuator, actuator_low is the minimum control value of the muscle actuator, and action_delta is the action difference. This is the activation value of the muscle driver.

[0008] In one possible implementation, the Markov decision process environment further includes an observation space; the method further includes: adding the characteristic expressions of the robot's left leg sinusoidal signal and right leg sinusoidal signal to the observation space; inputting the posture information into the policy output network to obtain the motion policy of the hydraulic artificial muscle includes: calculating the robot's current left leg sinusoidal signal based on the posture information and the characteristic expression of the left leg sinusoidal signal, and calculating the robot's current right leg sinusoidal signal based on the posture information and the characteristic expression of the right leg sinusoidal signal, wherein the phase difference between the left leg sinusoidal signal and the right leg sinusoidal signal is 90°; inputting the historical motion policy output by the policy output network in the previous step, the posture information, the left leg sinusoidal signal, and the right leg sinusoidal signal into the policy output network to obtain the motion policy.

[0009] In one possible implementation, the characteristic expression of the left leg sinusoidal signal includes: sin_pos = backend.sin(2 × backend.pi × phase); where sin_pos is the left leg sinusoidal signal, backend.pi is pi, and phase is the current gait phase value; the characteristic expression of the right leg sinusoidal signal includes: cos_pos = backend.cos(2 × backend.pi × phase); where cos_pos is the right leg sinusoidal signal.

[0010] In one possible implementation, the posture information includes the robot's center-of-mass velocity and the Euler angles of the torso posture; the center-of-mass velocity tracking reward is calculated according to the following formula: tracking_reward = tracking_reward_scale × exp(-k × (vel_local - goal_vel)^2); where tracking_reward is the center-of-mass velocity tracking reward, tracking_reward_scale is the reward weight, exp is the exponent e, k is the coefficient, vel_local is the center-of-mass velocity, and goal_vel is the preset target center-of-mass velocity; the upper limb torso balance reward is calculated according to the following formula: roll_pitch_reward = rp_scale × exp(-4 × (euler[:2])^2); where roll_pitch_reward is the target upper limb torso balance reward, rp_scale is the balance reward weight, and euler is the Euler angles of the torso posture.

[0011] In one possible implementation, the posture information includes the rotation matrix of the robot's pelvis in the world coordinate system, the first posture rotation matrix of the lower leg in the world coordinate system, the second posture rotation matrix of the thigh in the world coordinate system, and the third posture rotation matrix of the foot in the world coordinate system; the gait guidance reward includes the gait guidance reward of the lower leg, the gait guidance reward of the thigh, and the gait guidance reward of the foot; before calculating the gait guidance reward, the method further includes: calculating the inverse matrix of the rotation matrix, and transforming the first posture rotation matrix, the second posture rotation matrix, and the third posture rotation matrix into the pelvic coordinate system based on the inverse matrix; determining the first xyz Euler angle of the lower leg based on the transformed first posture rotation matrix, determining the second xyz Euler angle of the thigh based on the transformed second posture rotation matrix, and determining the third xyz Euler angle of the foot based on the transformed third posture matrix; and calculating the gait guidance reward according to the following formula: hip_pose_error = des_hip_pos-hip_rpy; cyclic_hip_reward = cyclic_hip_scale × exp(-4 × hip_pose_error^2); knee_pose_error = des_knee_pos-knee_rpy; cyclic_knee_reward = cyclic_knee_scale × exp(-4 × knee_pose_error^2); ankle_pose_error = des_ankle_pos-ankle_rpy; cyclic_ankle_reward = cyclic_ankle_scale × exp(-4 × ankle_pose_error^2); where hip_pose_error is the posture error of the thigh, des_hip_pos is the desired posture of the thigh, hip_rpy is the second xyz Euler angle, and cyclic_hip_reward is the gait guidance reward function of the thigh; knee_pose_error is the posture error of the lower leg, des_knee_pos is the desired posture of the lower leg, knee_rpy is the first xyz Euler angle, and cyclic_knee_reward is the gait guidance reward function of the lower leg; ankle_pose_error is the posture error of the foot, des_ankle_pos is the desired posture of the foot, ankle_rpy is the third xyz Euler angle, and cyclic_ankle_reward is the gait guidance reward function of the foot.

[0012] Secondly, embodiments of this application also provide a robot training device, wherein the robot is driven by a hydraulic artificial muscle. The device includes: a construction module for constructing a training network, wherein the training network includes a policy output network and an evaluation network, the policy output network being used to output the action policy of the hydraulic artificial muscle, and the evaluation network being used to detect whether the action policy meets preset driving requirements; a determination module for determining the reward function of the training network, wherein the reward function includes a center of mass velocity tracking reward function, an upper limb trunk balance reward function, and a gait guidance reward function; and an acquisition module for acquiring the current posture information of the robot to be trained and, based on the posture information... The system calculates the center-of-mass velocity tracking reward corresponding to the center-of-mass velocity tracking reward function, the upper limb-trunk balance reward corresponding to the upper limb-trunk balance reward function, and the gait guidance reward corresponding to the gait guidance reward function; a first input module is used to input the posture information into the strategy output network to obtain the motion strategy of the hydraulic artificial muscle; a second input module is used to input the center-of-mass velocity tracking reward, the upper limb-trunk balance reward, and the gait guidance reward into the evaluation network to obtain the evaluation result of the motion strategy; and a driving module is used to drive the hydraulic artificial muscle according to the motion strategy when the evaluation result indicates that the motion strategy meets the preset driving requirements.

[0013] Thirdly, embodiments of this application also provide an electronic device, the electronic device including a processor and a memory, the memory being used to store instructions, and the processor being used to call the instructions in the memory, causing the electronic device to execute the robot training method as described in the first aspect.

[0014] Fourthly, embodiments of this application also provide a computer-readable storage medium that stores computer instructions that, when executed on an electronic device, cause the electronic device to perform the robot training method as described in the first aspect.

[0015] Compared with related technologies, the embodiments of this application have at least the following advantages: By determining the reward function of the training network, on the one hand, the reward function only includes the center of mass velocity tracking reward function, the upper limb trunk balance reward function, and the gait guidance reward function, that is, all joint penalty rewards are removed, thereby improving the training efficiency of the robot and reducing the training difficulty; on the other hand, after obtaining the current posture information of the robot to be trained, the center of mass velocity tracking reward corresponding to the center of mass velocity tracking reward function, the upper limb trunk balance reward corresponding to the upper limb trunk balance reward function, and the gait guidance reward corresponding to the gait guidance reward function can be calculated based on the posture information. Thus, the center of mass velocity tracking reward, the upper limb trunk balance reward, and the gait guidance reward can be input into the evaluation network to obtain the evaluation result of the action strategy output by the strategy output network. Then, it is possible to detect whether the action strategy meets the preset driving requirements based on the evaluation result. When the evaluation result indicates that the action strategy meets the preset driving requirements, the hydraulic artificial muscle is driven according to the action strategy, thereby realizing the coordinated movement of the robot driven by the hydraulic artificial muscle.

[0016] The technical effects achieved by the second, third, and fourth aspects mentioned above are similar to those achieved by the corresponding technical means in the first aspect, and will not be repeated here. Attached Figure Description

[0017] Figure 1 A flowchart illustrating the steps of a robot training method provided in an embodiment of this application; Figure 2 A graph of tanh provided in one embodiment of this application; Figure 3 A flowchart illustrating another step of the robot training method provided in one embodiment of this application; Figure 4 A functional block diagram of a robot training device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] To better understand the above-mentioned objectives, features, and advantages of this application, the application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0019] The following description sets forth many specific details to provide a full understanding of this application. The described embodiments are only some, not all, of the embodiments of this application.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0021] It should be further noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0022] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and drawings of this application are used to distinguish similar objects, not to describe a specific order or sequence.

[0023] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0024] For ease of understanding, some concepts related to the embodiments of this application are illustrated and explained by way of example for reference.

[0025] MuJoCo_mjx: MuJoCo is a high-performance physics engine focused on robotics, biomechanics, and machine learning. Its core design integrates innovative technologies of dynamics simulation and contact mechanics. MuJoCo_mjx is a JAX-accelerated version of the MuJoCo physics engine, designed specifically for computationally intensive tasks such as reinforcement learning, and provides GPU acceleration support. MuJoCo_mjx's core features include: achieving efficient parallel computing through the JAX framework, significantly improving the running speed of large-scale simulation scenarios; supporting direct policy training in the simulation environment, suitable for applications such as robot motion control and dynamic environment modeling.

[0026] Hydraulic artificial muscle: This is a soft actuator that uses hydraulic or pneumatic pressure to drive flexible materials to simulate the contraction of biological muscles. Its core principle is to use fluid pressure to deform an elastic structure and generate mechanical movement. It consists of an elastic inner liner (rubber / silicone) and a braided sleeve. After fluid is injected, the internal pressure increases, and the sleeve constrains the axial contraction and outputs tensile force. The deformation rate can reach 20%-30%.

[0027] Muscle actuators: These are biomimetic actuators that mimic the contraction and relaxation mechanisms of biological muscles. They are mainly divided into two major directions: biological muscle drive principles and artificial muscle technology. The core goal is to provide robots, medical devices, and other devices with compliant and efficient power output that is close to that of biological muscles.

[0028] Actor-Critic is an algorithmic framework in reinforcement learning that combines policy optimization and value function approximation. It consists of two parts: a policy network (Actor) and a value network (Critic). The Actor (policy network) outputs action decisions based on the current state and optimizes the policy using policy gradient methods. The Critic (value network) evaluates the value of the current policy, updates the value estimate through temporal difference learning, and guides policy adjustments. Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of an embodiment of the robot training method of this application. Depending on different requirements, the order of the steps in this flowchart can be changed, and some steps can be omitted.

[0029] It should be noted that the robot training method of this application embodiment can be applied to robot training scenarios, and its execution entity can be a robot training device. For example, in a robot training scenario, reinforcement learning training of the robot can be achieved through a robot training device. Of course, the robot training method can also be applied to other scenarios that require robot training, and this application does not specifically limit it. It should also be noted that the robot trained in this application is driven by hydraulic artificial muscles. Due to the high nonlinearity and large time delay of hydraulic artificial muscles, as well as the control coupling caused by the high parallel connection of multiple actuators, traditional numerical optimization-based methods are difficult to model and control the robot as a whole. Reinforcement learning can handle such high-dimensional multi-input multi-output systems without precisely considering the intermediate modeling process. Therefore, the robot training method of this embodiment is applied to a simulation platform that provides a Markov decision process environment.

[0030] The specific process of this embodiment is as follows: Figure 1 As shown, it includes the following steps: S101, Construct a training network, which includes a policy output network and an evaluation network. The policy output network is used to output the motion strategy of the hydraulic artificial muscle, and the evaluation network is used to detect whether the motion strategy meets the preset driving requirements.

[0031] In some embodiments, the training network is Actor-Critic, that is, the policy output network is Actor and the evaluation network is Critic. Both Actor and Critic use two-layer MLP (Multilayer Perceptron).

[0032] Specifically, Actor-Critic is built on a simulation platform that provides a Markov decision process environment. In this embodiment, the simulation platform is MuJoCo_mjx, which provides the Markov decision process environment, including the action space and the observation space.

[0033] In some embodiments, after constructing the training network, the method further includes: determining an activation layer of the training network, wherein the activation layer is used to limit the mean of the action space output to the range of (-1, 1) and to limit the control range of the muscle actuator to the range of (0, 1).

[0034] Specifically, the activation layer is determined according to the following formula: action_mean=(actuator_high+actuator_low) / 2; action_delta=(actuator_high-actuator_low) / 2; Where, action_mean is the mean of the action, actuator_high is the maximum control value of the muscle actuator, actuator_low is the minimum control value of the muscle actuator, and action_delta is the action difference. This is the activation value for the muscle driver.

[0035] Understandably, importing the above three formulas into MuJoCo_mjx allows for the determination of the activation layers in the training network.

[0036] To facilitate understanding, the following will be combined with... Figure 2 For a detailed explanation of how the activation layer is determined in this embodiment, please refer to: Figure 2 The graph shown is a tanh curve provided in the embodiments of this application.

[0037] In this embodiment, tanh() is selected as the activation layer of Actor-Critic, so that the mean value of the action output in the action space is between (-1 and 1). Since the action sampling conforms to the normal distribution, and the control range of the hydraulic artificial muscle actuator in this application is (0 to 1), and since the normal distribution has symmetry, the range of action output is limited to (-1 to 1). Therefore, it is necessary to map the action output range and the actuator control range, that is, to perform the mapping through the above three formulas.

[0038] S102, determine the reward function for training the network, where the reward function includes the center of mass velocity tracking reward function, the upper limb trunk balance reward function, and the gait guidance reward function.

[0039] S103, Obtain the current posture information of the robot to be trained.

[0040] S104, calculate the center of mass velocity tracking reward corresponding to the center of mass velocity tracking reward function, the upper limb trunk balance reward corresponding to the upper limb trunk balance reward function, and the gait guidance reward corresponding to the gait guidance reward function based on the posture information.

[0041] Specifically, regarding S102 to S104, since the robot's individual joints are driven by multiple hydraulic artificial muscles, providing sufficient joint constraints, penalties related to motor-driven joints can be removed, thus retaining the most basic center of mass velocity tracking rewards and upper limb trunk posture balance rewards.

[0042] In some embodiments, the centroid velocity tracking reward is calculated according to the following formula: tracking_reward=tracking_reward_scale×exp(-k×(vel_local-goal_vel)^2); Where tracking_reward is the centroid velocity tracking reward, tracking_reward_scale is the reward weight, exp is the exponent e, k is the coefficient, vel_local is the centroid velocity, and goal_vel is the preset centroid target velocity.

[0043] In some embodiments, the upper limb trunk balance reward is calculated according to the following formula: roll_pitch_reward=rp_scale×exp(-4×(euler[:2])^2); Where roll_pitch_reward is the target upper limb trunk balance reward, rp_scale is the weight of the balance reward, and euler is the Euler angle of the trunk posture.

[0044] In some embodiments, since the robot's hip joint mimics the human skeletal hip joint, the rotation axes of the three degrees of freedom of the hip joint are not parallel to any coordinate axis in the right-handed coordinate system. Traditional gait guidance methods based on joint angles are unsuitable in this embodiment. Traditional joint guidance assumes that the pitch rotation axis of the hip joint is parallel to the pitch rotation axes of the knee and ankle joints, and also parallel to the pelvic coordinate system. However, the robot in this embodiment is not parallel. If the hip joint angle is calculated based on the femur end posture, this inverse kinematics lacks an analytical expression, and the numerical solution has large errors and takes a long time to optimize, making it unsuitable for inclusion in the reward function of reinforcement learning. Therefore, this application proposes link posture guidance, directly obtaining the postures of the thigh link, lower leg link, and foot link in the pelvic coordinate system from the kinematic model as feedback for given guidance. By using the actual postures of key links as the basis for reward calculation, training efficiency, robustness, and reliability of the reward signal are improved, and training problems caused by instability (drift) or time consumption of inverse kinematics numerical solutions are avoided.

[0045] Specifically, the posture information includes the rotation matrix of the robot's pelvis in the world coordinate system, the first posture rotation matrix of the lower leg in the world coordinate system, the second posture rotation matrix of the thigh in the world coordinate system, and the third posture rotation matrix of the foot in the world coordinate system; the gait guidance reward includes the gait guidance reward of the lower leg, the gait guidance reward of the thigh, and the gait guidance reward of the foot; before calculating the gait guidance reward, the following steps are also included: calculating the inverse matrix of the rotation matrix, and transforming the first posture rotation matrix, the second posture rotation matrix, and the third posture rotation matrix into the pelvic coordinate system based on the inverse matrix; determining the first xyz Euler angle of the lower leg based on the transformed first posture rotation matrix, determining the second xyz Euler angle of the thigh based on the transformed second posture rotation matrix, and determining the third xyz Euler angle of the foot based on the transformed third posture matrix.

[0046] Calculate the gait guidance reward using the following formula: hip_pose_error = des_hip_pos-hip_rpy; cyclic_hip_reward = cyclic_hip_scale × exp(-4 × hip_pose_error^2); knee_pose_error = des_knee_pos-knee_rpy; cyclic_knee_reward = cyclic_knee_scale × exp(-4 × knee_pose_error^2); ankle_pose_error = des_ankle_pos-ankle_rpy; cyclic_ankle_reward = cyclic_ankle_scale × exp(-4 × ankle_pose_error^2); Wherein, hip_pose_error is the pose error of the thigh, des_hip_pos is the desired pose of the thigh, hip_rpy is the second xyz Euler angle, and cyclic_hip_reward is the gait guidance reward function for the thigh; knee_pose_error is the pose error of the lower leg, des_knee_pos is the desired pose of the lower leg, knee_rpy is the first xyz Euler angle, and cyclic_knee_reward is the gait guidance reward function for the lower leg; ankle_pose_error is the pose error of the foot, des_ankle_pos is the desired pose of the foot, ankle_rpy is the third xyz Euler angle, and cyclic_ankle_reward is the gait guidance reward function for the foot.

[0047] S105, input the posture information into the strategy output network to obtain the motion strategy of the hydraulic artificial muscle.

[0048] S106 inputs the center of mass velocity tracking reward, upper limb trunk balance reward, and gait guidance reward into the evaluation network to obtain the evaluation results of the movement strategy.

[0049] S107, if the evaluation result shows that the motion strategy meets the preset driving requirements, drive the hydraulic artificial muscle according to the motion strategy.

[0050] In some embodiments, after inputting the center of mass velocity tracking reward, upper limb trunk balance reward, and gait guidance reward into the evaluation network, the evaluation network scores the action strategy. If the score is greater than the preset score, it indicates that the action strategy meets the preset driving requirements.

[0051] In some embodiments, if the score of the action strategy is low, i.e., the action strategy does not meet the preset driving requirements, the evaluation network outputs adjustment parameters, and the strategy output network outputs a new action strategy based on the adjustment parameters, until the score of the final output action strategy is greater than the preset score. In this way, the robot's motion coordination based on hydraulic artificial muscle actuation is ensured.

[0052] Compared with related technologies, the embodiments of this application have at least the following advantages: By determining the reward function of the training network, on the one hand, the reward function only includes the center of mass velocity tracking reward function, the upper limb trunk balance reward function, and the gait guidance reward function, that is, all joint penalty rewards are removed, thereby improving the training efficiency of the robot and reducing the training difficulty; on the other hand, after obtaining the current posture information of the robot to be trained, the center of mass velocity tracking reward corresponding to the center of mass velocity tracking reward function, the upper limb trunk balance reward corresponding to the upper limb trunk balance reward function, and the gait guidance reward corresponding to the gait guidance reward function can be calculated based on the posture information. Thus, the center of mass velocity tracking reward, the upper limb trunk balance reward, and the gait guidance reward can be input into the evaluation network to obtain the evaluation result of the action strategy output by the strategy output network. Then, it is possible to detect whether the action strategy meets the preset driving requirements based on the evaluation result. When the evaluation result indicates that the action strategy meets the preset driving requirements, the hydraulic artificial muscle is driven according to the action strategy, thereby realizing the coordinated movement of the robot driven by the hydraulic artificial muscle.

[0053] This embodiment is a further improvement upon the aforementioned embodiment. The main improvement lies in the following: In this embodiment, the characteristic expressions of the robot's left leg sinusoidal signal and right leg sinusoidal signal are added to the observation space. Based on the historical action strategy, posture information, left leg sinusoidal signal, and right leg sinusoidal signal output from the previous policy output network, the action strategy is obtained. This approach further improves the quality of action decision-making, thereby further enhancing the robot's motion coordination.

[0054] The specific process of this embodiment is as follows: Figure 3 As shown, it includes the following steps: S301, Construct a training network, which includes a policy output network and an evaluation network. The policy output network is used to output the motion strategy of the hydraulic artificial muscle, and the evaluation network is used to detect whether the motion strategy meets the preset driving requirements.

[0055] S302, Determine the reward function for training the network, where the reward function includes the center of mass velocity tracking reward function, the upper limb trunk balance reward function, and the gait guidance reward function.

[0056] S303: Obtain the current posture information of the robot to be trained.

[0057] S304, calculate the center of mass velocity tracking reward corresponding to the center of mass velocity tracking reward function, the upper limb trunk balance reward corresponding to the upper limb trunk balance reward function, and the gait guidance reward corresponding to the gait guidance reward function based on the posture information.

[0058] S305, add the characteristic expressions of the robot's left leg sinusoidal signal and right leg sinusoidal signal to the observation space.

[0059] S306, calculate the robot's current left leg sinusoidal signal based on the posture information and the characteristic expression of the left leg sinusoidal signal, and calculate the robot's current right leg sinusoidal signal based on the posture information and the characteristic expression of the right leg sinusoidal signal.

[0060] Specifically, regarding S305 and S306, in order to train the hydraulic artificial muscle driven robot, the observation space needs to be designed reasonably. In addition to the basic joint angles and angular velocities, it is also necessary to observe the previous action output to form a complete Markov decision process, so that the robot can better observe and understand its current state. At the same time, in order to better learn to walk on two legs, sinusoidal signals of the left and right legs are added to the observation space, with a 90° phase difference between the sinusoidal signals of the left and right legs.

[0061] More specifically, the characteristic expression of the left leg sinusoidal signal includes: sin_pos = backend.sin(2 × backend.pi × phase); where sin_pos is the sinusoidal signal of the left leg, backend.pi is pi, and phase is the current gait phase value, ranging from 0 to 1. The characteristic expression for the sinusoidal signal of the right leg includes: cos_pos = backend.cos(2 × backend.pi × phase); where cos_pos is the sinusoidal signal of the right leg.

[0062] S307: Input the historical action policy, posture information, left leg sine signal, and right leg sine signal from the previous output of the policy output network into the policy output network to obtain the action policy.

[0063] S308 inputs the center of mass velocity tracking reward, upper limb trunk balance reward, and gait guidance reward into the evaluation network to obtain the evaluation results of the movement strategy.

[0064] S309, when the evaluation result shows that the motion strategy meets the preset driving requirements, the hydraulic artificial muscle is driven according to the motion strategy.

[0065] Compared with related technologies, the embodiments of this application have at least the following advantages: By determining the reward function of the training network, on the one hand, the reward function only includes the center of mass velocity tracking reward function, the upper limb trunk balance reward function, and the gait guidance reward function, that is, all joint penalty rewards are removed, thereby improving the training efficiency of the robot and reducing the training difficulty; on the other hand, after obtaining the current posture information of the robot to be trained, the center of mass velocity tracking reward corresponding to the center of mass velocity tracking reward function, the upper limb trunk balance reward corresponding to the upper limb trunk balance reward function, and the gait guidance reward corresponding to the gait guidance reward function can be calculated based on the posture information. Thus, the center of mass velocity tracking reward, the upper limb trunk balance reward, and the gait guidance reward can be input into the evaluation network to obtain the evaluation result of the action strategy output by the strategy output network. Then, it is possible to detect whether the action strategy meets the preset driving requirements based on the evaluation result. When the evaluation result indicates that the action strategy meets the preset driving requirements, the hydraulic artificial muscle is driven according to the action strategy, thereby realizing the coordinated movement of the robot driven by the hydraulic artificial muscle.

[0066] Based on the same idea as the robot training method in the above embodiments, this application also provides a robot training device that can be used to execute the above robot training method. For ease of explanation, the structural schematic diagram of the robot training device embodiment only shows the parts related to the embodiments of this application. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0067] like Figure 4 As shown, the robot training device 40 includes a construction module 401, a determination module 402, an acquisition module 403, a first input module 404, a second input module 405, and a driving module 406. In some embodiments, the above modules can be programmable software instructions stored in memory and executable by a processor. It is understood that in other embodiments, the above modules can also be program instructions or firmware embedded in a processor.

[0068] The construction module 401 is used to construct a training network, wherein the training network includes a policy output network and an evaluation network, the policy output network is used to output the motion strategy of the hydraulic artificial muscle, and the evaluation network is used to detect whether the motion strategy meets the preset driving requirements. The determination module 402 is used to determine the reward function of the training network, wherein the reward function includes a center of mass velocity tracking reward function, an upper limb trunk balance reward function, and a gait guidance reward function; The acquisition module 403 is used to acquire the current posture information of the robot to be trained, and calculate the center of mass velocity tracking reward corresponding to the center of mass velocity tracking reward function, the upper limb trunk balance reward corresponding to the upper limb trunk balance reward function, and the gait guidance reward corresponding to the gait guidance reward function based on the posture information. The first input module 404 is used to input the posture information into the strategy output network to obtain the motion strategy of the hydraulic artificial muscle. The second input module 405 is used to input the center of mass velocity tracking reward, the upper limb trunk balance reward and the gait guidance reward into the evaluation network to obtain the evaluation result of the action strategy; The driving module 406 is used to drive the hydraulic artificial muscle according to the driving strategy when the evaluation result indicates that the driving strategy meets the preset driving requirements.

[0069] The robot training device 40 provided in the above embodiments can realize the technical solutions described in the above robot training method embodiments. The specific implementation principles of each module or unit can be found in the corresponding content in the above robot training method embodiments, and will not be repeated here.

[0070] Please refer to point 5. Figure 5 This is a schematic diagram of an embodiment of the electronic device of this application.

[0071] In some embodiments, processor 501 may be a central processing unit (CPU), microprocessor, or other data processing chip, used to run program code stored in memory 502 or process data, such as the robot training method of the present invention.

[0072] In some embodiments, processor 501 may be a single server or a group of servers. The server group may be centralized or distributed. In some embodiments, processor 501 may be local or remote. In some embodiments, processor 501 may be implemented on a cloud platform. In one embodiment, the cloud platform may include a private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, intranet, multi-cloud, etc., or any combination thereof.

[0073] In some embodiments, memory 502 may be an internal storage unit of electronic device 500, such as a hard disk or memory of electronic device 500. In other embodiments, memory 502 may also be an external storage device of electronic device 500, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 500.

[0074] Furthermore, the memory 502 may include both internal storage units of the electronic device 500 and external storage devices. The memory 502 is used to store application software and various types of data installed on the electronic device 500.

[0075] In some embodiments, display 503 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 503 is used to display information from electronic device 500 and to display a visual user interface. Components 501-503 of electronic device 500 communicate with each other via a system bus.

[0076] In one embodiment, when processor 501 executes the robot training program in memory 502, the following steps can be performed: A training network is constructed, comprising a policy output network and an evaluation network. The policy output network is used to output the motion strategy of the hydraulic artificial muscle, and the evaluation network is used to detect whether the motion strategy meets preset driving requirements. Determine the reward function of the training network, wherein the reward function includes a center of mass velocity tracking reward function, an upper limb trunk balance reward function, and a gait guidance reward function; Obtain the current posture information of the robot to be trained, and calculate the center of mass velocity tracking reward corresponding to the center of mass velocity tracking reward function, the upper limb trunk balance reward corresponding to the upper limb trunk balance reward function, and the gait guidance reward corresponding to the gait guidance reward function based on the posture information. The posture information is input into the strategy output network to obtain the motion strategy of the hydraulic artificial muscle; The center of mass velocity tracking reward, the upper limb trunk balance reward, and the gait guidance reward are input into the evaluation network to obtain the evaluation result of the movement strategy; If the evaluation result indicates that the motion strategy meets the preset driving requirements, the hydraulic artificial muscle is driven according to the motion strategy.

[0077] It should be understood that when the processor 501 executes the robot training program in the memory 502, in addition to the functions mentioned above, it can also perform other functions, as detailed in the description of the corresponding method embodiments above.

[0078] Furthermore, this embodiment of the invention does not specifically limit the type of electronic device 500 mentioned. Electronic device 500 can be a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, laptop computer, or other portable electronic device. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable electronic device can also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the invention, electronic device 500 may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0079] Accordingly, this application also provides a computer-readable storage medium for storing computer-readable programs or instructions. When the programs or instructions are executed by a processor, they can implement the steps or functions in the robot training methods provided in the above-described method embodiments.

[0080] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.), and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0081] The robot training method, apparatus, electronic device, and storage medium provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A robot training method, characterized in that, The robot is driven by hydraulic artificial muscles, and the method includes: A training network is constructed, comprising a policy output network and an evaluation network. The policy output network is used to output the motion strategy of the hydraulic artificial muscle, and the evaluation network is used to detect whether the motion strategy meets preset driving requirements. Determine the reward function of the training network, wherein the reward function includes a center of mass velocity tracking reward function, an upper limb trunk balance reward function, and a gait guidance reward function; Obtain the current posture information of the robot to be trained, and calculate the center of mass velocity tracking reward corresponding to the center of mass velocity tracking reward function, the upper limb trunk balance reward corresponding to the upper limb trunk balance reward function, and the gait guidance reward corresponding to the gait guidance reward function based on the posture information. The posture information is input into the strategy output network to obtain the motion strategy of the hydraulic artificial muscle; The center of mass velocity tracking reward, the upper limb trunk balance reward, and the gait guidance reward are input into the evaluation network to obtain the evaluation result of the movement strategy; If the evaluation result indicates that the motion strategy meets the preset driving requirements, the hydraulic artificial muscle is driven according to the motion strategy.

2. The robot training method according to claim 1, characterized in that, The training network is built on a simulation platform that provides a Markov decision process environment, which includes an action space; the robot includes muscle actuators for driving the hydraulic artificial muscles. Following the construction of the training network, the following is also included: The activation layer of the training network is determined, wherein the activation layer is used to limit the mean value of the action output in the action space to the range of (-1, 1) and to limit the control range of the muscle actuator to the range of (0, 1).

3. The robot training method according to claim 2, characterized in that, Determining the activation layers of the training network includes: The activation layer is determined according to the following formula: action_mean=(actuator_high+actuator_low) / 2; action_delta=(actuator_high-actuator_low) / 2; ； Wherein, action_mean is the mean of the action, actuator_high is the maximum control value of the muscle actuator, actuator_low is the minimum control value of the muscle actuator, and action_delta is the action difference. This is the activation value of the muscle driver.

4. The robot training method according to claim 2, characterized in that, The Markov decision process environment also includes the observation space; The method further includes: The characteristic expressions for the sinusoidal signals of the robot's left leg and right leg are added to the observation space; The step of inputting the posture information into the policy output network to obtain the motion strategy of the hydraulic artificial muscle includes: The robot's current left leg sinusoidal signal is calculated based on the posture information and the characteristic expression of the left leg sinusoidal signal, and the robot's current right leg sinusoidal signal is calculated based on the posture information and the characteristic expression of the right leg sinusoidal signal, wherein the phase difference between the left leg sinusoidal signal and the right leg sinusoidal signal is 90°; The historical action strategy, the posture information, the left leg sine signal, and the right leg sine signal are input into the strategy output network to obtain the action strategy.

5. The robot training method according to claim 4, characterized in that, The characteristic expression of the left leg sinusoidal signal includes: sin_pos = backend.sin(2 × backend.pi × phase); where sin_pos is the sinusoidal signal of the left leg, backend.pi is pi, and phase is the current gait phase value; The characteristic expression of the right leg sinusoidal signal includes: cos_pos = backend.cos(2×backend.pi×phase); where cos_pos is the right leg sine signal.

6. The robot training method according to claim 1, characterized in that, The attitude information includes the robot's center of mass velocity and the Euler angles of its torso posture; The centroid velocity tracking bonus is calculated using the following formula: tracking_reward=tracking_reward_scale×exp(-k×(vel_local-goal_vel)^2); Where tracking_reward is the centroid velocity tracking reward, tracking_reward_scale is the reward weight, exp is the exponent e, k is the coefficient, vel_local is the centroid velocity, and goal_vel is the preset centroid target velocity; The upper limb trunk balance reward is calculated using the following formula: roll_pitch_reward=rp_scale×exp(-4×(euler[:2])^2); Where roll_pitch_reward is the target upper limb trunk balance reward, rp_scale is the weight of the balance reward, and euler is the Euler angle of the trunk posture.

7. The robot training method according to claim 1, characterized in that, The posture information includes the rotation matrix of the robot's pelvis in the world coordinate system, the first posture rotation matrix of the lower leg in the world coordinate system, the second posture rotation matrix of the thigh in the world coordinate system, and the third posture rotation matrix of the foot in the world coordinate system; the gait guidance reward includes the gait guidance reward of the lower leg, the gait guidance reward of the thigh, and the gait guidance reward of the foot. Before calculating the gait guidance reward, the following is also included: Calculate the inverse of the rotation matrix, and transform the first posture rotation matrix, the second posture rotation matrix, and the third posture rotation matrix into the pelvic coordinate system based on the inverse matrix; The first xyz Euler angle of the lower leg is determined according to the first pose rotation matrix after transformation; the second xyz Euler angle of the thigh is determined according to the second pose rotation matrix after transformation; and the third xyz Euler angle of the foot is determined according to the third pose matrix after transformation. The gait guidance reward is calculated using the following formula: hip_pose_error = des_hip_pos-hip_rpy; cyclic_hip_reward = cyclic_hip_scale × exp(-4 × hip_pose_error^2); knee_pose_error = des_knee_pos-knee_rpy; cyclic_knee_reward = cyclic_knee_scale × exp(-4 × knee_pose_error^2); ankle_pose_error = des_ankle_pos-ankle_rpy; cyclic_ankle_reward = cyclic_ankle_scale × exp(-4 × ankle_pose_error^2); Wherein, hip_pose_error is the pose error of the thigh, des_hip_pos is the desired pose of the thigh, hip_rpy is the second xyz Euler angle, and cyclic_hip_reward is the gait guidance reward function of the thigh; knee_pose_error is the pose error of the lower leg, des_knee_pos is the desired pose of the lower leg, knee_rpy is the first xyz Euler angle, and cyclic_knee_reward is the gait guidance reward function of the lower leg; ankle_pose_error is the pose error of the foot, des_ankle_pos is the desired pose of the foot, ankle_rpy is the third xyz Euler angle, and cyclic_ankle_reward is the gait guidance reward function of the foot.

8. A robot training device, characterized in that, The robot is driven by hydraulic artificial muscles, and the device includes: A construction module is used to construct a training network, wherein the training network includes a policy output network and an evaluation network, the policy output network is used to output the motion strategy of the hydraulic artificial muscle, and the evaluation network is used to detect whether the motion strategy meets preset driving requirements; A determination module is used to determine the reward function of the training network, wherein the reward function includes a center of mass velocity tracking reward function, an upper limb trunk balance reward function, and a gait guidance reward function; The acquisition module is used to acquire the current posture information of the robot to be trained, and calculate the center of mass velocity tracking reward corresponding to the center of mass velocity tracking reward function, the upper limb trunk balance reward corresponding to the upper limb trunk balance reward function, and the gait guidance reward corresponding to the gait guidance reward function based on the posture information. The first input module is used to input the posture information into the strategy output network to obtain the motion strategy of the hydraulic artificial muscle. The second input module is used to input the center of mass velocity tracking reward, the upper limb trunk balance reward, and the gait guidance reward into the evaluation network to obtain the evaluation result of the movement strategy; A drive module is used to drive the hydraulic artificial muscle according to the motion strategy when the evaluation result indicates that the motion strategy meets the preset drive requirements.

9. An electronic device, the electronic device comprising a processor and a memory, characterized in that, The memory is used to store instructions, and the processor is used to call the instructions in the memory to cause the electronic device to execute the robot training method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on an electronic device, cause the electronic device to perform the robot training method as described in any one of claims 1 to 7.

Images