Method and device for training a motion control policy network of a robot based on reinforcement learning

Abstract

The application provides a training method and device of a motion control strategy network of a robot based on reinforcement learning, and relates to the technical field of robots. The method is applied to a robot comprising a lower body motion unit and an upper body motion unit spliced with each other, and comprises the following steps: acquiring current state data of the lower body motion unit, inputting the motion control strategy network to be trained, and obtaining lower limb joint control instructions; in the process of controlling the two legs to perform gait actions according to the lower limb joint control instructions, sending an arm swinging instruction to the upper body motion unit, so that the two arms perform arm swinging actions, and the disturbance generated by the arm swinging actions is used as a disturbance term for training the motion control strategy network; acquiring motion state data of the two legs after performing gait actions under the disturbance generated by the arm swinging actions, determining a training reward according to the motion state data, and updating parameters of the motion control strategy network based on the training reward. The application can improve the stability of gait motion of the robot after the upper body and the lower body are spliced.

Description

Technical Field

[0001] This invention relates to the field of robotics, and specifically to a training method and apparatus for a motion control strategy network based on reinforcement learning. Background Technology

[0002] Before implementing walking control, existing bipedal robots typically require training corresponding motion control strategies tailored to their structure and motion state. Taking a lower-body robot with two legs as an example, its structure and motion state are relatively fixed when operating independently. After repeated training, the motion control strategy can gradually adapt to the robot's motion characteristics in the current state and generate corresponding lower limb joint control commands based on its motion state, enabling the robot to maintain stable walking.

[0003] During product development, the applicant further developed a humanoid robot that can be spliced ​​together, combining a lower-body robot with legs and an upper-body robot with arms to form a complete humanoid robot. After the two robots are spliced ​​together, the lower-body robot still undertakes the main functions of walking and support, but compared with independent operation, the overall structure and operation of the robot have changed. Furthermore, during actual walking, the upper-body robot also performs corresponding actions along with the overall movement of the robot, making the movement state of the spliced ​​robot more complex.

[0004] Existing technologies have conducted relevant research on the coordinated movement of different limbs of a robot. For example, Chinese patent document CN100346941C discloses a robot and a robot posture control method, which processes the movement of the robot's movable parts as periodic motion and generates phase signals related to the periodic motion; when the robot has parts that perform periodic motion and other movable parts, corresponding controllers are set up to coordinate the control of different movable parts.

[0005] For example, the paper "A Method for Correcting Yaw Torque in Humanoid Robots Based on Arm Swinging" discloses a method for correcting the yaw torque generated during the walking process of a humanoid robot by swinging its arms. This method calculates the yaw torque generated during the robot's walking process and plans the swing angular trajectory of the arms according to the walking cycle, causing the arms to swing in accordance with the walking rhythm, thereby using the torque generated by the arm swinging to correct the yaw torque generated during the robot's walking process.

[0006] However, during the actual testing of the aforementioned humanoid robot developed by the applicant, it was found that the motion control strategy originally trained for the independent operation of the lower body robot was easily affected in terms of gait control when directly applied to the humanoid robot after the upper and lower bodies were spliced ​​together. This could lead to body swaying, decreased gait stability, or even instability. In particular, during the robot's walking process, the arm swinging movements of the upper body can also affect the gait of the lower body, making it even more difficult for the original motion control strategy to adapt to the actual operating state after the upper and lower bodies are spliced ​​together.

[0007] In summary, existing training methods for the independent operation of the lower body robot are insufficient to adapt to changes in the overall structure and operation of the robot after the upper and lower body robots are spliced ​​together, thus affecting the stable walking of the humanoid robot spliced ​​together. Summary of the Invention

[0008] In view of the technical problem that the motion control strategies trained for the independent operation of the lower body robot in the prior art are difficult to adapt to the changes in the overall structure and operation state of the robot after the upper body motion unit and the lower body motion unit are spliced ​​together, the present invention provides a training method and device for a motion control strategy network of a robot based on reinforcement learning, a motion control method and device for a robot based on reinforcement learning, an electronic device, a robot, and a computer-readable storage medium, aiming to enable the motion control strategy network to adapt to the changes in the overall structure and operation state of the robot after the upper and lower body motion units are spliced ​​together, thereby improving the walking stability of the robot spliced ​​together.

[0009] To achieve the above objectives, according to one aspect of the present invention, a training method for a motion control policy network of a robot based on reinforcement learning is provided. The robot includes a lower body motion unit and an upper body motion unit. The lower body motion unit includes two legs, and the upper body motion unit includes two arms. The lower body motion unit and the upper body motion unit are spliced ​​together. The method includes: The current state data of the lower body motion unit is obtained and input into the motion control strategy network to be trained to obtain lower limb joint control commands, which are used to control the legs to perform gait movements. During the process of controlling the legs to perform gait movements according to the lower limb joint control commands, the arm swing command is sent to the upper body motor unit so that the arms can perform arm swing movements. The disturbance generated by the arm swing movement is the disturbance term for training the motor control strategy network. The system acquires motion state data of both legs after performing gait movements under the disturbance of arm swing, determines training rewards based on the motion state data, and updates the parameters of the motion control policy network based on the training rewards. The training rewards are used to characterize the walking stability of the lower body motor unit.

[0010] Specifically, during the process of controlling the legs to perform gait movements according to lower limb joint control commands, arm swing commands are sent to the upper body motion unit to enable the arms to perform arm swing movements. This includes: acquiring preset training arm swing parameters, including swing amplitude, swing frequency, and initial swing phase; determining the arm swing phase corresponding to each control moment during training based on the swing frequency and initial swing phase, where the arm swing phase characterizes the swing stage of the arms at the corresponding control moment within a swing cycle; determining the target joint positions of the arms at the corresponding control moment based on each arm swing phase and swing amplitude, and generating a training arm swing trajectory based on the target joint positions at each control moment, where the training arm swing trajectory characterizes the change relationship of the target joint positions of the arms over time; generating arm swing commands based on the training arm swing trajectory, and sending the arm swing commands to the upper body motion unit during the gait movements of the legs to enable the arms to perform arm swing movements according to the training arm swing trajectory.

[0011] The method further includes: determining the training arm swing parameters for each training round in multiple training rounds, wherein the training arm swing parameters for different training rounds differ in at least one of the swing amplitude, swing frequency, and initial swing phase; generating the training arm swing trajectory for each training round based on the training arm swing parameters for each training round; controlling the arms to perform arm swing movements according to the corresponding training arm swing trajectory in each training round, so that the arm swing movements in different training rounds generate different perturbations; acquiring the motion state data of the legs after performing gait movements under the corresponding perturbation in each training round, determining the corresponding training reward based on the motion state data for each training round, and iteratively updating the parameters of the same motion control strategy network based on each training reward.

[0012] In this process, the training arm swing parameters for each training round are determined, including: obtaining a preset number of training rewards for completed training rounds and determining the average training reward based on the preset number of training rewards for completed training rounds; if the average training reward is greater than or equal to a preset reward threshold, it is determined that the walking stability of the lower body motor unit has reached a preset stability level, and the swing amplitude for subsequent training rounds is increased within a preset swing amplitude range; the training arm swing parameters for subsequent training rounds are determined based on the increased swing amplitude and the swing frequency and initial phase of the swing for subsequent training rounds.

[0013] The method further includes: during a reference training round, sending a hold command to the upper body motor unit to keep the arms in a preset reference posture, and acquiring reference motion state data after the legs perform gait movements. The preset reference posture is the posture in which the arms remain unchanged and do not produce arm swinging movements during the reference training round; determining a reference stability index based on the reference motion state data, which characterizes the walking stability of the lower body motor unit when no disturbance caused by arm swinging movements is applied; determining training rewards based on the motion state data, including: determining a perturbation stability index based on the motion state data, which characterizes the walking stability of the lower body motor unit under the perturbation caused by arm swinging movements; and determining training rewards based on the difference between the reference stability index and the perturbation stability index.

[0014] The process involves acquiring motion state data of both legs after performing gait movements under the disturbance of arm swinging motion, and determining training rewards based on the motion state data. This includes: acquiring motion state data of the lower body motor unit at multiple acquisition times within a preset time window during the arm swinging motion and leg gait movements, and assembling the data into a motion state data sequence according to the order of acquisition times; determining the degree of motion state fluctuation based on the changes between the motion state data at different acquisition times in the motion state data sequence, which characterizes the walking stability of the lower body motor unit when performing gait movements within the preset time window; and determining training rewards based on the degree of motion state fluctuation.

[0015] The method further includes: acquiring the mass attribute data of the upper body motion unit and the lower body motion unit, as well as the splicing pose relationship between the upper body motion unit and the lower body motion unit. The mass attribute data includes the mass, center of mass position, and moment of inertia of the corresponding motion unit. The splicing pose relationship is used to characterize the relative position and relative posture of the upper body motion unit and the lower body motion unit after splicing. Based on the mass attribute data of the upper body motion unit and the lower body motion unit and the splicing pose relationship, the overall mass attribute data of the spliced ​​robot is determined. The overall mass attribute data includes the total mass, overall center of mass position, and overall moment of inertia of the spliced ​​robot. Based on the overall mass attribute data, the dynamic parameters of the robot model used to characterize the spliced ​​robot in the reinforcement learning training environment are configured. In the reinforcement learning training environment, the motion control policy network is trained based on the robot model with configured dynamic parameters.

[0016] The method further includes the following steps before training the motion control strategy network to be trained on the robot based on the spliced ​​upper and lower body motion units: In the first training stage before the upper and lower body motion units are spliced, the state data of the lower body motion units is acquired and input into the initial motion control strategy network to obtain lower limb joint control commands; the legs are controlled to perform gait movements according to the lower limb joint control commands, and a pre-training reward is determined based on the motion state data after the legs perform gait movements. The pre-training reward is used to characterize the walking stability of the lower body motion units when they are not spliced ​​with the upper body motion units; the parameters of the initial motion control strategy network are updated based on the pre-training reward to obtain the pre-trained motion control strategy network; after completing the first training stage, the upper and lower body motion units are spliced, the motion control strategy network to be trained is initialized based on the parameters of the pre-trained motion control strategy network, and the motion control strategy network to be trained is continued to be trained based on the perturbations generated by the arm swinging movements.

[0017] According to another aspect of the present invention, a motion control method for a robot based on reinforcement learning is also provided. The robot includes a lower body motion unit and an upper body motion unit. The lower body motion unit includes two legs, and the upper body motion unit includes two arms. The lower body motion unit and the upper body motion unit are spliced ​​together. The method includes: The current state data of the lower body motion unit is obtained and input into the motion control strategy network trained by reinforcement learning to obtain lower limb joint control commands. The lower limb joint control commands control the legs to perform gait movements, and the gait signals of the lower body motor units are acquired during the gait movements. The gait signals are used to characterize the periodic motion state of the legs when performing gait movements. The arm swing trajectory of the upper body motion unit is generated based on the gait signal. The arm swing trajectory is used to characterize the relationship between the target position of the joint to be controlled in the two arms and the gait signal. The system generates joint control signals based on the arm swing trajectory and controls the arms to perform arm swing movements in accordance with these signals, so that the arm swing movements are coordinated with the gait movements of the legs.

[0018] The gait signal includes a gait phase signal, which is used to characterize the gait stage of both legs in the current gait cycle. The arm swing trajectory of the upper body movement unit is generated based on the gait signal, including: determining the target joint positions corresponding to the shoulder joint pitch degree of freedom and the elbow joint pitch degree of freedom in each gait stage based on the gait phase signal; generating a shoulder joint swing trajectory based on the target joint positions corresponding to the shoulder joint pitch degree of freedom in each gait stage, and generating an elbow joint swing trajectory based on the target joint positions corresponding to the elbow joint pitch degree of freedom in each gait stage; and generating an arm swing trajectory based on the shoulder joint swing trajectory and the elbow joint swing trajectory.

[0019] The process of determining the target joint position corresponding to the shoulder joint pitch degree of freedom in each gait phase includes: determining the gait frequency of the lower body movement unit based on the change of the gait phase signal, where the gait frequency is the frequency at which both legs repeat a complete gait cycle; determining the swing frequency of the shoulder joint pitch degree of freedom to be the same as the gait frequency; and determining the target joint position corresponding to the shoulder joint pitch degree of freedom in each gait phase based on the gait phase signal, the swing frequency of the shoulder joint pitch degree of freedom, and the preset shoulder joint swing amplitude.

[0020] The process of determining the target joint position corresponding to the elbow joint pitch degree of freedom in each gait phase includes: determining the swing period of the shoulder joint pitch degree of freedom based on the shoulder joint swing trajectory; determining the swing period of the elbow joint pitch degree of freedom based on the swing period of the shoulder joint pitch degree of freedom and a preset period relationship, wherein the preset period relationship is used to characterize the proportional relationship between the swing period of the elbow joint pitch degree of freedom and the swing period of the shoulder joint pitch degree of freedom; and determining the target joint position corresponding to the elbow joint pitch degree of freedom in each gait phase based on the gait phase signal, the swing period of the elbow joint pitch degree of freedom, and the preset elbow joint swing amplitude.

[0021] The process of generating an elbow joint swing trajectory based on the target joint position corresponding to the elbow joint pitch degree of freedom in each gait phase includes: dividing a complete swing cycle of the elbow joint pitch degree of freedom into a first half swing cycle and a second half swing cycle in chronological order; generating the first half cycle trajectory based on the target joint position of the elbow joint in the first half swing cycle, and generating the second half cycle trajectory based on the target joint position of the elbow joint in the second half swing cycle; and connecting the second half cycle trajectory and the first half cycle trajectory to obtain the elbow joint swing trajectory.

[0022] The process of generating an arm swing trajectory based on the shoulder joint swing trajectory and the elbow joint swing trajectory includes: determining the swing start time and swing end time corresponding to the shoulder joint swing trajectory and the elbow joint swing trajectory respectively according to a preset timing relationship, wherein the swing start time corresponding to the shoulder joint swing trajectory is earlier than the swing start time corresponding to the elbow joint swing trajectory, and the swing end time corresponding to the shoulder joint swing trajectory is later than the swing end time corresponding to the elbow joint swing trajectory; and generating the arm swing trajectory based on the shoulder joint swing trajectory and the elbow joint swing trajectory after determining the swing start time and the swing end time.

[0023] According to another aspect of the present invention, a training device for a motion control strategy network of a robot based on reinforcement learning is also provided. The robot includes a lower body motion unit and an upper body motion unit. The lower body motion unit includes two legs, and the upper body motion unit includes two arms. The lower body motion unit and the upper body motion unit are spliced ​​together. The training device includes: a control command generation module, used to acquire the current state data of the lower body motion unit and input the current state data into the motion control strategy network to be trained to obtain lower limb joint control commands, which are used to control the two legs to perform gait movements; a disturbance application module, used to send arm swing commands to the upper body motion unit during the process of controlling the two legs to perform gait movements according to the lower limb joint control commands, so that the two arms perform arm swing movements, and the disturbance generated by the arm swing movements is a disturbance term for training the motion control strategy network; and a training update module, used to acquire the motion state data of the two legs after performing gait movements under the disturbance generated by the arm swing movements, determine the training reward according to the motion state data, and update the parameters of the motion control strategy network based on the training reward, which is used to characterize the walking stability of the lower body motion unit.

[0024] According to another aspect of the present invention, a motion control device for a robot based on reinforcement learning is also provided. The robot includes a lower body motion unit and an upper body motion unit. The lower body motion unit includes two legs, and the upper body motion unit includes two arms. The lower body motion unit and the upper body motion unit are spliced ​​together. The motion control device includes: a lower limb control module, used to acquire the current state data of the lower body motion unit and input the current state data into a motion control strategy network trained by reinforcement learning to obtain lower limb joint control commands; and a gait signal acquisition module, used to acquire gait signals according to the lower limb joint control commands. The system controls both legs to perform gait movements and acquires gait signals from the lower body movement unit during these movements. These gait signals characterize the periodic motion state of the legs during gait movements. An arm swing trajectory generation module generates arm swing trajectories for the upper body movement unit based on the gait signals. These trajectories characterize the relationship between the target position of the joints to be controlled in the arms and the gait signals. An arm swing control module generates joint control signals based on the arm swing trajectories and controls the arms to perform arm swing movements according to these signals, ensuring that the arm swing movements coordinate with the leg gait movements.

[0025] According to another aspect of the present invention, an electronic device is also provided, comprising: a processor; and a memory storing computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, the training method for the motion control policy network of the robot based on reinforcement learning described above, or the motion control method for the robot based on reinforcement learning described above, is implemented.

[0026] According to another aspect of the present invention, a robot is also provided, comprising: a lower body motion unit including two legs; an upper body motion unit including two arms, wherein the upper body motion unit is spliced ​​with the lower body motion unit; and one or more controllers configured to implement the motion control method for a robot based on reinforcement learning as claimed in any one of claims 9 to 14.

[0027] One or more controllers include a lower body controller and an upper body controller. The lower body controller is located in the lower body motion unit, and the upper body controller is located in the upper body motion unit. The lower body controller and the upper body controller are communicatively connected. The lower body controller is used to control the legs to perform gait movements, and acquires gait signals from the lower body motion unit during the gait movements. It generates the arm swing trajectory of the upper body motion unit based on the gait signals, generates joint control signals based on the arm swing trajectory, and sends the joint control signals to the upper body controller. The upper body controller is used to control the arms to perform arm swing movements based on the joint control signals.

[0028] According to another aspect of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the above-described training method for the motion control policy network of a reinforcement learning-based robot, or implements the above-described motion control method for a reinforcement learning-based robot.

[0029] Compared with the prior art, the present invention has the following beneficial effects: This invention trains a motion control strategy network in a spliced ​​state between the lower and upper body motion units. During the gait movements of the lower body motion units, the network controls the arms to perform swinging motions. The disturbances generated by these swinging motions are used as perturbation terms in the training process of the motion control strategy network. This allows the network to adapt to the actual operating state after the two motion units are spliced ​​together and the dynamic influence of the upper body motion unit's movement on the lower body motion unit. Simultaneously, training rewards are determined based on the motion state data of the legs after performing gait movements under the perturbation, and the parameters of the motion control strategy network are updated based on these rewards. This allows the network to gradually learn to maintain stable walking of the lower body motion unit while the upper body motion unit is moving. Therefore, the trained motion control strategy can better adapt to changes in the overall structure and operating state of the robot after splicing, enabling the lower body motion unit to maintain a stable gait even when the upper body motion unit is continuously moving. This reduces body sway and gait fluctuations during robot walking, lowers the possibility of instability, and improves the walking stability of the spliced ​​robot.

[0030] Meanwhile, since the robot's motion state during arm swinging has been taken into account during the training process, the trained motion control strategy can take into account both the stable walking of the lower body and the arm swinging of the upper body during the actual walking process of the robot. This allows the robot to maintain gait stability while retaining the motion form of the arms swinging during the walking process, improving the coordination of the upper and lower body movements and making the robot's overall walking movements more natural. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0032] Figure 1 A schematic diagram of the system architecture according to an embodiment of the present invention is shown.

[0033] Figure 2 A flowchart illustrating a training method for a motion control strategy network according to an embodiment of the present invention is shown.

[0034] Figure 3 A flowchart illustrating the training process of a motion control strategy network in one embodiment of the present invention is shown.

[0035] Figure 4 This diagram illustrates a motion state of the lower body motion unit in an unassembled state according to an embodiment of the present invention.

[0036] Figure 5 This diagram illustrates a motion state of the lower body motion unit in an unassembled state according to another embodiment of the present invention.

[0037] Figure 6 A schematic diagram of the upper body movement unit in one embodiment of the present invention is shown.

[0038] Figure 7 This diagram illustrates the splicing of the upper body motion unit and the lower body motion unit in one embodiment of the present invention.

[0039] Figure 8 A schematic diagram of the robot structure after the upper body motion unit and the lower body motion unit are spliced ​​together is shown in one embodiment of the present invention.

[0040] Figure 9 A schematic diagram of the robot in a first motion state according to an embodiment of the present invention is shown.

[0041] Figure 10 A schematic diagram of the robot in a second motion state is shown in one embodiment of the present invention.

[0042] Figure 11 A schematic diagram of the robot in a third motion state according to an embodiment of the present invention is shown.

[0043] Figure 12 This diagram illustrates the data relationships during the training and deployment process of a motion control strategy network according to an embodiment of the present invention.

[0044] Figure 13 A flowchart illustrating a robot motion control method according to an embodiment of the present invention is shown.

[0045] Figure 14 The diagram illustrates a process of controlling the upper body movement unit to perform arm swinging movements based on the gait signals of the lower body movement unit in one embodiment of the present invention.

[0046] Figure 15 A schematic diagram of the swing trajectory of the shoulder and elbow joints in one embodiment of the present invention is shown.

[0047] Figure 16 A structural block diagram of a training device for a motion control strategy network according to an embodiment of the present invention is shown.

[0048] Figure 17 A structural block diagram of a robot motion control device according to an embodiment of the present invention is shown.

[0049] Figure 18 A schematic diagram of the structure of an electronic device according to an embodiment of the present invention is shown.

[0050] Figure 19 A schematic diagram of the structure of a robot according to an embodiment of the present invention is shown.

[0051] Figure 20 A schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present invention is shown. Detailed Implementation

[0052] In the description of the embodiments of this invention, the terms "first" and "second" are used for description only and do not indicate relative importance or imply the number of technical features. Therefore, the features referred to as "first" and "second" may explicitly or implicitly include at least one of those features. "A plurality of" means at least two, unless otherwise expressly defined.

[0053] The main technical terms involved in the embodiments of this invention are explained in the table below. The following description is for the purpose of facilitating understanding of the embodiments of this invention.

[0054] The specific values ​​of the above terms can be set according to the robot structure, training environment, and control configuration. The training process of the motion control strategy network and the motion control process after robot deployment are explained below with reference to the accompanying drawings.

[0055] Figure 1 A system architecture 100 according to an embodiment of the present invention is shown. For example... Figure 1 As shown, the system architecture 100 may include a terminal device 101, a device node 102, a communication network 103, and a server 104. The terminal device 101, device node 102, and server 104 can interact with each other through the communication network 103, which can be a wired communication network or a wireless communication network.

[0056] In some embodiments, device node 102 or server 104 can be used to execute the training process of the motion control policy network and store the trained motion control policy network parameters. After training is completed, the corresponding network parameters can be deployed to one or more controllers on the robot side for robot motion control. Terminal device 101 can be used for training task configuration, running status viewing, or related data interaction. Figure 1 The specific functions of each device shown can be configured according to the actual training environment and the deployment method of the robot control system.

[0057] The robot is formed by splicing together a lower body motion unit and an upper body motion unit. The lower body motion unit includes the two legs, which are responsible for supporting the entire robot and gait movement; the upper body motion unit includes the two arms. After the upper body motion unit is spliced ​​with the lower body motion unit, the mass of the upper body itself will change the total mass, center of mass position and moment of inertia of the entire robot, and the movement of the two arms will also transmit the dynamic effects that change over time to the lower body through the splicing structure.

[0058] The training phase takes place on a robot model where the upper and lower body are assembled. The training environment periodically acquires the current state data of the lower body motion units and inputs this data into the motion control strategy network to be trained. The motion control strategy network outputs lower limb joint control commands, and the training environment drives the legs to perform gait movements according to these commands.

[0059] After the upper and lower body are spliced ​​together, the overall mass, center of mass position, and moment of inertia of the robot will change. When the arms swing, the acceleration, deceleration, and reversal of the linkages such as the upper arm and forearm will also generate inertial effects that vary over time. The training environment advances the state according to the spliced ​​complete robot model, so that the above-mentioned changes in mass attributes and the movements of the arms are all involved in the training of the lower body motion control strategy network.

[0060] Within a training round, the lower body motor unit continuously executes gait movements according to the lower limb joint control commands given by the motor control strategy network, while the upper body motor unit moves according to the training arm swing trajectory corresponding to the current training round. The effects of the arm movements are reflected in the posture, velocity, joint state, and foot contact state at the next control moment, and the motor control strategy network performs subsequent control based on the continuously updated state data.

[0061] While the legs perform gait movements, the training environment controls the upper body motor unit's arms to swing slightly according to a pre-determined training arm swing trajectory. The upper body arm swing ability is not trained separately during the training phase; the dynamic effect generated by the arm swing serves as a perturbation term during the training of the lower body motor control strategy network.

[0062] After the legs execute gait movements under the influence of arm swing disturbances, the training environment acquires the corresponding motion state data and determines the training reward based on this data. The training reward is used to evaluate the walking stability of the lower body motor unit under the current disturbance conditions, and is subsequently used to update the parameters of the motor control policy network. As training rounds continue, the motor control policy network gradually learns to maintain a stable gait under conditions where the upper body is already spliced ​​and the arms are in continuous motion.

[0063] After training is completed, the deployment phase begins. During deployment, the controller acquires the current state data of the lower body motion unit, calls the trained motion control strategy network to obtain lower limb joint control commands, and controls walking based on these commands. Simultaneously, the controller acquires the gait signals of the lower body motion unit and generates the arm swing trajectory of the upper body motion unit in real time based on these signals. During deployment, training rewards are no longer calculated, and the motion control strategy network parameters are no longer updated.

[0064] The use of upper body arm swing differs between the training and deployment phases. In the training phase, the arms are controlled according to a set training arm swing trajectory, using the arm movements to create training disturbances for the lower body; in the deployment phase, the arm swing trajectory is generated based on the current gait signal of the lower body, causing the arms to move in accordance with the actual gait rhythm of the lower body.

[0065] Figure 2 The method for training a motion control policy network according to this embodiment is shown, which includes steps S201 to S203.

[0066] Step S201: Obtain the current state data of the lower body motion unit, input the current state data into the motion control strategy network to be trained, obtain the lower limb joint control commands, and control the legs to perform gait movements according to the lower limb joint control commands.

[0067] In step S202, during the gait movement of both legs, the arm swing command is sent to the upper body motion unit to make the arms swing, and the dynamic influence generated by the arm swing is used as a perturbation term for training the motion control strategy network.

[0068] Step S203: Obtain motion state data of both legs after performing gait movements under the disturbance of arm swing, determine training rewards based on the motion state data, and update the parameters of the motion control strategy network based on the training rewards.

[0069] Steps S201 to S203 can be repeated across multiple training rounds. Each training round may include multiple control cycles. Within each control cycle, network inference is first performed based on the current state data, and then the training environment executes the corresponding lower limb joint control commands and arm swing commands to advance the robot's state. Training rewards can be calculated per control cycle, or they can be calculated cumulatively within a preset time window or at the end of the training round.

[0070] In step S201, the current state data is used to represent the motion state of the lower body motion unit at the current control moment. The current state data may include one or more of the following: lower body posture, linear velocity, angular velocity, joint positions and velocities of both legs, and foot contact state. Specific input items can reuse the state variables used when the lower body motion unit was trained in the unstitched state, so that existing network parameters can continue to be used in subsequent stitching training.

[0071] The motion control strategy network outputs lower limb joint control commands for each joint to be controlled in both legs based on the current state data. Depending on the underlying joint control method, the lower limb joint control commands can be represented as target joint angles, target joint velocities, target joint torques, or motion quantities used to further calculate the aforementioned control quantities. After the training environment executes the control command, it updates the robot's state at the next control moment.

[0072] In step S202, the upper body motion unit has been mechanically connected to the lower body motion unit. The mass of the upper body motion unit itself constitutes the additional load that the lower body needs to continuously bear; when the arms swing, the acceleration and deceleration of the linkages such as the upper arm and forearm will also generate inertial force, which is transmitted to the lower body through the connection structure between the upper and lower body. This dynamic effect changes with the swing phase and swing speed of the arms and is not a fixed external force.

[0073] For example, when the arms swing forward and decelerate near the swing endpoint, the inertia of the upper body's corresponding linkage is transmitted to the lower body through the torso and splicing position; after the arms change their swing direction, the direction and magnitude of the above-mentioned effect also change accordingly. This effect is superimposed on the changes in body posture and center of mass movement caused by the original gait motion. The training environment directly executes upper limb joint movements, causing the disturbance to change continuously with the actual swinging process of the arms.

[0074] The same arm swing trajectory can have different effects on the lower body at different gait stages. For example, the lower body's response to the upper limb's dynamic action differs when supporting the robot on two legs, supporting it on one leg, or when the swinging foot is about to land. The training environment continuously advances the complete robot dynamic state, allowing these differences to be naturally reflected in the motion state data at each control moment. Based on this, the motion control policy network learns to maintain stability at different gait stages.

[0075] During training, the upper body motor unit can move directly according to the trained arm swing trajectory without the need for a separate reinforcement learning strategy to generate the arm swing movement. The parameters of the trained arm swing trajectory can be preset and adjusted in different training rounds to change the perturbation of the arms on the lower body.

[0076] In step S203, motion state data can be selected from lower body posture, velocity, angular velocity, joint state, and foot contact state. Training rewards are determined based on this motion state data and are used to reflect whether the lower body motor unit can maintain the expected gait under the influence of arm perturbations. After the network updates its parameters based on the training rewards, it can gradually reduce body swaying, gait fluctuations, or instability caused by similar perturbations.

[0077] In one implementation, the current motion state can be evaluated from three aspects: fuselage stability, speed following, and foot support. For example, when the fuselage posture deviates from a preset range or the fuselage angular velocity fluctuates significantly, the corresponding stability evaluation decreases; when the deviation between the actual motion speed and the target motion speed increases, the corresponding speed following evaluation decreases; and when the foot contact relationship is abnormal or the support switching is unstable, the corresponding support evaluation decreases. Each evaluation result can constitute a reward item individually, or they can be combined to obtain training rewards for updating the motion control strategy network. The specific reward format can be set according to the existing reinforcement learning training framework for the lower body.

[0078] The swinging of the arms themselves does not need to be considered an independent observation in the motion control policy network. After the arms move within the complete robot dynamics environment, their impact on the robot's posture, velocity, and foot contact state will be reflected in the current state data at the next control moment, and further affect the training reward.

[0079] When trained in the above manner, the changes in mass properties resulting from the splicing of the upper body and the dynamic effects generated by the movement of the arms are all involved in the training process. Therefore, the trained motion control strategy network can be used for the robot after the upper and lower bodies are spliced ​​together, and is not limited to the case of the lower body operating alone.

[0080] Figure 3 This further illustrates a specific process for training a motion control strategy network. Step S301 acquires the mass attribute data and splicing pose relationship of the upper and lower body motion units, and configures the dynamic parameters of the spliced ​​robot model accordingly. Step S302 pre-trains the lower body motion control strategy network before the upper and lower bodies are spliced, obtaining a pre-trained motion control strategy network. Step S303 splices the upper and lower body motion units, and initializes the motion control strategy network to be trained using the pre-trained network parameters. After splicing, Step S304 acquires the current state data and obtains lower limb joint control commands through the motion control strategy network. Step S305 determines the swing amplitude, swing frequency, and initial swing phase used in the current training round. Step S306 determines the arm swing phase and target joint position at each control moment and generates the training arm swing trajectory accordingly.

[0081] Step S307 controls the legs to perform gait movements and the arms to perform arm swing movements according to the training arm swing trajectory; Step S308 acquires motion state data at multiple collection points within a preset time window, forming a motion state data sequence; Step S309 determines the training reward based on the motion state data, and determines the average training reward based on the training rewards corresponding to a preset number of completed training rounds. Step S310 determines whether the average training reward reaches a preset reward threshold; if it reaches the preset reward threshold, then proceed to step S311, increasing the swing amplitude in subsequent training rounds within a preset swing amplitude range, and proceed to step S312, updating the parameters of the motion control strategy network based on the training reward; if it does not reach the preset reward threshold, then maintain the current swing amplitude, and directly proceed to step S312, updating the parameters of the motion control strategy network based on the training reward. If the preset reward threshold is not reached, the current swing amplitude can be maintained and training can continue. Figure 3 It is used to represent the processing relationship between two-stage training, arm swing perturbation generation, stability evaluation, and gradual adjustment of swing amplitude.

[0082] In one implementation, the training arm swing parameters include swing amplitude, swing frequency, and initial swing phase. Swing amplitude is used to determine the range of swing of the arms or the upper limb joint to be controlled relative to a reference position; swing frequency is used to determine the speed of the swing cycle; and initial swing phase is used to determine which phase of the swing cycle the arms are in at the start of a training round.

[0083] At each control moment, the current arm swing phase can be determined based on the swing frequency and the time elapsed since the start of the training round, and combined with the initial swing phase to obtain the actual arm swing phase corresponding to this control moment. Then, the target joint position of the joint to be controlled in both arms is determined based on the current arm swing phase and the swing amplitude. The target joint positions corresponding to multiple consecutive control moments are arranged in chronological order to form the training arm swing trajectory.

[0084] The training arm trajectory can be generated online using a periodic function, or a discrete target joint position within a swing cycle can be pre-generated and then queried or interpolated according to the current arm phase. As long as the arms can move according to the set swing amplitude, swing frequency, and initial swing phase, this embodiment does not limit the specific trajectory expression form.

[0085] Once the arm swing trajectory is determined, the training environment generates an arm swing command based on the target joint position at the current control moment and sends the command to the upper body motion unit. The arms continuously execute the arm swing command during walking, resulting in corresponding upper limb dynamic actions at each control moment.

[0086] By employing the aforementioned parametric swing arm method, the amplitude of the disturbance, the rate of change of the disturbance, and the initial state when the disturbance occurs can be changed separately. For example, when the swing amplitude is increased under the condition of the same swing frequency, the influence of the double arm movement on the overall attitude and center of gravity of the machine usually increases accordingly; when the swing frequency is changed under the condition of the same swing amplitude, the rate of change of the dynamic disturbance also changes accordingly.

[0087] In terms of disturbance intensity, swing amplitude mainly affects the range of motion of the arms relative to the reference posture; in terms of disturbance change rate, swing frequency determines the speed of joint movement changes within the same swing range; in terms of the timing of disturbance occurrence, the initial phase of the swing determines whether the arms are in different phases such as forward swing, backward swing, or change of direction at the beginning of the training round. These three types of parameters can be adjusted individually or in combination to create upper limb dynamic influences of different intensities, rhythms, and starting states in the training environment.

[0088] For both arms, the same swing amplitude can be used with corresponding phase relationships, or the swing parameters can be set separately. The training phase primarily utilizes different arm swing movements to create different disturbances; therefore, the training arm swing trajectory can differ from the arm swing trajectory used in the deployment phase.

[0089] Different training arm swing parameters can be set for each training round. At least one of the following—swing amplitude, swing frequency, and initial swing phase—will differ between training rounds, and corresponding training arm swing trajectories will be generated based on these parameters. The same lower body gait can correspond to different upper limb swing states in different training rounds.

[0090] For example, the swing frequency can be kept relatively constant over several training rounds while the swing amplitude is gradually changed; alternatively, the swing frequency or the initial phase of the swing can be changed while keeping the swing amplitude constant. For the same motion control policy network, the training rewards generated in each training round are used for iterative parameter updates, enabling the network to learn together under different perturbation conditions.

[0091] After adjusting the training arm swing parameters in different training rounds, the lower body motion control strategy network can access the bi-arm motion states with different amplitudes, frequencies, and initial phases, thereby increasing the coverage of upper limb perturbations during training.

[0092] In one progressive training implementation, the swing amplitude of subsequent training rounds can be adjusted based on the training rewards of completed training rounds. Specifically, a preset number of training rewards corresponding to completed training rounds are obtained, and the average training reward is calculated. When the average training reward is greater than or equal to a preset reward threshold, the lower body motor unit is considered to have reached a preset level of stability under the current perturbation intensity.

[0093] A preset number of training rounds can form a statistical interval for evaluating the current training phase. By calculating the average training reward across multiple training rounds and then adjusting the swing amplitude accordingly, reward fluctuations caused by initial state, contact state, or random factors in a single training round can be reduced.

[0094] The amplitude of the swing can be increased in stages, such as moving to the next amplitude level after the average training reward corresponding to the current amplitude reaches a threshold; or it can be increased continuously in preset increments, but not exceeding the preset swing amplitude range. Regardless of the method used, the training difficulty should be increased only after the lower body has been able to withstand the current disturbance.

[0095] Once a preset level of stability is reached, the swing amplitude in subsequent training rounds is increased within a preset swing amplitude range. A new training arm swing trajectory is generated by combining the swing frequency and initial phase of the swing in each subsequent training round. If the average training reward has not yet reached the preset reward threshold, the current swing amplitude can be maintained for training until the lower body movement reaches the required level.

[0096] During training, the swing amplitude can be gradually adjusted according to the stability of the lower body. In the early stages of training, a smaller swing amplitude is used. After the average training reward reaches the preset reward threshold, the swing amplitude in subsequent training rounds can be increased to reduce frequent instability due to excessive disturbance in the early stages of training.

[0097] In this way, training can be completed first with smaller arm swing disturbances, and then the swing amplitude can be gradually increased so that the motor control strategy network can continue to adapt to stronger upper limb dynamics.

[0098] In another implementation, a reference training round can be set. In the reference training round, a hold command is sent to the upper body movement unit to keep the arms in a preset reference posture without performing arm swinging movements, while the legs are controlled to perform gait movements according to the same or similar lower body movement task as the perturbation training round.

[0099] After the reference training rounds, a reference stability index is determined based on the reference motion data after the legs perform gait movements. The reference stability index represents the walking stability of the lower body motor unit without additional arm swing disturbances. During the training rounds involving arm swing movements, a perturbation stability index is determined based on the corresponding motion data.

[0100] Reference stability indices and perturbation stability indices can use the same state variables and calculation methods. For example, changes in body posture, angular velocity fluctuations, velocity deviations, or changes in foot contact state can be calculated separately. After determining the training reward based on the difference between the two, the additional effects of arm swing movements can be distinguished from the basic gait fluctuations of the lower body itself.

[0101] When the same stability index is used to evaluate the reference training round and the arm swing disturbance training round, the difference in index between the two types of training rounds can be used to reflect the impact of arm swing on lower body stability, and training rewards can be determined accordingly.

[0102] In another embodiment, during the process of the arms swinging and the legs gaiting, multiple motion state data corresponding to the acquisition time are continuously acquired within a preset time window, and the motion state data sequence is formed according to the order of acquisition time.

[0103] By analyzing the changes in motion state data at different acquisition times within a motion state data sequence, the degree of motion state fluctuation can be determined. Specifically, the maximum variation, variance, root mean square error, or variation between adjacent acquisition times of a certain state variable can be calculated. Alternatively, multiple state variables can be calculated separately and then combined. The greater the motion state fluctuation, the worse the walking stability of the lower body within that time window.

[0104] Using multiple motion state data within a preset time window to determine training rewards can reflect the continuous changes in body posture, angular velocity, or foot contact state during the continuous swinging of the arms, rather than just taking the state at a certain control moment.

[0105] The robot model in the training environment is configured according to the overall dynamic parameters of the spliced ​​upper and lower body motion units. Specifically, the mass, center of mass position, and moment of inertia of the upper and lower body motion units can be obtained separately, and the relative position and relative attitude of the spliced ​​units can be obtained.

[0106] Based on the splicing pose relationship, the mass attribute data of the two motion units are converted to a unified reference coordinate system, thereby determining the total mass, overall center of mass position, and overall inertia of the spliced ​​robot. Subsequently, the robot model in the reinforcement learning training environment is configured using the aforementioned overall mass attribute data. During the movement of the two arms, the dynamic effects of each upper limb link changing with the joint state are calculated by the robot model according to the current motion state.

[0107] The unified reference coordinate system can be selected from the lower body base coordinate system, the robot torso coordinate system, or the world coordinate system in the training environment. When the splicing position or splicing posture of the upper and lower body changes, the mass properties of each part of the upper body can be recalculated according to the current splicing relationship, thereby obtaining the corresponding overall dynamic parameters of the robot. When the location of the newly added mass changes, the position of the overall center of mass and the moment of inertia are also recalculated according to the same splicing relationship.

[0108] For movable structures such as dual arms, the position of each link relative to the whole machine changes with joint movement. After the training environment executes the arm swing command, the complete robot model calculates the dynamic response at the next control moment based on the new joint state, with mass attributes and upper limb movements jointly participating in the state update.

[0109] After configuring the robot model according to the overall quality attribute data after splicing, the training environment can simultaneously reflect the load changes after the upper body is increased and the dynamic inertial effect generated by the movement of the two arms.

[0110] In one two-stage training implementation, the lower body can be pre-trained before the upper and lower body motor units are spliced ​​together. In the first training stage, the state data of the lower body motor units are input into the initial motion control strategy network to obtain lower limb joint control commands; the pre-training reward is determined based on the motion state data after the legs perform gait movements, and the initial motion control strategy network is updated using the pre-training reward to obtain the pre-trained motion control strategy network.

[0111] After the first training phase is completed, the upper body motion units are spliced ​​together with the lower body motion units, and the motion control policy network to be trained is initialized using the parameters of the pre-trained motion control policy network. Then, the training process in the spliced ​​state is executed again, so that the perturbations generated by the arm swings participate in the subsequent network parameter updates.

[0112] When using a two-stage training approach, the lower body can first develop basic standing and walking abilities, and then the upper body can be trained to further adapt the network parameters to the dynamic effects of the added load and arm swing.

[0113] The first and second training phases can use the same lower body state data definition and lower limb motion space. In the second training phase, the training object is replaced with a complete robot model consisting of the upper and lower body segments, and a training arm trajectory is added, allowing the pre-trained network parameters to be used for subsequent training.

[0114] If training starts directly from random network parameters under conditions of a fully spliced ​​model and strong arm swing perturbations, the network needs to learn both basic gait and stable control under perturbation conditions simultaneously. Pre-training the lower body first, followed by continued training after splicing, can reduce the amount of basic gait content that needs to be relearned in the second training phase.

[0115] Figure 4 and Figure 5 This illustrates different motion states of the lower body motor unit when the upper body motor unit is not spliced ​​together. The first training phase can be conducted in the unspliced ​​upper body motor unit state, based on... Figure 4 or Figure 5 The lower body motor units shown are pre-trained to develop basic motor skills.

[0116] Figure 4 The figure illustrates the independent motion state of a lower body motor unit in a stepped environment. At this point, the upper body motor unit has not yet been spliced ​​together. The training environment can perform network inference, joint movements, and pre-training reward calculations based on the state data of the lower body motor unit, and update the initial motor control policy network based on the pre-training reward to obtain the pre-trained motor control policy network.

[0117] Figure 5 This illustrates a state of the lower body movement unit with a wheeled walking structure in a low-lying, deployed posture. For lower body movement units with different structures, basic movement training can be completed first in the unassembled state, followed by training after the upper and lower body are assembled.

[0118] Figure 6 This diagram illustrates a structural state of the upper body motion unit. The upper body motion unit comprises a torso and two arms, with joints in the arms capable of swinging according to a trained arm swing trajectory. During the training phase, the upper body motion unit itself is not an output object of the lower body motion control strategy network; after the upper and lower bodies are assembled, the mass properties of the upper body and the dynamic effects generated by the arm movements are calculated using the complete robot dynamics model. Figure 6 This only illustrates the movable state of the upper body movement unit and the arms, and does not indicate that the upper and lower body have been assembled.

[0119] Figure 7 This illustrates one state when the upper body motion unit and the lower body motion unit are spliced ​​together. For example... Figure 7As shown, the upper body motion unit is located above the lower body motion unit and moves towards the lower body motion unit in the direction indicated by the arrow in the figure, so that the lower connection position of the upper body motion unit corresponds to the upper connection position of the lower body motion unit. After the position is aligned, the upper body motion unit and the lower body motion unit can be fixedly connected by fasteners to form a complete robot after being spliced ​​together. Figure 7 The arrows in the diagram are used to indicate the splicing direction of the upper and lower body motion units, and do not limit the specific movement method of the upper and lower body motion units during the actual assembly process.

[0120] Figure 8 The diagram shows the robot structure after the upper and lower body motion units are assembled. (Example) Figure 8 As shown, the upper body motion unit is positioned above the lower body motion unit. The upper body motion unit includes the torso and both arms, while the lower body motion unit includes both legs. After assembly, the upper and lower body motion units form a complete robot. The mass of the upper body motion unit constitutes the additional load that the lower body motion unit needs to bear. The dynamic effects generated by the arm movements can also be transmitted to the lower body motion unit through the assembly point. The second training phase can continue to train the motion control strategy network based on this assembled complete robot model.

[0121] Figures 9 to 11 This diagram illustrates several motion states of the robot when performing gait movements in a simulation environment after the upper and lower body motion units are assembled. Taking any one of these motion states as an example, the training environment acquires the current state data of the lower body at that control moment and inputs it into the motion control strategy network to obtain the lower limb joint control commands corresponding to this control cycle. Simultaneously, based on the training arm swing parameters of the current training round, the target position of the upper body joints to be controlled is determined, and the movement of both arms is controlled. After the complete robot dynamics model executes the lower limb and upper limb movements, the robot state at the next control moment is obtained.

[0122] Figures 9 to 11 The changes in the robot's posture correspond to the state of the complete robot in different gait stages. During training, the motion control strategy network outputs lower limb joint control commands based on the current state of the lower body, while the upper body moves according to the training arm swing trajectory of the current training round. If the arm swing causes changes in the robot's posture, speed, or foot contact state, these changes will be included in subsequent state data and training rewards, and will participate in the parameter updates of the lower body motion control strategy network.

[0123] Figures 9 to 11 The motion states in the figures can be taken from different sampling times of the same training task, or from different training rounds. The attached figures are used to illustrate that the assembled complete robot can perform lower body gait training under upper limb movement conditions, and it is not required that adjacent figures correspond to adjacent control cycles.

[0124] During the simulation training process described above, the training environment continuously acquires motion state data of the assembled robot. Different training arm parameters can be set for different training rounds, enabling... Figures 9 to 11 The complete robot shown performs gait movements under different upper limb perturbation conditions.

[0125] Figure 12 The data processing relationship between the training and deployment phases is shown. Figure 12 The upper part represents the training phase: current state data is input into the motion control strategy network to be trained to obtain lower limb joint control commands, which the legs then execute gait movements accordingly; the arm swing trajectory is trained to control the arms to generate arm swing perturbations, and the resulting motion state data and training rewards are returned to the network parameter update process. After training is completed, the trained network parameters are used in the deployment phase. Figure 12 The lower part is the deployment phase: the current state data is input into the trained motion control strategy network to obtain lower limb joint control commands, the legs execute gait movements and generate gait signals; the gait signals are used to generate arm swing trajectories, and further generate joint control signals to control the arms to perform arm swing movements in coordination with the leg gait.

[0126] Figure 13 This embodiment illustrates a robot motion control method, which includes steps S1301 to S1304.

[0127] Step S1301: Obtain the current state data of the lower body motion unit, input the current state data into the motion control strategy network trained by reinforcement learning, and obtain the lower limb joint control commands.

[0128] Step S1302: Control both legs to perform gait movements according to the lower limb joint control commands, and acquire gait signals that characterize the periodic movement state.

[0129] Step S1303: Generate the arm swing trajectory of the upper body motion unit based on the gait signal. The arm swing trajectory is used to represent the relationship between the target position of the joint to be controlled in both arms and the gait progression.

[0130] Step S1304: Generate joint control signals based on the arm swing trajectory, and control the arms to perform arm swing movements in coordination with the leg gait movements based on the joint control signals.

[0131] Steps S1301 to S1304 can be repeated according to the control cycle. After the lower body movement state changes, the motion control strategy network outputs new lower limb joint control commands; the gait signal changes with the progression of gait, and the target position of the upper body joint to be controlled is continuously updated accordingly.

[0132] The current state data and lower limb joint control commands used in the deployment phase can use the same data definitions as in the training phase. Therefore, the trained motion control strategy network can be directly used for the lower body motion control of the assembled robot without having to retrain the network in the deployment phase.

[0133] Figure 14 A further refined approach to the robot motion control process is presented. Step S1401: Acquire current state data; the trained motion control strategy network obtains lower limb joint control commands. Step S1402: Control both legs to perform gait movements according to the lower limb joint control commands and acquire gait phase signals. Step S1403: Determine the gait frequency based on the gait phase signals and set the shoulder joint swing frequency to be the same as the gait frequency. Step S1404: Determine the target joint position of the shoulder joint in each gait stage and generate the shoulder joint swing trajectory.

[0134] exist Figure 14 In the process shown, step S1405 determines the elbow joint swing cycle and the target joint position for each gait stage based on the shoulder joint swing cycle and the preset cycle relationship; step S1406 divides the elbow joint swing cycle into the first half swing cycle and the second half swing cycle, and generates and connects the corresponding trajectories; step S1407 generates the arm swing trajectory according to the preset timing relationship of the shoulder joint starting early and ending late; step S1408 then generates joint control signals based on the arm swing trajectory to control both arms to perform the arm swing action. Figure 14 Therefore, this corresponds to the complete control process of determining the shoulder joint frequency, determining the elbow joint cycle, connecting half cycles, and controlling the start and end timing of the shoulder and elbow joints.

[0135] When gait signals are represented in phase form, a complete gait cycle can be mapped to a preset phase interval, and the target position of each upper limb joint to be controlled can be queried or calculated based on the current phase. When gait signals are obtained from foot contact state or gait timing information, they can be converted into the corresponding gait stage or gait phase first, and then the same trajectory generation process can be performed.

[0136] Gait signals can be directly output by the lower body motion control program, or determined based on lower limb joint status, foot contact status, or gait timing information. This signal indicates the current stage of a complete gait cycle in which the legs are positioned and updates with the actual gait rhythm.

[0137] In one specific implementation, the lower body controller generates the arm swing trajectory of the upper body motion unit in real time based on the gait signals of the lower body motion unit, and generates corresponding joint control signals at a frequency of 500 Hz. 500 Hz corresponds to a 2 ms joint control cycle. In each joint control cycle, the lower body controller determines the target joint position of the arms to be controlled based on the current gait signals, generates joint control signals based on the target joint positions, and sends the joint control signals to the upper body controller. The upper body controller controls the corresponding joint movement based on the received joint control signals.

[0138] 500 Hz is used to illustrate that the update frequency of the upper body joint control signal can be higher than the macroscopic frequency of gait state changes. Even if a gait cycle lasts for hundreds of milliseconds, the lower body controller can still continuously update the target positions of the shoulder and elbow joints at 2 ms intervals within that cycle, so that the discrete control moments determined by the gait phase form a continuous arm swing motion in time.

[0139] A complete gait cycle typically includes multiple 2 ms joint control cycles, thus generating multiple target joint positions consecutively within the same arm swing cycle. As the gait phase changes, the target positions corresponding to shoulder_pitch and elbow_pitch are gradually updated, forming a continuous arm swing trajectory.

[0140] When the lower body gait frequency changes, the rate of change of the gait signal also changes accordingly. The lower body controller redetermines the target joint position at each control moment based on the gait signal, so that the arm swing rhythm is adjusted with the current gait.

[0141] Figure 15 This diagram illustrates the swing trajectories of the shoulder and elbow joints during walking on flat ground. The arm swing primarily controls the shoulder joint pitch (shoulder_pitch) and elbow joint pitch (elbow_pitch) degrees of freedom. The lower body controller, based on the gait signals from the lower body motion units, determines the target joint positions for the shoulder and elbow joint pitch degrees of freedom at the corresponding gait stages, and generates the shoulder and elbow swing trajectories accordingly.

[0142] Figure 15 The horizontal axis is used to represent the process of arm swing over time, while the shoulder joint swing curve and elbow joint swing curve are used to illustrate the periodic changes in the position of the corresponding target joint. Figure 15 It also schematically illustrates the start and end times of the swing of the shoulder and elbow joints, as well as the connection between the first half of the swing cycle, the second half of the swing cycle, and adjacent trajectories during the elbow joint swing process. Figure 15It is mainly used to illustrate the periodic coordination and start-end sequence between the swing trajectories of the shoulder and elbow joints, without limiting the actual time length, number of cycles, or cycle ratio of each trajectory segment in the figure.

[0143] For shoulder_pitch, the gait frequency of the lower body movement unit can be determined based on the gait phase signal, and the swing frequency of shoulder_pitch can be set to be the same as the gait frequency. Based on the gait phase signal, the swing frequency of shoulder_pitch, and the preset shoulder joint swing amplitude, the target joint position of shoulder_pitch corresponding to each gait stage can be determined, and a shoulder joint swing trajectory can be formed in chronological order. When the gait frequency of the lower body movement unit changes, the swing frequency of shoulder_pitch is adjusted accordingly, so that the shoulder joint swing rhythm changes with the lower body gait.

[0144] For the elbow_pitch, the swing period of the elbow joint pitch degree of freedom can be determined based on the swing period of the shoulder joint pitch degree of freedom and a preset period relationship. The preset period relationship characterizes the proportional relationship between the swing period of the elbow joint pitch degree of freedom and the swing period of the shoulder joint pitch degree of freedom; this proportional relationship can be set according to the robot's upper limb structure and arm control requirements. The lower body controller determines the target elbow_pitch joint position corresponding to the appropriate gait stage based on the gait phase signal, the swing period of the elbow joint pitch degree of freedom, and a preset elbow joint swing amplitude.

[0145] In some embodiments, a complete swing cycle of elbow_pitch is divided into a first half swing cycle and a second half swing cycle in chronological order. The first half swing cycle trajectory is formed based on the target elbow joint position within the first half swing cycle, and the second half swing cycle trajectory is formed based on the target elbow joint position within the second half swing cycle. During continuous arm swinging, temporally adjacent second half swing cycle trajectories and first half swing cycle trajectories can be connected to form a continuous elbow swing trajectory. Figure 15 The markings for the first half of the oscillation period, the second half of the oscillation period, and the trajectory connection position are only used to indicate the above trajectory processing relationship and do not indicate that the horizontal length of the corresponding curve segment in the figure is proportional to the actual half-cycle duration.

[0146] The shoulder joint swing trajectory and the elbow joint swing trajectory can also have different swing start and end times. Specifically, the shoulder_pitch starts moving before the elbow_pitch, and continues moving until the end time of the corresponding swing of the shoulder_pitch after the elbow_pitch finishes its swing. Therefore, the swing start time corresponding to the shoulder joint swing trajectory is earlier than the swing start time corresponding to the elbow joint swing trajectory, and the swing end time corresponding to the shoulder joint swing trajectory is later than the swing end time corresponding to the elbow joint swing trajectory.

[0147] During deployment, the controller continuously updates the target joint positions corresponding to shoulder_pitch and elbow_pitch based on the current gait signal, and generates the arm swing trajectory of the upper body motion unit based on the shoulder joint swing trajectory and elbow joint swing trajectory, so as to control the arms to perform arm swinging movements in coordination with the lower body gait.

[0148] Figure 16 A training apparatus for a motion control strategy network according to an embodiment of the present invention is shown. The training apparatus 1600 includes a control command generation module 1601, a disturbance application module 1602, and a training update module 1603.

[0149] Figure 16 In the process, the control command generation module 1601 is used to generate lower limb joint control commands, the disturbance application module 1602 is used to generate arm swing commands and control the upper body motion unit to perform arm swing movements to form training disturbances, and the training update module 1603 calculates training rewards and updates motion control strategy network parameters based on the motion state data after execution.

[0150] The control command generation module 1601 is used to acquire the current state data of the lower body motion unit and input the current state data into the motion control strategy network to be trained to obtain lower limb joint control commands. The disturbance application module 1602 is used to send arm swing commands to the upper body motion unit during the gait movements of the legs, so that the arms perform arm swing movements according to the training arm swing trajectory and form training disturbances.

[0151] The training update module 1603 is used to acquire motion state data under the action of arm swing perturbation, determine training rewards based on the motion state data, and update the motion control policy network parameters based on the training rewards. The specific processing procedures of the above modules correspond to the aforementioned training method.

[0152] By combining the above modules, the training method for the aforementioned motion control strategy network can be realized, and the same or corresponding technical effects as those in the aforementioned training method embodiments can be achieved.

[0153] Figure 17This invention illustrates a robot motion control device according to an embodiment of the present invention. The motion control device 1700 includes a lower limb control module 1701, a gait signal acquisition module 1702, a swing arm trajectory generation module 1703, and a swing arm control module 1704.

[0154] Figure 17 In the process, the lower limb control module 1701 generates lower limb joint control commands based on the current state data; the gait signal acquisition module 1702 acquires the current gait signal of the lower body; the arm swing trajectory generation module 1703 determines the target position of the upper limb joints such as the shoulder and elbow based on the gait signal; and the arm swing control module 1704 outputs joint control signals according to the corresponding control frequency.

[0155] The lower limb control module 1701 is used to obtain lower limb joint control commands based on the current state data of the lower body motion unit and the trained motion control strategy network; the gait signal acquisition module 1702 is used to acquire gait signals during the gait movements of both legs; the arm swing trajectory generation module 1703 is used to generate arm swing trajectory based on gait signals; and the arm swing control module 1704 is used to generate joint control signals based on arm swing trajectory and control the movement of both arms.

[0156] The motion control device 1700 can be implemented by one or more controllers on the robot side. When the one or more controllers include a lower body controller and an upper body controller, the lower limb control module 1701, gait signal acquisition module 1702, and arm swing trajectory generation module 1703 can be implemented by the lower body controller; the arm swing control module 1704 can be implemented by the lower body controller and the upper body controller working together, wherein the lower body controller generates joint control signals based on the arm swing trajectory and sends the joint control signals to the upper body controller, and the upper body controller controls both arms to perform arm swinging movements based on the received joint control signals.

[0157] By combining the above modules, the aforementioned robot motion control method can be realized, and the same or corresponding technical effects as the aforementioned robot motion control method embodiments can be achieved.

[0158] Figure 18 An electronic device 1800 according to an embodiment of the present invention is shown. The electronic device 1800 includes a processor 1801 and a memory 1802, the memory 1802 storing a computer program. In some embodiments, the memory 1802 may also store parameters of a trained motion control strategy network. When the processor 1801 executes the computer program, it implements a method for training the aforementioned motion control strategy network or a robot motion control method.

[0159] The electronic device 1800 can serve as a computing device during the training phase or as a robot controller during the deployment phase. Depending on the specific needs, the electronic device can also be equipped with a communication interface for data exchange with the training environment, upper body motion units, lower body motion units, or joint controllers.

[0160] When the processor 1801 executes the computer program, it can implement the corresponding processing procedures in the foregoing method embodiments, and the technical effects can be found in the foregoing method embodiments.

[0161] Figure 19 A robot 1900 according to an embodiment of the present invention is shown. The robot 1900 includes a lower body motion unit 1901, an upper body motion unit 1902, and one or more controllers. The lower body motion unit 1901 includes two legs, and the upper body motion unit 1902 includes two arms; the upper body motion unit 1902 is connected to the lower body motion unit 1901 vertically. Figure 19 In the illustrated embodiment, the one or more controllers include a lower body controller 1903 and an upper body controller 1904. The lower body controller 1903 is disposed in the lower body motion unit 1901, and the upper body controller 1904 is disposed in the upper body motion unit 1902. The lower body controller 1903 and the upper body controller 1904 are communicatively connected.

[0162] In one implementation, the lower body controller 1903 acquires the current state data of the lower body motion unit 1901 and inputs the current state data into a motion control strategy network trained through reinforcement learning to obtain lower limb joint control commands. Based on these commands, the controller controls both legs to perform gait movements. During the gait movements, the lower body controller 1903 acquires the gait signals of the lower body motion unit 1901, generates the arm swing trajectory of the upper body motion unit 1902 based on the gait signals, generates joint control signals based on the arm swing trajectory, and sends the joint control signals to the upper body controller 1904. The upper body controller 1904 then controls both arms to perform arm swing movements based on the received joint control signals.

[0163] In some embodiments, the upper body motion unit is positioned above the lower body motion unit and has two arms. The lower body motion unit is responsible for overall robot support and gait movements. After the upper and lower body motion units are assembled, they form a complete robot. The specific assembly position and connection structure can be set according to the robot's body structure.

[0164] When the upper body motion unit is positioned above the lower body motion unit, its mass and the dynamic effects generated by the arm movements are transmitted to the lower body through the connection point. During training, a spliced, integrated robot model can be used, allowing the upper body's mounting position, mass attributes, and arm movements to all participate in robot dynamics calculations.

[0165] During deployment, the lower body controller 1903 can generate a target position sequence of the joints to be controlled in both arms based on the gait signal of the lower body motion unit 1901, and generate joint control signals based on the target position sequence, and send the joint control signals to the upper body controller 1904; the upper body controller 1904 controls the corresponding joint drive mechanism in the upper body motion unit 1902 according to the received joint control signals, so that the arms perform arm swinging movements.

[0166] During the deployment phase, the shoulder and elbow pitches of both arms can be primarily controlled. The lower body controller 1903 generates target position sequences for the two joints based on the gait signals from the lower body motion unit 1901, and generates corresponding joint control signals based on these target position sequences. The upper body controller 1904 controls the corresponding joint drive mechanisms to execute based on the received joint control signals. The upper body motion unit may also include other joint degrees of freedom, which can maintain preset postures or be controlled separately according to the overall movement requirements.

[0167] The upper body motion unit and the lower body motion unit can be fixedly connected by fasteners. After assembly, the two motion units form a stable mechanical connection, and the mechanical force generated by the upper body movement can be transmitted to the lower body through the connection point. The lower body controller 1903 and the upper body controller 1904 can transmit joint control signals or other control information via wired or wireless communication.

[0168] The lower body controller 1903 controls the movement of the legs and generates joint control signals based on the gait signals of the lower body motion unit 1901. The joint control signals are then sent to the upper body controller 1904, which controls the arms to perform arm swinging movements. This achieves the aforementioned robot motion control method, and its technical effects can be found in the aforementioned robot motion control method embodiment.

[0169] Figure 20 A computer-readable storage medium 2000 according to an embodiment of the present invention is shown, wherein a computer program is stored. When executed by a processor, the program can implement the aforementioned motion control policy network training method or robot motion control method, and can also be used to load the parameters of the trained motion control policy network and perform network inference.

[0170] Computer-readable storage media 2000 can take the form of non-transient storage, such as semiconductor memory, solid-state memory, magnetic storage media or optical storage media, and can also be used as part of the memory of electronic devices.

[0171] When a computer program is executed by a processor, it can implement the corresponding processing procedures in the aforementioned method embodiments, and its technical effects can be found in the aforementioned method embodiments.

[0172] The above implementation method is illustrated using upper body arm swing control during flat walking as an example. Parameters such as swing amplitude, swing frequency, initial swing phase, reward threshold, and time window during the training phase, as well as shoulder-elbow cycle relationship, swing amplitude, and joint control signal frequency during the deployment phase, can be set according to the robot structure, gait, and control requirements.

[0173] The above embodiments are used to illustrate the technical solution of the present invention. Those skilled in the art can make corresponding adjustments to the form of state data, trajectory generation method, training parameters, and implementation method of each functional module according to the actual robot structure and control system.

Claims

1. A training method for a motion control policy network for a robot based on reinforcement learning, characterized in that, The robot includes a lower body motion unit and an upper body motion unit. The lower body motion unit includes two legs, and the upper body motion unit includes two arms. The lower body motion unit and the upper body motion unit are spliced ​​together. The method includes: The current state data of the lower body motion unit is obtained and input into the motion control strategy network to be trained to obtain lower limb joint control commands, which are used to control the legs to perform gait movements. During the process of controlling the legs to perform the gait movement according to the lower limb joint control command, the arm swing command is sent to the upper body motion unit so that the arms perform the arm swing movement. The perturbation generated by the arm swing movement is the perturbation term for training the motion control strategy network. The motion state data of the legs after performing the gait action under the disturbance generated by the arm swing is obtained. The training reward is determined based on the motion state data, and the parameters of the motion control strategy network are updated based on the training reward. The training reward is used to characterize the walking stability of the lower body movement unit.

2. The training method for the motion control policy network of a robot based on reinforcement learning according to claim 1, characterized in that, During the process of controlling the legs to perform the gait movement according to the lower limb joint control command, sending the arm swing command to the upper body motion unit to cause the arms to perform the arm swing movement includes: Obtain preset training arm parameters, which include swing amplitude, swing frequency, and initial swing phase; Based on the swing frequency and the initial swing phase, the swing phase of the two arms at each control moment during the training process is determined. The swing phase is used to characterize the swing stage of the two arms at the corresponding control moment in one swing cycle. Based on the phase of each swing arm and the swing amplitude, the target joint position of the two arms at the corresponding control time is determined, and a training swing arm trajectory is generated based on the target joint position corresponding to each control time. The training swing arm trajectory is used to characterize the relationship between the target joint position of the two arms and time. An arm swing command is generated based on the training arm swing trajectory, and the arm swing command is sent to the upper body motion unit during the gait movement of the legs, so that the arms perform the arm swing movement according to the training arm swing trajectory.

3. The training method for the motion control policy network of a robot based on reinforcement learning according to claim 2, characterized in that, The method further includes: In multiple training rounds, the training arm parameters corresponding to each training round are determined respectively, and the training arm parameters corresponding to different training rounds are different in at least one of the swing amplitude, the swing frequency and the swing initial phase; Based on the training arm swing parameters corresponding to each training round, generate the training arm swing trajectory for the corresponding training round; In each training round, the arms are controlled to perform arm swinging movements according to the corresponding training arm swinging trajectory, so that the arm swinging movements in different training rounds produce different perturbations. The motion state data of the two legs after performing the gait action under the corresponding disturbance in each training round is obtained. The corresponding training reward is determined according to the motion state data of each training round, and the parameters of the same motion control strategy network are iteratively updated based on each training reward.

4. The training method for the motion control policy network of a robot based on reinforcement learning according to claim 3, characterized in that, The step of determining the training arm parameters for each training round in multiple training rounds includes: In multiple training rounds, obtain the training rewards corresponding to a preset number of completed training rounds, and determine the average training reward based on the training rewards corresponding to the preset number of completed training rounds. If the average training reward is greater than or equal to the preset reward threshold, then the walking stability of the lower body motor unit is determined to have reached the preset stability level, and the swing amplitude corresponding to the subsequent training rounds is increased within the preset swing amplitude range. Based on the increased swing amplitude and the swing frequency and initial phase of the swing corresponding to the subsequent training rounds, the training arm parameters corresponding to the subsequent training rounds are determined.

5. The training method for the motion control policy network of a robot based on reinforcement learning according to claim 1, characterized in that, The method further includes: During the reference training round, a hold command is sent to the upper body motion unit to keep the arms in a preset reference posture, and reference motion state data of the legs after performing the gait action is obtained. The preset reference posture is the posture in which the arms remain unchanged and do not produce arm swinging movements during the reference training round. A reference stability index is determined based on the reference motion state data. The reference stability index is used to characterize the walking stability of the lower body movement unit when no disturbance caused by the arm swinging action is applied. The step of determining the training reward based on the motion state data includes: The disturbance stability index is determined based on the motion state data. The disturbance stability index is used to characterize the walking stability of the lower body motion unit under the disturbance generated by the arm swinging motion. The training reward is determined based on the difference between the reference stability index and the perturbation stability index.

6. The training method for the motion control policy network of a robot based on reinforcement learning according to claim 1, characterized in that, The step of acquiring motion state data of the legs after performing the gait movement under the disturbance generated by the arm swing motion, and determining training rewards based on the motion state data, includes: During the process of the arms performing the arm swinging action and the legs performing the gait action, the motion state data of the lower body motion unit at multiple acquisition times are acquired within a preset time window, and the motion state data sequence is formed according to the order of acquisition times. Based on the changes in motion state data at different acquisition times in the motion state data sequence, the degree of motion state fluctuation is determined. The degree of motion state fluctuation is used to characterize the walking stability of the lower body movement unit when performing the gait action within the preset time window. Training rewards are determined based on the degree of fluctuation in the described motion state.

7. The training method for the motion control policy network of a robot based on reinforcement learning according to claim 1, characterized in that, The method further includes: The mass attribute data of the upper body motion unit and the lower body motion unit are obtained, as well as the splicing pose relationship between the upper body motion unit and the lower body motion unit. The mass attribute data includes the mass, center of mass position and moment of inertia of the corresponding motion unit. The splicing pose relationship is used to characterize the relative position and relative posture of the upper body motion unit and the lower body motion unit after splicing. Based on the mass attribute data of the upper body motion unit and the lower body motion unit, as well as the splicing pose relationship, the overall mass attribute data of the spliced ​​robot is determined. The overall mass attribute data includes the total mass, overall center of mass position, and overall inertia of the spliced ​​robot. Based on the overall quality attribute data, the dynamic parameters of the robot model used to characterize the spliced ​​robot in the reinforcement learning training environment are configured; In the reinforcement learning training environment, the motion control policy network is trained based on the robot model configured with the dynamic parameters.

8. The training method for the motion control policy network of a robot based on reinforcement learning according to claim 1, characterized in that, Before training the motion control policy network to be trained on the robot based on the spliced ​​upper body motion unit and the lower body motion unit, the method further includes: In the first training phase before the upper body movement unit and the lower body movement unit are spliced ​​together, the state data of the lower body movement unit is acquired and the state data is input into the initial motion control strategy network to obtain lower limb joint control commands. The lower limb joint control commands control the legs to perform gait movements, and the pre-training reward is determined based on the motion state data of the legs after performing the gait movements. The pre-training reward is used to characterize the walking stability of the lower body motor unit when it is not spliced ​​with the upper body motor unit. The parameters of the initial motion control policy network are updated based on the pre-training rewards to obtain the pre-trained motion control policy network; After completing the first training phase, the upper body motion unit and the lower body motion unit are spliced ​​together, the motion control strategy network to be trained is initialized based on the parameters of the pre-trained motion control strategy network, and the motion control strategy network to be trained continues to be trained based on the perturbation generated by the arm swinging motion.

9. A motion control method for a robot based on reinforcement learning, characterized in that, The robot includes a lower body motion unit and an upper body motion unit. The lower body motion unit includes two legs, and the upper body motion unit includes two arms. The lower body motion unit and the upper body motion unit are spliced ​​together. The method includes: The current state data of the lower body motion unit is obtained, and the current state data is input into the motion control strategy network trained by reinforcement learning to obtain lower limb joint control commands; The lower limb joint control command controls the legs to perform gait movements, and during the gait movements, the gait signal of the lower body motion unit is acquired. The gait signal is used to characterize the periodic motion state of the legs when performing the gait movements. The upper body motion unit's arm swing trajectory is generated based on the gait signal. The arm swing trajectory is used to characterize the relationship between the target position of the joint to be controlled in the two arms and the gait signal. The joint control signal is generated based on the arm swing trajectory, and the arms are controlled to perform arm swinging movements based on the joint control signal, so that the arm swinging movements are coordinated with the gait movements of the legs.

10. The motion control method for a robot based on reinforcement learning according to claim 9, characterized in that, The gait signal includes a gait phase signal, which characterizes the gait phase of the legs in the current gait cycle; generating the arm swing trajectory of the upper body movement unit based on the gait signal includes: Based on the gait phase signal, the target joint positions corresponding to the shoulder joint pitch degree of freedom and the elbow joint pitch degree of freedom in each gait phase are determined respectively. Based on the target joint position corresponding to the shoulder joint pitch degree of freedom in each gait stage, a shoulder joint swing trajectory is generated, and based on the target joint position corresponding to the elbow joint pitch degree of freedom in each gait stage, an elbow joint swing trajectory is generated. An arm swing trajectory is generated based on the shoulder joint swing trajectory and the elbow joint swing trajectory.

11. The motion control method for a robot based on reinforcement learning according to claim 10, characterized in that, The process of determining the target joint position corresponding to the shoulder joint pitch degree of freedom in each gait phase includes: The gait frequency of the lower body movement unit is determined based on the change in the gait phase signal. The gait frequency is the frequency at which the two legs repeat a complete gait cycle. The swing frequency of the shoulder joint pitch freedom is determined to be the same as the gait frequency; Based on the gait phase signal, the swing frequency of the shoulder joint pitch degree of freedom, and the preset shoulder joint swing amplitude, the target joint position corresponding to the shoulder joint pitch degree of freedom in each gait phase is determined.

12. The motion control method for a robot based on reinforcement learning according to claim 10, characterized in that, The process of determining the target joint position corresponding to the elbow joint pitch degree of freedom in each gait phase includes: The swing period of the shoulder joint pitch degree of freedom is determined based on the swing trajectory of the shoulder joint. Based on the swing period of the shoulder joint pitch degree of freedom and the preset period relationship, the swing period of the elbow joint pitch degree of freedom is determined. The preset period relationship is used to characterize the proportional relationship between the swing period of the elbow joint pitch degree of freedom and the swing period of the shoulder joint pitch degree of freedom. Based on the gait phase signal, the swing period of the elbow joint pitch degree of freedom, and the preset elbow joint swing amplitude, the target joint position corresponding to the elbow joint pitch degree of freedom in each gait phase is determined.

13. The motion control method for a robot based on reinforcement learning according to claim 12, characterized in that, The step of generating the elbow joint swing trajectory based on the target joint position corresponding to the elbow joint pitch degree of freedom in each gait phase includes: The complete swing cycle of the elbow joint pitch freedom is divided into the first half swing cycle and the second half swing cycle according to the time sequence. The first half-cycle trajectory is generated based on the target elbow joint position within the first half-swing cycle, and the second half-cycle trajectory is generated based on the target elbow joint position within the second half-swing cycle. By connecting the second half-cycle trajectory and the first half-cycle trajectory, the elbow joint swing trajectory is obtained.

14. The motion control method for a robot based on reinforcement learning according to claim 10, characterized in that, The step of generating the arm swing trajectory based on the shoulder joint swing trajectory and the elbow joint swing trajectory includes: The start and end times of the swing of the shoulder joint and the elbow joint are determined according to a preset timing relationship. The start time of the swing of the shoulder joint is earlier than the start time of the swing of the elbow joint, and the end time of the swing of the shoulder joint is later than the end time of the swing of the elbow joint. Based on the shoulder joint swing trajectory and the elbow joint swing trajectory after determining the start and end times of the swing, the arm swing trajectory is generated.

15. A training device for a motion control policy network for a robot based on reinforcement learning, characterized in that, The robot includes a lower body motion unit and an upper body motion unit. The lower body motion unit includes two legs, and the upper body motion unit includes two arms. The lower body motion unit and the upper body motion unit are spliced ​​together. The training device includes: The control command generation module is used to acquire the current state data of the lower body motion unit and input the current state data into the motion control strategy network to be trained to obtain lower limb joint control commands, which are used to control the legs to perform gait movements. The perturbation application module is used to send an arm swing command to the upper body motion unit during the process of controlling the two legs to perform the gait action according to the lower limb joint control command, so that the two arms perform an arm swing action, and the perturbation generated by the arm swing action is a perturbation term for training the motion control strategy network. The training update module is used to acquire motion state data of the legs after performing the gait action under the disturbance generated by the arm swing action, determine the training reward based on the motion state data, and update the parameters of the motion control strategy network based on the training reward. The training reward is used to characterize the walking stability of the lower body movement unit.

16. A motion control device for a robot based on reinforcement learning, characterized in that, The robot includes a lower body motion unit and an upper body motion unit. The lower body motion unit includes two legs, and the upper body motion unit includes two arms. The lower body motion unit and the upper body motion unit are spliced ​​together. The motion control device includes: The lower limb control module is used to acquire the current state data of the lower body motion unit and input the current state data into the motion control strategy network trained by reinforcement learning to obtain lower limb joint control commands. The gait signal acquisition module is used to control the legs to perform gait movements according to the lower limb joint control command, and to acquire the gait signal of the lower body movement unit during the gait movements performed by the legs. The gait signal is used to characterize the periodic movement state of the legs when performing the gait movements. The arm swing trajectory generation module is used to generate the arm swing trajectory of the upper body motion unit based on the gait signal. The arm swing trajectory is used to characterize the relationship between the target position of the joint to be controlled in the two arms and the gait signal. The arm swing control module is used to generate joint control signals based on the arm swing trajectory, and control the arms to perform arm swing movements based on the joint control signals, so that the arm swing movements are coordinated with the gait movements of the legs.

17. An electronic device, characterized in that, include: processor; as well as The memory stores computer-readable instructions, which, when executed by the processor, implement the training method for the motion control policy network of a robot based on reinforcement learning as described in any one of claims 1 to 8, or implement the motion control method for a robot based on reinforcement learning as described in any one of claims 9 to 14.

18. A robot, characterized in that, include: A lower body movement unit, comprising both legs; An upper body movement unit, comprising two arms, and the upper body movement unit being spliced ​​with the lower body movement unit; One or more controllers, the one or more controllers being configured to implement the motion control method for a robot based on reinforcement learning as described in any one of claims 9 to 14.

19. The robot according to claim 18, characterized in that, The one or more controllers include a lower body controller and an upper body controller. The lower body controller is disposed in the lower body motion unit, and the upper body controller is disposed in the upper body motion unit. The lower body controller and the upper body controller are communicatively connected. The lower body controller is used to control the legs to perform gait movements, and to acquire the gait signal of the lower body motion unit during the gait movements of the legs, generate the arm swing trajectory of the upper body motion unit based on the gait signal, generate joint control signals based on the arm swing trajectory, and send the joint control signals to the upper body controller. The upper body controller is used to control the arms to perform arm swinging movements according to the joint control signals.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the training method for the motion control policy network of a robot based on reinforcement learning as described in any one of claims 1 to 8, or implements the motion control method for a robot based on reinforcement learning as described in any one of claims 9 to 14.

Images