Operation control model training method, robot operation control method and system

Abstract

This invention provides a method for training a motion control model, a robot motion control method, and a system. The method includes: training an initial policy network using reinforcement learning based on the first sample state information of the simulated robot and the reward function corresponding to each action task, to obtain a target policy network corresponding to each action task; training the initial motion control model using transfer learning, reinforcement learning, and imitation learning in conjunction with the second sample state information of the simulated robot, the task label corresponding to the preset training task, and the expected action information corresponding to the second sample state information output by the target policy network corresponding to each action task; and constructing a motion control model based on the trained initial motion control model. This invention achieves multi-task compatibility processing of the robot through joint training of multiple training tasks, while ensuring smooth switching between different action tasks, effectively improving the performance of robot motion control.

Description

Technical Field

[0001] This invention relates to the field of automation control technology, and in particular to a method for training an operation control model, a method for controlling robot operation, and a system. Background Technology

[0002] Humanoid robots possess high versatility, capable of mimicking human movements and performing a wide range of operations through operational control. This makes them adept at autonomously executing tasks in various complex scenarios. Therefore, how to effectively control robot operation is an important research topic urgently needing to be addressed in this field.

[0003] In related technologies, extensive robot simulation training is typically performed in a simulator. This involves designing and debugging numerous reward value functions to train a specific model, enabling the robot to perform a specific action based on that model. However, this method usually trains a specific model for a specific task to learn a specific action. The actions generated by a specific model can only be used for that specific task, leading to sudden state changes when switching between different models, resulting in unstable actions and reduced stability of robot operation and control. Summary of the Invention

[0004] This invention provides a method for training a running control model, a method for controlling robot operation, and a system to address the shortcomings of existing technologies where different models are prone to sudden state changes during switching, resulting in poor stability of robot operation control, thereby improving the stability of robot operation control.

[0005] This invention provides a method for training a runtime control model, comprising:

[0006] Based on the first sample state information of the simulated robot and the reward function corresponding to each action task, the initial policy network is trained by reinforcement learning to obtain the target policy network corresponding to each action task.

[0007] Based on the second sample state information of the simulated robot, the task label corresponding to the preset training task, and the expected action information corresponding to the second sample state information output by the target policy network corresponding to each action task, the initial operation control model is jointly trained by transfer learning, reinforcement learning training and imitation learning.

[0008] Based on the initial operational control model after training, an operational control model is constructed. The operational control model is used to control the physical robot to perform different motion control actions under different action tasks.

[0009] The sample status information in the first sample status information and the second sample status information each includes sample body information and sample speed command.

[0010] According to the present invention, a method for training a running control model is provided, wherein the action task includes a standing task or a walking task;

[0011] The step of training the initial policy network through reinforcement learning based on the first sample state information of the simulated robot and the reward function corresponding to each action task to obtain the target policy network corresponding to each action task includes:

[0012] Based on the first sample state information and the reward function corresponding to the standing task, the initial policy network is trained by reinforcement learning to obtain the target policy network corresponding to the standing task;

[0013] Based on the first sample state information and the reward function corresponding to the walking task, the initial policy network is trained by reinforcement learning to obtain the target policy network corresponding to the walking task;

[0014] The reward function for the walking task includes a first reward function and a second reward function, and the reward function for the standing task includes the first reward function. The first reward function is a reward function constructed based on the motion parameters of the simulated robot, and the second reward function is a reward function constructed based on the environmental interaction information of the simulated robot.

[0015] According to the present invention, a method for training a running control model is provided, wherein the first reward function includes at least one of an energy consumption penalty function, a running posture constraint function, a speed tracking reward function, a joint motion penalty function, a fuselage motion penalty function, a motion smoothing function, and a fall penalty function;

[0016] The energy consumption penalty function includes a function for constraining the energy consumption generated by joint movements; the running posture constraint function includes a function for constraining the alignment of the feet, knees, and hips with the fuselage orientation and / or optimizing the upright state of the leg joints; the velocity tracking reward function includes a function for constraining the command tracking performance corresponding to linear velocity and / or angular velocity; the joint motion penalty function includes a function for constraining at least one joint motion parameter among joint acceleration, joint torque, joint limit position, and joint limit force; and the fuselage motion penalty function includes a function for constraining the fuselage attitude and / or fuselage motion state.

[0017] According to the present invention, a method for training a running control model is provided, wherein the second reward function includes a path tracking reward function and / or a ground contact force penalty function;

[0018] The path tracking reward function includes a function for constraining the path tracking performance of the simulated robot; the ground contact force penalty function is a function for constraining the contact pressure between the simulated robot and the ground.

[0019] According to the present invention, a method for training a running control model is provided, wherein the initial running control model is jointly trained by transfer learning, reinforcement learning, and imitation learning based on the second sample state information of the simulated robot, the task label corresponding to the preset training task, and the expected action information corresponding to the second sample state information output by the target policy network corresponding to each action task, including:

[0020] The initial operation control model is constructed by selecting the target policy network corresponding to the target action task from the target policy networks corresponding to the multiple action tasks.

[0021] The initial operation control model is trained by transfer learning to obtain the operation control model to be trained;

[0022] The second sample state information and the task label are input into the running control model to be trained to obtain the predicted action information corresponding to the task label;

[0023] Based on the task label, the target action information corresponding to the preset training task is obtained from the expected action information corresponding to the second sample state information output by the target policy network corresponding to the multiple action tasks.

[0024] Based on the predicted action information and the target action information, as well as the reward function corresponding to the preset training task, a target loss function is constructed.

[0025] Based on the target loss function, the training model to be trained is subjected to joint training of reinforcement learning and imitation learning.

[0026] According to the present invention, a method for training a runtime control model is provided, wherein constructing a target loss function based on the predicted action information, the target action information, and the reward function corresponding to the preset training task includes:

[0027] The imitation learning loss function is determined based on the probability distribution difference between the predicted action information and the target action information;

[0028] The reinforcement learning loss function is determined based on the predicted action information and the reward function corresponding to the preset training task;

[0029] The target loss function is obtained by weighted summing of the imitation learning loss function and the reinforcement learning loss function;

[0030] The weight coefficients between the imitation learning loss function and the reinforcement learning loss function are dynamically updated based on the number of iterations during the iterative training process.

[0031] This invention provides a method for training an operation control model, wherein constructing an operation control model based on a trained initial operation control model includes:

[0032] At least one disturbance control mechanism is added to the simulated robot;

[0033] Obtain the third sample state information of the simulated robot under each of the disturbance controls, and perform reinforcement learning training on the trained initial operation control model based on each of the third sample state information and the task label corresponding to the preset training task to obtain the operation control model;

[0034] The disturbance control includes disturbance control of physical parameters and / or disturbance control of environmental parameters.

[0035] The present invention also provides a robot operation control method, comprising:

[0036] Obtain the current status information and task tag of the current task of the physical robot; the current status information includes the current body information and the current speed command;

[0037] The current status information and the task label of the current task are input into the operation control model to obtain the current action information corresponding to the task label of the current task;

[0038] The torque of the physical robot is obtained by calculating the torque based on the current motion information.

[0039] Based on the joint torque, the physical robot is controlled to perform motion control actions under the current task;

[0040] The operation control model is trained based on the operation control model training method described in any of the above-mentioned methods.

[0041] The present invention also provides a system for training a runtime control model, comprising:

[0042] The first training unit is used to perform reinforcement learning training on the initial policy network based on the first sample state information of the simulated robot and the reward function corresponding to each action task, so as to obtain the target policy network corresponding to each action task.

[0043] The second training unit is used to perform joint training of transfer learning, reinforcement learning training and imitation learning on the initial operation control model based on the second sample state information of the simulated robot, the task label corresponding to the preset training task, and the expected action information corresponding to the second sample state information output by the target policy network corresponding to each action task.

[0044] The model building unit is used to build an operation control model based on the trained initial operation control model. The operation control model is used to control the physical robot to perform different motion control actions under different action tasks.

[0045] The sample status information in the first sample status information and the second sample status information each includes sample body information and sample speed command.

[0046] The present invention also provides a robot operation control system, comprising:

[0047] The acquisition unit is used to acquire the current state information and task tag of the current task of the physical robot; the current state information includes the current body information and the current speed command;

[0048] The prediction unit is used to input the current state information and the task label of the current task into the operation control model to obtain the current action information corresponding to the task label of the current task;

[0049] A calculation unit is used to calculate the torque of the current motion information to obtain the joint torque of the physical robot;

[0050] The control unit is used to control the physical robot to perform motion control actions under the current task based on the joint torque.

[0051] The operation control model is trained based on the operation control model training method described in any of the above-mentioned methods.

[0052] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the operation control model training method as described above, or the robot operation control method.

[0053] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the operation control model training method as described above, or the robot operation control method.

[0054] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the operation control model training method or the robot operation control method described above.

[0055] The operation control model training method, robot operation control method, and system provided by this invention first train an initial policy network using reinforcement learning based on the first sample state information of the simulated robot and the reward function corresponding to each action task, thereby obtaining the target policy network corresponding to each action task. Then, based on the second sample state information of the simulated robot, the task label corresponding to the preset training task, and the expected action information corresponding to the second sample state information output by the target policy network corresponding to each action task, the initial operation control model is jointly trained using transfer learning, reinforcement learning, and imitation learning to construct an operation control model with an integrated policy network compatible with multi-action task processing. This enables the robot to handle multiple tasks simultaneously and achieve smooth switching between different action tasks, effectively improving the robot's stable multi-gait walking performance and saving the robot's energy consumption, thereby ensuring the performance of robot operation control. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0057] Figure 1 This is one of the flowcharts illustrating the operation control model training method provided by the present invention.

[0058] Figure 2 This is a structural schematic diagram of the humanoid robot provided by the present invention.

[0059] Figure 3 This is the second flowchart of the operation control model training method provided by the present invention.

[0060] Figure 4 This is one of the flowcharts illustrating the robot operation control method provided by the present invention.

[0061] Figure 5 This is the second flowchart of the robot operation control method provided by the present invention.

[0062] Figure 6 This is a schematic diagram of the structure of the operation control model training system provided by the present invention.

[0063] Figure 7 This is a schematic diagram of the robot operation control system provided by the present invention.

[0064] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0066] The operation control model training method, robot operation control method and system provided in this application can be applied to various robot control scenarios, such as humanoid robots (also known as humanoid robots). The following description uses a humanoid robot as an example to elaborate on the operation control model training method, robot operation control method and system provided in this application.

[0067] Humanoid robots possess high versatility, capable of mimicking human movements and performing a wide range of operations, making them adept at autonomously executing tasks in various complex scenarios. However, the structure of humanoid robots and the environments they face are typically very complex, and controlling them remains a significant challenge.

[0068] Early humanoid robot control algorithms primarily relied on classical motion control algorithms. Mainstream classical motion control algorithms include Zero Moment Point (ZMP), Model Predictive Control (MPC), and Whole Body Control (WBC). Because humanoid robots have higher degrees of freedom and more complex dynamics, classical motion control algorithms often simplify the mathematical models when modeling and solving the dynamics and kinematics of humanoid robots to simplify computation. This limits the algorithm's performance, resulting in stable walking in specific environments, such as specific terrains and actions, but insufficient robustness. This leads to significant challenges in adapting to new environments and tasks, specifically instability on unstructured terrain.

[0069] With the rapid development of deep learning, especially deep reinforcement learning, and the implementation of numerous simulators, robot motion control technology based on deep reinforcement learning has developed rapidly and is gradually becoming the mainstream algorithm for humanoid robot control. Motion control algorithms based on reinforcement learning typically undergo extensive robot simulation training in a simulator. By designing and debugging numerous reward value functions, the robot is made to perform a specific action. Finally, the converged model is deployed to a real machine for operation.

[0070] A representative study in this field is the application of Deep Reinforcement Learning (DRL) to the bipedal robot platform Cassie. The movement of a bipedal robot can be viewed as a feedback control problem, which can be solved by finding the optimal policy through a Markov decision process. By combining reinforcement learning with supervised learning with a limited sample size, a robust walking controller can be effectively learned in a simulated environment. Domain randomization techniques are added during training to reduce the gap between simulation and real-world operation. Aesthetically pleasing humanoid gait is achieved by integrating periodic gaits. Compared to Cassie's bipedal structure, when a humanoid robot has upper limbs, its body weight increases, its center of gravity shifts, and the motion control of a fully humanoid robot becomes more complex, leading to poor adaptability of the control algorithm.

[0071] Furthermore, motion control algorithms based on reinforcement learning exhibit stronger robustness. However, this technique typically involves extensive robot simulation training in a simulator. It involves designing and debugging numerous reward value functions to train a specific model, enabling the robot to perform a specific action based on that model. This approach usually trains a specific model for a specific task to learn a specific action. The actions generated by this specific model can only be used for that specific task. For example, to enable a robot to walk and stand, separate walking and standing models need to be trained. Furthermore, the walking model can only perform walking control, and the standing model can only perform standing control. This leads to sudden state changes during the transition between different models, resulting in unstable actions and reduced stability of the robot's operational control.

[0072] To address the shortcomings of existing technologies, this application proposes a training method for a motion control model. This method is a multi-task deep reinforcement learning approach based on deep reinforcement learning, imitation learning, and transfer learning. First, a straight-leg walking model and a straight-leg standing model are trained separately, and a straight-leg movement is achieved through reward function design. Then, combining transfer learning, reinforcement learning, and imitation learning methods, the actions output by a well-trained single-task policy network are used as expert data. By introducing imitation learning loss constraints on top of the deep reinforcement learning loss function, joint training of the multi-task policy network using transfer learning, deep reinforcement learning, and imitation learning is achieved. This results in an integrated policy network, i.e., a motion control model, that can handle both walking and standing tasks. This enables the robot to perform both straight-leg walking and standing tasks simultaneously, allowing for smooth switching between walking and standing, improving the stability of robot motion control, saving robot energy consumption, and thus ensuring the performance of robot motion control.

[0073] Figure 1This is one of the flowcharts of the operation control model training method provided by the present invention. The execution subject of this method can be an operation controller with machine learning model training and application, so as to realize the robot's standing, walking and other operation control through machine learning model training and application.

[0074] like Figure 1 As shown, the method includes steps 110, 120 and 130.

[0075] Step 110: Based on the first sample state information of the simulated robot and the reward function corresponding to each action task, perform reinforcement learning training on the initial policy network to obtain the target policy network corresponding to each action task.

[0076] The simulated robot here can be a humanoid machine model built in a simulation environment.

[0077] Figure 2 This is a structural schematic diagram of the humanoid robot provided by the present invention. Figure 2 As shown, the humanoid robot has a total of 51 degrees of freedom (denoted as q), with 6 degrees of freedom for each leg, 1 degree of freedom for the waist, 7 degrees of freedom for each arm, and 12 degrees of freedom for each hand. Three cameras are mounted on the robot body for performing tasks related to localization, grasping, and walking. The height and weight of the humanoid robot can be set according to actual needs, such as setting the height to 176 cm and the overall weight to 64 kg based on the average adult height. The parameters of the joint units can also be set according to actual needs, such as setting the maximum torque of the joint units to approximately 360 N·m at the knee joint, 220 N·m at the hip joint, and 45 N·m at the ankle joint. For the arms, the torque ranges from 20 N·m to 120 N·m depending on the specific joint.

[0078] Figure 3 This is the second flowchart illustrating the operation control model training method provided by this invention. For example... Figure 3 As shown, during the training of the operational control model, skill learning training for each action task can be performed first to obtain the target policy network corresponding to each action task. The specific implementation steps include:

[0079] First, obtain the initial sample state information of the simulated robot in the simulation environment.

[0080] The first sample state information here includes sample ontology information and sample velocity command. The sample ontology information is the body perception information of the simulated robot, which can be obtained through simulation in a simulated environment based on simulated sensors in the simulated robot, such as simulated joint encoders and simulated inertial measurement units. This includes, but is not limited to, the position, orientation, linear velocity, and angular velocity of the pelvis, as well as the position, joint velocity, body posture, and desired position of all joints in the robot body, which is not specifically limited in this embodiment. The sample velocity command refers to the velocity command formed by simulated user input, which includes the linear velocity information and angular velocity information desired by the user. The linear velocity information includes the linear velocity relative to each coordinate axis in the robot body coordinate system, and the angular velocity information includes the turning velocity relative to the robot body coordinate system.

[0081] Here, reward functions for each action task are also needed. Since the task labels for different action tasks are pre-associated with different reward functions—for example, the reward function for the straight-leg standing action task includes, but is not limited to, the reward function constructed from the motion parameters of the simulated robot; the reward function for the straight-leg walking task includes, but is not limited to, the reward function constructed from the motion parameters of the simulated robot and the reward function constructed from environmental interaction information—the reward function for each action task can be obtained by associating it with the task labels of each action task.

[0082] The action task here refers to the action task of controlling the robot to move, which can be a task of standing with straight legs, walking with straight legs, squatting, etc. This embodiment does not specifically limit it. For the sake of simplicity, the following description will take the action task including the task of standing with straight legs or walking with straight legs as an example.

[0083] Subsequently, based on the first sample state information of the simulated robot and the reward function corresponding to each action task, deep reinforcement learning can be used to train the initial policy network separately to obtain the target policy network corresponding to each action task. The target policy network can accurately perform individual operation control of the corresponding action task.

[0084] It should be noted that, since humanoid robot control tasks are typical continuous control tasks,

[0085] Proximal Policy Optimization (PPO) is a general and effective choice for handling continuous control problems. Therefore, the training of the target policy network corresponding to each action task in this implementation can be achieved through optimization training based on the PPO algorithm.

[0086] The PPO algorithm consists of two parts: an initial policy network (Actor) and an initial value network (Critic). The input to the initial policy network includes the first sample state information, which is limited to sample ontology information and sample velocity commands, and is partially observable. The output may contain the desired action information corresponding to the first sample state information. This desired action information can be a set of motion information of the simulated robot's joints, such as the offset of joint angles relative to the default joint positions. This is used to simulate the interaction between the robot and its environment. The default joint position describes the default joint position of the simulated robot. Therefore, the formulas for calculating the desired motion information and the target joint position of the simulated robot after interaction with the environment can be implemented based on the following joint position calculation formula:

[0087] ;

[0088] in, This indicates the pre-set default joint position. It is the output of the Actor network. This represents the target position of the joint at time t.

[0089] Furthermore, once the target joint position is determined, the joint torque can be calculated using a proportional-derivative (PD) control formula. The motors are fed to obtain the corresponding actions of the simulated robot at each moment. The formula for calculating the joint torque is as follows:

[0090] ;

[0091] in, ,and The gain parameter of the PD. and This represents the current joint angles and joint velocities of the simulated robot.

[0092] The input to the initial value network can be based on the input to the policy network, with additional observations added to integrate and form fully observable information. These observations can be environmental information, such as terrain elevation, etc., but this embodiment does not specifically limit their application.

[0093] The output of the initial value network can include value results used to evaluate the desired actions provided by the initial policy network.

[0094] Deep reinforcement learning is a process of learning through interaction between an agent and its environment. After selecting an action, the agent receives a corresponding state and reward from the environment. Through continuous interaction, the agent eventually obtains the optimal control policy. The interaction between the agent and the environment is modeled as a Markov Decision Process (MDP), which is typically represented by a quintuple. ,in Represents a set of states. Represents a set of executable actions. Let represent the state transition function, which represents the probability that the agent will transition to the next state after taking an action in a given state. Let represent the reward function, which represents the immediate reward for the agent to transition to the next state after taking an action in a given state. This is the discount factor for the reward. The goal of deep reinforcement learning is to train a policy network such that the expected cumulative discounted reward of its output action information is maximized over a finite time T.

[0095] Step 120: Based on the second sample state information of the simulated robot, the task labels corresponding to the preset training tasks, and the expected action information corresponding to the second sample state information output by the target policy network for each action task, the initial operation control model is jointly trained using transfer learning, reinforcement learning, and imitation learning. Each sample state information in both the first and second sample state information includes sample ontology information and sample velocity commands.

[0096] like Figure 3 As shown, after training the target policy network corresponding to each action task using reinforcement learning in step 110, the second sample state information can be input into the target policy network corresponding to each action task to obtain the expected action information corresponding to the second sample state information output by the target policy network corresponding to each action task. A task label corresponding to the training task is preset, and different expert data are selected from the expected action information corresponding to the second sample state information output by the target policy networks corresponding to multiple action tasks based on the preset task label. Different reward functions are also selected based on the preset task label. By jointly selecting the expert data, the selected reward function, the second sample state information, and the preset task label, the initial operation control model undergoes joint training of transfer learning, reinforcement learning, and imitation learning. This results in an integrated model capable of simultaneously processing multiple action tasks, ensuring that the integrated model can control and generate diverse gait trajectories to smoothly switch between various gaits of the humanoid robot, effectively improving the operation control performance of the humanoid robot.

[0097] The task labels corresponding to the preset training tasks here are determined by using different tasks to achieve multi-task training.

[0098] Here, the specific implementation of joint training can be as follows: after selecting a target policy network as the initial running control model from the target policy networks corresponding to multiple action tasks for transfer learning training, a combination of imitation learning and reinforcement learning is used to conduct retraining based on the transfer learning. This embodiment does not specifically limit this.

[0099] Step 130: Based on the trained initial operation control model, construct an operation control model, which is used to control the physical robot to perform different motion control actions under different action tasks.

[0100] Optionally, after obtaining the integrated model capable of multi-action task processing through step 120, i.e. the trained initial operation control model, an operation control model that can synchronously control the physical robot to perform different motion control actions under different action tasks can be constructed based on the trained initial operation control model. This allows for the generation of control over diverse gait trajectories through the operation control model, thereby controlling the smooth switching between various gaits of the physical robot and improving the operation control effect of the physical robot.

[0101] The construction here can be based directly on the initial operational control model after training to form an operational control model; or it can be based on the task labels corresponding to the preset training tasks to select different reward functions to perform reinforcement learning training on the initial operational control model after training to increase the generalization ability of the model and then construct an operational control model. This embodiment does not specifically limit this.

[0102] The method provided in this embodiment first trains the initial policy network using reinforcement learning based on the first sample state information of the simulated robot and the reward function corresponding to each action task, thereby obtaining the target policy network corresponding to each action task. Then, based on the second sample state information of the simulated robot, the task label corresponding to the preset training task, and the expected action information corresponding to the second sample state information output by the target policy network corresponding to each action task, the initial operation control model is jointly trained using transfer learning, reinforcement learning, and imitation learning to construct an operation control model with an integrated policy network that is compatible with multi-action task processing. This enables the robot to handle multiple tasks simultaneously and achieves smooth switching between different action tasks, effectively improving the robot's stable multi-gait walking performance and saving the robot's energy consumption, thereby ensuring the performance of the robot's operation control.

[0103] In some embodiments, the motion task includes a standing task or a walking task;

[0104] Step 110 specifically includes:

[0105] Based on the first sample state information and the reward function corresponding to the standing task, the initial policy network is trained by reinforcement learning to obtain the target policy network corresponding to the standing task;

[0106] Based on the first sample state information and the reward function corresponding to the walking task, the initial policy network is trained by reinforcement learning to obtain the target policy network corresponding to the walking task;

[0107] The reward function for the walking task includes a first reward function and a second reward function, and the reward function for the standing task includes the first reward function. The first reward function is a reward function constructed based on the motion parameters of the simulated robot, and the second reward function is a reward function constructed based on the environmental interaction information of the simulated robot.

[0108] Optionally, when the action task includes standing or walking, reinforcement learning training is required to train a target policy network for the standing task that can accurately control standing, and a target policy network for the walking task that can accurately control walking gait. The specific implementation steps are as follows:

[0109] Using the first sample state information and the reward function corresponding to the standing task, the initial policy network is trained using reinforcement learning to obtain the target policy network for the standing task, which can accurately control standing. For example, the first sample state information can be input into the initial policy network to obtain the predicted standing action information. The value of the reward function corresponding to the standing task can be calculated based on the predicted standing action information. The value of the reward function corresponding to the standing task can then be used to obtain the reinforcement learning loss function corresponding to the standing task. The initial policy network can then be iteratively trained based on the reinforcement learning loss function corresponding to the standing task to obtain the target policy network for the standing task.

[0110] Similarly, using the first sample state information and the reward function corresponding to the walking task, the initial policy network is trained using reinforcement learning to obtain the target policy network for the walking task, which can accurately control walking. For example, the first sample state information can be input into the initial policy network to obtain predicted walking action information. The value of the reward function corresponding to the walking task can be calculated based on the predicted walking action information, and the reinforcement learning loss function corresponding to the walking task can be obtained using the value of the reward function. The initial policy network can then be iteratively trained based on the reinforcement learning loss function corresponding to the walking task to obtain the target policy network for the walking task.

[0111] The reward functions for the walking task and the standing task are set differently. The reward function for the walking task can include a first reward function constructed based on the motion parameters of the simulated robot and a second reward function constructed based on the environmental interaction information of the simulated robot; the reward function for the standing task can include only the first reward function constructed based on the motion parameters of the simulated robot.

[0112] The first reward function, which is constructed based on the motion parameters of the simulated robot, includes at least one function used to constrain the robot's performance.

[0113] For example, in some embodiments, the first reward function includes at least one of an energy consumption penalty function, an operating posture constraint function, a speed tracking reward function, a joint motion penalty function, a fuselage motion penalty function, a motion smoothing function, and a fall penalty function;

[0114] The energy consumption penalty function includes a function for constraining the energy consumption generated by joint movements; the running posture constraint function includes a function for constraining the alignment of the feet, knees, and hips with the fuselage orientation and / or optimizing the upright state of the leg joints; the velocity tracking reward function includes a function for constraining the command tracking performance corresponding to linear velocity and / or angular velocity; the joint motion penalty function includes a function for constraining at least one joint motion parameter among joint acceleration, joint torque, joint limit position, and joint limit force; and the fuselage motion penalty function includes a function for constraining the fuselage attitude and / or fuselage motion state.

[0115] Optionally, the energy consumption penalty function can be calculated based on the joint torque and joint angular velocity generated by the model output during model training. By setting the energy consumption penalty function, the energy required for robot operation control can be minimized. The specific calculation formula for the energy consumption penalty function is as follows:

[0116] ;

[0117] In the formula, The value of the energy consumption penalty function; This refers to the joint torque generated by the model output during model training. This represents the joint angular velocity.

[0118] The running posture constraint function includes functions for constraining the alignment of the two feet, two knees, two hips with the body and / or optimizing the upright state of the leg joints to maintain posture consistency during robot operation, reduce joint wear, improve motion accuracy and efficiency, and thus optimize gait aesthetics and stability during operation.

[0119] The function that constrains the alignment of the feet, knees, and hips with the body orientation (also referred to as the alignment constraint function between the feet, knees, hips, and body orientation) can be calculated using the z-values ​​of the quaternions in the body, feet, knees, and hip coordinate systems. It is used to ensure the aesthetic gait of the robot when walking. The specific calculation formula is as follows:

[0120] ;

[0121] In the formula, The values ​​of the consistency constraint function between the two feet, two knees, two hips and the fuselage orientation are taken; , , , These represent the z-coordinate values ​​in the quaternions of the body, bipedal, biknee, and bihip coordinate systems generated by the model output during model training. This reward function primarily ensures the aesthetic gait of the robot during walking.

[0122] The function that optimizes the upright position of the leg joints (also known as the straight-leg reward function) can be calculated based on the current joint position and plantar pressure. The specific calculation formula is as follows:

[0123] ;

[0124] In the formula, The value of the straight-leg reward function is used to encourage the robot's leg joints to approach an upright position; It is the current joint position generated by the model output during model training; This is the default joint position when walking and standing with straight legs. The value approaches the straight leg position. This represents the plantar pressure generated by the model output during model training; it should be noted that the straight-leg reward function only takes effect when the sole of the foot is in contact with the ground, and does not take effect when lifting the leg to walk.

[0125] The velocity tracking reward function includes functions for constraining the command tracking performance corresponding to linear velocity and / or angular velocity, respectively, to optimize robot tracking performance.

[0126] The function for command tracking performance corresponding to the constrained linear velocity (also known as the linear velocity tracking reward function) can be calculated based on the linear velocity generated by the model output during model training and the linear velocity command in the sample velocity command. The specific calculation formula is as follows:

[0127] ;

[0128] In the formula, The value of the linear velocity tracking reward function; This represents the linear velocity of the fuselage in the x and y directions in the fuselage coordinate system generated by the model output during model training. This refers to the fuselage linear velocity command in the x and y directions of the fuselage coordinate system within the sample velocity command.

[0129] The function representing the command tracking performance (referred to as the angular velocity tracking reward function) corresponding to angular velocity can be calculated based on the angular velocity generated by the model output during model training and the angular velocity command in the sample velocity command. The specific calculation formula is as follows:

[0130] ;

[0131] In the formula, The value of the angular velocity tracking reward function; This represents the fuselage angular velocity in the fuselage coordinate system generated by the model output during model training. This refers to the fuselage angular velocity command in the fuselage coordinate system within the sample velocity command.

[0132] The joint motion penalty function includes functions for constraining at least one of the joint motion parameters, such as joint acceleration, joint torque, joint limit position, and joint limit force, to ensure joint motion performance and thus guarantee smooth robot motion.

[0133] The function constraining joint acceleration (referred to as the joint acceleration penalty function) can be calculated based on the joint angular acceleration generated by the model output during model training. The specific calculation formula is as follows:

[0134] ;

[0135] In the formula, The value of the joint acceleration penalty function; This represents the joint angular acceleration generated by the model output during model training.

[0136] The function constraining joint torques (referred to as the joint torque penalty function) can be calculated based on the joint torques generated by the model output during model training. The specific calculation formula is as follows:

[0137] ;

[0138] In the formula, This represents the value of the joint torque penalty function. This represents the joint torque generated by the model output during model training, specifically derived from the expected joint position output by the policy network and converted by the PD controller.

[0139] The function constraining the extreme joint positions (referred to as the joint extreme position penalty function) can be calculated based on the joint positions generated by the model output during model training. The specific calculation formula is as follows:

[0140] ;

[0141] In the formula, This represents the value of the joint limit position penalty function; This represents the joint positions generated by the model output during model training. and Indicates the upper and lower limits of joint position.

[0142] The function constraining the joint limit force (referred to as the joint limit torque penalty function) can be calculated based on the joint torque generated by the model output during model training. The specific calculation formula is as follows:

[0143] ;

[0144] In the formula, This represents the value of the joint limit torque penalty function; This represents the joint torque generated by the model output during model training. and This indicates the upper and lower limits of the joint torque. The joint limit torque penalty function is used to constrain the motor from exceeding the maximum torque, thus protecting the motor.

[0145] The body motion penalty function includes functions used to constrain the body attitude and / or body motion state to ensure body motion performance and thus ensure smooth robot motion.

[0146] The function constraining fuselage attitude (referred to as the fuselage attitude penalty function) is calculated based on the projected gravity vector generated by the model output during model training. It is used to penalize fuselage tilt, and the specific calculation formula is as follows:

[0147] ;

[0148] In the formula, The value of the fuselage attitude penalty function; This represents the components of the projected gravity vector in the x and y directions generated by the model output during model training.

[0149] The function that constrains the robot's motion state (referred to as the roll and pitch penalty function) is calculated based on the roll and pitch angular velocities generated by the model output during model training. It is used to penalize the robot for swaying back and forth and left and right when walking, keeping the robot vertical. The specific calculation formula is as follows:

[0150] ;

[0151] In the formula, This represents the values ​​of the penalty functions for fuselage roll and pitch motions; This represents the roll and pitch angular velocity of the fuselage generated by the model output during model training.

[0152] The dynamic smoothing function (also known as the motion smoothing term function) is calculated based on the expected joint movements at different times generated by the model output during model training. It is used to penalize the robot for performing sudden, non-smooth movements. The specific calculation formula is as follows:

[0153] ;

[0154] In the formula, The values ​​of the motion smoothing term function; , , These represent the expected joint actions output by the policy network at times t, t-1, and t-2, respectively.

[0155] The fall penalty function is calculated based on the collision state determined by the expected action information generated from the model output during model training. It is used to penalize the robot for falling. The specific calculation formula is as follows:

[0156] ;

[0157] In the formula, This represents the value of the fall penalty function; This means that if the model output determines that the simulated robot will collide with the ground at any part of its upper limb, torso, pelvis, or thigh, a large penalty value of -1 will be added.

[0158] The second reward function, which is constructed from the environmental interaction information of the simulated robot, includes at least one function used to constrain the robot's tracking performance.

[0159] For example, in some embodiments, the second reward function includes a path tracking reward function and / or a ground contact force penalty function;

[0160] The path tracking reward function includes a function for constraining the path tracking performance of the simulated robot; the ground contact force penalty function is a function for constraining the contact pressure between the simulated robot and the ground.

[0161] Optionally, the ground contact force penalty function includes a function to constrain the contact pressure between the simulated robot and the ground. This function can be calculated based on the plantar pressure generated at different time steps through the model output during model training. The specific calculation formula is as follows:

[0162] ;

[0163] In the formula, The value of the path tracking reward function; The plantar pressure at the current time step is generated by the model output during model training; The plantar pressure at the previous time step is generated by the model output during model training. It is the threshold of the contact force rate, which can be set according to actual needs, such as setting it so that the change within 20ms does not exceed 50N; These are the weighting coefficients.

[0164] The path tracking reward function includes a function used to constrain the path tracking performance of the simulated robot. This function can be calculated based on the difference between the current position generated by the model output during model training and the pre-set target position. It is used to reward or punish the robot based on the input positioning information to prevent the robot from deviating from the target running direction when walking.

[0165] The specific calculation formula is as follows:

[0166] ;

[0167] In the formula, The value of the path tracking reward function; This is the difference between the current position generated by the model output during model training and the pre-set target position; This is a hyperparameter used to limit the sensitivity of the path tracking reward function to errors. The specific value can be set according to actual needs.

[0168] It should be noted that when training the humanoid robot to walk, the forward and backward movement and the left and right movement speed are randomly sampled between -1m / s and 1m / s. The reward function for the walking task can be obtained by weighted summing of all functions contained in the first reward function and all functions contained in the second reward function. When training the robot to stand, the robot speed is only tracked to zero. Therefore, during the training for the standing task, the functions contained in the second reward function are set to 0, such as the ground contact force penalty function and the path tracking reward weight are set to 0, and the reward function for the standing task is obtained by weighted summing of all functions contained in the first reward function.

[0169] In contrast, existing technologies typically control humanoid robots to walk by bending their knees to achieve more stable operation. However, this approach requires a large torque from the knee joint motor, and walking with bent knees can result in unsightly gait, reduced robot height, high energy consumption, and increased susceptibility to obstacles.

[0170] The method provided in this embodiment uses multi-dimensional reward functions, such as energy consumption, orientation consistency, leg joint upright state, speed tracking, joint motion, body motion, motion smoothing and fall penalty, as well as path tracking and ground contact force, to accurately perform deep reinforcement learning training on the target policy networks corresponding to the standing and straight-leg walking tasks of the humanoid robot. This enables the trained target policy networks corresponding to the standing and walking tasks to effectively achieve upright walking in operation control, improve gait aesthetics, stability and energy efficiency, while avoiding collisions and deviations from the path, thus comprehensively improving the performance of robot operation control.

[0171] In some embodiments, step 120 specifically includes:

[0172] The initial operation control model is constructed by selecting the target policy network corresponding to the target action task from the target policy networks corresponding to the multiple action tasks.

[0173] The initial operation control model is trained by transfer learning to obtain the operation control model to be trained;

[0174] The second sample state information and the task label are input into the running control model to be trained to obtain the predicted action information corresponding to the task label;

[0175] Based on the task label, the target action information corresponding to the preset training task is obtained from the expected action information corresponding to the second sample state information output by the target policy network corresponding to the multiple action tasks.

[0176] Based on the predicted action information and the target action information, as well as the reward function corresponding to the preset training task, a target loss function is constructed.

[0177] Based on the target loss function, the training model to be trained is subjected to joint training of reinforcement learning and imitation learning.

[0178] Optionally, after training the target policy network for each action task separately in the simulator, such as the walking model and the standing model, although each target policy network can effectively process each action task independently, such as controlling the robot to walk and stand independently, if the robot is directly controlled to perform different action tasks in action task switching scenarios, the robot will be unstable at the moment of action switching. In order to ensure that the robot can smoothly switch between multi-gait actions, a task label can be added to the separately trained target policy networks for each action task to perform joint training of transfer learning, reinforcement learning training and imitation learning of the multi-task policy network. This makes the trained multi-task policy network compatible with the task processing performance of the target policy networks corresponding to different action tasks, thereby achieving compatible processing of different action tasks, ensuring smooth switching between multi-gait actions, and improving the stability of robot operation control.

[0179] The joint training process here specifically includes the following steps:

[0180] First, from the target policy networks corresponding to each action task obtained through pre-training, the target policy network corresponding to the target action task is selected to construct the initial operation control model for knowledge transfer. For example, the target policy network corresponding to the standing task (also known as the pre-trained standing model) is selected to construct the initial operation control model for knowledge transfer. This embodiment does not specifically limit this. The initial operation control model determined in this way is not a randomly initialized model, but an initialization model constructed using a pre-trained high-performance model, which can effectively ensure that the model can converge to a better motion effect more quickly.

[0181] Once the initial operation control model is obtained, it can be trained through transfer learning to obtain a new operation control model with knowledge of the operation control domain.

[0182] Next, the target policy networks corresponding to multiple action tasks are used as teacher models. Using task labels corresponding to preset training tasks, the expected action information corresponding to the second sample state information output by the target policy networks for multiple action tasks is sampled to obtain the target action information corresponding to the preset training tasks. The second sample state information and task labels are then input into the operation control model to be trained to obtain the predicted action information corresponding to the task labels. Using the predicted action information, target action information, and the reward function corresponding to the preset training tasks, a target loss function is constructed. Thus, the operation control model to be trained undergoes joint training of reinforcement learning and imitation learning with the goal of minimizing the target loss function. This integrates the target policy networks corresponding to multiple action tasks into a single model through knowledge distillation, achieving a smooth transition between different skills. This results in an integrated model that simultaneously processes multiple action tasks. It also learns the action task processing advantages of the target policy networks corresponding to each action task trained in the first stage of reinforcement learning, ensuring that the integrated model can control and generate diverse gait trajectories to smoothly switch between various gaits of the humanoid robot, effectively improving the operation control performance of the humanoid robot.

[0183] The target loss function here can be calculated by combining the imitation learning loss function and the reinforcement learning loss function after determining the imitation learning loss function and the reinforcement learning loss function using the predicted action information, the target action information, and the reward function corresponding to the preset training task.

[0184] For example, in some embodiments, the construction step of the target loss function in step 120 specifically includes:

[0185] The imitation learning loss function is determined based on the probability distribution difference between the predicted action information and the target action information;

[0186] The reinforcement learning loss function is determined based on the predicted action information and the reward function corresponding to the preset training task;

[0187] The target loss function is obtained by weighted summing of the imitation learning loss function and the reinforcement learning loss function;

[0188] The weight coefficients between the imitation learning loss function and the reinforcement learning loss function are dynamically updated based on the number of iterations during the iterative training process.

[0189] Optionally, the specific formula for calculating the target loss function is as follows:

[0190] ;

[0191] ;

[0192] In the formula, The value of the target loss function; The weighting coefficients between the imitation learning loss function and the reinforcement learning loss function can be dynamically updated based on the number of iterations during the iterative training process. For example, at the beginning of training, the proportion of imitation learning is large, and as training progresses, the proportion of imitation learning loss gradually decreases. After a certain number of training steps, only pure reinforcement learning is used for generalization. To mimic the learning loss function's value; To determine the value of the loss function for reinforcement learning; The predicted action information output by the control model to be trained. The expected action information output from different expert skill models (i.e., target policy networks corresponding to different action tasks).

[0193] The method provided in this embodiment achieves compatible processing of multiple action tasks by using a joint training approach of transfer learning, reinforcement learning, and imitation learning on the basis of the target policy network corresponding to each action task that has been pre-trained. This ensures smooth action transitions when switching between different gait actions, improves the stability and reliability of robot operation control in different task scenarios, and enables the robot to have the upright running performance and path tracking performance corresponding to the target policy network for each action task. This effectively reduces energy consumption and knee collisions, improves path tracking performance, and thus enhances the robot's adaptability and stability in various complex tasks, achieving stable multi-gait walking, robust resistance to external interference, and the ability to adapt to changes in hardware and environmental conditions.

[0194] In some embodiments, step 130 specifically includes:

[0195] At least one disturbance control mechanism is added to the simulated robot;

[0196] Obtain the third sample state information of the simulated robot under each of the disturbance controls, and perform reinforcement learning training on the trained initial operation control model based on each of the third sample state information and the task label corresponding to the preset training task to obtain the operation control model;

[0197] The disturbance control includes disturbance control of physical parameters and / or disturbance control of environmental parameters.

[0198] Optionally, after obtaining a high-performance initial operational control model through joint training that can simultaneously handle different action tasks in a simulation environment, reinforcement learning can be used to further train the initial operational control model to improve its generalization ability. This minimizes the performance gap between the model in the simulation scenario and the model in the real scenario, thereby improving the model's performance in real-world scenarios. This allows for effective control of the physical robot to successfully perform various action tasks in real-world scenarios. The specific implementation steps are as follows:

[0199] First, in order to transfer the initial operational control model trained on the simulated robot to the physical robot, domain generalization can be performed on the simulation training data. This domain generalization can involve adding perturbation control to the simulated robot's physical parameters and / or environmental parameters, such as adding perturbations to the joint positions, joint velocities, external force disturbances, motor delays, body weight, center of mass position, ground friction, and gravity vector values ​​within different ranges.

[0200] Then, the third sample state information of the simulated robot under various disturbance control is obtained. Based on the third sample state information under various disturbance control and the task label corresponding to the preset training task, the initial operation control model after training is subjected to reinforcement learning training. This makes the trained operation control model more generalizable and robust. As a result, when the operation control model is deployed on the physical robot, it can have a strong anti-interference ability and achieve more robust control, thereby effectively realizing the smooth switching of different gait actions of the physical robot.

[0201] The method provided in this embodiment effectively improves the robustness of Sim2real when deployed on a real machine by adding multiple generalization training parameters in the simulation training, reduces the difference between the real machine effect and the simulation effect, and thus has a strong anti-interference ability when the operation control model trained in this way is deployed on the physical robot, achieving more robust control, and thus effectively realizing the smooth switching of different gait movements of the physical robot.

[0202] In some embodiments, this embodiment also provides a robot operation control method.

[0203] Figure 4 This is one of the flowcharts illustrating the robot operation control method provided by the present invention. Figure 5 This is the second flowchart of the robot operation control method provided by the present invention.

[0204] like Figure 4 As shown, the method includes steps 410, 420, 430 and 440.

[0205] Step 410: Obtain the current status information and task tag of the current task of the physical robot; the current status information includes the current body information and the current speed command;

[0206] Step 420: Input the current status information and the task tag of the current task into the operation control model to obtain the current action information corresponding to the task tag of the current task;

[0207] Step 430: Calculate the torque based on the current motion information to obtain the joint torque of the physical robot;

[0208] Step 440: Based on the joint torque, control the physical robot to perform the motion control action under the current task;

[0209] The operation control model is trained based on the operation control model training method provided in the above embodiments.

[0210] like Figure 5 As shown, in practical applications, it can be deployed and used via physical robots. Figure 1 The training process shown shows a well-trained multi-task policy model (also known as a runtime control model or an integrated policy network).

[0211] When the operational control model of the physical robot is determined after deployment, the task label for the current task, generated via user input, and the current speed command can be obtained for triggering operations on the physical robot. The task label indicates the task type, specifically whether the current task is a walking or standing task. The current speed command includes, but is not limited to, the desired linear velocity and the desired angular velocity.

[0212] The user input here can be information input through command line interface, graphical interface, touch input, drop-down selection input, voice input, gesture input, visual input, brain-computer input, etc. This embodiment does not specifically limit it.

[0213] In addition, it is necessary to acquire physical robot body sensors, such as joint encoders and inertial measurement units (IMU) attitude sensors, to obtain the current body information of the physical robot in the real environment. The body information here includes, but is not limited to, the position, orientation, linear velocity, and angular velocity of the physical robot's pelvis, as well as the position, joint velocity, body posture, and the expected position of the humanoid robot in the previous cycle. This embodiment does not specifically limit these.

[0214] Subsequently, the task label, current velocity command, and current ontology information of the current task are input into the integrated policy network as model input, so that the integrated policy network can calculate the current action information corresponding to the task label of the current task. The current action information here is used to characterize the gait trajectory, and it can include the offset of the robot's current joint position relative to the default joint position. .

[0215] Subsequently, based on the current motion information, torque is calculated to obtain the joint torque (also known as motor torque) of the physical robot, which is then used to execute the motor. This yields the corresponding motion command for the physical robot at the current moment, enabling the physical robot to perform motion control actions under the current task according to the corresponding motion command.

[0216] The specific steps for calculating joint torque include:

[0217] First set the offset The target joint position is obtained by adding the target position to the default joint position. For details, please refer to the joint position calculation formula described in the model training process above. It will not be repeated here.

[0218] Then, by applying the PD control formula to the current target joint position, the corresponding joint torque can be calculated. For details, please refer to the joint torque calculation formula described in the above model training process. It will not be repeated here.

[0219] The method provided in this embodiment constructs an integrated policy network operation control model that is compatible with multi-action task processing by acquiring a multi-objective reward function and performing joint training of transfer learning, reinforcement learning training and imitation learning. The operation control model is then used to control the operation of the physical robot, which can realize the robot's multi-task compatibility processing and the smooth switching between different action tasks. This effectively improves the robot's stable multi-gait walking performance and saves the robot's energy consumption, thereby ensuring the performance of the robot's operation control.

[0220] The operation control model training system provided by the present invention is described below. The operation control model training system described below can be referred to in correspondence with the operation control model training method described above.

[0221] Figure 6 This is a schematic diagram of the structure of the operation control model training system provided by the present invention; as shown below. Figure 6 As shown, the system includes:

[0222] The first training unit 610 is used to perform reinforcement learning training on the initial policy network according to the first sample state information of the simulated robot and the reward function corresponding to each action task, so as to obtain the target policy network corresponding to each action task.

[0223] The second training unit 620 is used to perform joint training of transfer learning, reinforcement learning training and imitation learning on the initial operation control model based on the second sample state information of the simulated robot, the task label corresponding to the preset training task, and the expected action information corresponding to the second sample state information output by the target policy network corresponding to each action task.

[0224] The model building unit 630 is used to build an operation control model based on the trained initial operation control model. The operation control model is used to control the physical robot to perform different motion control actions under different action tasks.

[0225] The sample status information in the first sample status information and the second sample status information each includes sample body information and sample speed command.

[0226] The system provided in this embodiment first trains the initial policy network using reinforcement learning based on the first sample state information of the simulated robot and the reward function corresponding to each action task, thereby obtaining the target policy network corresponding to each action task. Then, based on the second sample state information of the simulated robot, the task label corresponding to the preset training task, and the expected action information corresponding to the second sample state information output by the target policy network corresponding to each action task, the initial operation control model is jointly trained using transfer learning, reinforcement learning, and imitation learning to construct an operation control model with an integrated policy network that is compatible with multi-action task processing. This enables the robot to handle multiple tasks simultaneously and achieves smooth switching between different action tasks, effectively improving the robot's stable multi-gait walking performance and saving the robot's energy consumption, thereby ensuring the performance of the robot's operation control.

[0227] In some embodiments, the motion task includes a standing task or a walking task;

[0228] The first training unit is specifically used for:

[0229] Based on the first sample state information and the reward function corresponding to the standing task, the initial policy network is trained by reinforcement learning to obtain the target policy network corresponding to the standing task;

[0230] Based on the first sample state information and the reward function corresponding to the walking task, the initial policy network is trained by reinforcement learning to obtain the target policy network corresponding to the walking task;

[0231] The reward function for the walking task includes a first reward function and a second reward function, and the reward function for the standing task includes the first reward function. The first reward function is a reward function constructed based on the motion parameters of the simulated robot, and the second reward function is a reward function constructed based on the environmental interaction information of the simulated robot.

[0232] In some embodiments, the first reward function includes at least one of an energy consumption penalty function, an operating posture constraint function, a speed tracking reward function, a joint motion penalty function, a fuselage motion penalty function, a motion smoothing function, and a fall penalty function;

[0233] The energy consumption penalty function includes a function for constraining the energy consumption generated by joint movements; the running posture constraint function includes a function for constraining the alignment of the feet, knees, and hips with the fuselage orientation and / or optimizing the upright state of the leg joints; the velocity tracking reward function includes a function for constraining the command tracking performance corresponding to linear velocity and / or angular velocity; the joint motion penalty function includes a function for constraining at least one joint motion parameter among joint acceleration, joint torque, joint limit position, and joint limit force; and the fuselage motion penalty function includes a function for constraining the fuselage attitude and / or fuselage motion state.

[0234] In some embodiments, the second reward function includes a path tracking reward function and / or a ground contact force penalty function;

[0235] The path tracking reward function includes a function for constraining the path tracking performance of the simulated robot; the ground contact force penalty function is a function for constraining the contact pressure between the simulated robot and the ground.

[0236] In some embodiments, the second training unit is specifically used for:

[0237] The initial operation control model is constructed by selecting the target policy network corresponding to the target action task from the target policy networks corresponding to the multiple action tasks.

[0238] The initial operation control model is trained by transfer learning to obtain the operation control model to be trained;

[0239] The second sample state information and the task label are input into the running control model to be trained to obtain the predicted action information corresponding to the task label;

[0240] Based on the task label, the target action information corresponding to the preset training task is obtained from the expected action information corresponding to the second sample state information output by the target policy network corresponding to the multiple action tasks.

[0241] Based on the predicted action information and the target action information, as well as the reward function corresponding to the preset training task, a target loss function is constructed.

[0242] Based on the target loss function, the training model to be trained is subjected to joint training of reinforcement learning and imitation learning.

[0243] In some embodiments, the second training unit is further configured to:

[0244] The imitation learning loss function is determined based on the probability distribution difference between the predicted action information and the target action information;

[0245] The reinforcement learning loss function is determined based on the predicted action information and the reward function corresponding to the preset training task;

[0246] The target loss function is obtained by weighted summing of the imitation learning loss function and the reinforcement learning loss function;

[0247] The weight coefficients between the imitation learning loss function and the reinforcement learning loss function are dynamically updated based on the number of iterations during the iterative training process.

[0248] In some embodiments, the model building unit is specifically used for:

[0249] At least one disturbance control mechanism is added to the simulated robot;

[0250] Obtain the third sample state information of the simulated robot under each of the disturbance controls, and perform reinforcement learning training on the trained initial operation control model based on each of the third sample state information and the task label corresponding to the preset training task to obtain the operation control model;

[0251] The disturbance control includes disturbance control of physical parameters and / or disturbance control of environmental parameters.

[0252] In some embodiments, this application also provides a robot operation control system.

[0253] Figure 7 This is a schematic diagram of the robot operation control system provided by the present invention, as shown below. Figure 7 As shown, the system includes:

[0254] The acquisition unit 710 is used to acquire the current status information and task tag of the current task of the physical robot; the current status information includes the current body information and the current speed command;

[0255] The prediction unit 720 is used to input the current state information and the task label of the current task into the operation control model to obtain the current action information corresponding to the task label of the current task;

[0256] The calculation unit 730 is used to calculate the torque of the current motion information to obtain the joint torque of the physical robot;

[0257] The control unit 740 is used to control the physical robot to perform motion control actions under the current task based on the joint torque;

[0258] The operation control model is trained based on the operation control model training method provided in the above embodiments, as detailed in the following documents. Figure 1 The training process shown is not repeated here.

[0259] The system provided in this embodiment acquires a multi-objective reward function and then performs joint training through transfer learning, reinforcement learning, and imitation learning to construct an integrated policy network operation control model compatible with multi-action task processing. This operation control model is then used to control the operation of a physical robot, enabling multi-task compatibility while allowing smooth switching between different action tasks. This effectively improves the robot's stable multi-gait walking performance, saves energy, and thus ensures the performance of the robot's operation control.

[0260] The system provided by this invention is used to execute the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.

[0261] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communications bus 840. The processor 810 can call logic instructions in the memory 830 to execute a running control model training method. This method includes: performing reinforcement learning training on an initial policy network based on the first sample state information of the simulated robot and the reward function corresponding to each action task, to obtain a target policy network corresponding to each action task; performing joint training of the initial running control model using transfer learning, reinforcement learning, and imitation learning based on the second sample state information of the simulated robot, the task label corresponding to a preset training task, and the expected action information corresponding to the second sample state information output by the target policy network corresponding to each action task; and constructing a running control model based on the trained initial running control model. This running control model is used to control the physical robot to perform different motion control actions under different action tasks. Each sample state information in the first and second sample state information includes sample ontology information and sample velocity commands. Alternatively, a robot operation control method may be implemented, comprising: acquiring the current state information and task label of the current task of the physical robot; the current state information including current body information and current speed command; inputting the current state information and the task label of the current task into an operation control model to obtain the current action information corresponding to the task label of the current task; performing torque calculation on the current action information to obtain the joint torque of the physical robot; and controlling the physical robot to perform motion control actions under the current task according to the joint torque; wherein the operation control model is trained based on the operation control model training method.

[0262] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0263] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the operation control model training method provided by the above methods. The method includes: performing reinforcement learning training on an initial policy network based on the first sample state information of the simulated robot and the reward function corresponding to each action task to obtain a target policy network corresponding to each action task; performing joint training of transfer learning, reinforcement learning training, and imitation learning on the initial operation control model based on the second sample state information of the simulated robot, the task label corresponding to the preset training task, and the expected action information corresponding to the second sample state information output by the target policy network corresponding to each action task; and constructing an operation control model based on the trained initial operation control model. The operation control model is used to control the physical robot to perform different motion control actions under different action tasks. Each sample state information in the first sample state information and the second sample state information includes sample ontology information and sample velocity commands. Alternatively, a robot operation control method may be implemented, comprising: acquiring the current state information and task label of the current task of the physical robot; the current state information including current body information and current speed command; inputting the current state information and the task label of the current task into an operation control model to obtain the current action information corresponding to the task label of the current task; performing torque calculation on the current action information to obtain the joint torque of the physical robot; and controlling the physical robot to perform motion control actions under the current task according to the joint torque; wherein the operation control model is trained based on the operation control model training method.

[0264] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a method for training a running control model provided by the above-described methods. This method includes: performing reinforcement learning training on an initial policy network based on a first sample state information of the simulated robot and a reward function corresponding to each action task to obtain a target policy network corresponding to each action task; performing joint training of the initial running control model using transfer learning, reinforcement learning, and imitation learning based on a second sample state information of the simulated robot, a task label corresponding to a preset training task, and expected action information corresponding to the second sample state information output by the target policy network corresponding to each action task; and constructing a running control model based on the trained initial running control model. The running control model is used to control the physical robot to perform different motion control actions under different action tasks. Each sample state information in the first and second sample state information includes sample ontology information and sample velocity commands. Alternatively, a robot operation control method may be implemented, comprising: acquiring the current state information and task label of the current task of the physical robot; the current state information including current body information and current speed command; inputting the current state information and the task label of the current task into an operation control model to obtain the current action information corresponding to the task label of the current task; performing torque calculation on the current action information to obtain the joint torque of the physical robot; and controlling the physical robot to perform motion control actions under the current task according to the joint torque; wherein the operation control model is trained based on the operation control model training method.

[0265] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0266] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0267] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training a runtime control model, characterized in that, include: Based on the first sample state information of the simulated robot and the reward function corresponding to each action task, the initial policy network is trained by reinforcement learning to obtain the target policy network corresponding to each action task; the action task includes a standing task or a walking task, the reward function corresponding to the walking task includes a first reward function and a second reward function, and the reward function corresponding to the standing task includes the first reward function; the first reward function is a reward function constructed based on the action parameters corresponding to the simulated robot, and the second reward function is a reward function constructed based on the environmental interaction information corresponding to the simulated robot; Based on the second sample state information of the simulated robot, the task label corresponding to the preset training task, and the expected action information corresponding to the second sample state information output by the target policy network corresponding to each action task, the initial operation control model is jointly trained by transfer learning, reinforcement learning training and imitation learning. Based on the initial operational control model after training, an operational control model is constructed. The operational control model is used to control the physical robot to perform different motion control actions under different action tasks. The sample status information in the first sample status information and the second sample status information each includes sample body information and sample speed command.

2. The operation control model training method according to claim 1, characterized by, The step of training the initial policy network through reinforcement learning based on the first sample state information of the simulated robot and the reward function corresponding to each action task to obtain the target policy network corresponding to each action task includes: Based on the first sample state information and the reward function corresponding to the standing task, the initial policy network is trained by reinforcement learning to obtain the target policy network corresponding to the standing task; Based on the first sample state information and the reward function corresponding to the walking task, the initial policy network is trained using reinforcement learning to obtain the target policy network corresponding to the walking task.

3. The method for training a runtime control model according to claim 2, characterized in that, The first reward function includes at least one of the following: energy consumption penalty function, running posture constraint function, speed tracking reward function, joint motion penalty function, fuselage motion penalty function, motion smoothing function, and fall penalty function; The energy consumption penalty function includes a function for constraining the energy consumption generated by joint movements; the running posture constraint function includes a function for constraining the alignment of the feet, knees, and hips with the fuselage orientation and / or optimizing the upright state of the leg joints; the velocity tracking reward function includes a function for constraining the command tracking performance corresponding to linear velocity and / or angular velocity; the joint motion penalty function includes a function for constraining at least one joint motion parameter among joint acceleration, joint torque, joint limit position, and joint limit force; and the fuselage motion penalty function includes a function for constraining the fuselage attitude and / or fuselage motion state.

4. The method for training a runtime control model according to claim 2, characterized in that, The second reward function includes a path tracking reward function and / or a ground contact force penalty function; The path tracking reward function includes a function for constraining the path tracking performance of the simulated robot; the ground contact force penalty function is a function for constraining the contact pressure between the simulated robot and the ground.

5. The method for training an operation control model according to any one of claims 1-4, characterized in that, The step of performing joint training of the initial operation control model using transfer learning, reinforcement learning, and imitation learning based on the second sample state information of the simulated robot, the task label corresponding to the preset training task, and the expected action information corresponding to the second sample state information output by the target policy network corresponding to each action task includes: The initial operation control model is constructed by selecting the target policy network corresponding to the target action task from the target policy networks corresponding to the multiple action tasks. The initial operation control model is trained by transfer learning to obtain the operation control model to be trained; The second sample state information and the task label are input into the running control model to be trained to obtain the predicted action information corresponding to the task label; Based on the task label, the target action information corresponding to the preset training task is obtained from the expected action information corresponding to the second sample state information output by the target policy network corresponding to the multiple action tasks. Based on the predicted action information and the target action information, as well as the reward function corresponding to the preset training task, a target loss function is constructed. Based on the target loss function, the training model to be trained is subjected to joint training of reinforcement learning and imitation learning.

6. The method for training a runtime control model according to claim 5, characterized in that, The step of constructing a target loss function based on the predicted action information, the target action information, and the reward function corresponding to the preset training task includes: The imitation learning loss function is determined based on the probability distribution difference between the predicted action information and the target action information; The reinforcement learning loss function is determined based on the predicted action information and the reward function corresponding to the preset training task; The target loss function is obtained by weighted summing of the imitation learning loss function and the reinforcement learning loss function; The weight coefficients between the imitation learning loss function and the reinforcement learning loss function are dynamically updated based on the number of iterations during the iterative training process.

7. The method for training an operation control model according to any one of claims 1-4, characterized in that, The step of constructing an operation control model based on the trained initial operation control model includes: At least one disturbance control mechanism is added to the simulated robot; Obtain the third sample state information of the simulated robot under each of the disturbance controls, and perform reinforcement learning training on the trained initial operation control model based on each of the third sample state information and the task label corresponding to the preset training task to obtain the operation control model. The disturbance control includes disturbance control of physical parameters and / or disturbance control of environmental parameters.

8. A robot operation control method, characterized in that, include: Obtain the current status information and task label of the physical robot; The current status information includes current body information and current speed command; The current status information and the task label of the current task are input into the operation control model to obtain the current action information corresponding to the task label of the current task; The torque of the physical robot is obtained by calculating the torque based on the current motion information. Based on the joint torque, the physical robot is controlled to perform motion control actions under the current task; The operation control model is trained based on the operation control model training method as described in any one of claims 1 to 7.

9. A training system for a runtime control model, characterized in that, include: The first training unit is used to perform reinforcement learning training on the initial policy network based on the first sample state information of the simulated robot and the reward function corresponding to each action task, to obtain the target policy network corresponding to each action task; the action task includes a standing task or a walking task, the reward function corresponding to the walking task includes a first reward function and a second reward function, and the reward function corresponding to the standing task includes the first reward function; the first reward function is a reward function constructed based on the action parameters corresponding to the simulated robot, and the second reward function is a reward function constructed based on the environmental interaction information corresponding to the simulated robot; The second training unit is used to perform joint training of transfer learning, reinforcement learning training and imitation learning on the initial operation control model based on the second sample state information of the simulated robot, the task label corresponding to the preset training task, and the expected action information corresponding to the second sample state information output by the target policy network corresponding to each action task. The model building unit is used to build an operation control model based on the trained initial operation control model. The operation control model is used to control the physical robot to perform different motion control actions under different action tasks. The sample status information in the first sample status information and the second sample status information each includes sample body information and sample speed command.

10. A robot operation control system, characterized in that, include: The acquisition unit is used to acquire the current state information and task label of the current task of the physical robot. The current status information includes current body information and current speed command; The prediction unit is used to input the current state information and the task label of the current task into the operation control model to obtain the current action information corresponding to the task label of the current task; A calculation unit is used to calculate the torque of the current motion information to obtain the joint torque of the physical robot; The control unit is used to control the physical robot to perform motion control actions under the current task based on the joint torque. The operation control model is trained based on the operation control model training method as described in any one of claims 1 to 7.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the operation control model training method as described in any one of claims 1 to 7, or the robot operation control method as described in claim 8.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the operation control model training method as described in any one of claims 1 to 7, or the robot operation control method as described in claim 8.

Images