Operation control model training method and robot operation control method and system

Abstract

The invention provides an operation control model training method and a robot operation control method and system, and the method comprises the steps: carrying out the reinforcement learning training of an initial strategy network according to the first sample state information of a simulation robot and a reward function corresponding to each action task, and obtaining a target strategy network corresponding to each action task; according to second sample state information of the simulation robot, a task label corresponding to a preset training task and expected action information corresponding to the second sample state information output by a target strategy network corresponding to each action task, carrying out joint training of transfer learning, reinforcement learning training and imitation learning on the initial operation control model; and constructing an operation control model according to the trained initial operation control model. Through joint training of multiple training tasks, smooth switching among different action tasks of the robot can be ensured while multi-task compatible processing of the robot can be realized, and the operation control performance of the robot is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of automation control technology, and in particular to an operation control model training method, a robot operation control method and a system. Background Art

[0002] Humanoid robots are highly versatile and can imitate human actions and perform a wide range of operations through operation control, which makes them good at autonomously performing tasks in various complex scenarios. Therefore, how to effectively control the operation of robots is an important topic that needs to be studied in this field.

[0003] In the related art, a large number of robot simulation trainings are usually carried out under the simulator, and a large number of reward value functions are designed and debugged to train a specific model so that the robot can perform a specific action based on the specific model. However, this method usually trains a specific model for a specific task to learn a specific action. The action generated by the specific model can only be used for a specific task, which leads to state mutations at the moment of switching between different models, unstable actions, and reduced stability of robot operation control. Summary of the invention

[0004] The present invention provides an operation control model training method, a robot operation control method and a system, which are used to solve the defects in the prior art that different models are prone to state mutation at the moment of switching and the robot operation control stability is poor, thereby improving the stability of the robot operation control.

[0005] The present invention provides an operation control model training method, comprising: According to the first sample state information of the simulation robot and the reward function corresponding to each action task, the initial strategy network is respectively subjected to reinforcement learning training to obtain the target strategy network corresponding to each action task; Performing joint training of transfer learning, reinforcement learning and imitation learning on the initial operation control model according to the second sample state information of the simulation robot, the task label corresponding to the preset training task, and the expected action information corresponding to the second sample state information output by the target strategy network corresponding to each of the action tasks; According to the trained initial operation control model, an operation control model is constructed, wherein the operation control model is used to control the physical robot to perform different motion control actions under different action tasks; Wherein, each sample status information in the first sample status information and the second sample status information includes sample body information and sample speed command.

[0006] According to the present invention, a method for training an operation control model is provided, wherein the motion task includes a standing task or a walking task; According to the first sample state information of the simulation robot and the reward function corresponding to each action task, the initial strategy network is respectively subjected to reinforcement learning training to obtain the target strategy network corresponding to each action task, including: According to the first sample state information and the reward function corresponding to the standing task, the initial policy network is subjected to reinforcement learning training to obtain a target policy network corresponding to the standing task; According to the first sample state information and the reward function corresponding to the walking task, the initial policy network is subjected to reinforcement learning training to obtain a target policy network corresponding to the walking task; Among them, the reward function corresponding to the walking task includes a first reward function and a second reward function, and the reward function corresponding to the standing task includes the first reward function; the first reward function is a reward function constructed based on the action parameters corresponding to the simulation robot, and the second reward function is a reward function constructed based on the environmental interaction information corresponding to the simulation robot.

[0007] According to the present invention, a method for training an operation control model is provided, wherein the first reward function includes at least one of an energy consumption penalty function, an operation posture constraint function, a speed tracking reward function, a joint motion penalty function, a body motion penalty function, an action smoothing function, and a fall penalty function; Among them, the energy consumption penalty function includes a function for constraining the energy consumption generated by joint movements; the running posture constraint function includes a function for constraining the consistency of the orientation of the two feet, two knees, two hips and the fuselage and / or optimizing the upright state of the leg joints; the speed tracking reward function includes a function for constraining the instruction tracking performance corresponding to the linear velocity and / or angular velocity respectively; the joint motion penalty function includes a function for constraining at least one joint motion parameter of the joint acceleration, joint torque, joint extreme position, and joint extreme force; the fuselage motion penalty function includes a function for constraining the fuselage posture and / or fuselage motion state.

[0008] According to the present invention, there is provided a method for training an operation control model, wherein the second reward function comprises a path tracking reward function and / or a ground contact force penalty function; The path tracking reward function includes a function for constraining the path tracking performance of the simulated robot; and the ground contact force penalty function is a function for constraining the contact pressure between the simulated robot and the ground.

[0009] According to the present invention, a method for training an operation control model is provided, wherein the initial operation control model is jointly trained by transfer learning, reinforcement learning training and imitation learning according to the second sample state information of the simulation robot, the task label corresponding to the preset training task, and the expected action information corresponding to the second sample state information output by the target strategy network corresponding to each action task, including: Selecting a target strategy network corresponding to a target action task from among the target strategy networks corresponding to the plurality of action tasks to construct the initial operation control model; Performing transfer learning training on the initial operation control model to obtain an operation control model to be trained; Inputting the second sample state information and the task label into the to-be-trained operation control model to obtain predicted action information corresponding to the task label; According to the task label, obtaining the target action information corresponding to the preset training task from the expected action information corresponding to the second sample state information output by the target strategy network corresponding to the plurality of action tasks; Constructing a target loss function according to the predicted action information and the target action information, and a reward function corresponding to the preset training task; According to the target loss function, the operation control model to be trained is jointly trained by reinforcement learning training and imitation learning.

[0010] According to the present invention, a method for training an operation control model is provided, wherein a target loss function is constructed according to the predicted action information and the target action information, and a reward function corresponding to the preset training task, including: Determining an imitation learning loss function according to a probability distribution difference between the predicted action information and the target action information; Determining a reinforcement learning loss function according to the predicted action information and the reward function corresponding to the preset training task; Performing weighted addition on the imitation learning loss function and the reinforcement learning loss function to obtain the target loss function; The weight coefficient between the imitation learning loss function and the reinforcement learning loss function is obtained by dynamically updating according to the number of iterations in the iterative training process.

[0011] According to the present invention, a method for training an operation control model is provided, wherein the operation control model is constructed according to the trained initial operation control model, comprising: Adding at least one disturbance control to the simulation robot; Acquire the third sample state information of the simulation robot under each disturbance control, and perform reinforcement learning training on the trained initial operation control model according to each third sample state information and the task label corresponding to the preset training task to obtain the operation control model; The disturbance control includes disturbance control of physical parameters and / or disturbance control of environmental parameters.

[0012] The present invention also provides a robot operation control method, comprising: Obtaining the current state information of the physical robot and the task label of the current task; the current state information includes the current body information and the current speed command; Input the current state information and the task tag of the current task into the operation control model to obtain the current action information corresponding to the task tag of the current task; Performing torque calculation on the current motion information to obtain the joint torque of the physical robot; According to the joint torque, controlling the physical robot to perform the motion control action under the current task; Wherein, the operation control model is trained based on any of the operation control model training methods described above.

[0013] The present invention also provides an operation control model training system, comprising: A first training unit is used to perform reinforcement learning training on the initial strategy network according to the first sample state information of the simulation robot and the reward function corresponding to each action task, so as to obtain a target strategy network corresponding to each action task; A second training unit is used to perform joint training of transfer learning, reinforcement learning training and imitation learning on the initial operation control model according to the second sample state information of the simulation robot, the task label corresponding to the preset training task, and the expected action information corresponding to the second sample state information output by the target strategy network corresponding to each of the action tasks; A model building unit, used to build an operation control model according to the trained initial operation control model, wherein the operation control model is used to control the physical robot to perform different motion control actions under different action tasks; Wherein, each sample status information in the first sample status information and the second sample status information includes sample body information and sample speed command.

[0014] The present invention also provides a robot operation control system, comprising: An acquisition unit, used to acquire the current state information of the physical robot and the task label of the current task; the current state information includes current body information and current speed command; A prediction unit, used to input the current state information and the task label of the current task into the operation control model to obtain current action information corresponding to the task label of the current task; A calculation unit, used to perform torque calculation on the current motion information to obtain the joint torque of the physical robot; A control unit, used for controlling the physical robot to perform the motion control action under the current task according to the joint torque; Wherein, the operation control model is trained based on any of the operation control model training methods described above.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements any one of the above-mentioned operation control model training methods or the robot operation control methods.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements any of the above-mentioned operation control model training methods or the robot operation control methods.

[0017] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned operation control model training methods or the robot operation control methods.

[0018] The operation control model training method, robot operation control method and system provided by the present invention first perform reinforcement learning training on the initial policy network according to the first sample state information of the simulated robot and the reward function corresponding to each action task, so as to obtain the target policy network corresponding to each action task; then, according to the second sample state information of the simulated robot, the task label corresponding to the preset training task, and the expected action information corresponding to the second sample state information output by the target policy network corresponding to each action task, the initial operation control model is jointly trained by transfer learning, reinforcement learning training and imitation learning to construct an operation control model of an integrated strategy network compatible with multi-action task processing, so as to realize the multi-task compatible processing of the robot and realize smooth switching between different action tasks of the robot, effectively improve the stable multi-gait walking performance of the robot, save the energy consumption of the robot, thereby ensuring the performance of the robot's operation control. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is one of the flow charts of the operation control model training method provided by the present invention.

[0021] Figure 2 It is a structural schematic diagram of the humanoid robot provided by the present invention.

[0022] Figure 3 This is the second flow chart of the operation control model training method provided by the present invention.

[0023] Figure 4 It is one of the flow charts of the robot operation control method provided by the present invention.

[0024] Figure 5 This is the second flow chart of the robot operation control method provided by the present invention.

[0025] Figure 6 It is a structural schematic diagram of the operation control model training system provided by the present invention.

[0026] Figure 7 It is a structural schematic diagram of the robot operation control system provided by the present invention.

[0027] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] The operation control model training method, robot operation control method and system provided in the present application can be applied to various robot control scenarios, such as humanoid robots (also known as humanoid robots), etc. The following takes a humanoid robot as an example to describe the operation control model training method, robot operation control method and system provided in the present application.

[0030] Humanoid robots are highly versatile and can imitate human movements and perform a wide range of operations, which makes them good at autonomously performing tasks in various complex scenarios. The structure of humanoid robots and the environmental scenarios they face are usually very complex, and there are still considerable difficulties in controlling humanoid robots.

[0031] Early humanoid robot control algorithms were mainly classical motion control algorithms. The mainstream classical motion control algorithms include Zero Moment Point (ZMP), Model Predictive Control (MPC), Whole Body Control (WBC) and other control algorithms. Since humanoid robots have higher degrees of freedom and more complex dynamics, in order to simplify calculations, classical motion control algorithms often simplify mathematical models when modeling and solving the dynamics and kinematics of humanoid robots. This limits the performance of the algorithm, resulting in humanoid robots being able to walk stably in specific environments, such as specific terrains and actions, but lacking robustness, which leads to major challenges in adapting to new environments and tasks, that is, unstable performance on unstructured terrains.

[0032] With the rapid development of deep learning, especially deep reinforcement learning, and the realization of a large number of simulators, robot motion control technology based on deep reinforcement learning has developed rapidly and gradually become the mainstream algorithm for humanoid robot control. Motion control algorithms based on reinforcement learning generally conduct a large number of robot simulation training under the simulator, and realize the robot to perform a specific action by designing and debugging a large number of reward value functions. Finally, the converged model is deployed to the real machine for operation.

[0033] Representative research in this field is the use of deep reinforcement learning (DRL) on the bipedal robot platform Cassie. The movement of a bipedal robot can be viewed as a feedback control problem, which can be solved by finding the optimal strategy for a Markov decision process. By combining reinforcement learning with supervised learning with a limited sample size, a robust walking controller can be effectively learned in a simulated environment. The gap between simulation and real machines is reduced by adding domain randomization technology during the training process. By integrating periodic gaits, beautiful walking with humanoid gaits is achieved. Compared with cassise, which only has a bipedal structure, when the humanoid robot has upper limbs, its body weight will increase, the center of gravity will move up, and the motion control of the full humanoid will be more complicated, resulting in poor adaptability of the control algorithm.

[0034] The motion control algorithm based on reinforcement learning has stronger robustness, but this technology usually uses a large number of robot simulation training under the simulator, and designs and debugs a large number of reward value functions to train a specific model so that the robot can perform a specific action based on the specific model. This method usually trains a specific model for a specific task to learn a specific action. The action generated by the specific model can only be used for a specific task. For example, in order to realize the robot walking and standing, it is necessary to train the walking model and the standing model separately, and the walking model can only realize walking control, and the standing model can only realize standing control. As a result, different models are prone to state mutation at the moment of switching, and the action is unstable, which reduces the stability of the robot's operation control.

[0035] In view of the shortcomings of the above-mentioned prior art, the present application proposes a method for training an operation control model, which is a multi-task deep reinforcement learning method based on deep reinforcement learning, imitation learning, and transfer learning. The method first trains a straight-leg walking model and a straight-leg standing model respectively, and realizes straight-knee movement through reward function design. Secondly, by combining transfer learning, reinforcement learning and imitation learning methods, the actions output by a well-trained single-task policy network are used as expert data. By introducing imitation learning loss constraints on the basis of deep reinforcement learning loss functions, transfer learning of multi-task policy networks and joint training of deep reinforcement learning and imitation learning are realized, and an integrated strategy network that can process walking tasks and standing tasks compatibly is obtained, that is, an operation control model, so as to realize straight-leg walking and straight-leg standing of the robot, and realize smooth switching between walking and standing of the robot, thereby improving the stability of the robot's operation control and saving the robot's energy consumption, thereby ensuring the performance of the robot's operation control.

[0036] Figure 1 It is one of the flow charts of the operation control model training method provided by the present invention. The executor of the method can be an operation controller with machine learning model training and application, so as to realize the operation control of the robot such as standing and walking through machine learning model training and application.

[0037] like Figure 1 As shown, the method includes step 110 , step 120 and step 130 .

[0038] Step 110, based on the first sample state information of the simulation robot and the reward function corresponding to each action task, respectively perform reinforcement learning training on the initial policy network to obtain the target policy network corresponding to each of the action tasks.

[0039] The simulated robot here can be a humanoid machine model simulated and constructed in a simulated environment.

[0040] Figure 2Schematic diagram of the structure of the humanoid robot provided by the present invention. Figure 2 As shown in the figure, the humanoid robot has a total of 51 degrees of freedom (marked as q), of which each leg has 6 degrees of freedom, the waist has 1 degree of freedom, each arm has 7 degrees of freedom, and each hand has 12 degrees of freedom. Three cameras are installed on the body to perform tasks related to positioning, grasping perception, and walking perception. The height and weight of the humanoid robot can be set according to actual needs, such as setting the height to 176 cm and the overall weight to 64 kg based on the average value of an adult. The parameters of the joint unit can also be set according to actual needs, such as setting the maximum torque of the joint unit to about 360 N·m at the knee joint, 220 N·m at the hip joint, and 45 N·m at the ankle joint. For the arm, depending on the specific joint, the torque range is between 20 N·m and 120 N·m.

[0041] Figure 3 This is the second flow chart of the operation control model training method provided by the present invention. Figure 3 As shown, during the operation control model training process, skill learning training of each action task can be performed first to obtain the target strategy network corresponding to each action task. The specific implementation steps include: First, first sample state information of the simulation robot is obtained in the simulation environment.

[0042] The first sample state information here includes sample proprioceptive information and sample speed command; the sample proprioceptive information is the proprioceptive perception information of the simulated robot, which can be obtained based on the simulated sensors in the simulated robot, such as simulated joint encoders and simulated inertial measurement units in a simulated environment, including but not limited to the position, direction, linear velocity and angular velocity of the pelvis, as well as the position, joint velocity, body posture and expected position of all joints of the fuselage in the previous cycle, which is not specifically limited in this embodiment. The sample speed command refers to the speed command formed by the simulated user input, which includes the linear velocity information and angular velocity information expected by the user. The linear velocity information includes the linear velocity relative to each coordinate axis in the fuselage coordinate system, and the angular velocity information includes the steering velocity relative to the fuselage coordinate system.

[0043] Here, the reward function corresponding to each action task is also required; since the task labels of different action tasks are pre-associated with different reward functions, for example, the reward function corresponding to the straight-leg standing action task includes but is not limited to the reward function constructed by the action parameters corresponding to the simulation robot; the reward function corresponding to the straight-leg walking task includes but is not limited to the reward function constructed by the action parameters corresponding to the simulation robot and the reward function constructed by the environmental interaction information. Therefore, the reward function corresponding to each action task can be obtained according to the association of the task labels of each action task.

[0044] The action task here refers to the action task of controlling the robot to move, which may be a straight-leg standing task or a straight-leg walking task, a squatting task, etc., which is not specifically limited in this embodiment. In order to simplify the description, the following description is based on the example that the action task includes a straight-leg standing task or a straight-leg walking task.

[0045] Then, based on the first sample state information of the simulated robot and the reward function corresponding to each action task, deep reinforcement learning can be used to train the initial policy network separately to obtain the target policy network corresponding to each action task, among which the target policy network can accurately perform the separate operation control of the corresponding action task.

[0046] It should be noted that since the humanoid robot control task is a typical continuous control task, The Proximal Policy Optimization (PPO) algorithm is a general and effective choice for dealing with continuous control problems. Therefore, the training of the target policy network corresponding to each action task in this implementation can be implemented based on the PPO algorithm optimization training.

[0047] The PPO algorithm consists of two parts: the initial policy network (Actor) and the initial value network (Critic). The input of the initial policy network here includes the first sample state information, that is, limited to the sample body information and sample speed command, which are partially observable. The output can include the expected action information corresponding to the first sample state information, and the expected action information can be a set of action information of the joints of the simulated robot, such as the offset of the joint angle relative to the default joint position. , which is used for the interaction between the simulated robot and the environment. Among them, the default joint position is used to describe the default joint position of the simulated robot. Therefore, the calculation formula of the expected action information and the target joint position of the simulated robot after interacting with the environment can be implemented based on the following joint position calculation formula: ; in, Indicates the preset default joint position. is the Actor network output, Represents the target position of the joint at time t.

[0048] Furthermore, after the joint target position is determined, the joint torque can be calculated using the proportional-derivative (PD) control formula. Execute the motor to get the corresponding action of the simulated robot at each moment, where the joint torque calculation formula is as follows: ; in, ,and is the gain parameter of PD, and is the current joint angle and joint velocity of the simulated robot.

[0049] The input of the initial value network can add additional observations based on the input of the strategy network to integrate and form fully observable information. The observations here can be environmental information, such as terrain height, etc., which is not specifically limited in this embodiment.

[0050] The output of the initial value network may include a value result for evaluating the desired action provided by the initial policy network.

[0051] The process of deep reinforcement learning is a process of learning through the interaction between the agent and the environment. After the agent selects an action, it will get the corresponding state and reward from the environment. Through continuous learning in the interactive process, the optimal control strategy can be obtained. The interaction between the agent and the environment is modeled as a Markov decision process (MDP), which is usually represented by a five-tuple ,in Represents a state set, Represents a set of executable actions. Represents the state transition function, which indicates the probability that the agent will transition to the next state after taking an action in the state. Represents the reward function, which represents the immediate reward for the agent to move to the next state after taking an action in the state. is the discount factor of the reward. The goal of deep reinforcement learning is to train a policy network so that the expected cumulative discounted reward of the expected action information it outputs within a limited time T can be maximized.

[0052] Step 120, based on the second sample state information of the simulation robot, the task label corresponding to the preset training task, and the expected action information corresponding to the second sample state information output by the target strategy network corresponding to each action task, the initial operation control model is jointly trained by transfer learning, reinforcement learning training and imitation learning. Wherein, each sample state information in the first sample state information and the second sample state information includes sample body information and sample speed command.

[0053] like Figure 3As shown, after the target policy network corresponding to each action task is obtained by training using the reinforcement learning method in step 110, the second sample state information can be input into the target policy network corresponding to each action task to obtain the expected action information corresponding to the second sample state information output by the target policy network corresponding to each action task, and the task label corresponding to the training task is preset, so as to select different expert data from the expected action information corresponding to the second sample state information output by the target policy network corresponding to multiple action tasks according to the task label corresponding to the preset training task, and select different reward functions according to the task label corresponding to the preset training task, so as to perform transfer learning, reinforcement learning training and imitation learning joint training on the initial operation control model by jointly selecting the expert data, the selected reward function, the second sample state information and the task label corresponding to the preset training task, thereby obtaining an integrated model that can perform multiple action tasks simultaneously, thereby ensuring that the integrated model can realize the control generation of various gait action trajectories, so as to control the smooth switching between various gaits of the humanoid robot, thereby effectively improving the operation control effect of the humanoid robot.

[0054] The task labels corresponding to the preset training tasks here are determined by using different tasks to achieve multi-task training.

[0055] Here, the specific implementation of joint training can be that after selecting a target policy network as the initial operation control model among the target policy networks corresponding to multiple action tasks for transfer learning training, a method combining imitation learning and reinforcement learning is used to perform re-training on the basis of transfer learning, etc. This embodiment does not specifically limit this.

[0056] Step 130, constructing an operation control model according to the trained initial operation control model, wherein the operation control model is used to control the physical robot to perform different motion control actions under different action tasks.

[0057] Optionally, after obtaining an integrated model capable of processing multiple motion tasks through training in step 120, that is, the trained initial operation control model, an operation control model capable of synchronously controlling the physical robot to perform different motion control actions under different motion tasks can be constructed based on the trained initial operation control model, so as to realize the control generation of various gait motion trajectories through the operation control model, and then control the smooth switching between various gaits of the physical robot, so as to improve the operation control effect of the physical robot.

[0058] The construction here can be to directly construct an operation control model based on the trained initial operation control model; or it can be to select different reward functions according to the task labels corresponding to the preset training tasks to perform reinforcement learning training on the trained initial operation control model again to increase the generalization ability of the model, and then construct the operation control model. This embodiment does not make specific limitations on this.

[0059] The method provided in this embodiment first performs reinforcement learning training on the initial policy network according to the first sample state information of the simulation robot and the reward function corresponding to each action task, so as to obtain the target policy network corresponding to each action task; then, according to the second sample state information of the simulation robot, the task label corresponding to the preset training task, and the expected action information corresponding to the second sample state information output by the target policy network corresponding to each action task, the initial operation control model is jointly trained by transfer learning, reinforcement learning training and imitation learning to construct an operation control model of an integrated strategy network compatible with multi-action task processing, so as to realize the multi-task compatible processing of the robot and the smooth switching between different action tasks of the robot, effectively improving the stable multi-gait walking performance of the robot and saving the energy consumption of the robot, thereby ensuring the performance of the robot's operation control.

[0060] In some embodiments, the motion task includes a standing task or a walking task; Step 110 specifically includes: According to the first sample state information and the reward function corresponding to the standing task, the initial policy network is subjected to reinforcement learning training to obtain a target policy network corresponding to the standing task; According to the first sample state information and the reward function corresponding to the walking task, the initial policy network is subjected to reinforcement learning training to obtain a target policy network corresponding to the walking task; Among them, the reward function corresponding to the walking task includes a first reward function and a second reward function, and the reward function corresponding to the standing task includes the first reward function; the first reward function is a reward function constructed based on the action parameters corresponding to the simulation robot, and the second reward function is a reward function constructed based on the environmental interaction information corresponding to the simulation robot.

[0061] Optionally, when the action task includes a standing task or a walking task, reinforcement learning training is required to respectively train a target policy network for the standing task that can accurately perform standing control, and a target policy network corresponding to the walking task that can accurately perform walking gait control. The specific implementation steps are as follows: The first sample state information and the reward function corresponding to the standing task are used to perform reinforcement learning training on the initial policy network to obtain a target policy network for the standing task that can accurately perform standing control; for example, the first sample state information can be input into the initial policy network to obtain predicted standing action information, the value of the reward function corresponding to the standing task is calculated according to the predicted standing action information, and the reinforcement learning loss function corresponding to the standing task is obtained using the value of the reward function corresponding to the standing task, so as to iteratively train the initial policy network based on the reinforcement learning loss function corresponding to the standing task to obtain the target policy network for the standing task.

[0062] Similarly, the first sample state information and the reward function corresponding to the walking task are used to perform reinforcement learning training on the initial policy network to obtain a target policy network for the walking task that can accurately control walking. For example, the first sample state information may be input into the initial policy network to obtain predicted walking action information, the value of the reward function corresponding to the walking task is calculated based on the predicted walking action information, and the value of the reward function corresponding to the walking task is used to obtain the reinforcement learning loss function corresponding to the walking task, so as to iteratively train the initial policy network based on the reinforcement learning loss function corresponding to the walking task to obtain the target policy network for the walking task.

[0063] The reward function corresponding to the walking task and the reward function corresponding to the standing task are set differently. The reward function corresponding to the walking task may be a first reward function formed based on the action parameters corresponding to the simulation robot and a second reward function formed based on the environment interaction information corresponding to the simulation robot; the reward function corresponding to the standing task may be a first reward function formed based only on the action parameters corresponding to the simulation robot.

[0064] The first reward function constructed based on the action parameters corresponding to the simulation robot includes at least one function for constraining the robot's operating performance.

[0065] Exemplarily, in some embodiments, the first reward function includes at least one of an energy consumption penalty function, a running posture constraint function, a speed tracking reward function, a joint motion penalty function, a body motion penalty function, an action smoothness function, and a fall penalty function; Among them, the energy consumption penalty function includes a function for constraining the energy consumption generated by joint movements; the running posture constraint function includes a function for constraining the consistency of the orientation of the two feet, two knees, two hips and the fuselage and / or optimizing the upright state of the leg joints; the speed tracking reward function includes a function for constraining the instruction tracking performance corresponding to the linear velocity and / or angular velocity respectively; the joint motion penalty function includes a function for constraining at least one joint motion parameter of the joint acceleration, joint torque, joint extreme position, and joint extreme force; the fuselage motion penalty function includes a function for constraining the fuselage posture and / or fuselage motion state.

[0066] Optionally, the energy penalty function can be calculated based on the joint torque and joint angular velocity generated by the model output during the model training process. By setting the energy penalty function, the energy of the robot during operation control can be minimized. The specific calculation formula of the energy penalty function is as follows: ; In the formula, is the value of the energy consumption penalty function; is the joint torque generated by the model output during model training, is the joint angular velocity.

[0067] The running posture constraint functions include functions for constraining the consistency of the orientation of the two feet, two knees, two hips and the body and / or optimizing the upright state of the leg joints, so as to maintain the posture consistency during the operation of the robot, reduce joint wear, improve movement accuracy and efficiency, and thus optimize the gait beauty and stability during operation.

[0068] The function constraining the consistency of the orientation of the two feet, two knees, two hips and the body (also referred to as the consistency constraint function of the orientation of the two feet, two knees, two hips and the body) can be calculated by taking the z value in the quaternion of the body, two feet, two knees, two hips coordinate system. It is used to ensure the beautiful gait of the robot when walking. The specific calculation formula is as follows: ; In the formula, is the value of the consistency constraint function between the two feet, two knees, two hips and the body orientation; , , , They represent the z coordinates of the quaternions of the body, feet, knees, and hips generated by the model output during model training. This reward function is mainly used to ensure the robot's beautiful gait when walking.

[0069] The function for optimizing the upright state of the leg joints (also referred to as the straight leg reward function) can be calculated based on the current position of the joints and the plantar pressure. The specific calculation formula is as follows: ; In the formula, is the value of the straight leg reward function, which is used to encourage the robot's leg joints to approach an upright state; It is the current position of the joint generated by the model output during model training; is the default position of the joint when walking and standing with straight legs. The value of is close to the straight leg position. It represents the plantar pressure generated by the model output during model training. It should be noted that the straight leg reward function is only effective when the sole of the foot is in contact with the ground, and is not effective when walking with the leg raised.

[0070] The velocity tracking reward function includes functions for constraining the instruction tracking performance corresponding to the linear velocity and / or angular velocity, respectively, to optimize the robot tracking performance.

[0071] The function of the instruction tracking performance corresponding to the constraint line speed (also called the line speed tracking reward function) can be calculated based on the line speed generated by the model output during the model training process and the line speed command in the sample speed command. The specific calculation formula is as follows: ; In the formula, Track the value of the reward function for linear velocity; It represents the linear velocity of the fuselage in the x and y directions in the fuselage coordinate system generated by the model output during the model training process. Indicates the fuselage linear velocity command in the x and y directions of the fuselage coordinate system in the sample velocity command.

[0072] The function of the command tracking performance corresponding to the angular velocity (hereinafter referred to as the angular velocity tracking reward function) can be calculated based on the angular velocity generated by the model output during the model training process and the angular velocity command in the sample velocity command. The specific calculation formula is as follows: ; In the formula, Tracking reward function for angular velocity; It represents the angular velocity of the fuselage in the fuselage coordinate system generated by the model output during the model training process. Represents the body angular velocity command in the body coordinate system in the sample velocity command.

[0073] The joint motion penalty function includes a function for constraining at least one joint motion parameter of joint acceleration, joint torque, joint limit position, and joint limit force to ensure joint motion performance and thus ensure smooth robot motion.

[0074] The function constraining the joint acceleration (referred to as the joint acceleration penalty function) can be calculated based on the joint angular acceleration generated by the model output during the model training process. The specific calculation formula is as follows: ; In the formula, is the value of the joint acceleration penalty function; Represents the joint angular acceleration generated by the model output during model training.

[0075] The function constraining the joint torque (referred to as the joint torque penalty function) can be calculated based on the joint torque generated by the model output during the model training process. The specific calculation formula is as follows: ; In the formula, represents the value of the joint torque penalty function, It represents the joint torque generated by the model output during the model training process, which is specifically converted from the desired joint position output by the strategy network through the PD controller.

[0076] The function constraining the joint limit position (referred to as the joint limit position penalty function) can be calculated based on the joint position generated by the model output during the model training process. The specific calculation formula is as follows: ; In the formula, Represents the value of the joint limit position penalty function; represents the joint position generated by the model output during model training, and Indicates the upper and lower limits of the joint position.

[0077] The function constraining the joint limit force (referred to as the joint limit torque penalty function) can be calculated based on the joint torque generated by the model output during the model training process. The specific calculation formula is as follows: ; In the formula, Represents the value of the joint limit torque penalty function; represents the joint torque generated by the model output during model training, and Indicates the upper and lower limits of the joint torque. The joint limit torque penalty function is used to constrain the motor from exceeding the maximum torque and protect the motor.

[0078] The body motion penalty function includes a function for constraining the body posture and / or the body motion state to ensure the body motion performance and thus ensure the smooth motion of the robot.

[0079] The function constraining the fuselage attitude (referred to as the fuselage attitude penalty function) is calculated based on the projected gravity vector generated by the model output during the model training process. It is used to penalize the fuselage tilt. The specific calculation formula is as follows: ; In the formula, is the value of the fuselage attitude penalty function; Represents the components of the projected gravity vector in the x and y directions generated by the model output during model training.

[0080] The function constraining the motion state of the fuselage (referred to as the fuselage roll and pitch motion penalty function) is calculated based on the roll and pitch angular velocity generated by the model output during the model training process. It is used to punish the robot's shaking back and forth and left and right when walking to keep the fuselage upright. The specific calculation formula is as follows: ; In the formula, Indicates the value of the penalty function for the roll and pitch motion of the fuselage; It represents the roll and pitch angular velocity of the fuselage generated by the model output during the model training process.

[0081] The dynamic smoothing function (also called the action smoothing function) can be calculated based on the expected joint actions at different times generated by the model output during the model training process. It is used to punish the robot for performing sudden non-smooth actions. The specific calculation formula is as follows: ; In the formula, is the value of the action smoothing function; , , They represent the expected joint actions output by the policy network at time t, time t-1, and time t-2 respectively.

[0082] The fall penalty function is calculated based on the collision state determined by the expected action information generated by the model output during the model training process. It is used to punish the robot for falling. The specific calculation formula is as follows: ; In the formula, Represents the value of the fall penalty function; It means that if the expected action information generated by the model output determines that any part of the simulated robot's upper limbs, torso, pelvis, or thigh collides with the ground, a large penalty value, that is, a penalty of -1, will be added.

[0083] The second reward function constructed by the environment interaction information corresponding to the simulated robot includes at least one function for constraining the tracking performance of the robot.

[0084] Exemplarily, in some embodiments, the second reward function includes a path tracking reward function and / or a ground contact force penalty function; The path tracking reward function includes a function for constraining the path tracking performance of the simulated robot; and the ground contact force penalty function is a function for constraining the contact pressure between the simulated robot and the ground.

[0085] Optionally, the ground contact force penalty function includes a function for constraining the contact pressure between the simulated robot and the ground. The function may be calculated based on the plantar pressure at different time steps generated by the model output during the model training process. The specific calculation formula is as follows: ; In the formula, is the value of the path tracking reward function; It is the plantar pressure at the current time step generated by the model output during model training; It is the plantar pressure at the previous time step generated by the model output during the model training process; It is the threshold of contact force rate, which can be set according to actual needs, such as setting the change within 20ms not to exceed 50N; is the weight coefficient.

[0086] The path tracking reward function includes a function for constraining the path tracking performance of the simulated robot. The function can be calculated based on the difference between the current position generated by the model output during the model training process and the pre-set target position. It is used to perform path tracking rewards and penalties based on the input positioning information to prevent the robot from deviating from the target running direction when walking.

[0087] The specific calculation formula is as follows: ; In the formula, is the value of the path tracking reward function; It is the difference between the current position generated by the model output during the model training process and the preset target position; It is a hyperparameter used to limit the sensitivity of the path tracking reward function to errors. The specific value can be set according to actual needs.

[0088] It should be noted that when training the humanoid robot to walk, the body moves forward and backward, and the speed tracking of the left and right movement is set to be randomly sampled between -1m / s and 1m / s. The reward function corresponding to the walking task can be obtained by weighted summing all functions included in the first reward function and all functions included in the second reward function; when training to stand, the body speed only tracks zero speed. Therefore, when training the standing task, the functions included in the second reward function are set to 0, such as the ground contact force penalty function and the path tracking reward weight are set to 0, and the reward function corresponding to the standing task is obtained by weighted summing all functions included in the first reward function.

[0089] In comparison, in the prior art, in order to keep the humanoid robot walking stably, the robot is usually controlled to walk with bent knees to obtain more stable operation control. However, the knee joint motor requires a large torque, and walking with bent knees has problems such as unsightly walking, lowering the height of the body, high energy consumption, and easy to hit obstacles.

[0090] The method provided in this embodiment accurately performs deep reinforcement learning training of the target policy networks corresponding to the standing and straight-leg walking tasks of the humanoid robot through multi-dimensional reward functions such as energy consumption, orientation consistency, upright state of leg joints, speed tracking, joint movement, body movement, motion smoothness and fall penalty, as well as path tracking, ground contact force, etc., so that the target policy networks corresponding to the standing task and walking task obtained by training can effectively realize upright walking in operation control, improve the aesthetics, stability and energy efficiency of gait, avoid collision and deviation from the path, and more comprehensively improve the performance of robot operation control.

[0091] In some embodiments, step 120 specifically includes: Selecting a target strategy network corresponding to a target action task from among the target strategy networks corresponding to the plurality of action tasks to construct the initial operation control model; Performing transfer learning training on the initial operation control model to obtain an operation control model to be trained; Inputting the second sample state information and the task label into the to-be-trained operation control model to obtain predicted action information corresponding to the task label; According to the task label, obtaining the target action information corresponding to the preset training task from the expected action information corresponding to the second sample state information output by the target strategy network corresponding to the plurality of action tasks; Constructing a target loss function according to the predicted action information and the target action information, and a reward function corresponding to the preset training task; According to the target loss function, the operation control model to be trained is jointly trained by reinforcement learning training and imitation learning.

[0092] Optionally, after the target policy network corresponding to each action task is trained separately under the simulator, such as the walking model and the standing model, although the target policy network corresponding to each action task can effectively process each action task separately, such as controlling the robot to walk and stand separately. However, if the robot is directly controlled to perform different action tasks by switching models in the action task switching scenario, the robot will be unstable at the moment of switching actions. In order to ensure that the robot can switch smoothly between multiple gait actions, it is possible to further perform transfer learning, reinforcement learning training and imitation learning joint training of the multi-task policy network on the basis of training the target policy network corresponding to each action task separately, by adding a task marker, so that the trained multi-task policy network can be compatible with the task processing performance of the target policy network corresponding to different action tasks, thereby achieving compatible processing of different action tasks, ensuring smooth switching between multiple gait actions, and improving the stability of robot operation control.

[0093] The steps of joint training here specifically include the following steps: First, the target policy network corresponding to the target action task is selected from the target policy networks corresponding to the pre-trained action tasks to construct the initial operation control model for knowledge transfer, such as selecting the target policy network corresponding to the standing task (also called the pre-trained standing model) to construct the initial operation control model for knowledge transfer, etc. This embodiment does not specifically limit this. The initial operation control model determined in this way is not a randomly initialized model, but an initialized model constructed using a pre-trained high-performance model, which can effectively ensure that the model can converge to a better motion effect faster.

[0094] After the initial operation control model is obtained, transfer learning training may be performed on the initial operation control model to obtain an operation control model to be trained that has operation control domain knowledge.

[0095] Next, the target policy network corresponding to the multiple action tasks is used as the teacher model. The expected action information corresponding to the second sample state information output by the target policy network corresponding to the multiple action tasks is sampled by setting the task label corresponding to the preset training task to obtain the target action information corresponding to the preset training task, and the second sample state information and the task label are input into the operation control model to be trained to obtain the predicted action information corresponding to the task label, so as to construct the target loss function by using the predicted action information and the target action information, and the reward function corresponding to the preset training task. Thus, the operation control model to be trained is jointly trained by reinforcement learning training and imitation learning with the goal of minimizing the target loss function, so as to integrate the target policy networks corresponding to the multiple action tasks into one model through knowledge distillation, so as to achieve a smooth transition between different skills, thereby obtaining an integrated model for simultaneous multi-action task processing, and at the same time being able to learn the action task processing advantages of the target policy network corresponding to each action task of the first stage reinforcement learning training, thereby ensuring that the integrated model can realize the control generation of various gait action trajectories, so as to control the smooth switching between various gaits of the humanoid robot, and effectively improve the operation control effect of the humanoid robot.

[0096] The target loss function here can be obtained by determining the imitation learning loss function and the reinforcement learning loss function by using the predicted action information and the target action information, as well as the reward function corresponding to the preset training task, and then jointly calculating the imitation learning loss function and the reinforcement learning loss function.

[0097] Exemplarily, in some embodiments, the step of constructing the target loss function in step 120 specifically includes: Determining an imitation learning loss function according to a probability distribution difference between the predicted action information and the target action information; Determining a reinforcement learning loss function according to the predicted action information and the reward function corresponding to the preset training task; Performing weighted addition on the imitation learning loss function and the reinforcement learning loss function to obtain the target loss function; The weight coefficient between the imitation learning loss function and the reinforcement learning loss function is obtained by dynamically updating according to the number of iterations in the iterative training process.

[0098] Optionally, the calculation formula of the objective loss function is as follows: ; ; In the formula, is the value of the target loss function; is the weight coefficient between the imitation learning loss function and the reinforcement learning loss function; it can be dynamically updated according to the number of iterations in the iterative training process. For example, at the beginning of training, the proportion of imitation learning is very large. As the training progresses, the proportion of imitation learning loss gradually decreases. After a certain number of training steps, only pure reinforcement learning is used for generalization. is the value of the imitation learning loss function; is the value of the reinforcement learning loss function; is the predicted action information output by the running control model to be trained, It is the expected action information output by different expert skill models (that is, the target policy network corresponding to different action tasks).

[0099] The method provided in this embodiment realizes compatible processing of multiple action tasks by using a joint training method of transfer learning, reinforcement learning and imitation learning on the basis of a pre-trained target strategy network corresponding to each action task, thereby ensuring smooth action conversion when switching between different gait actions, improving the stability and reliability of the robot's operation control in different task scenarios, and enabling the robot to have the upright operation performance and path tracking performance corresponding to the target strategy network corresponding to each action task, effectively reducing energy consumption and knee collisions, and improving path tracking performance, thereby improving the robot's adaptability and stability in various complex tasks, achieving stable multi-gait walking, robust resistance to external interference, and the ability to adapt to changes in hardware and environmental conditions.

[0100] In some embodiments, step 130 specifically includes: Adding at least one disturbance control to the simulation robot; Acquire the third sample state information of the simulation robot under each disturbance control, and perform reinforcement learning training on the trained initial operation control model according to each third sample state information and the task label corresponding to the preset training task to obtain the operation control model; The disturbance control includes disturbance control of physical parameters and / or disturbance control of environmental parameters.

[0101] Optionally, after obtaining a trained initial operation control model that can simultaneously handle different motion tasks in a simulation environment through joint training, the trained initial operation control model can be further reinforced by reinforcement learning to further improve the generalization ability of the model, thereby minimizing the gap between the performance of the model in the simulation scenario and the performance of the model in the real scenario, thereby improving the performance of the model in the real scenario, so as to effectively control the physical robot to successfully perform various motion tasks in the real scenario. The specific implementation steps are as follows: First, in order to transfer the initial operation control model trained under the simulation robot to the physical robot, the simulation training data can be generalized. The domain generalization here can be to add disturbance control of physical parameters and / or disturbance control of environmental parameters to the simulation robot, such as adding body joint position, joint speed, external force interference, motor delay, body weight, center of mass position, ground friction, gravity vector value disturbance in different ranges.

[0102] Then, the third sample state information of the simulated robot under each disturbance control is obtained, and reinforcement learning training is performed on the trained initial operation control model according to the third sample state information under each disturbance control and the task label corresponding to the preset training task, so that the trained operation control model has stronger generalization and robustness, so that when the operation control model is deployed on the physical robot, it can have stronger anti-interference ability and realize more robust control, thereby effectively realizing the smooth switching of different gait movements of the physical robot.

[0103] The method provided in this embodiment effectively improves the robustness of Sim2real when deployed on a real machine by adding multiple generalization training parameters in simulation training, reduces the difference between the real machine effect and the simulation effect, and then when the operation control model trained thereby is deployed on a physical robot, it can have a stronger anti-interference ability and achieve more robust control, thereby effectively realizing the smooth switching of different gait movements of the physical robot.

[0104] In some embodiments, this embodiment also provides a robot operation control method.

[0105] Figure 4 It is one of the flow charts of the robot operation control method provided by the present invention. Figure 5 This is the second flow chart of the robot operation control method provided by the present invention.

[0106] like Figure 4 As shown, the method includes step 410 , step 420 , step 430 and step 440 .

[0107] Step 410, obtaining the current state information of the physical robot and the task label of the current task; the current state information includes the current body information and the current speed command; Step 420, input the current state information and the task tag of the current task into the operation control model to obtain the current action information corresponding to the task tag of the current task; Step 430, performing torque calculation on the current motion information to obtain the joint torque of the physical robot; Step 440, controlling the physical robot to perform the motion control action under the current task according to the joint torque; Wherein, the operation control model is obtained by training based on the operation control model training method provided in the above embodiments.

[0108] like Figure 5 As shown, in actual applications, it can be deployed on physical robots using Figure 1 The multi-task policy model (also called the operational control model or integrated policy network) is well trained using the training process shown.

[0109] When it is determined that the physical robot has completed the deployment of the running control model, the task label of the current task generated by the user input when the physical robot is triggered can be obtained, as well as the current speed command; wherein the task label of the current task is a label used to indicate the task type of the current task, that is, a label used to indicate whether the current task belongs to a walking task or a standing task. The current speed command includes but is not limited to the expected linear speed and the expected angular speed.

[0110] The user input here can be information input through a command line interface, a graphical interface input, touch input, a drop-down selection input, voice input, gesture input, visual input, brain-computer input, etc. This embodiment does not specifically limit this.

[0111] In addition, it is also necessary to obtain the body sensors of the physical robot, such as joint encoders and inertial measurement unit (IMU) posture sensors, to obtain the current body information of the physical robot in the real environment. The body information here includes but is not limited to the position, direction, linear velocity and angular velocity of the pelvis of the physical robot, as well as the position, joint velocity, body posture of all joints of the body and the expected position of the humanoid robot in the previous cycle, which is not specifically limited in this embodiment.

[0112] Then, the task label of the current task, the current speed command and the current body information are input into the integrated strategy network as the model input, so that the current action information corresponding to the task label of the current task can be calculated by the integrated strategy network. The current action information here is the information used to characterize the gait action trajectory, which may include the offset of the current joint position of the physical robot relative to the default joint position. .

[0113] Then, the torque is calculated based on the current action information to obtain the joint torque (also called motor torque) of the physical robot and give it to the motor for execution, so as to obtain the corresponding action instructions of the physical robot at the current moment, so that the physical robot can perform the motion control action under the current task according to the corresponding action instructions.

[0114] Among them, the specific calculation steps of joint torque include: First, the offset Add it to the default joint position to get the current joint target position. For details, please refer to the joint position calculation formula described in the above model training process for calculation, which will not be repeated here.

[0115] Then, the current joint target position is passed through the PD control formula to calculate the corresponding joint torque , which can be calculated by referring to the joint torque calculation formula described in the above model training process, and will not be repeated here.

[0116] The method provided in this embodiment obtains a multi-objective reward function and conducts joint training of transfer learning, reinforcement learning and imitation learning to construct an operation control model of an integrated strategy network compatible with multi-action task processing. The operation control model is used to control the operation of the physical robot. While realizing the multi-task compatible processing of the robot, it can also realize smooth switching between different action tasks of the robot, effectively improving the stable multi-gait walking performance of the robot and saving the energy consumption of the robot, thereby ensuring the performance of the robot's operation control.

[0117] The operation control model training system provided by the present invention is described below. The operation control model training system described below and the operation control model training method described above can be referenced to each other.

[0118] Figure 6 is a schematic diagram of the structure of the operation control model training system provided by the present invention; Figure 6 As shown, the system includes: The first training unit 610 is used to perform reinforcement learning training on the initial strategy network according to the first sample state information of the simulation robot and the reward function corresponding to each action task, so as to obtain the target strategy network corresponding to each action task; The second training unit 620 is used to perform transfer learning, reinforcement learning training and imitation learning joint training on the initial operation control model according to the second sample state information of the simulation robot, the task label corresponding to the preset training task, and the expected action information corresponding to the second sample state information output by the target strategy network corresponding to each action task; The model building unit 630 is used to build an operation control model according to the trained initial operation control model, wherein the operation control model is used to control the physical robot to perform different motion control actions under different action tasks; Wherein, each sample status information in the first sample status information and the second sample status information includes sample body information and sample speed command.

[0119] The system provided in this embodiment first performs reinforcement learning training on the initial policy network according to the first sample state information of the simulated robot and the reward function corresponding to each action task, so as to obtain the target policy network corresponding to each action task; then, according to the second sample state information of the simulated robot, the task label corresponding to the preset training task, and the expected action information corresponding to the second sample state information output by the target policy network corresponding to each action task, the initial operation control model is jointly trained by transfer learning, reinforcement learning training and imitation learning, so as to construct an operation control model of the integrated strategy network compatible with multi-action task processing, so as to realize the multi-task compatible processing of the robot and the smooth switching between different action tasks of the robot, thereby effectively improving the stable multi-gait walking performance of the robot and saving the energy consumption of the robot, thereby ensuring the performance of the robot's operation control.

[0120] In some embodiments, the motion task includes a standing task or a walking task; The first training unit is specifically used for: According to the first sample state information and the reward function corresponding to the standing task, the initial policy network is subjected to reinforcement learning training to obtain a target policy network corresponding to the standing task; According to the first sample state information and the reward function corresponding to the walking task, the initial policy network is subjected to reinforcement learning training to obtain a target policy network corresponding to the walking task; Among them, the reward function corresponding to the walking task includes a first reward function and a second reward function, and the reward function corresponding to the standing task includes the first reward function; the first reward function is a reward function constructed based on the action parameters corresponding to the simulation robot, and the second reward function is a reward function constructed based on the environmental interaction information corresponding to the simulation robot.

[0121] In some embodiments, the first reward function includes at least one of an energy consumption penalty function, a running posture constraint function, a speed tracking reward function, a joint motion penalty function, a body motion penalty function, an action smoothness function, and a fall penalty function; Among them, the energy consumption penalty function includes a function for constraining the energy consumption generated by joint movements; the running posture constraint function includes a function for constraining the consistency of the orientation of the two feet, two knees, two hips and the fuselage and / or optimizing the upright state of the leg joints; the speed tracking reward function includes a function for constraining the instruction tracking performance corresponding to the linear velocity and / or angular velocity respectively; the joint motion penalty function includes a function for constraining at least one joint motion parameter of the joint acceleration, joint torque, joint extreme position, and joint extreme force; the fuselage motion penalty function includes a function for constraining the fuselage posture and / or fuselage motion state.

[0122] In some embodiments, the second reward function includes a path tracking reward function and / or a ground contact force penalty function; The path tracking reward function includes a function for constraining the path tracking performance of the simulated robot; and the ground contact force penalty function is a function for constraining the contact pressure between the simulated robot and the ground.

[0123] In some embodiments, the second training unit is specifically configured to: Selecting a target strategy network corresponding to a target action task from among the target strategy networks corresponding to the plurality of action tasks to construct the initial operation control model; Performing transfer learning training on the initial operation control model to obtain an operation control model to be trained; Inputting the second sample state information and the task label into the to-be-trained operation control model to obtain predicted action information corresponding to the task label; According to the task label, obtaining the target action information corresponding to the preset training task from the expected action information corresponding to the second sample state information output by the target strategy network corresponding to the plurality of action tasks; Constructing a target loss function according to the predicted action information and the target action information, and a reward function corresponding to the preset training task; According to the target loss function, the operation control model to be trained is jointly trained by reinforcement learning training and imitation learning.

[0124] In some embodiments, the second training unit is further configured to: Determining an imitation learning loss function according to a probability distribution difference between the predicted action information and the target action information; Determining a reinforcement learning loss function according to the predicted action information and the reward function corresponding to the preset training task; Performing weighted addition on the imitation learning loss function and the reinforcement learning loss function to obtain the target loss function; The weight coefficient between the imitation learning loss function and the reinforcement learning loss function is obtained by dynamically updating according to the number of iterations in the iterative training process.

[0125] In some embodiments, the model building unit is specifically configured to: Adding at least one disturbance control to the simulation robot; Acquire the third sample state information of the simulation robot under each disturbance control, and perform reinforcement learning training on the trained initial operation control model according to each third sample state information and the task label corresponding to the preset training task to obtain the operation control model; The disturbance control includes disturbance control of physical parameters and / or disturbance control of environmental parameters.

[0126] In some embodiments, the present application also provides a robot operation control system.

[0127] Figure 7 It is a schematic diagram of the structure of the robot operation control system provided by the present invention. Figure 7 As shown, the system includes: The acquisition unit 710 is used to acquire the current state information of the physical robot and the task label of the current task; the current state information includes the current body information and the current speed command; The prediction unit 720 is used to input the current state information and the task label of the current task into the operation control model to obtain the current action information corresponding to the task label of the current task; The calculation unit 730 is used to perform torque calculation on the current motion information to obtain the joint torque of the physical robot; The control unit 740 is used to control the physical robot to perform the motion control action under the current task according to the joint torque; The operation control model is obtained by training based on the operation control model training method provided in the above embodiments. Figure 1 The training process shown will not be repeated here.

[0128] The system provided in this embodiment obtains a multi-objective reward function and performs joint training of transfer learning, reinforcement learning training and imitation learning to construct an operation control model of an integrated strategy network compatible with multi-action task processing. The operation control model is used to control the operation of the physical robot. While realizing the multi-task compatible processing of the robot, it can also realize smooth switching between different action tasks of the robot, effectively improving the stable multi-gait walking performance of the robot and saving the energy consumption of the robot, thereby ensuring the performance of the robot's operation control. The system provided by the present invention is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for the specific processes and detailed contents, which will not be repeated here.

[0129] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8As shown, the electronic device may include: a processor (processor) 810, a communication interface (Communications Interface) 820, a memory (memory) 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call the logic instructions in the memory 830 to execute the operation control model training method, which includes: according to the first sample state information of the simulated robot and the reward function corresponding to each action task, respectively performing reinforcement learning training on the initial policy network to obtain the target policy network corresponding to each action task; according to the second sample state information of the simulated robot, the task label corresponding to the preset training task, and the expected action information corresponding to the second sample state information output by the target policy network corresponding to each action task, the initial operation control model is jointly trained by transfer learning, reinforcement learning training and imitation learning; according to the trained initial operation control model, an operation control model is constructed, and the operation control model is used to control the physical robot to perform different motion control actions under different action tasks; wherein, each sample state information in the first sample state information and the second sample state information contains sample body information and sample speed commands. Alternatively, a robot operation control method is executed, the method comprising: obtaining current state information of the physical robot and a task label of the current task; the current state information comprises current body information and a current speed command; the current state information and the task label of the current task are input into an operation control model to obtain current action information corresponding to the task label of the current task; torque calculation is performed on the current action information to obtain the joint torque of the physical robot; according to the joint torque, the physical robot is controlled to perform the motion control action under the current task; wherein the operation control model is trained based on the operation control model training method.

[0130] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0131] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the operation control model training method provided by the above-mentioned methods, which method includes: according to the first sample state information of the simulated robot and the reward function corresponding to each action task, respectively performing reinforcement learning training on the initial policy network to obtain the target policy network corresponding to each action task; according to the second sample state information of the simulated robot, the task label corresponding to the preset training task, and the expected action information corresponding to the second sample state information output by the target policy network corresponding to each action task, the initial operation control model is jointly trained by transfer learning, reinforcement learning training and imitation learning; according to the trained initial operation control model, an operation control model is constructed, and the operation control model is used to control the physical robot to perform different motion control actions under different action tasks; wherein, each sample state information in the first sample state information and the second sample state information contains sample body information and sample speed commands. Alternatively, a robot operation control method is executed, the method comprising: obtaining current state information of the physical robot and a task label of the current task; the current state information comprises current body information and a current speed command; the current state information and the task label of the current task are input into an operation control model to obtain current action information corresponding to the task label of the current task; torque calculation is performed on the current action information to obtain the joint torque of the physical robot; according to the joint torque, the physical robot is controlled to perform the motion control action under the current task; wherein the operation control model is trained based on the operation control model training method.

[0132] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the operation control model training method provided by the above-mentioned methods, the method comprising: performing reinforcement learning training on the initial policy network according to the first sample state information of the simulated robot and the reward function corresponding to each action task, so as to obtain the target policy network corresponding to each of the action tasks; performing transfer learning, reinforcement learning training and imitation learning joint training on the initial operation control model according to the second sample state information of the simulated robot, the task label corresponding to the preset training task, and the expected action information corresponding to the second sample state information output by the target policy network corresponding to each of the action tasks; constructing an operation control model according to the trained initial operation control model, the operation control model being used to control the physical robot to perform different motion control actions under different of the action tasks; wherein each sample state information in the first sample state information and the second sample state information contains sample body information and sample speed commands. Alternatively, a robot operation control method is executed, the method comprising: obtaining current state information of the physical robot and a task label of the current task; the current state information comprises current body information and a current speed command; the current state information and the task label of the current task are input into an operation control model to obtain current action information corresponding to the task label of the current task; torque calculation is performed on the current action information to obtain the joint torque of the physical robot; according to the joint torque, the physical robot is controlled to perform the motion control action under the current task; wherein the operation control model is trained based on the operation control model training method.

[0133] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0134] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training an operation control model, characterized in that: include: According to the first sample state information of the simulation robot and the reward function corresponding to each action task, the initial strategy network is respectively subjected to reinforcement learning training to obtain the target strategy network corresponding to each action task; Performing joint training of transfer learning, reinforcement learning and imitation learning on the initial operation control model according to the second sample state information of the simulation robot, the task label corresponding to the preset training task, and the expected action information corresponding to the second sample state information output by the target strategy network corresponding to each of the action tasks; According to the trained initial operation control model, an operation control model is constructed, wherein the operation control model is used to control the physical robot to perform different motion control actions under different action tasks; Wherein, each sample status information in the first sample status information and the second sample status information includes sample body information and sample speed command.

2. The operation control model training method according to claim 1, characterized in that: The action task includes a standing task or a walking task; According to the first sample state information of the simulation robot and the reward function corresponding to each action task, the initial strategy network is respectively subjected to reinforcement learning training to obtain the target strategy network corresponding to each action task, including: According to the first sample state information and the reward function corresponding to the standing task, the initial policy network is subjected to reinforcement learning training to obtain a target policy network corresponding to the standing task; According to the first sample state information and the reward function corresponding to the walking task, the initial policy network is subjected to reinforcement learning training to obtain a target policy network corresponding to the walking task; Among them, the reward function corresponding to the walking task includes a first reward function and a second reward function, and the reward function corresponding to the standing task includes the first reward function; the first reward function is a reward function constructed based on the action parameters corresponding to the simulation robot, and the second reward function is a reward function constructed based on the environmental interaction information corresponding to the simulation robot.

3. The operation control model training method according to claim 2, characterized in that: The first reward function includes at least one of an energy consumption penalty function, a running posture constraint function, a speed tracking reward function, a joint motion penalty function, a body motion penalty function, an action smoothness function, and a fall penalty function; Among them, the energy consumption penalty function includes a function for constraining the energy consumption generated by joint movements; the running posture constraint function includes a function for constraining the consistency of the orientation of the two feet, two knees, two hips and the fuselage and / or optimizing the upright state of the leg joints; the speed tracking reward function includes a function for constraining the instruction tracking performance corresponding to the linear velocity and / or angular velocity respectively; the joint motion penalty function includes a function for constraining at least one joint motion parameter of the joint acceleration, joint torque, joint extreme position, and joint extreme force; the fuselage motion penalty function includes a function for constraining the fuselage posture and / or fuselage motion state.

4. The operation control model training method according to claim 2, characterized in that: The second reward function includes a path tracking reward function and / or a ground contact force penalty function; The path tracking reward function includes a function for constraining the path tracking performance of the simulated robot; and the ground contact force penalty function is a function for constraining the contact pressure between the simulated robot and the ground.

5. The operation control model training method according to any one of claims 1 to 4, characterized in that: The method includes performing transfer learning, reinforcement learning training and imitation learning joint training on the initial operation control model according to the second sample state information of the simulation robot, the task label corresponding to the preset training task, and the expected action information corresponding to the second sample state information output by the target strategy network corresponding to each action task, including: Selecting a target strategy network corresponding to a target action task from among the target strategy networks corresponding to the plurality of action tasks to construct the initial operation control model; Performing transfer learning training on the initial operation control model to obtain an operation control model to be trained; Inputting the second sample state information and the task label into the to-be-trained operation control model to obtain predicted action information corresponding to the task label; According to the task label, obtaining the target action information corresponding to the preset training task from the expected action information corresponding to the second sample state information output by the target strategy network corresponding to the plurality of action tasks; Constructing a target loss function according to the predicted action information and the target action information, and a reward function corresponding to the preset training task; According to the target loss function, the operation control model to be trained is jointly trained by reinforcement learning training and imitation learning.

6. The operation control model training method according to claim 5, characterized in that: The constructing a target loss function according to the predicted action information and the target action information, and the reward function corresponding to the preset training task, includes: Determining an imitation learning loss function according to a probability distribution difference between the predicted action information and the target action information; Determining a reinforcement learning loss function according to the predicted action information and the reward function corresponding to the preset training task; Performing weighted addition on the imitation learning loss function and the reinforcement learning loss function to obtain the target loss function; The weight coefficient between the imitation learning loss function and the reinforcement learning loss function is obtained by dynamically updating according to the number of iterations in the iterative training process.

7. The operation control model training method according to any one of claims 1 to 4, characterized in that: The step of constructing the operation control model according to the trained initial operation control model includes: Adding at least one disturbance control to the simulation robot; Acquire the third sample state information of the simulation robot under each disturbance control, and perform reinforcement learning training on the trained initial operation control model according to each third sample state information and the task label corresponding to the preset training task to obtain the operation control model; The disturbance control includes disturbance control of physical parameters and / or disturbance control of environmental parameters.

8. A robot operation control method, characterized in that: include: Get the current status information of the physical robot and the task label of the current task; The current state information includes current body information and current speed command; Input the current state information and the task tag of the current task into the operation control model to obtain the current action information corresponding to the task tag of the current task; Performing torque calculation on the current motion information to obtain the joint torque of the physical robot; According to the joint torque, controlling the physical robot to perform the motion control action under the current task; Wherein, the operation control model is trained based on the operation control model training method according to any one of claims 1 to 7.

9. A running control model training system, characterized in that: include: A first training unit is used to perform reinforcement learning training on the initial strategy network according to the first sample state information of the simulation robot and the reward function corresponding to each action task, so as to obtain a target strategy network corresponding to each action task; A second training unit is used to perform transfer learning, reinforcement learning training and imitation learning joint training on the initial operation control model according to the second sample state information of the simulation robot, the task label corresponding to the preset training task, and the expected action information corresponding to the second sample state information output by the target strategy network corresponding to each of the action tasks; A model building unit, used to build an operation control model according to the trained initial operation control model, wherein the operation control model is used to control the physical robot to perform different motion control actions under different action tasks; Wherein, each sample status information in the first sample status information and the second sample status information includes sample body information and sample speed command.

10. A robot operation control system, characterized in that: include: An acquisition unit, used to obtain the current state information of the physical robot and the task label of the current task; The current state information includes current body information and current speed command; A prediction unit, used to input the current state information and the task label of the current task into the operation control model to obtain current action information corresponding to the task label of the current task; A calculation unit, used to perform torque calculation on the current motion information to obtain the joint torque of the physical robot; A control unit, used for controlling the physical robot to perform the motion control action under the current task according to the joint torque; Wherein, the operation control model is trained based on the operation control model training method according to any one of claims 1 to 7.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the operation control model training method as described in any one of claims 1 to 7, or the robot operation control method as described in claim 8 is implemented.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the operation control model training method as described in any one of claims 1 to 7, or the robot operation control method as described in claim 8 is implemented.

Images