Methods, devices, equipment, robots, and media for training humanoid robot in-situ motion control strategy networks based on deep reinforcement learning.

Abstract

This disclosure provides a method, apparatus, device, robot, and medium for training a robot in-situ motion control policy network based on deep reinforcement learning, relating to the field of robotics. It includes: constructing a first input for controlling the height of a humanoid robot, a second input for controlling linear velocity, a third input for controlling angular velocity, and a fourth input for indicating the type of training task; setting the first input to a random value, setting the second and third inputs to zero, and setting the fourth input to an in-situ task, and executing a first training task; setting the first, second, and third inputs to fixed values, and setting the fourth input to a walking task, and executing a second training task; updating the in-situ motion control policy network based on the motion policies output from the first and second training tasks and their corresponding reward function sub-items. This disclosure avoids the problem of motion output jitter caused by over-reliance on inhibitory constraints during training.

Description

Technical Field

[0001] This disclosure relates to the field of robotics technology, and to a method, apparatus, device, robot, and medium for training a humanoid robot's in-situ motion control strategy network based on deep reinforcement learning. Background Technology

[0002] In-situ tasks refer to tasks performed by the robot through the movement of its joints, while maintaining a relatively constant contact position between the robot's feet or other supporting ends and the ground or other supporting surfaces, without any overall displacement. Examples include standing still, swinging the arm in place, grasping objects in place, and operating equipment in place. However, humanoid robots trained using deep reinforcement learning often exhibit joint tremors and unstable control when performing in-situ tasks.

[0003] The paper "Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion Control" (Zhongyu Li et al., August 26, 2024) proposes a general motion control framework for bipedal robots that can achieve various dynamic motion skills, including standing, walking, running, and jumping. Although this framework supports accepting different task commands as input, enabling the policy network to learn multiple task types under the same structure and achieve smooth switching between walking and standing, this scheme only reflects different task types through command parameters during training and does not distinguish task types from the input configuration, which easily leads to the risk of underfitting during training. At the same time, the inventors found in practice that although the scheme adds multiple smoothing penalty terms to the reward function to suppress the sharp fluctuations of the policy, it fails to fundamentally alleviate the motion jitter problem from the perspective of training sample structure or input method.

[0004] In some existing technologies, such as the Chinese patent application with publication number CN119748447A, an integrated policy network capable of handling multiple action tasks is constructed to achieve multi-task compatibility for robots. However, this approach obtains independent target policy networks corresponding to different action tasks. Each target policy network is in a single task state during training, but it does not provide an effective strategy to address the underfitting of the policy caused by insufficient changes in the single task state. At the same time, the inventors found in practice that this approach relies heavily on inhibitory constraints to limit the policy output during training. When environmental disturbances exceed the coverage of the training data or when there are conflicts between constraints, the policy output is still prone to oscillation, which in turn leads to jitter in the action output. Summary of the Invention

[0005] This disclosure provides a method, apparatus, device, robot, and medium for training a humanoid robot's in-situ motion control strategy network based on deep reinforcement learning. This enables the robot to perform transfer learning between different motion modes, avoiding joint tremor problems caused by single-task training and improving the stability of in-situ motion control.

[0006] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure.

[0007] According to a first aspect of this disclosure, a method for training a robot in-situ motion control policy network based on deep reinforcement learning is provided, comprising:

[0008] Construct a first input for controlling the height of the humanoid robot's body, a second input for controlling the linear velocity, a third input for controlling the angular velocity, and a fourth input for indicating the type of training task;

[0009] Set the first input to a random value, set the second and third inputs to zero, and set the fourth input to the in-place task, and input them into the in-place motion control policy network to execute the first training task;

[0010] Set the first input to a fixed value, set the second and third inputs to fixed values, and set the fourth input to the walking task, and input them into the stationary motion control policy network to execute the second training task;

[0011] Update the stationary motion control policy network based on the action policies output from the first and second training tasks, as well as the reward function sub-items corresponding to the stationary and walking tasks.

[0012] In one exemplary embodiment of this disclosure, the in-place motion control policy network is updated based on the motion policies output from the first training task and the second training task, as well as the reward function sub-items corresponding to the in-place task and the walking task, respectively, including:

[0013] Calculate the reward function sub-item corresponding to the stationary task based on the action policy output by the first training task, and calculate the reward function sub-item corresponding to the walking task based on the action policy output by the second training task;

[0014] Construct a joint training loss based on the reward function sub-item corresponding to the first training task and the reward function sub-item corresponding to the second training task;

[0015] The execution parameters in the in-situ motion control policy network are updated based on the joint training loss.

[0016] In one exemplary embodiment of this disclosure, calculating the reward function sub-item corresponding to the in-place task based on the action policy output by the first training task includes:

[0017] The humanoid robot is controlled to perform stationary tasks based on the motion strategy output from the first training task, and the angular velocity change data of each driven joint and the posture angle change data of each body part are collected when the humanoid robot performs stationary tasks.

[0018] The stability score for the first training task is determined based on angular velocity change data and attitude angle change data.

[0019] The reward function sub-item corresponding to the first training task is determined based on the stability score.

[0020] In one exemplary embodiment of this disclosure, determining a stability score for a first training task based on angular velocity change data and attitude angle change data includes:

[0021] The average change value of the angular velocity change data corresponding to each drive joint is weighted and calculated to obtain the first change characteristic value, and the average change value of the attitude angle change data corresponding to each body part is weighted and calculated to obtain the second change characteristic value.

[0022] The first change feature value and the second change feature value are weighted and summed according to preset weights to obtain a comprehensive change feature value;

[0023] The stability score corresponding to the comprehensive change feature value is determined by a preset monotonically decreasing mapping relationship.

[0024] In one exemplary embodiment of this disclosure, determining the reward function sub-item corresponding to the first training task based on the stability score includes:

[0025] The stability score is normalized based on the preset upper and lower limits to obtain the normalized score.

[0026] When the normalized score is greater than the first threshold, the reward function sub-item corresponding to the first training task is determined based on the linear growth model.

[0027] When the normalized score is less than or equal to the first threshold and greater than or equal to the second threshold, the reward function sub-item corresponding to the first training task is determined based on the linear interpolation model.

[0028] When the normalized score is less than the second threshold, the reward function sub-item corresponding to the first training task is determined to be zero; where the first threshold is greater than the second threshold.

[0029] In one exemplary embodiment of this disclosure, when the normalized score is greater than a first threshold, a reward function sub-item corresponding to the first training task is determined based on a linear growth model, including:

[0030]

[0031] in, This represents the reward function sub-item corresponding to the first training task. This represents the preset base reward value. This represents the normalized score. Indicates the first reward coefficient. This represents the first threshold.

[0032] In one exemplary embodiment of this disclosure, when the normalized score is less than or equal to a first threshold and greater than or equal to a second threshold, a reward function sub-item corresponding to the first training task is determined based on a linear interpolation model, including:

[0033] The reward function sub-item corresponding to the first training task is determined according to the following formula:

[0034]

[0035] in, This represents the reward function sub-item corresponding to the first training task. This represents the preset base reward value. This represents the normalized score. Indicates the first threshold. This indicates the second threshold.

[0036] In one exemplary embodiment of this disclosure, calculating the reward function sub-item corresponding to the walking task based on the action policy output by the second training task includes:

[0037] The humanoid robot is controlled to perform a walking task based on the motion strategy output from the second training task, and the angle error between the actual walking direction and the target walking direction of the humanoid robot during the walking task is collected.

[0038] Calculate the reward function sub-item corresponding to the walking task based on the included angle error.

[0039] In one exemplary embodiment of this disclosure, calculating the reward function sub-item corresponding to the walking task based on the included angle error includes:

[0040] When the included angle error is less than the error threshold, the reward function sub-item corresponding to the walking task is determined according to the following function:

[0041]

[0042] in, This represents the reward function sub-item corresponding to the second training task. Indicates the included angle error. Indicates the error threshold. Indicates the second reward coefficient;

[0043] When the included angle error is greater than or equal to the error threshold, the reward function sub-item corresponding to the walking task is set to zero.

[0044] In one exemplary embodiment of this disclosure, a joint training loss is constructed based on the reward function sub-item corresponding to the first training task and the reward function sub-item corresponding to the second training task, including:

[0045] Based on the reward function sub-item corresponding to the first training task, and the log probability of the action policy output in the first training task in the action space, the first training loss is constructed.

[0046] Based on the reward function sub-item corresponding to the second training task, and the log probability of the action policy output in the second training task in the action space, a second training loss is constructed.

[0047] The joint training loss is determined by the weighted sum of the first training loss and the second training loss.

[0048] In one exemplary embodiment of this disclosure, constructing a first input for controlling the height of the humanoid robot's body, a second input for controlling the linear velocity, a third input for controlling the angular velocity, and a fourth input for indicating the type of training task includes:

[0049] Determine the input construction rules for the in-place task and the walking task respectively. The input construction rules include the data type, value range and logical relationship of the first input, the second input, the third input and the fourth input respectively.

[0050] In one exemplary embodiment of this disclosure, the method further includes:

[0051] Determine the ratio of input data corresponding to the first training task and the second training task;

[0052] Every preset training period, determine the average rate of change of the reward function sub-items corresponding to the first training task and the second training task within the preset training period;

[0053] The proportion of input data is dynamically adjusted based on the rate of change of the average value.

[0054] In one exemplary embodiment of this disclosure, the in-situ motion control policy network includes:

[0055] The state coding module is used to receive a first input for controlling the height of the humanoid robot's body, a second input for controlling the linear velocity, a third input for controlling the angular velocity, and a fourth input for indicating the type of training task, and constructs an input vector.

[0056] The task modulation module is used to control the in-situ motion control strategy network to execute the first training task or the second training task according to the input vector based on the fourth input.

[0057] The strategy update module is used to update the stationary motion control policy network based on the action policies output in the first and second training tasks, as well as the reward function sub-items corresponding to the stationary and walking tasks, respectively.

[0058] In one exemplary embodiment of this disclosure, controlling the in-situ motion control policy network to perform a first training task or a second training task based on the input vector, according to a fourth input, includes:

[0059] The current training task type is determined based on the fourth input. The current training task type is either the first training task or the second training task.

[0060] Determine the execution parameters corresponding to the current training task type in the in-situ motion control policy network;

[0061] The reward function sub-item corresponding to the current training task type is determined based on the execution parameters corresponding to the current training task type, and the execution parameters are updated based on the reward function sub-item corresponding to the current training task type.

[0062] According to a second aspect of this disclosure, a method for in-situ motion control of a humanoid robot based on deep reinforcement learning is provided, comprising:

[0063] Determine a first input for controlling the height of the humanoid robot's body, a second input for controlling the linear velocity, a third input for controlling the angular velocity, and a fourth input for indicating the task type;

[0064] The first, second, third, and fourth inputs are fed into a pre-trained in-place action control policy network, and the control policy for the in-place task is determined based on the in-place action control policy network.

[0065] The in-situ motion control policy network is obtained according to the training method for the in-situ motion control policy network of humanoid robots based on deep reinforcement learning described above.

[0066] According to a third aspect of this disclosure, a training device for a humanoid robot in-situ motion control policy network based on deep reinforcement learning is provided, comprising:

[0067] The module is used to construct a first input for controlling the height of the humanoid robot's body, a second input for controlling the linear velocity, a third input for controlling the angular velocity, and a fourth input for indicating the type of training task.

[0068] The first training module is used to set the first input to a random value, set the second and third inputs to zero, and set the fourth input to the in-place task, and input them into the in-place motion control policy network to execute the first training task.

[0069] The second training module is used to set the first input to a fixed value, set the second and third inputs to fixed values, and set the fourth input to a walking task, and input them into the stationary motion control strategy network to execute the second training task.

[0070] The update module is used to update the stationary motion control policy network based on the action policies output in the first and second training tasks, as well as the reward function sub-items corresponding to the stationary and walking tasks, respectively.

[0071] According to a fourth aspect of this disclosure, a humanoid robot in-situ motion control device based on deep reinforcement learning is provided, comprising:

[0072] The module determines a first input for controlling the height of the humanoid robot's body, a second input for controlling the linear velocity, a third input for controlling the angular velocity, and a fourth input for indicating the task type.

[0073] The control module is used to input the first input, the second input, the third input, and the fourth input together into a pre-trained in-situ motion control policy network, and determine the control policy for the in-situ task based on the in-situ motion control policy network; wherein, the in-situ motion control policy network is obtained according to the training method for humanoid robot in-situ motion control policy network based on deep reinforcement learning described above.

[0074] According to a fifth aspect of this disclosure, an electronic device is provided, comprising:

[0075] Processor; and

[0076] A memory that stores computer-readable instructions, which, when executed by a processor, implement the method as described in the above embodiments.

[0077] According to a sixth aspect of this disclosure, a robot is provided, comprising:

[0078] Processor; and

[0079] A memory that stores computer-readable instructions, which, when executed by a processor, implement the method as described in the above embodiments.

[0080] In one exemplary embodiment of this disclosure, the robot includes any one of legged robots, quadruped robots, bipedal robots, wheeled robots, wheel-legged robots, quadrupedal robots, humanoid robots, cleaning robots, transport robots, mobile robots, and robotic arms.

[0081] According to a seventh aspect of this disclosure, a computer-readable storage medium is provided that stores computer program code instructions, which, when invoked by a robot's processor, cause the robot to perform the method as described in the above embodiments.

[0082] As can be seen from the above technical solution, this disclosure possesses at least one of the following advantages and positive effects:

[0083] This disclosure constructs four types of input structures—body height, linear velocity, angular velocity, and task type—within a single in-situ motion control policy network. It also configures diverse input data within the network based on different task types, exposing the network to richer posture variations and dynamic response distributions. This avoids underfitting caused by a single training state. By applying the reward function sub-terms for walking and in-situ tasks to the same set of network parameters, the same in-situ motion control policy network learns and switches between in-situ and walking tasks. Furthermore, the walking training task provides additional gradient constraints and state exploration for the in-situ motion control policy network to perform in-situ tasks, ultimately improving the smoothness and stability of in-situ task execution and avoiding the jitter in motion output caused by reliance on inhibitory constraints during training in traditional methods. Attached Figure Description

[0084] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0085] Figure 1 A system architecture diagram of a robot in-situ motion control policy network training method based on deep reinforcement learning according to an embodiment of this disclosure is shown.

[0086] Figure 2 A flowchart illustrating the robot in-situ motion control policy network training method based on deep reinforcement learning in an embodiment of this disclosure is shown.

[0087] Figure 3 The diagram illustrates the input construction rules corresponding to the in-place task and the walking task in the embodiments of this disclosure.

[0088] Figure 4 A schematic diagram of the structure of the in-situ motion control strategy network in an embodiment of this disclosure is shown.

[0089] Figure 5 A flowchart illustrating the method for calculating the reward function sub-item corresponding to the in-place task in an embodiment of this disclosure is shown.

[0090] Figure 6 A flowchart illustrating a method for determining the stability score of a first training task in an embodiment of this disclosure is shown.

[0091] Figure 7 A flowchart illustrating the method for determining the reward function sub-item corresponding to the walking task in an embodiment of this disclosure is shown.

[0092] Figure 8 A flowchart illustrating a robot in-situ motion control method based on deep reinforcement learning in an embodiment of this disclosure is shown.

[0093] Figure 9 A block diagram of a robot in-situ motion control policy network training device based on deep reinforcement learning is shown in an embodiment of this disclosure.

[0094] Figure 10 A block diagram of a robot in-situ motion control device based on deep reinforcement learning is shown in an embodiment of this disclosure.

[0095] Figure 11 A schematic diagram of a type of robot is shown, illustrating a robot in-situ motion control method based on deep reinforcement learning according to an embodiment of this disclosure.

[0096] Figure 12 A schematic diagram of another type of robot is shown in the embodiments of the present disclosure, illustrating the robot in-situ motion control method based on deep reinforcement learning.

[0097] Figure 13 A schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure is shown.

[0098] Figure 14 A schematic diagram of a computer-readable storage medium according to an embodiment of the present disclosure is shown. Detailed Implementation

[0099] In this disclosure, the terms "first" and "second" are used for description only and do not indicate relative importance or imply the number of technical features. Therefore, the features referred to as "first" or "second" may explicitly or implicitly include at least one of those features. "A plurality of" means at least two, unless otherwise expressly defined.

[0100] First, the relevant terms used in the exemplary embodiments of this disclosure will be explained:

[0101] Deep Reinforcement Learning (DRL) is a machine learning method that combines reinforcement learning decision-making mechanisms with deep neural networks. By enabling an agent to interact with the environment in the training environment, the neural network parameters are iteratively updated based on the reward signals from the environment, thereby learning the mapping relationship from state to action.

[0102] Control strategy network: refers to a neural network model trained through deep reinforcement learning, which is used to output actions or control commands to control the robot's movement based on the robot's current state information.

[0103] Motion strategy: refers to the control results output by the control strategy network for controlling the robot's motion, including but not limited to joint control commands, force or torque control commands, gait parameters or posture adjustment commands, which are used to drive the robot to complete motion control.

[0104] Reward function / reward term: refers to the function used to calculate the reward value based on the robot's motion state, action output, and environmental feedback during deep reinforcement learning training; reward term refers to the various sub-terms that make up the reward function, used to constrain different motion characteristics or control objectives respectively.

[0105] Figure 1 A system architecture diagram is shown that can be applied to the deep reinforcement learning-based humanoid robot in-situ motion control policy network training method in the embodiments of this disclosure.

[0106] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a humanoid robot 102, a network 103, and a server 104. The terminal device 101 can be a desktop computer, a laptop computer, a smartphone, or a tablet computer, etc. The terminal device 101 can serve as an interactive interface, providing visualization functions to display information such as the robot 102's body height, forward translation speed, and rotational speed around the vertical axis when performing stationary tasks. It also supports sending stationary motion control commands to the robot 102.

[0107] Robot 102 is equipped with various sensors to collect its own state information during stationary tasks. Under the control of server 104, it can execute stationary motion strategies to complete corresponding training actions or tasks. Server 104 is deployed with a deep reinforcement learning-based training framework. During training, server 104 receives data from robot 102 on its body height, linear velocity, angular velocity, and training task type, and inputs this data into the stationary motion control strategy network. Combining the output motion strategy with the reward function sub-items corresponding to stationary and walking tasks, the parameters of the stationary motion control strategy network are optimized using deep reinforcement learning algorithms. Server 104 also supports multi-concurrent training task management, performing task scheduling and policy transfer.

[0108] After training, server 104 can deploy the trained in-situ motion control strategy network to robot 102, enabling it to autonomously reason and execute motion strategies in actual operation. Terminal device 101 can also be used to call and deploy motion strategies, monitor robot motion status, or remotely schedule and control the robot.

[0109] Network 103 serves as the medium for providing a communication link between terminal device 101, robot 102, and server 104. Network 103 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc. It should be understood that... Figure 1 The number and type of terminal devices, robots, networks, and servers shown are merely illustrative. Depending on implementation needs, any number and type of terminal devices, robots, networks, and servers can be included.

[0110] This disclosure provides an exemplary embodiment of a training method for a humanoid robot's in-situ motion control policy network based on deep reinforcement learning. This in-situ motion control policy network can receive the robot's current control inputs, including a first input for adjusting body height, a second input representing the target linear velocity, a third input representing the target angular velocity, and a fourth input identifying the current task type: in-situ or walking. During training, the in-situ motion control policy network can dynamically adjust its execution parameters according to the task type corresponding to the fourth input, enabling task switching control between in-situ and walking tasks, while avoiding feature interference between different training tasks and improving the stability of the control policy. In the policy training phase, this embodiment executes a first training task and a second training task respectively, utilizing corresponding reward function sub-terms and iteratively optimizing the network parameters through a deep reinforcement learning algorithm. This training method does not rely on traditional inhibitory constraint policies based on penalty terms, but instead achieves a more stable control policy output by introducing multi-task state perturbations and a unified network structure, alleviating the problem of policy output oscillations in in-situ tasks that lead to jitter in the movements. In the exemplary embodiments of this disclosure, deep reinforcement learning includes, but is not limited to, algorithms such as PPO (Proximal Policy Optimization), DDPG (Deep Deterministic Policy Gradient), and SAC (Soft Actor-Critic). This disclosure does not limit the specific algorithm type of deep reinforcement learning.

[0111] In some example implementations, references Figure 2 As shown, the training method for the in-situ motion control policy network of a humanoid robot based on deep reinforcement learning includes steps S201-S204:

[0112] Step S201: Construct a first input for controlling the height of the humanoid robot's body, a second input for controlling the linear velocity, a third input for controlling the angular velocity, and a fourth input for indicating the type of training task.

[0113] In some example implementations, the first input can be used to indicate the humanoid robot's body height under the current training task; the second input can be used to indicate the humanoid robot's translational speed in the horizontal direction; the third input can be used to indicate the angular velocity of the humanoid robot's entire body rotating about a vertical axis; and the fourth input can be used to indicate the current training task type, which may include stationary tasks and walking tasks. Stationary tasks represent actions such as standing still with both feet stationary and the body's center of gravity controlled within a certain range, such as standing in place, bending knees in place, or standing up in place. In stationary tasks, the linear velocity of the humanoid robot's translation in the horizontal direction and the angular velocity of the entire body rotating about a vertical axis are typically zero or minimal. Walking tasks represent humanoid robots performing translational motion on the ground, completing forward, lateral, or curved path walking tasks. In walking tasks, the linear velocity and / or angular velocity are typically non-zero values.

[0114] By using the above method, body height, linear velocity, angular velocity, and task type information are uniformly constructed into input data, enabling the in-place motion control strategy network to receive input data from different tasks at the same input interface, thus providing a foundation for subsequent training of different training task types.

[0115] Step S202: Set the first input to a random value, set the second and third inputs to zero, and set the fourth input to the in-place task, and input them into the in-place motion control policy network to execute the first training task.

[0116] In some example implementations, when training the first training task, the first input can be set to a random value randomly sampled within a preset height range, so that the in-situ motion control strategy network receives data at different body heights during training. The second and third inputs can both be set to zero to constrain the humanoid robot from producing horizontal or rotational movements in the first training task. The fourth input can be set as an identifier to characterize the in-situ task, indicating that the training task type corresponding to the current input data is an in-situ task. In each training cycle of the first training task, different height values ​​can be randomly generated based on the preset height range. Since the target height is continuously changing, the in-situ control strategy network needs to output a set of motion strategies including the hip, knee, and ankle joints in real time to drive the robot to perform a dynamic displacement process that switches the current height.

[0117] This input setting method allows the stationary motion control strategy network to focus on learning control strategies for maintaining stable standing or performing stationary movements under different body height conditions during training, without being disturbed by motion commands for non-stationary tasks such as walking or turning.

[0118] Step S203: Set the first input to a fixed value, set the second and third inputs to fixed values, and set the fourth input to a walking task, and input them into the stationary motion control strategy network to execute the second training task.

[0119] In some example implementations, when performing the second training task, the first input can be set to a preset body height to simulate a standard walking posture; the second and third inputs can be set to preset linear velocity and angular velocity values, respectively, to instruct the robot to perform walking or turning actions; and the fourth input can be set as an identifier to characterize the walking task, indicating that the training task type corresponding to the current input data is a walking task.

[0120] By inputting different types of tasks into the same policy network in a uniform format during training and providing instructions based on task type parameters, the in-situ action control policy network can be guided to identify and adapt to different action output requirements while ensuring the consistency of the input structure. This effectively improves the generalization ability of the training policy to multi-task scenarios and reduces model instability caused by differences in input structure.

[0121] Step S204: Update the stationary motion control policy network based on the action policies output from the first training task and the second training task, as well as the reward function sub-items corresponding to the stationary task and the walking task, respectively.

[0122] In some example implementations, during the first training task, the height of the humanoid robot changes, causing unexpected swaying of its drive joints or body parts due to inertia. Therefore, by collecting data on the changes in the humanoid robot's angular velocity and posture angles, a reward function sub-term for this dynamic process can be calculated to evaluate the action strategy output by the in-situ motion control strategy network. Similarly, during the second training task, the action strategy output by the in-situ motion control strategy network for the walking task can be evaluated based on the reward function sub-term for the walking task. The parameters of the in-situ motion control strategy network can be jointly optimized and updated based on the evaluation results of the two training tasks.

[0123] By constructing targeted reward function sub-items for different task types and performing differentiated evaluation of policy outputs, the problem of inter-task interference in multi-task reinforcement learning can be effectively alleviated, and the adaptability of the policy network to state space switching can be improved. Furthermore, by leveraging differentiated reward feedback and a unified parameter update mechanism, the same set of network execution parameters can accommodate both stationary dynamic height adjustment and fixed-height directional walking tasks, improving the humanoid robot's overall coordination and generalization ability under complex conditions. This can reduce the risk of policy underfitting caused by a single training state and improve the naturalness and stability of movements.

[0124] In some example implementations, such as Figure 3 As shown, a first input for controlling the height of the humanoid robot's body, a second input for controlling the linear velocity, a third input for controlling the angular velocity, and a fourth input for indicating the type of training task can be constructed. Specifically, input construction rules can be determined for the first, second, third, and fourth inputs for stationary and walking tasks, respectively. These input construction rules include data types, value ranges, and logical relationships.

[0125] As an example, the data type of the first input can be configured as a one-dimensional floating-point number to represent the height of the robot's torso relative to the ground; the data type of the second input can be configured as a one-dimensional floating-point number to represent the translational speed of the robot in the horizontal direction; the data type of the third input can be configured as a one-dimensional floating-point number to represent the rotational speed of the robot about the vertical axis; and the data type of the fourth input can be configured as a one-dimensional Boolean value, where a value of 0 indicates that the current training task is a stationary task, and a value of 1 indicates that the current training task is a walking task.

[0126] As an example, the value ranges of the first, second, third, and fourth inputs can be determined according to the training task type. For example, for stationary tasks, the value range of the first input can be set to the minimum and maximum height allowed by the humanoid robot's physical structure, the second and third inputs can be set to the value 0, and the fourth input can be set to a Boolean value of 0. For walking tasks, the value range of the first input can be a fixed value between the minimum and maximum height allowed by the humanoid robot's physical structure; the value range of the second input can be between the maximum walking speed in the positive and negative directions, for example, between -1.0 m / s and +1.0 m / s; the value range of the third input can be between the maximum rotational angular velocity in the positive and negative directions; and the fourth input can be set to a Boolean value of 1. By limiting the value ranges, invalid or out-of-bounds inputs can be effectively prevented from interfering with the training process. The specific values ​​for these ranges are not specifically limited in this disclosure.

[0127] When the fourth input indicates a stationary task, the second and third inputs can be fixed to zero to ensure the robot remains stationary. The first input can be set to vary randomly within its corresponding value range to train the stationary motion control strategy network to learn the control strategy for stationary tasks. When the fourth input indicates a walking task, the first, second, and third inputs can each be set to a fixed value within their corresponding value range. This logical constraint ensures the rationality of the input data and guarantees the accuracy of the stationary motion control strategy network training process.

[0128] In some example implementations, the structural diagram of the in-situ motion control strategy network can be as follows: Figure 4 As shown. In Figure 4 The in-situ motion control strategy network may include: a state encoding module, used to receive a first input for controlling the height of the humanoid robot, a second input for controlling the linear velocity, a third input for controlling the angular velocity, and a fourth input for indicating the type of training task, and construct an input vector; a task modulation module, used to control the in-situ motion control strategy network to execute a first training task or a second training task according to the input vector based on the fourth input; and a policy update module, used to update the in-situ motion control strategy network according to the motion policies output in the first and second training tasks, and the reward function sub-items corresponding to the in-situ task and the walking task, respectively.

[0129] The in-situ action control policy network can be structured as a multi-layer feedforward neural network, with each layer containing several neurons and employing a non-linear activation function to enhance feature modeling capabilities. Parameter sharing is maintained between in-situ and walking tasks; that is, independent network paths are not set for different tasks, but policy switching is achieved within a unified network through a task modulation module. This structure helps improve the model's parameter utilization and enables policy transfer between different training tasks. During model training, training samples corresponding to different task types can be alternately input into this in-situ action control policy network. The fourth input identifies the current training task type, allowing switching between different training task types based on this fourth input, thus achieving unified training of action policies for different training tasks using the same network structure.

[0130] The state encoding module can encode the received first, second, third, and fourth input information with the current state data of the humanoid robot to construct a corresponding input vector. This state data includes body state information and joint state information. The body state information includes the posture information of various body parts of the humanoid robot, such as the posture angle data of the chest, pelvis, and head. The joint state information includes the joint angle information of each driven joint, such as the rotation angle information of the knee, ankle, and hip joints. In addition, the state data may also include the position and velocity information of the humanoid robot's end effector, such as the spatial coordinate information of the ends of the feet or soles, and the current body height information of the humanoid robot. Specific state data can be determined according to actual conditions, and this disclosure does not impose any special limitations.

[0131] The task modulation module controls the in-place motion control policy network to execute the execution parameters of the first training task when the fourth input indicates that the current training task type is an in-place task, and to execute the execution parameters of the second training task when the fourth input indicates that the current training task type is a walking task. The execution parameters include a set of parameters used to selectively update the parameters of the in-place motion control policy network under different training tasks. The task modulation module can employ a conditional coding-based gating mechanism, that is, dynamically adjusting the feature activation values ​​of the intermediate layers of the backbone network according to the task type represented by the fourth input. This allows the response intensity of some feature pathways to be suppressed when the fourth input indicates an in-place task, reducing the impact of translational behavior on motion strategy generation; while when the fourth input indicates a walking task, the responsiveness of linear velocity and angular velocity direction features can be enhanced to promote the network's learning of walking motion patterns.

[0132] The first and second training tasks can be executed alternately. When executing the first training task (the in-place task), the in-place action control policy network outputs a corresponding action policy based on the current input vector. The policy update module evaluates this action policy only based on the reward function sub-item corresponding to the in-place task. Similarly, when executing the second training task (the walking task), the policy network outputs an action policy, and the policy update module evaluates this action policy only based on the reward function sub-item corresponding to the walking task. Subsequently, the policy update module uses the reward function sub-items corresponding to different training task types to update the execution parameters of the first and second training tasks, allowing the network to simultaneously receive training feedback from both in-place and walking tasks within the same parameter space. In this way, the reward function sub-items of different tasks alternately act on the same in-place action control policy network in the time dimension, thereby enabling the in-place action control policy network to gradually develop an action control capability that balances static stability and dynamic coordination.

[0133] Through the above structural design, the in-situ motion control strategy network can take into account the strategy modeling requirements of both in-situ and walking tasks under a shared parameter structure, and dynamically adjust the network behavior through a task modulation mechanism to improve the coherence and stability of the overall motion output.

[0134] In some example implementations, the in-situ motion control policy network is controlled to perform a first training task or a second training task based on the input vector according to the fourth input, including: determining the current training task type based on the fourth input, wherein the current training task type is either the first training task or the second training task; determining the execution parameters corresponding to the current training task type in the in-situ motion control policy network; and training the current training task type based on the execution parameters.

[0135] In each training cycle, after determining the current training task type based on the fourth input in the input vector, execution parameters corresponding to that training task type can be determined within the in-place motion control policy network. These execution parameters may include a set of parameters used to selectively update the parameters of the in-place motion control policy network under different training tasks. This set of parameters can be dynamically updated during subsequent training based on the feedback results of the reward function sub-items of the in-place and walking tasks.

[0136] In this way, the in-situ action control policy network, under the premise of parameter sharing, achieves alternating optimization of multiple training tasks through the selective execution mechanism guided by the fourth input. This improves the network's adaptability to multimodal behavior switching, reduces the instability of policy output caused by task confusion, and enhances the overall control accuracy without increasing the complexity of the model structure.

[0137] In some example implementations, the in-place motion control policy network is updated based on the action policies output from the first training task and the second training task, as well as the reward function sub-items corresponding to the in-place task and the walking task, respectively. This includes: calculating the reward function sub-item corresponding to the in-place task based on the action policy output from the first training task; calculating the reward function sub-item corresponding to the walking task based on the action policy output from the second training task; constructing a joint training loss based on the reward function sub-item corresponding to the in-place task and the reward function sub-item corresponding to the walking task; and updating the execution parameters in the in-place motion control policy network based on the joint training loss.

[0138] The in-place action control policy network outputs the action policy for the in-place task after performing the first training task, and outputs the action policy for the walking task after performing the second training task. A joint training loss can be constructed based on the outputs of the reward function sub-items corresponding to the in-place and walking action policies. This joint training loss can be achieved by weighted fusion of the reward function sub-items corresponding to the two tasks, allowing the in-place action control policy network to simultaneously consider the standing stability of the in-place task and the dynamic coordination of the walking task during the update and optimization process. This enables the network to activate the optimal policy path and execute the updated parameters when performing different tasks.

[0139] By extracting reward function sub-items from two training tasks, the stability of policy output in in-place tasks can be improved, and the motion jitter caused by over-regularization or penalties can be reduced. Furthermore, this method avoids parameter redundancy and deployment complexity associated with using multiple independent policy networks.

[0140] In some example implementations, such as Figure 5As shown, the reward function sub-item corresponding to the in-place task is calculated based on the action policy output by the first training task, including steps S501-S502:

[0141] Step S501: Control the humanoid robot to perform a stationary task according to the motion strategy output by the first training task, and collect the angular velocity change data of each driven joint and the posture angle change data of the body parts when the humanoid robot performs the stationary task.

[0142] When controlling the humanoid robot to perform a stationary task according to the motion strategy output from the first training task, the angular velocity change data of the humanoid robot's drive joints and the posture angle change data of its body parts can be obtained.

[0143] As an example, a continuous sampling sequence of the rotational speed corresponding to each drive joint can be obtained within a set sampling period, and the change amplitude sequence of the rotational speed corresponding to each drive joint over time can be determined based on the continuous sampling sequence. The change amplitude sequence corresponding to each drive joint can be determined as angular velocity change data. This angular velocity change data can be used to reflect the stability of the drive joints during stationary movements. The drive joints may include, but are not limited to, shoulder joints, elbow joints, hip joints, knee joints, and ankle joints.

[0144] A continuous sampling sequence of spatial attitude angles corresponding to each body part can be acquired within a set sampling period. Based on these continuous sampling sequences, the change in attitude amplitude over time for each body part can be determined. This change in attitude amplitude sequence for each body part can be defined as attitude angle change data. This attitude angle change data can be used to characterize the dynamic stability of the humanoid robot's overall body posture during in-situ tasks. Body parts can include the torso, head, limbs, etc. The sampling period can be determined based on actual conditions; no specific value is specifically limited in this disclosure.

[0145] Step S502: Determine the stability score of the first training task based on the angular velocity change data and attitude angle change data; determine the reward function sub-item corresponding to the in-place task based on the stability score.

[0146] The stability score of the first training task can be determined based on the angular velocity change data and the attitude angle change data. This stability score is a score value that quantifies the smoothness and coordination of the in-place task during execution. The higher the score value, the more stable the motion state under the control of the current action strategy.

[0147] After obtaining the stability score, it can be mapped to a reward value for feedback on the performance of the action strategy control, and this reward value can be determined as a sub-item of the reward function corresponding to the in-place task.

[0148] By constructing a stability reward term based on the angular velocity of the driving joints and the posture angle data of the body parts during the actual process of performing stationary tasks, the above method can provide accurate feedback on minor joint vibrations and posture instability, thereby improving the stability of the stationary task control strategy.

[0149] In some example implementations, such as Figure 6 As shown, determining the stability score of the first training task based on angular velocity change data and attitude angle change data may include steps S601-S602:

[0150] Step S601: The average change value of the angular velocity change data corresponding to each drive joint is weighted and calculated to obtain the first change feature value; and the average change value of the attitude angle change data corresponding to each body part is weighted and calculated to obtain the second change feature value.

[0151] Within a set sampling period, the corresponding angular velocity change data can be segmented and statistically processed. For example, the average change value of the angular velocity change amplitude sequence of the joint can be calculated. This average change value reflects the fluctuation intensity of the joint per unit time. After obtaining the average change value of the angular velocity corresponding to each drive joint, a corresponding weight value can be set for each drive joint according to the degree of influence of different drive joints on the stability of in-situ movement. The average change value of the angular velocity of each drive joint is weighted and calculated with its corresponding weight, and the weighted results of all drive joints are summarized to obtain the first change feature value. The weight value corresponding to each drive joint can be determined according to the actual situation, and the specific value is not specifically limited in this disclosure.

[0152] Among them, the larger the first characteristic value of change is, the more obvious the fluctuation of the overall angular velocity of the driving joint is, and the less stable the movement process is; conversely, the smaller the first characteristic value of change is, the smoother the movement of the driving joint is, and the higher the stability at the joint level is.

[0153] By performing statistical and weighted processing on the angular velocity change data based on the time dimension, we can avoid the interference of outliers at a single moment on the stability judgment, so that the first change characteristic value can truly reflect the dynamic stability of the drive joint during the entire stationary movement process.

[0154] Step S602: The first change feature value and the second change feature value are weighted and summed according to preset weights to obtain the comprehensive change feature value.

[0155] For each body part, its posture angle change data can be statistically processed within a set sampling period. For example, the average change value of the sequence of changes in posture angles can be calculated throughout the entire sampling period. This change level is used to characterize whether the body part experiences continuous shaking, tilting, or unnecessary posture adjustments during in-place task execution. After obtaining the average change value for each body part, the average change values ​​for each body part can be aggregated, for example, by setting corresponding weight values ​​for each body part. The average change value of each body part is weighted and calculated with its corresponding weight, and the weighted results for all body parts are aggregated to obtain a comprehensive change feature value representing the overall dynamic change of body posture. A larger comprehensive change feature value indicates more frequent changes in the robot's body posture during in-place movements, resulting in poorer stability; a smaller comprehensive change feature value indicates a more stable body posture.

[0156] By conducting a comprehensive statistical analysis of the posture angle change sequence, stability can be assessed from the perspective of whole-body posture coordination, avoiding the problem of relying solely on local joint information while ignoring the overall body balance.

[0157] After obtaining the first and second change feature values, corresponding fusion weights can be pre-set based on the relative importance of joint stability and overall body posture stability in the current training task. Then, the first change feature value and its corresponding weight are weighted and calculated, and the second change feature value and its corresponding weight are weighted and calculated. The two weighted results are then summarized to obtain the final comprehensive change feature value. This comprehensive change feature value can be used to characterize the overall motion stability of the robot during the execution of the first training task; a higher comprehensive change feature value indicates a more unstable motion, while a lower comprehensive change feature value indicates a more stable motion. Furthermore, the first and second change feature values ​​can be normalized before summarizing to eliminate the influence of dimensional differences on the scoring results.

[0158] Step S602: Determine the stability score corresponding to the comprehensive change feature value through a preset monotonically decreasing mapping relationship.

[0159] As an example, the comprehensive change feature value can be used as an input parameter and fed into a pre-defined monotonically decreasing mapping function. Based on the monotonically decreasing mapping relationship corresponding to this function, a stability score corresponding to the comprehensive change feature value is obtained. This monotonically decreasing mapping function describes the inverse correlation between the comprehensive change feature value and the stability score; when the comprehensive change feature value is large, the corresponding stability score is low, and when the comprehensive change feature value is small, the corresponding stability score is high.

[0160] In one embodiment, the monotonically decreasing mapping relationship can be constructed based on a linear function. This is achieved by normalizing the comprehensive change feature value with a preset upper limit value for the change feature, and then linearly scaling the stability score based on the normalization result. This method allows the stability score to change continuously within a preset range, maintaining a monotonically decreasing trend as the comprehensive change feature value increases.

[0161] In another implementation, the monotonically decreasing mapping relationship can also be constructed based on an exponential decay function, a piecewise function, or a lookup table mapping method. This is used to suppress the comprehensive change feature value, thereby reducing the weight of the influence of the comprehensive change feature value on the stability score when it is large, thus avoiding drastic changes in the stability score under extreme fluctuation conditions.

[0162] By adopting the above monotonically decreasing mapping method, the comprehensive change feature value representing the degree of instability is mapped to the stability score representing the degree of stability. This can achieve the unification of the directionality of the stability index while maintaining the change characteristics of the underlying physical quantity. This allows the magnitude of the stability score to intuitively reflect the stability of the humanoid robot during the execution of the first training task, and provides a reliable evaluation basis for constructing reward function sub-items based on the stability score.

[0163] By integrating the dynamic stability characteristics at the joint level with the stability characteristics at the overall body posture level, the stability of stationary movements can be comprehensively evaluated from both the micro-driving behavior and macro-body posture levels, providing a reliable basis for subsequent reward function construction or strategy optimization.

[0164] In some example implementations, the reward function sub-item corresponding to the first training task is determined based on the stability score. This can be achieved by normalizing the stability score using preset upper and lower limits to obtain a normalized score. When the normalized score is greater than a first threshold, the reward function sub-item corresponding to the first training task is determined based on a linear growth model. When the normalized score is less than or equal to the first threshold and greater than or equal to a second threshold, the reward function sub-item corresponding to the first training task is determined based on a linear interpolation model. When the normalized score is less than the second threshold, the reward function sub-item corresponding to the first training task is determined to be zero. The reward function sub-item corresponding to the first training task is limited to a non-negative value not less than zero.

[0165] As an example, a normalized interval can be constructed based on preset upper and lower limits. For instance, if the upper limit of the score is set to 1.0 and the lower limit to 0.0, then any actual stability score can be mapped to the interval [0,1] using linear normalization to obtain a normalized score value. This normalized score value can be used to uniformly process stability score results generated under different tasks or sampling conditions.

[0166] After the normalized score is determined, a three-stage reward generation model can be constructed by combining the first threshold and the second threshold. The first threshold is greater than the second threshold, and their specific values ​​can be determined based on actual circumstances; this disclosure does not impose any special limitations. These values ​​are used to divide the system into three intervals: high stability, medium stability, and low stability.

[0167] When the normalized score is greater than the first threshold, indicating a high stability range, a linear growth model can be directly applied. This model constructs a linear growth model with the normalized score as the independent variable, outputting the corresponding reward function sub-term. By setting a fixed slope and intercept, this model ensures that higher scores result in larger reward function sub-terms, thus prioritizing the maintenance of high stability in the action policy.

[0168] When the normalized score is less than or equal to the first threshold and greater than or equal to the second threshold, falling between the first and second thresholds—that is, within the moderate stability range—a linear interpolation model can be used for reward calculation. Specifically, this range can be considered a transition region. By linearly interpolating the reward values ​​at the two endpoints of the range, a moderate level of reward value is smoothly assigned to the moderately stable strategy. This interpolation function can avoid reward jumps caused by score fluctuations, improving the stability and continuity of the training process.

[0169] When the normalized score is less than the second threshold, it indicates that the in-place action performs poorly in terms of stability. In this case, the reward function term can be directly set to zero. This can be used to penalize unstable control policies, suppress their accumulated expected value during reinforcement learning training, and thus encourage the model to optimize towards better stability.

[0170] By setting two thresholds, reward strategies corresponding to different stability levels can be clearly distinguished, avoiding interference with model learning when scores are ambiguous. Simultaneously, it allows reward values ​​to change smoothly with scores, preventing drastic oscillations in policy updates caused by sudden reward changes; providing higher incentives for highly stable action strategies helps improve the training process's preference for high-performance strategies; and setting rewards to zero in low-score ranges enables rapid filtering of unstable actions in the early training stages, saving training resources.

[0171] In some example implementations, when the normalized score is greater than a first threshold, the reward function sub-item corresponding to the first training task is determined based on a linear growth model, including:

[0172]

[0173] in, This represents the reward function sub-item corresponding to the first training task. This represents the preset base reward value. This represents the normalized score. Indicates the first reward coefficient. This represents the first threshold. , and The value can be determined according to the actual situation, and this disclosure does not impose any special restrictions.

[0174] The above method allows for flexible adjustment of the reward function sub-terms corresponding to action policies with high stability scores, and smoothly enhances the rewards for high-scoring actions, ensuring gradient stability during reinforcement learning.

[0175] In some example implementations, when the normalized score is less than or equal to a first threshold and greater than or equal to a second threshold, the reward function sub-item corresponding to the first training task is determined based on a linear interpolation model, including: determining the reward function sub-item corresponding to the first training task according to the following formula:

[0176]

[0177] in, This represents the reward function sub-item corresponding to the first training task. This represents the preset base reward value. This represents the normalized score. Indicates the first threshold. This indicates the second threshold. The value can be determined according to the actual situation, and this disclosure does not impose any special restrictions.

[0178] Because this linear interpolation model can guarantee that... to The gradient continuity within the interval can avoid policy oscillation or learning instability; at the same time, the model only involves simple addition, subtraction and multiplication and division operations, without the need for complex nonlinear activation functions or lookup table operations, making it suitable for deployment in resource-constrained robot embedded systems; furthermore, the reward function sub-terms can be flexibly configured by adjusting the parameters of this linear interpolation model to meet the training requirements of different stability indices.

[0179] In some example implementations, such as Figure 7 As shown, calculating the reward function sub-item corresponding to the walking task based on the action strategy output by the second training task may include steps S701-S702:

[0180] Step S701: Control the humanoid robot to perform a walking task based on the action strategy output from the second training task, and collect the angle error between the actual walking direction and the target walking direction of the humanoid robot when performing the walking task.

[0181] The humanoid robot's leg drive joints can be controlled to perform the walking task via the motion strategy output from the second training phase. This motion strategy is a continuous control vector output based on the current state during the iterative process of the in-situ motion control strategy network, typically including the desired position or angular velocity values ​​of multiple joints. When the humanoid robot actually executes this motion strategy, the actual movement direction vector of each step, i.e., the actual walking direction, can be obtained through the inertial navigation unit and the chassis encoder, while simultaneously obtaining the target walking direction vector in the set task, i.e., the target walking direction.

[0182] Step S702: Calculate the reward function sub-item corresponding to the second training task based on the included angle error.

[0183] The angular error between the actual walking direction and the target walking direction can be calculated using the vector angle formula. This angular error can be used to measure whether the humanoid robot deviates from its walking task. The reward function sub-term corresponding to the walking task can be determined based on this angular error, thus keeping the training process sensitive to directional deviations. The training direction of the action strategy can be dynamically adjusted based on the reward function sub-term, prompting the humanoid robot to gradually reduce directional deviations during walking tasks and achieve higher control precision.

[0184] In some example implementations, in step S701, when calculating the reward function sub-item corresponding to the walking task based on the included angle error, if the included angle error is greater than or equal to the error threshold, the reward function sub-item corresponding to the walking task can be set to zero. If the included angle error is less than the error threshold, the reward function sub-item corresponding to the walking task can be determined according to the following function:

[0185]

[0186] in, This represents the reward function sub-item corresponding to the second training task. Indicates the included angle error. Indicates the error threshold. This represents the second reward coefficient. The value can be determined according to the actual situation, and this disclosure does not impose any special restrictions.

[0187] This method allows for the determination of function reward items in real time based on the angular error of the humanoid robot. The larger the angular error, the smaller the reward; the smaller the angular error, the larger the reward, which is beneficial for training to obtain more accurate action strategies.

[0188] In some example implementations, in step S702, when constructing the joint training loss based on the reward function sub-item corresponding to the first training task and the reward function sub-item corresponding to the second training task, the first training loss can be constructed based on the reward function sub-item corresponding to the first training task and the log probability of the action policy output in the first training task in the action space; the second training loss can be constructed based on the reward function sub-item corresponding to the second training task and the log probability of the action policy output in the second training task in the action space; and the joint training loss is determined based on the weighted sum of the first training loss and the second training loss.

[0189] As an example, in the first training task, the policy gradient loss function can be constructed by multiplying the negative log probability of the action policy in the action policy space by the corresponding reward function term, thus obtaining the training loss for the first training task, as shown in the following formula:

[0190]

[0191] in, Indicates the first training loss. This represents the reward function sub-item corresponding to the first training task. Let represent the probability distribution function of the in-place motion control policy network under the first training task. This represents the action policy output by the first training task. This indicates the current state data of the humanoid robot. The current state data may include the posture information of the robot's current body parts, the angles of each driven joint, and the robot's body height, etc. The specific state data can be determined according to the actual situation, and this disclosure does not impose any special limitations.

[0192] In the second training task, the log probability value corresponding to the control policy output by the second training task can also be obtained; using the reward function sub-term corresponding to the walking task and the log probability, a policy gradient loss function is constructed, and thus the second training loss term is obtained, as shown in the following formula:

[0193]

[0194] in, Indicates the second training loss. This represents the reward function sub-item corresponding to the second training task. Let represent the probability distribution function of the in-place motion control policy network under the second training task. This represents the action policy output by the second training task. This indicates the current state data of the humanoid robot. The state data includes the current posture information of the humanoid robot's body parts, the angles of each driven joint, and the body height, etc. The specific state data can be determined according to the actual situation, and this disclosure does not impose any special limitations.

[0195] The joint training loss can be determined by the weighted sum of the first training loss and the second training loss, as shown in the following formula:

[0196]

[0197] in, Indicates joint training losses, This represents the weighting coefficient of the first training loss. Indicates the first training loss. This represents the second training loss. The specific weighting coefficients can be determined based on actual conditions, and this disclosure does not impose any special limitations. This joint training loss can be used to update the parameters of the in-situ action control policy network in deep reinforcement learning. Through this method, it is possible to simultaneously consider the stability objective of in-situ actions and the directional objective of walking actions within the same network, effectively improving the robot's policy generalization ability and control stability.

[0198] In some example implementations, the training method for the in-situ motion control strategy network of a humanoid robot based on deep reinforcement learning further includes: determining the input data ratio corresponding to the first training task and the second training task; determining the average rate of change of the reward function sub-items corresponding to the first training task and the second training task within the preset training period every preset training period; and dynamically adjusting the input data ratio according to the average rate of change.

[0199] The input data ratio represents the proportion of input data corresponding to the first and second training tasks in the in-place motion control strategy network training process within a preset training period. At the beginning of training, the input data ratio between the in-place and walking tasks can be set to a preset initial value, such as 1:1, meaning that the input ratio for each task is equal in each training period. This initial ratio can be set empirically, and this disclosure does not impose any special limitations.

[0200] The training effect can be evaluated at preset training cycles, such as every 50 parameter updates. The average value of the reward function sub-items obtained for the first training task and the average value of the reward function sub-items for the second training task can be calculated separately within that cycle. Then, the rate of change of the average value for each training task type can be calculated. A larger rate of change indicates a faster performance improvement for the in-place action control policy network on that training task type; a rate of change approaching zero indicates that the performance of the in-place action control policy network on that training task type has stabilized; and a negative rate of change indicates a performance degradation for the in-place action control policy network on that training task type.

[0201] The input data ratio for the next cycle can be dynamically adjusted based on the rate of change of the average values ​​of the two training task types: if the rate of change of the average value corresponding to the stationary task is lower than a preset convergence threshold, and the rate of change of the average value corresponding to the walking task is higher than a preset convergence threshold, then the input data ratio of the first training task is reduced, and the input data ratio of the second training task is increased, such as adjusting the ratio to 1:2 or 1:3, which can be determined based on the actual situation, and this disclosure does not impose any special limitations; if the rate of change of the average value of the stationary task or the average value of the walking task is negative, then the corresponding input data ratio is increased. As an example, if the ratio of the input data for the stationary task and the walking task is adjusted to 2:8, and the total batch size is 128, then approximately 25 sets of input data for the stationary task and 103 sets of input data for the walking task can be generated. In the subsequent joint loss calculation, the first training loss for the in-place task and the second training loss for the walking task can be determined based on the average training losses for the in-place task and the walking task in each batch. The weighting coefficients for the first and second training losses can then be re-determined based on the proportions of the in-place and walking tasks in the adjusted input data. For example, the weighting coefficients for the first and second training losses can be determined using the following formulas:

[0202]

[0203]

[0204] in, This represents the weighting coefficient of the first training loss before the input data ratio adjustment. This indicates the rate of decrease or increase in the in-place task compared to before the adjustment, after the input data ratio was adjusted. This indicates the rate of decrease or increase in the walking task after the input data ratio adjustment compared to before the adjustment. This represents the weighting coefficient of the first training loss after adjusting the input data ratio. This represents the weighting coefficient of the second training loss after adjusting the input data ratio. The specific method for adjusting the weighting coefficient is not specifically limited in this disclosure.

[0205] Through the above dynamic adjustments, excessive computing power can be avoided on already trained training task types, and the training focus can be automatically shifted to training tasks with poor training results, thereby improving the stability of in-place task execution and avoiding the problem of motion output jitter caused by over-reliance on inhibitory constraints during training.

[0206] The following will provide a detailed description of the in-situ motion control strategy method for humanoid robots based on deep reinforcement learning in this example embodiment. Figure 8 As shown, the in-situ motion control strategy method for humanoid robots based on deep reinforcement learning includes steps S801-S802:

[0207] Step S801: Determine a first input for controlling the height of the humanoid robot's body, a second input for controlling the linear velocity, a third input for controlling the angular velocity, and a fourth input for indicating the task type.

[0208] If the current task type is a stationary task, the first input of body height can be randomly determined within a preset height range to control the body height of the humanoid robot, so as to drive the humanoid robot to perform stationary tasks such as squatting, standing, or adjusting the center of gravity; the second and third inputs are set to 0 respectively to ensure that the robot remains stationary under this task; the fourth input is determined as the identifier corresponding to the stationary task.

[0209] If the current task type is a walking task, the body height can be determined as a fixed value within a preset height range to clarify the height in the walking task; the second input is configured as the desired horizontal translation speed, which includes the speed direction and speed magnitude, to indicate the speed and displacement direction of the humanoid robot; the third input is configured as the desired angular velocity of the entire humanoid robot body rotating around the vertical axis, to indicate the magnitude and direction of the humanoid robot's turn; and the fourth input is determined as the identifier corresponding to the walking task.

[0210] By using the above method, the corresponding task type can be accurately distinguished and responded to based on the fourth input, and the in-situ action control policy network can be guided to generate a control policy corresponding to the task type, effectively solving the confusion problem in multi-task control policies.

[0211] Step S802: Input the first input, the second input, the third input, and the fourth input together into the pre-trained in-place motion control policy network, and determine the control policy for the in-place task based on the in-place motion control policy network.

[0212] The first, second, third, and fourth inputs, along with the current state data of the humanoid robot, can be encoded to construct corresponding input vectors. This state data includes the humanoid robot's current body state information and joint state information. The body state information includes the posture information of various body parts, such as the posture angles of the chest, pelvis, and head. The joint state information includes the joint angle information of each driven joint, such as the rotation angles of the knee, ankle, and hip joints. Furthermore, the state data may also include the position and velocity information of the humanoid robot's end effector, such as the spatial coordinates of the ends of the feet or soles, and the current body height information of the humanoid robot. Specific state data can be determined according to actual circumstances, and this disclosure does not impose any special limitations.

[0213] The input vector is fed into a pre-trained in-situ motion control policy network. This network, trained using deep reinforcement learning, is capable of generating motion policies for each driven joint of the humanoid robot based on the input vector, such as target joint position sequences or feedforward torques. The underlying controller receives this motion policy and drives the motors to execute it, thereby controlling the humanoid robot to perform in-situ or walking movements that match the fourth input while maintaining dynamic balance.

[0214] By using a fourth input and differentiated rules for angular velocity, linear velocity, and height configuration, the system can accurately distinguish between two distinct task types: stationary tasks and walking tasks. This effectively avoids logical confusion in the policy network during multi-task switching, such as accidentally triggering a stepping action when crouching in place. Introducing random height input in stationary tasks, combined with static velocity constraints, enables the robot to adapt to significant changes in center of gravity without losing stability. In walking tasks, the introduction of independent angular velocity control gives the robot the ability to move in all directions and turn flexibly, significantly improving the generalization and practicality of the policy.

[0215] In an exemplary embodiment of this disclosure, a training device for a humanoid robot's in-situ motion control policy network based on deep reinforcement learning is also provided. (See reference...) Figure 9 As shown, the humanoid robot in-situ motion control policy network training device 900 based on deep reinforcement learning includes a construction module 901, a first training module 902, a second training module 903, and an update module 904, wherein:

[0216] The construction module 901 is used to construct a first input for controlling the height of the humanoid robot's body, a second input for controlling the linear velocity, a third input for controlling the angular velocity, and a fourth input for indicating the type of training task.

[0217] The first training module 902 is used to set the first input to a random value, set the second and third inputs to zero, and set the fourth input to an in-place task, and input them to the in-place motion control policy network to execute the first training task.

[0218] The second training module 903 is used to set the first input to a fixed value, set the second and third inputs to fixed values, and set the fourth input to a walking task, and input them into the stationary motion control strategy network to execute the second training task.

[0219] The update module 904 is used to update the stationary motion control policy network based on the action policies output in the first training task and the second training task, as well as the reward function sub-items corresponding to the stationary task and the walking task, respectively.

[0220] The specific details of each module in the above-mentioned training device for humanoid robot in-situ motion control strategy network based on deep reinforcement learning have been described in detail in the corresponding training method for humanoid robot in-situ motion control strategy network based on deep reinforcement learning, so they will not be repeated here.

[0221] In an exemplary embodiment of this disclosure, a humanoid robot in-situ motion control device based on deep reinforcement learning is also provided. (See reference...) Figure 10 The humanoid robot in-situ motion control device 1000 based on deep reinforcement learning shown includes a determination module 1001 and a control module 1002, wherein:

[0222] The module 1001 is used to determine a first input for controlling the height of the humanoid robot's body, a second input for controlling the linear velocity, a third input for controlling the angular velocity, and a fourth input for indicating the task type.

[0223] The control module 1002 is used to input the first input, the second input, the third input and the fourth input together into the pre-trained in-situ motion control policy network, and determine the control policy for the in-situ task based on the in-situ motion control policy network; wherein, the in-situ motion control policy network is obtained by the in-situ motion control policy network training method for humanoid robots based on deep reinforcement learning.

[0224] The specific details of each module in the above-mentioned deep reinforcement learning-based humanoid robot in-situ motion control strategy device have been described in detail in the corresponding deep reinforcement learning-based humanoid robot in-situ motion control method, so they will not be repeated here.

[0225] In an exemplary embodiment of this disclosure, a robot is also provided, which includes any one of legged robots, quadruped robots, bipedal robots, wheeled robots, wheel-legged robots, quadrupedal robots, humanoid robots, cleaning robots, transport robots, mobile robots, and robotic arms. Of course, other types of mobile robots may also be included, and this disclosure does not limit them.

[0226] refer to Figure 11 The diagram illustrates a type of robot to which the deep reinforcement learning-based robot in-situ motion control method described in this disclosure can be applied. Figure 11 It can be seen that this type of robot is a humanoid robot.

[0227] refer to Figure 12 As shown in Figure 1201, another type of robot schematic diagram is illustrated, to which the deep reinforcement learning-based robot in-situ motion control method of the present disclosure embodiment can be applied.

[0228] refer to Figure 13 As shown, an electronic device capable of implementing the above method is also provided. The electronic device 1300 includes a processor 1301 and a memory 1302. The memory 1302 stores computer-readable instructions, which, when executed by the processor 1301, implement the method of this disclosure.

[0229] In an exemplary embodiment of this disclosure, a computer-readable storage medium is also provided, having stored thereon computer program code instructions that, when invoked by a robot's processor, cause the robot to perform the method described in the embodiments.

[0230] refer to Figure 14 As shown, a program product 1400 for implementing the above-described method according to an embodiment of the present disclosure is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0231] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0232] Finally, the above preferred embodiments are only used to illustrate the technical solutions of this disclosure and are not restrictive. Although this disclosure has been described in detail, those skilled in the art should understand that changes in form and detail can be made without departing from the scope defined by the claims of this disclosure. The dimensions in the drawings are not related to the specific physical object, and the physical object dimensions can be arbitrarily changed.

Claims

1. A method for training a deep reinforcement learning based humanoid robot motion-in-place control policy network, characterized in that, The method comprises: constructing a first input for controlling the height of the humanoid robot body, a second input for controlling the linear velocity, a third input for controlling the angular velocity, and a fourth input for indicating the type of training task; setting the first input to a random value, setting the second input and the third input to zero, and setting the fourth input to a stationary task, and inputting to a stationary action control policy network to perform a first training task; setting the first input to a fixed value, setting the second input and the third input to a fixed value, and setting the fourth input to a walking task, and inputting to the stationary action control policy network to perform a second training task; updating the stationary action control policy network according to the action policy output in the first training task and the second training task, and the reward function subterm corresponding to the stationary task and the walking task respectively.

2. The deep reinforcement learning based humanoid robot motion-in-place control policy network training method according to claim 1, wherein, The updating of the stationary action control policy network according to the action policy output in the first training task and the second training task, and the reward function subterm corresponding to the stationary task and the walking task respectively, comprises: calculating the reward function subterm corresponding to the stationary task according to the action policy output in the first training task, and calculating the reward function subterm corresponding to the walking task according to the action policy output in the second training task; constructing a joint training loss according to the reward function subterm corresponding to the first training task and the reward function subterm corresponding to the second training task; updating the execution parameters in the stationary action control policy network based on the joint training loss.

3. The deep reinforcement learning based humanoid robot motion-in-place control policy network training method according to claim 2, characterized in that, The calculation of the reward function subterm corresponding to the stationary task according to the action policy output in the first training task comprises: controlling the humanoid robot to perform the stationary task based on the action policy output in the first training task, and collecting the angular velocity change data of each driving joint and the attitude angle change data of each body part of the humanoid robot when performing the stationary task; determining a stability score of the first training task based on the angular velocity change data and the attitude angle change data; determining the reward function subterm corresponding to the first training task according to the stability score.

4. The deep reinforcement learning based humanoid robot motion-in-place control policy network training method according to claim 3, characterized in that, The determination of the stability score of the first training task based on the angular velocity change data and the attitude angle change data comprises: weighting the average change value of the angular velocity change data corresponding to each driving joint to obtain a first change characteristic value, and weighting the average change value of the attitude angle change data corresponding to each body part to obtain a second change characteristic value; weighting and summing the first change characteristic value and the second change characteristic value according to a preset weight to obtain a comprehensive change characteristic value; determining the stability score corresponding to the comprehensive change characteristic value through a preset monotonically decreasing mapping relationship.

5. The deep reinforcement learning based humanoid robot motion-in-place control policy network training method according to claim 3, characterized in that, The determination of the reward function subterm corresponding to the first training task according to the stability score comprises: normalizing the stability score according to a preset upper limit value and a preset lower limit value to obtain a normalized score value; determining the reward function sub-item corresponding to the first training task based on a linear growth model when the normalized score value is greater than a first threshold value; determining the reward function sub-item corresponding to the first training task based on a linear interpolation model when the normalized score value is less than or equal to the first threshold value and greater than or equal to a second threshold value; determining the reward function sub-item corresponding to the first training task as zero when the normalized score value is less than the second threshold value; wherein the first threshold value is greater than the second threshold value.

6. The deep reinforcement learning based humanoid robot motion-in-place control policy network training method according to claim 5, characterized in that, The method further comprises: wherein, represents a reward function sub-item corresponding to the first training task, represents a preset basic reward value, represents the normalized score value, represents a first reward coefficient, represents the first threshold value.

7. The deep reinforcement learning based humanoid robot motion control policy network training method according to claim 5, characterized in that, The method further comprises: The method further comprises: wherein, represents a reward function sub-item corresponding to the first training task, represents a preset basic reward value, represents the normalized score value, represents the first threshold value, represents the second threshold value.

8. The deep reinforcement learning based humanoid robot motion control policy network training method according to claim 2, characterized in that, The method further comprises: The method further comprises: The method further comprises:

9. The deep reinforcement learning based humanoid robot motion control policy network training method according to claim 8, characterized in that, The method further comprises: The method further comprises: wherein, represents a reward function subterm corresponding to the second training task, represents the included angle error, represents an error threshold, represents a second reward coefficient; The method further comprises:

10. The deep reinforcement learning based humanoid robot motion control policy network training method according to claim 2, wherein, The method further comprises: The method further comprises: The method further comprises: The method further comprises:

11. The deep reinforcement learning based humanoid robot motion control policy network training method according to claim 1, wherein, The method further comprises: The method further comprises:

12. The deep reinforcement learning based humanoid robot motion control policy network training method according to claim 1, wherein, The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method Determine average value change rates of reward function sub-items corresponding to the first training task and the second training task respectively in a preset training period every preset training period; Dynamically adjust the input data proportion according to the average value change rates.

13. The deep reinforcement learning based humanoid robot motion control policy network training method of claim 1, wherein, The in-place action control policy network comprises: A state encoding module configured to receive a first input for controlling a body height of the humanoid robot, a second input for controlling a linear velocity, a third input for controlling an angular velocity, and a fourth input for indicating a training task type, and construct an input vector; A task modulation module configured to control the in-place action control policy network to perform a first training task or a second training task according to the input vector based on the fourth input respectively; A policy updating module configured to update the in-place action control policy network according to action policies output in the first training task and the second training task and reward function sub-items corresponding to the in-place task and the walking task respectively.

14. The deep reinforcement learning based humanoid robot motion-in-place control policy network training method according to claim 13, characterized in that, The controlling the in-place action control policy network to perform a first training task or a second training task according to the input vector based on the fourth input comprises: Determining a current training task type according to the fourth input, the current training task type being the first training task or the second training task; Determining an execution parameter corresponding to the current training task type in the in-place action control policy network; Determining a reward function sub-item corresponding to the current training task type based on the execution parameter corresponding to the current training task type, and updating the execution parameter based on the reward function sub-item corresponding to the current training task type.

15. A method for in-place motion control of a humanoid robot based on deep reinforcement learning, characterized by, Comprise: Determining a first input for controlling a body height of the humanoid robot, a second input for controlling a linear velocity, a third input for controlling an angular velocity, and a fourth input for indicating a task type; Inputting the first input, the second input, the third input, and the fourth input into a pre-trained in-place action control policy network together to determine a control policy of an in-place task according to the in-place action control policy network; The in-place action control policy network is obtained according to the humanoid robot in-place action control policy network training method based on deep reinforcement learning of any one of claims 1-14.

16. A deep reinforcement learning based humanoid robot motion-in-place control policy network training apparatus, characterized by, Comprise: A constructing module configured to construct a first input for controlling a body height of the humanoid robot, a second input for controlling a linear velocity, a third input for controlling an angular velocity, and a fourth input for indicating a training task type; A first training module configured to set the first input as a random value, set the second input and the third input as zero, set the fourth input as an in-place task, and input into the in-place action control policy network to perform a first training task; A second training module configured to set the first input as a fixed value, set the second input and the third input as fixed values, set the fourth input as a walking task, and input into the in-place action control policy network to perform a second training task; An updating module is configured to update the standing action control policy network according to the first training task, the action policy output in the second training task, and the reward function sub-items corresponding to the standing task and the walking task, respectively.

17. An in-place motion control apparatus for a humanoid robot based on deep reinforcement learning, characterized by, The method comprises the following steps: A determining module is configured to determine a first input for controlling the body height of the humanoid robot, a second input for controlling the linear velocity, a third input for controlling the angular velocity, and a fourth input for indicating the task type. A control module is configured to input the first input, the second input, the third input, and the fourth input into a pre-trained standing action control policy network, and determine the control policy of the standing task according to the standing action control policy network.

18. An electronic device, comprising: The method comprises the following steps: A processor; and A memory having computer readable instructions stored thereon, wherein the computer readable instructions, when executed by the processor, implement the method according to any one of claims 1-15.

19. A robot, characterized in that The method comprises the following steps: A processor; and A memory having computer readable instructions stored thereon, wherein the computer readable instructions, when executed by the processor, implement the method according to any one of claims 1-15.

20. The robot of claim 19, wherein, The robot comprises a mobile robot.

21. The robot of claim 19, wherein, The robot comprises a humanoid robot.

22. The robot of claim 19, wherein, The robot comprises one of a legged robot and a wheeled robot.

23. The robot of claim 19, wherein, The robot comprises a wheeled-legged robot.

24. The robot of claim 19, wherein, The robot comprises one of a quadruped robot, a biped robot, and a quadruped-wheeled robot.

25. A computer readable storage medium, characterized in that, The computer readable storage medium has computer program code instructions stored thereon, wherein when the computer program code instructions are called by the processor of the robot, the robot executes the method according to any one of claims 1-15.

Images