A method and device for training a model for stopping control of a biped robot based on deep reinforcement learning

Abstract

The present disclosure provides a kind of based on deep reinforcement learning's double wheel foot robot stay control model training method and device, it is related to sensor and robot technical field.The method comprises: in each control cycle, the current position value of the wheel foot joint of double wheel foot robot is acquired;The joint position difference value between wheel foot joint in current control cycle and last control cycle is calculated;Under the condition that control target is zero speed stay, wheel foot joint position difference value is used as observation, and input data for training control strategy network is constructed;The target position difference value of wheel foot joint output by control strategy network is compared with actual position difference value, the control torque of wheel foot joint is calculated, and the parameters of control strategy network are updated in combination with the interaction result of control torque and simulation environment.The present disclosure can effectively avoid wheel slip cumulative error, realize the stable control of double wheel foot robot zero speed stay.

Description

Technical Field

[0001] This disclosure relates to the fields of sensor and robotics technology, and to a training method and apparatus for a stationary control model of a bipedal robot based on deep reinforcement learning. Background Technology

[0002] With the development of robotics technology, biwheeled legged robots, combining the advantages of both wheeled and legged structures, have gradually become an important research direction in related fields. Specifically, biwheeled legged robots can achieve high-speed, low-energy translational movement through their wheels, while simultaneously adapting to complex terrain and maintaining stable posture control through their leg joints. However, because the wheeled leg structure has the characteristics of both continuous rotational joints and multi-degree-of-freedom joints, its control in stationary states is extremely complex.

[0003] In related technologies, the control system of a bipedal robot may be based on reinforcement learning (RL), employing either speed control or absolute position control modes for policy network training, and using the trained policy network to adjust wheel motion during the operational phase. In speed control mode, the speed returned by the encoders of the bipedal robot wheels is usually calculated using position differential, which is more susceptible to noise interference compared to position feedback. To avoid noise amplification, the control parameters used for speed adjustment are generally set relatively small, resulting in insufficient wheel driving force. Under external disturbances, the bipedal robot is prone to continuous forward or backward drift. In contrast, the absolute position control mode controls the movement of the bipedal robot by setting absolute position coordinates, which can reduce forward or backward drift during stationary periods to some extent. However, since wheeled structures often experience slippage during actual movement, accumulated errors can easily occur with increasing operating time, resulting in a significant deviation between the stationary position output by the policy network and the target position, increasing the risk of control failure.

[0004] Therefore, there is an urgent need to provide a new control scheme for bipedal robots to achieve stable control of bipedal robots at zero speed.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] This disclosure provides a training method and apparatus for a dwell control model of a bipedal robot based on deep reinforcement learning, in order to overcome the problem in related technologies where the cumulative error caused by wheel slippage results in a large deviation between the dwell position output by the policy network and the target position, thus affecting the zero-speed dwell stability of the bipedal robot.

[0007] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure.

[0008] According to a first aspect of this disclosure, a method for training a stationary control model for a bipedal robot based on deep reinforcement learning is provided, comprising:

[0009] Within each control cycle, acquire the current position value of the wheel joints of the bipedal robot;

[0010] Calculate the difference in joint position between the wheel-foot joint in the current control cycle and the previous control cycle;

[0011] Under the condition that the control objective is to stop at zero speed, the joint position difference is used as an observation to construct the input data for training the control policy network;

[0012] The target position difference of the wheel-foot joint output by the control strategy network is compared with the actual position difference. The control torque of the wheel-foot joint is calculated, and the parameters of the control strategy network are updated by combining the interaction results of the control torque of the wheel-foot joint and the simulation environment.

[0013] In one exemplary embodiment of this disclosure, the target position difference of the wheel-foot joint output by the control strategy network is compared with the actual position difference to calculate the control torque of the wheel-foot joint, including:

[0014] The target position difference of the wheel-foot joint output by the control strategy network is compared with the actual position difference to obtain the position difference error of the wheel-foot joint.

[0015] Calculate the speed error of the wheel-foot joint based on the current speed of the wheel-foot joint and with zero speed as the target speed;

[0016] The control torque of the wheel-foot joint is calculated based on the position difference error and velocity error.

[0017] In one exemplary embodiment of this disclosure, the control torque of the wheel-foot joint is calculated based on the position difference error and the velocity error, including:

[0018] Based on proportional-derivative control, the control torque of the wheel-foot joint is calculated using position difference error and velocity error.

[0019] In one exemplary embodiment of this disclosure, the control torque of the wheel-foot joint is calculated based on proportional-derivative control, utilizing position difference error and velocity error, including:

[0020]

[0021] in, For the control torque of the wheel joint, This is the position difference error. For speed error, This represents the difference in the target position of the wheel joint. This represents the actual position difference of the wheel joint. This represents the current velocity of the wheel joint. For the proportional gain of the wheel-foot joint. The differential gain of the wheel joint.

[0022] In one exemplary embodiment of this disclosure, calculating the joint position difference of the wheel-foot joint between the current control cycle and the previous control cycle includes:

[0023] Noise is added to the current position value of the wheel joint in the current control cycle and the actual position value in the previous control cycle to obtain the current position value with noise and the actual position value with noise.

[0024] The joint position difference is calculated based on the current position value with noise and the actual position value with noise.

[0025] In one exemplary embodiment of this disclosure, the same noise is added to the current position value of the wheel joint in the current control cycle and the actual position value in the previous control cycle.

[0026] In one exemplary embodiment of this disclosure, the method further includes:

[0027] The noise intensity is adjusted according to a preset noise scaling factor.

[0028] In one exemplary embodiment of this disclosure, the method further includes:

[0029] Obtain the current position value of each leg joint of the bipedal robot;

[0030] Calculate the first difference between the current position value and the dwell position value for each leg joint;

[0031] Under the condition that the control objective is zero-speed dwell, the first difference is used as an observation to construct the input data for training the control policy network;

[0032] The target position value of the leg joint output by the control strategy network is compared with the current position value, and the control torque of each leg joint is calculated.

[0033] In one exemplary embodiment of this disclosure, the target position value of the leg joint output by the control policy network is compared with the current position value, and the control torque of each leg joint is calculated, including:

[0034] The target position value of the leg joint output by the control strategy network is compared with the current position value to obtain the position error of each leg joint;

[0035] Calculate the velocity error of each leg joint based on the current velocity of each leg joint and with zero velocity as the target velocity.

[0036] Based on proportional-derivative control, the control torque of each leg joint is calculated using position error and velocity error.

[0037] In one exemplary embodiment of this disclosure, based on proportional-derivative control, the control torque for each leg joint is calculated using position error and velocity error, including:

[0038]

[0039] in, For the control torque of the leg joint, This refers to the positional error of the leg joints. This represents the target position value for the leg joint. This represents the current position value of the leg joint. For the speed error of the leg joint, This represents the current velocity of the leg joint. The proportional gain for the leg joints. The differential gain is for the leg joint.

[0040] In one exemplary embodiment of this disclosure, after calculating the control torque of the wheel-foot joint, the method further includes:

[0041] The joint torque is obtained by combining the control torque of the wheel joint and the control torque of each leg joint;

[0042] The parameters of the control strategy network are updated based on the interaction between joint torque and the simulation environment.

[0043] In one exemplary embodiment of this disclosure, updating the parameters of the control strategy network based on the merged control torque further includes:

[0044] The joint torques are constrained and scaled to update the parameters of the control strategy network based on the interaction between the processed joint torques and the simulation environment.

[0045] In one exemplary embodiment of this disclosure, the interaction result includes a status and a reward;

[0046] The parameters of the control strategy network are updated based on the interaction between joint torques and the simulation environment, including:

[0047] Based on the state and reward obtained from joint torque interaction in the simulation environment, a loss function is constructed.

[0048] The parameters of the control policy network are updated based on the loss function.

[0049] In one exemplary embodiment of this disclosure, the loss function further includes a gradient penalty term;

[0050] The loss function is:

[0051]

[0052] in, For loss function, To control the parameters of the policy network, To reinforce the learning loss term, the state... s ,action a With rewards r Build, The gradient penalty coefficient is... To control the policy network in the state s The following action output, The output of the control policy network is relative to the parameters gradient, It is the square of the gradient norm, i.e., the regularization term.

[0053] In one exemplary embodiment of this disclosure, the input data for training the control policy network includes:

[0054] Obtain the base angular velocity and gravity projection of the bipedal robot in the current control cycle;

[0055] Acquire the joint motion data of the bipedal robot output by the control strategy network in the previous control cycle; wherein, the joint motion data includes the target position value of each leg joint and the target position difference of the wheel joints;

[0056] The input data is constructed based on the base angular velocity, gravity projection, joint motion data, the current velocity of each leg joint, the current velocity of the wheel joint, the calculated joint position value of each leg joint, and the joint position difference of the wheel joint.

[0057] According to a second aspect of this disclosure, a method for training a stationary control model for a bipedal robot based on deep reinforcement learning is provided, comprising:

[0058] Obtain the current position values ​​of the wheel joints of the bipedal robot;

[0059] Calculate the second difference in joint position between the current position value and the dwell position value of the wheel joint;

[0060] Under the condition that the control objective is zero-speed dwell, the second difference is used as an observation to construct the input data for training the control policy network;

[0061] The target position value of the wheel joint output by the control strategy network is compared with the current position value. The control torque of the wheel joint is calculated, and the parameters of the control strategy network are updated by combining the interaction results of the control torque of the wheel joint and the simulation environment.

[0062] According to a third aspect of this disclosure, a method for controlling the stationary position of a bipedal robot is provided, comprising:

[0063] Within each control cycle, acquire the current position value of the wheel joints of the bipedal robot;

[0064] Calculate the difference in joint position between the wheel-foot joint in the current control cycle and the previous control cycle;

[0065] The joint position difference is input into the pre-trained control strategy network to obtain the target position difference of the wheel joint in the current control cycle.

[0066] The control torque of the wheel-foot joint is calculated based on the current position value of the wheel-foot joint and the difference between the target position and the current position value in the current control cycle.

[0067] The control torque is sent to the joint actuators of the bipedal robot so that the joint actuators can control the stopping of the wheel joints according to the control torque;

[0068] The control strategy network is obtained using the deep reinforcement learning-based stationary control model training method for bipedal robots in the first aspect of this disclosure.

[0069] In one exemplary embodiment of this disclosure, the control torque of the wheel-foot joint is calculated based on the current position value of the wheel-foot joint in the current control cycle and the target position difference, including:

[0070] The target position value of the wheel-foot joint in the current control cycle is calculated based on the difference between the target positions of the wheel-foot joint in the current control cycle and the target position value in the previous control cycle.

[0071] The control torque of the wheel-foot joint is calculated based on the target position value, current position value, and current velocity of the wheel-foot joint in the current control cycle.

[0072] In one exemplary embodiment of this disclosure, calculating the target position value of the wheel-foot joint in the current control cycle based on the target position difference of the wheel-foot joint in the current control cycle and the target position value in the previous control cycle includes:

[0073] The target position difference is scaled according to the preset position scaling factor to obtain the scaled target position difference.

[0074] The target position value is calculated based on the scaled target position difference and the target position value of the previous control cycle.

[0075] In one exemplary embodiment of this disclosure, the control torque of the wheel-foot joint is calculated based on the target position value, current position value, and current velocity of the wheel-foot joint in the current control cycle, including:

[0076] The target position value of the wheel joint in the current control cycle is compared with the current position value to obtain the position error of the wheel joint.

[0077] Obtain the current velocity of the wheel joint, and use zero velocity as the target velocity to obtain the velocity error of the wheel joint;

[0078] Based on proportional-derivative control, the control torque of the wheel joint is calculated using position error and velocity error.

[0079] According to a fourth aspect of this disclosure, a method for controlling the stationary position of a bipedal robot is provided, comprising:

[0080] Obtain the current position values ​​of the wheel joints of the bipedal robot;

[0081] Calculate the difference in joint position between the current position value and the dwell position value of the wheel joint;

[0082] The difference in joint position is input into a pre-trained control strategy network to obtain the target position value of the wheel joint;

[0083] Calculate the control torque of the wheel-foot joint based on the current position value and the target position value;

[0084] The control torque is sent to the joint actuators of the bipedal robot so that the joint actuators can control the stopping of the wheel joints according to the control torque;

[0085] The control strategy network is obtained according to the deep reinforcement learning-based stationary control model training method for bipedal robots in the second aspect of this disclosure.

[0086] According to a fifth aspect of this disclosure, a training device for a stationary control model of a bipedal robot based on deep reinforcement learning is provided, comprising:

[0087] The first position acquisition module is used to acquire the current position value of the wheel joints of the bipedal robot in each control cycle.

[0088] The first difference determination module is used to calculate the joint position difference of the wheel-foot joint between the current control cycle and the previous control cycle.

[0089] The first input construction module is used to construct input data for training the control strategy network by using the joint position difference as an observation under the condition that the control target is zero-speed dwell.

[0090] The first network training module is used to compare the target position difference of the wheel-foot joint output by the control strategy network with the actual position difference, calculate the control torque of the wheel-foot joint, and update the parameters of the control strategy network by combining the interaction results of the control torque of the wheel-foot joint with the simulation environment.

[0091] According to a sixth aspect of this disclosure, a training device for a stationary control model of a bipedal robot based on deep reinforcement learning is provided, comprising:

[0092] The second position acquisition module is used to acquire the current position value of the wheel joints of the bipedal robot;

[0093] The second difference determination module is used to calculate the second difference between the current position value and the dwell position value of the wheel joint;

[0094] The second input construction module is used to construct input data for training the control policy network by taking the second difference of the joint position as an observation under the condition that the control target is zero-speed dwell.

[0095] The second network training module is used to compare the target position value of the wheel joint output by the control strategy network with the current position value, calculate the control torque of the wheel joint, and update the parameters of the control strategy network by combining the interaction results of the control torque of the wheel joint and the simulation environment.

[0096] According to a seventh aspect of this disclosure, a dwell control device for a bipedal robot is provided, comprising:

[0097] The third position acquisition module is used to acquire the current position value of the wheel joints of the bipedal robot in each control cycle.

[0098] The third difference determination module is used to calculate the joint position difference between the wheel-foot joint in the current control cycle and the previous control cycle.

[0099] The relative position determination module is used to input the joint position difference into the pre-trained control strategy network to obtain the target position difference of the wheel joint in the current control cycle;

[0100] The first torque determination module is used to calculate the control torque of the wheel-foot joint based on the current position value of the wheel-foot joint in the current control cycle and the difference between the target position and the current position value.

[0101] The first dwell control module is used to send control torque to the joint actuator of the bipedal robot so that the joint actuator can control the dwell of the wheel joints according to the control torque;

[0102] The control strategy network is obtained according to the deep reinforcement learning-based stationary control model training method for bipedal robots of the first aspect of this disclosure.

[0103] According to the eighth aspect of this disclosure, a dwell control device for a bipedal robot is provided, comprising:

[0104] The fourth position acquisition module is used to acquire the current position value of the wheel joints of the bipedal robot;

[0105] The fourth difference determination module is used to calculate the third difference between the current position value and the dwell position value of the wheel joint;

[0106] The absolute position determination module is used to input the third difference into the pre-trained control strategy network to obtain the target position value of the wheel joint;

[0107] The second torque determination module is used to calculate the control torque of the wheel-foot joint based on the current position value and the target position value of the wheel-foot joint.

[0108] The second dwell control module is used to send control torque to the joint actuator of the bipedal robot so that the joint actuator can control the dwell of the wheel joints according to the control torque;

[0109] The control strategy network is obtained according to the deep reinforcement learning-based stationary control model training method for bipedal robots of the second aspect of this disclosure.

[0110] According to a ninth aspect of this disclosure, an electronic device is provided, comprising:

[0111] Processor; and

[0112] A memory that stores computer-readable instructions, which, when executed by a processor, implement the method as described in the above embodiments.

[0113] According to a tenth aspect of this disclosure, a bipedal robot is provided, comprising:

[0114] Processor; and

[0115] A memory that stores computer-readable instructions, which, when executed by a processor, implement the method as described in the above embodiments.

[0116] According to the eleventh aspect of this disclosure, a computer-readable storage medium is provided, on which computer program code instructions are stored, which, when invoked by the processor of a bipedal robot, cause the robot to perform the method as described in the above embodiments.

[0117] As can be seen from the above technical solution, this disclosure possesses at least one of the following advantages and positive effects:

[0118] This disclosure uses relative position difference, i.e., joint position difference, as the control variable during training. This allows the control strategy to be adjusted based on the actual changes in the wheel-leg joints within adjacent control cycles, more accurately reflecting the motion state during the stationary phase, thereby avoiding the cumulative error caused by wheel slippage in absolute position control. Therefore, this method can reduce stationary positioning deviations caused by error accumulation, thus improving the stability of the bipedal robot in maintaining zero-speed stationary positions under complex terrain and long-term operating conditions. Attached Figure Description

[0119] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0120] Figure 1 The diagram illustrates the system architecture of the deep reinforcement learning-based bipedal robot dwell control model training method and the bipedal robot dwell control method applicable to embodiments of this disclosure.

[0121] Figure 2 A flowchart illustrating a training method for a dwell control model of a bipedal robot based on deep reinforcement learning is shown in an embodiment of this disclosure.

[0122] Figure 3 A flowchart illustrating a method for calculating the control torque of a wheel joint according to an embodiment of this disclosure is shown.

[0123] Figure 4 A schematic flowchart illustrating a method for calculating the control torque of each leg joint in an embodiment of this disclosure is shown.

[0124] Figure 5 A flowchart illustrating another method for training a dwell control model for a bipedal robot based on deep reinforcement learning, as shown in an embodiment of this disclosure, is presented.

[0125] Figure 6 A flowchart illustrating a method for controlling the stationary position of a bipedal robot according to an embodiment of this disclosure is shown.

[0126] Figure 7 A schematic flowchart illustrating another embodiment of the present disclosure for calculating the control torque of the wheel joint is shown.

[0127] Figure 8 A flowchart illustrating another method for controlling the stationary position of a bipedal robot according to an embodiment of this disclosure is shown.

[0128] Figure 9A block diagram of a training device for a bipedal robot stationary control model based on deep reinforcement learning is shown in an embodiment of this disclosure.

[0129] Figure 10 A block diagram of another deep reinforcement learning-based bipedal robot dwell control model training device is shown in an embodiment of this disclosure.

[0130] Figure 11 A block diagram of a bipedal robot stationary control device according to an embodiment of the present disclosure is shown.

[0131] Figure 12 A block diagram of another bipedal robot stationary control device is shown in an embodiment of this disclosure.

[0132] Figure 13 A schematic diagram of a bipedal robot according to an embodiment of this disclosure is shown.

[0133] Figure 14 A schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure is shown.

[0134] Figure 15 A schematic diagram of a computer-readable storage medium according to an embodiment of the present disclosure is shown. Detailed Implementation

[0135] In this disclosure, the terms "first" and "second" are used for description only and do not indicate relative importance or imply the number of technical features. Therefore, the features referred to as "first" or "second" may explicitly or implicitly include at least one of those features. "A plurality of" means at least two, unless otherwise expressly defined.

[0136] First, the relevant terms used in the exemplary embodiments of this disclosure will be explained:

[0137] Wheel-foot joint position: refers to the joint angle between the wheel and the leg link in a biwheeled robot, used to characterize the posture state of the wheel-foot joint at the current moment.

[0138] Zero-speed stationary position: During the control process, the overall linear velocity and angular velocity of the biwheeled robot approach zero, reaching a stable state. In this state, the biwheeled robot can maintain balance near the target stationary position without significant displacement or tipping.

[0139] Policy Network: A control model trained within a reinforcement learning framework, disclosed herein used to map the state information collected by the biwheeled robot in each control cycle to the control output. The policy network is typically implemented using a deep neural network, and through interaction with the simulation environment, it continuously updates the model parameters based on the control objective of zero-speed dwell, thereby achieving stable dwell control of the biwheeled robot.

[0140] Control torque: The torque signal output by the main controller of the bi-wheeled robot and applied to the joint actuator is used to adjust the rotation of the wheel joints and realize the stationary control of the bi-wheeled robot.

[0141] Proportional-Derivative (PD) control is a commonly used closed-loop control method that generates a control signal by calculating the proportional term of the current error and the derivative term of the error's rate of change. The proportional term is used to quickly respond to deviations, while the derivative term is used to suppress dynamic oscillations, thereby achieving a smooth and stable control effect.

[0142] Control cycle: refers to the time interval between state acquisition and control output in a complete control loop. Specifically, the control cycle during the training phase corresponds to the time step of the training environment, describing the time interval between joint position acquisition, joint position difference calculation, policy network output, and parameter updates during training. The control cycle during the deployment phase corresponds to the refresh cycle of the main controller, referring to the time interval between the main controller acquiring joint data, calculating control torque, and outputting it to the joint actuators. The control cycles of both phases are kept consistent on the time scale to ensure the consistency of the model's dynamic response during training and actual operation.

[0143] Figure 1 The diagram illustrates a training method for a deep reinforcement learning-based bipedal robot dwell control model and a system architecture diagram for the dwell control method of a bipedal robot, applicable to embodiments of this disclosure. Figure 1 As shown, the system architecture 100 may include a terminal device 101, a bipedal robot 102, a network 103, and a server 104.

[0144] The terminal device 101 includes, but is not limited to, desktop computers, laptops, smartphones, and tablets. The terminal device 101 is equipped with a graphical user interface, providing visualization capabilities to display the real-time posture information, wheel joint motion status, and control strategy output results of the bipedal robot 102 during training. Furthermore, the terminal device 101 also provides parameter configuration functions, allowing users to set training parameters, control cycles, and zero-speed dwell thresholds for the control strategy network, and to collaborate with the server 104 to perform visual analysis and model management of training data.

[0145] The bipedal robot 102 is equipped with various sensors, such as inertial measurement units and joint angle encoders, to collect real-time operational status information, including the current position values ​​of the wheel and leg joints. It is important to note that the bipedal robot 102 includes at least a main controller and joint actuators. For example, the main controller can be used to run a deployed control strategy network. In each control cycle, it receives sensor sampling data, calculates the joint position difference between the current and previous control cycles, and uses this joint position difference as an observation input to the control strategy network to predict the target position difference of the wheel and leg joints. Then, based on the deviation between the predicted target position difference and the actual position difference, it calculates the control torque of the wheel and leg joints and sends the control torque to the joint actuators. The joint actuators drive the wheel and leg joints to perform corresponding actions based on the control torque to achieve dwell control.

[0146] Server 104 is equipped with a training module for a control strategy network. This module receives sensor data, such as joint position values, uploaded by the bipedal robot 102. After preprocessing the joint position values, it trains the control strategy network based on a deep reinforcement learning framework. During training, server 104 calculates the control torque based on the zero-speed stop control objective and updates the parameters of the control strategy network by combining the interaction results of the control torque and the simulation environment. After training, server 104 can distribute the trained control strategy network to the bipedal robot 102 for use in executing stop control during operation.

[0147] Network 103 serves as the medium for providing a communication link between terminal device 101, bipedal robot 102, and server 104. Network 103 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc. It should be understood that... Figure 1 The number and type of terminal devices, bipedal robots, networks, and servers shown are merely illustrative. Depending on implementation needs, there can be any number and type of terminal devices, bipedal robots, networks, and servers; for example, the servers could be cloud servers.

[0148] Through the coordinated operation of various parts in the system architecture 100, a complete process can be realized, from wheel joint position acquisition, joint position difference calculation, control strategy network training to actual dwell control execution, thereby improving the control accuracy and stability of the bipedal robot in zero-speed dwell tasks.

[0149] This disclosure provides an example implementation of a method for training a dwell control model for a bipedal robot based on deep reinforcement learning. (See reference...) Figure 2 As shown, the method may include the following steps S201 to S204:

[0150] Step S201: In each control cycle, obtain the current position value of the wheel joints of the bipedal robot;

[0151] Step S202: Calculate the joint position difference of the wheel-foot joint between the current control cycle and the previous control cycle;

[0152] Step S203: Under the condition that the control objective is to stop at zero speed, the joint position difference is used as an observation to construct input data for training the control strategy network.

[0153] Step S204: Compare the target position difference of the wheel-foot joint output by the control strategy network with the actual position difference, calculate the control torque of the wheel-foot joint, and update the parameters of the control strategy network by combining the interaction results of the control torque of the wheel-foot joint and the simulation environment.

[0154] The deep reinforcement learning-based method for training a bipedal robot's stationary control model, as provided in this exemplary embodiment, uses relative position differences (i.e., joint position differences) as control variables during training. This allows the control strategy to adjust based on the actual changes in the wheel and leg joints within adjacent control cycles, more accurately reflecting the motion state during stationary operation and avoiding the cumulative errors caused by wheel slippage in absolute position control. Therefore, this method reduces stationary positioning deviations due to error accumulation, thereby improving the stability of the bipedal robot in maintaining zero-speed stationary operation under complex terrain and long-term running conditions.

[0155] The training method for the dwell control model of a bipedal robot based on deep reinforcement learning in this example embodiment will be described in detail below.

[0156] In step S201, the current position value of the wheel joints of the bipedal robot is obtained in each control cycle.

[0157] In the training environment, the current position values ​​of the wheel joints of the bipedal robot can be measured in real time using angle encoders, inertial measurement units, or other angle detection sensors installed at each joint. For example, at the beginning of each control cycle, the current joint angle data is collected by sensors to obtain the current position value of the corresponding wheel joint. The current position value of the wheel joint reflects the relative motion relationship between the wheel and the leg linkage, and is an important basic data for calculating the joint position difference.

[0158] By acquiring the current position value of the wheel-foot joint in real time during each control cycle, the changes in the position of the wheel-foot joint can be determined, providing basic data support for subsequent calculation of the joint position difference between adjacent control cycles.

[0159] The control cycle can be flexibly set according to the system control requirements, for example, within the range of 10 to 20 milliseconds, to ensure that the sensor sampling has sufficient time resolution, thereby improving the accuracy and response performance of subsequent difference calculation and control output.

[0160] In step S202, the difference in joint position between the wheel-foot joint in the current control cycle and the previous control cycle is calculated.

[0161] Specifically, the wheel joint position values ​​of the current control cycle are compared with those of the previous control cycle to calculate the difference between them. This joint position difference characterizes the relative motion changes of the wheel joints within adjacent control cycles, reflecting the posture adjustment trend and dynamic response of the wheel structure during the stationary phase of the biwheeled robot.

[0162] By calculating the joint position difference between adjacent control cycles, the minute displacement changes of the wheel-foot joints can be effectively captured, providing input information reflecting instantaneous attitude changes to the subsequent control strategy network, thereby achieving dynamic adjustment of the dwell process.

[0163] In some example implementations, to enhance the stability and robustness of the control strategy network under different operating conditions, perturbation processing can be introduced into the collected wheel joint position data before calculating the joint position difference.

[0164] Specifically, noise can be added to the current position value of the wheel joint in the current control cycle and the actual position value in the previous control cycle to obtain the current position value with noise and the actual position value with noise. The noise can be random noise or a disturbance signal generated based on a preset distribution (such as uniform distribution, normal distribution, etc.) to simulate sensor measurement errors, mechanical vibrations, or external interference, thereby introducing a certain degree of uncertainty into the training process.

[0165] Subsequently, the joint position difference is calculated based on the noisy current position value and the noisy actual position value. This method allows the training data to more closely approximate actual operating conditions, enabling the control strategy network to maintain a stable response even in the presence of measurement noise or external disturbances. This, in turn, improves the generalization performance and control stability of the control strategy network in complex terrain and long-term operating scenarios.

[0166] It should be noted that, in this example implementation, the noise added to the current position value of the wheel joint in the current control cycle is the same as the actual position value in the previous control cycle. By introducing the same noise into the joint position values ​​of the two control cycles, it can be ensured that the noise terms cancel each other out when calculating the difference between the two, thereby avoiding fluctuations in the joint position difference caused by random disturbance differences and maintaining the stability and consistency of the joint position difference calculation.

[0167] Furthermore, to simulate different levels of measurement errors or external disturbances, the noise intensity can be adjusted according to a preset noise scaling factor. This noise scaling factor can be dynamically set based on the task requirements or environmental complexity during the training phase, controlling the range of noise signal intensity. This allows the control strategy network to gradually adapt to different training conditions, from low to high noise, thereby improving the robustness and convergence stability of the control strategy network in complex operating scenarios.

[0168] Taking random noise as an example, a random noise signal with a value range of [-1, 1] can be generated in each control cycle, and its dimension is the same as the position data of the wheel-foot joint. Before calculating the joint position difference, the same random noise term is superimposed on the current position value of the wheel-foot joint in the current control cycle and the actual position value in the previous control cycle, respectively, to obtain the noisy current position value and the noisy actual position value.

[0169] The intensity of random noise can be adjusted by weighting according to a preset noise scaling vector to control the disturbance amplitude of different wheel joints. It should be noted that the noise scaling vector can be composed of multiple noise scaling factors, each corresponding to a wheel joint, used to determine the noise weight of that wheel joint.

[0170] In addition, the overall noise signal can be uniformly scaled by setting a global amplification factor, which can be used to control the overall intensity of the disturbance signal during the training process.

[0171] By combining the noise scaling factor and the global amplification coefficient, fine control of the noise intensity of different wheel joints can be achieved while maintaining the consistency of the overall perturbation range, thereby improving the stability and generalization performance of the training process.

[0172] Subsequently, the joint position difference between adjacent control cycles is calculated based on the current position value with noise and the actual position value with noise. That is, the difference between the current position value with noise and the actual position value with noise is calculated to obtain the joint position difference.

[0173] This approach allows for the introduction of simulated measurement errors and environmental disturbances during the training phase, making the training data distribution closer to actual operating conditions. As a result, even after model deployment, the control strategy network can maintain stable attitude adjustment capabilities and high dwell control accuracy despite external interference such as sensor noise or uneven terrain.

[0174] In step S203, under the condition that the control objective is to stop at zero speed, the joint position difference is used as an observation to construct input data for training the control strategy network.

[0175] By using the joint position difference as an observation, the dynamic equilibrium process of the bipedal robot in a zero-speed stationary state can be accurately reflected, thus providing continuous and measurable state input for the control strategy network.

[0176] When constructing the training input, the joint position difference can be used as an observation, or it can be combined with other state information to form a complete observation vector, which is used to characterize the overall posture state of the bipedal robot in the current control cycle. Subsequently, the observation or observation vector is input into the control policy network to predict the target position difference of each wheel joint in the next control cycle.

[0177] By training the control strategy network with joint position difference as the primary observation under zero-speed stationary conditions, the control strategy network can learn the posture adjustment rules under different posture disturbances, thereby improving the balance maintenance ability and stability of the bipedal robot in a stationary state.

[0178] In some example implementations, when constructing input data for training the control policy network, the base angular velocity and gravity projection of the biwheeled robot in the current control cycle can be obtained. The base angular velocity characterizes the overall rotation rate of the biwheeled robot, while the gravity projection describes the components of gravity in different directions within the robot's body coordinate system, reflecting the degree of deviation of the robot's posture relative to the direction of gravity. Together, they reflect the posture changes of the biwheeled robot in space, helping to assess its stability and responsiveness to external disturbances.

[0179] It can also acquire joint motion data output by the control strategy network from the previous control cycle, including the target position value of each leg joint and the target position difference of the wheel joints. The target position value of each leg joint represents the ideal angular position generated by the control strategy network for the leg joint, used to guide the bi-wheeled robot to maintain or adjust to the desired posture during control. The target position difference of the wheel joints represents the angular change of the wheel joints relative to the previous control cycle, used to fine-tune the posture of the bi-wheeled robot during stationary states to maintain a stable stationary state.

[0180] Simultaneously, the current velocity of each leg joint and the current velocity of the wheel joint are acquired, and the joint position value of each leg joint and the joint position difference of the wheel joint are calculated. Among them, the joint velocity and joint position together characterize the motion state of the bipedal robot in the current control cycle, while the joint position difference of the wheel joint reflects the small angular changes of the wheel joint between adjacent control cycles, which is an important basis for achieving fine adjustment of dwell control.

[0181] Finally, the acquired information, including base angular velocity, gravity projection, joint motion data, joint velocity, and joint position, is integrated into a set of multidimensional input data for training the control policy network. For example, various types of state information can be scaled or standardized separately to ensure that data with different dimensions and value ranges are within similar numerical ranges, thereby ensuring that the impact of various types of state information on network parameter updates remains balanced during training.

[0182] In this example, the constructed input data can provide comprehensive state information of the bipedal robot in each control cycle, providing high-quality training samples for the control policy network. This helps the control policy network learn how to cope with different posture adjustment tasks and optimize the balance control of the bipedal robot during zero-speed dwell.

[0183] In step S204, the target position difference of the wheel joint output by the control strategy network is compared with the actual position difference, the control torque of the wheel joint is calculated, and the parameters of the control strategy network are updated by combining the interaction results of the control torque of the wheel joint and the simulation environment.

[0184] Specifically, within each control cycle, the target position difference of the wheel-leg joints output by the control strategy network is compared with the actual position difference, and the control error signal is calculated based on the deviation between the two. The actual position difference refers to the actual angular change of the wheel-leg joint between two adjacent control cycles, characterizing the amplitude of angular change generated by the wheel-leg joint during actual movement and reflecting the posture adjustment of the bi-wheeled robot during stationary states.

[0185] Furthermore, based on the control error signal, combined with other control laws such as proportional-derivative control, nonlinear damping control, adaptive gain control, fuzzy PD control, and neural network approximation control, the corresponding control torque is calculated to drive the wheel joints to perform corresponding compensation actions, thereby realizing real-time adjustment and stable control of the bipedal robot's posture.

[0186] During the training phase, the calculated control torques of the wheel-leg joints can be input as execution signals to the simulation environment to drive the bipedal robot to produce interactive behaviors. The simulation environment outputs corresponding environmental states and reward signals based on the control torques, forming the interactive results of the simulation environment. The control strategy network receives the environmental states and reward signals, evaluates the current control effect, and updates the parameters of the control strategy network based on the evaluation results.

[0187] Specifically, the effectiveness of the control strategy network's output is determined by comparing the deviation between the state change caused by the current control output and the zero-speed dwell target. If the deviation is large, the network parameters are gradient-corrected based on the reward signal to guide the strategy network to generate control torques that better match the dwell target in subsequent training. Through continuous iteration of the above process, the control strategy network can gradually learn the correlation between the wheel joint control torques and the state feedback of the simulation environment, thereby adaptively outputting stable control torques under different disturbance conditions, achieving steady-state control and attitude maintenance of the bipedal robot in a zero-speed state.

[0188] In some example implementations, references Figure 3 As shown, the control torque of the wheel-foot joint can be calculated according to the following steps S301 to S303:

[0189] Step S301: Compare the target position difference of the wheel joint output by the control strategy network with the actual position difference to obtain the position difference error of the wheel joint. The position difference error is used to reflect the degree of deviation between the actual angle change of the wheel joint and the expected output of the control strategy network.

[0190] Step S302: Based on the current speed of the wheel joint and with zero speed as the target speed, calculate the speed error of the wheel joint. The speed error characterizes the dynamic offset of the wheel joint relative to the stationary state during the stop control process. It can be used to compensate for attitude changes caused by slight wheel slippage or external force disturbances, providing a dynamic compensation basis for the control torque.

[0191] Step S303: Calculate the control torque of the wheel joint based on the position difference error and velocity error.

[0192] For example, the position difference error and the velocity error can be weighted and combined to obtain the control torque used to drive the wheel joint. For instance, based on proportional-derivative control, the control torque of the wheel joint is calculated using the position difference error and the velocity error, i.e.:

[0193] (1)

[0194] in, For the control torque of the wheel joint, This is the position difference error. For speed error, This represents the difference in the target position of the wheel joint. This represents the actual position difference of the wheel joint. This represents the current velocity of the wheel joint. For the proportional gain of the wheel-foot joint. The differential gain of the wheel joint.

[0195] In formula (1), the proportional term is used to correct the error between the target position difference and the actual position difference of the wheel-foot joint, so as to achieve rapid tracking of the target attitude of the wheel-foot joint. The differential term is used to suppress the instantaneous fluctuations caused by speed changes and smooth the control output, so as to achieve stable differential following even when the wheel slips.

[0196] In some example implementations, when controlling the stationary position of a bipedal robot, its balance depends not only on the adjustment of the ground support force by the wheel joints but also on the coordinated action of the leg joints to counteract the shift in the center of gravity caused by changes in upper body posture. Therefore, a control mechanism for the leg joints can be further introduced to enhance the overall stability and disturbance resistance during stationary control by finely adjusting the leg posture.

[0197] refer to Figure 4 As shown, the control torque of each leg joint can be calculated according to steps S401 to S404 to establish coordinated control between the leg joints and the wheel foot joints during the hold control training process, so that the control strategy network can simultaneously learn the coordination relationship between leg support and wheel foot posture adjustment:

[0198] Step S401: Obtain the current position value of each leg joint of the bipedal robot.

[0199] Understandably, the current position value of the leg joint is the angle value of the leg joint, which can be collected in real time by an angle encoder installed at each leg joint to characterize the posture state of the leg structure of the two-wheeled robot during the current control cycle.

[0200] Step S402: Calculate the first difference between the current position value and the dwell position value of each leg joint.

[0201] The dwell position value in this step refers to the target angle value of each leg joint of the bipedal robot in a zero-speed dwell state. It represents the ideal posture reference position set to maintain overall posture stability and center of gravity balance. The dwell position value can be obtained through static posture planning or simulation calibration, and is used to represent the standard joint posture that the bipedal robot should maintain in a dwell state. The first difference value reflects the angular deviation of the leg joints between the current posture and the target dwell posture. It is an important input parameter for control strategy network training and control calculation, and is used to describe the posture error and deviation degree of the bipedal robot during the dwell process.

[0202] Step S403: Under the condition that the control objective is to stop at zero speed, the first difference is used as an observation to construct input data for training the control policy network.

[0203] Optionally, when constructing input data, the first difference of the leg joints, the joint position difference of the wheel joints, the base angular velocity, and the gravity projection can be used as input to guide the control strategy network to learn how to coordinate the linkage control of the leg joints and the wheel joints, so as to achieve the overall balance and posture stability of the bipedal robot.

[0204] Step S404: Compare the target position value of the leg joint output by the control strategy network with the current position value, and calculate the control torque of each leg joint.

[0205] For example, firstly, the target position value of the leg joints output by the control policy network is compared with the current position value to obtain the position error of each leg joint. Here, the target position value is the desired angle dynamically calculated by the control policy network based on the current state, used to guide the leg joint movement to approach the dwell position value. Next, based on the current velocity of each leg joint, and using zero velocity as the target velocity, the velocity error of each leg joint is calculated. Finally, based on proportional-derivative control, the control torque of each leg joint is calculated using the position error and velocity error.

[0206] For example, when calculating the control torque of each leg joint, it can be based on:

[0207] (2)

[0208] in, For the control torque of the leg joint, This refers to the positional error of the leg joints. This represents the target position value for the leg joint. This represents the current position value of the leg joint. For the speed error of the leg joint, This represents the current velocity of the leg joint. The proportional gain for the leg joints. The differential gain is for the leg joint.

[0209] In formula (2), the proportional term is used to correct the error between the target position and the actual position of the leg joint, so as to achieve rapid response and accurate tracking of the target posture of the leg joint. The differential term is used to suppress the velocity deviation of the leg joint, reduce the oscillation caused by inertia or disturbance during the dynamic process, and ensure that the leg joint reaches the target position smoothly.

[0210] By calculating the control torque of each leg joint, the posture of the biwheeled robot's legs can be corrected in real time, coordinating with the movement of the wheel joints, thereby improving the posture stability and disturbance resistance of the biwheeled robot in stationary tasks.

[0211] Furthermore, after calculating the control torques of the wheel-foot joints and each leg joint, these torques can be combined to obtain the joint torques, thus forming a complete control vector covering the joints related to the stationary control. This control vector serves as the output of the current control cycle, driving the coordinated movements of the bipedal robot in its stationary state to achieve stable overall posture maintenance.

[0212] In some example implementations, when updating the parameters of the control policy network based on the interaction results between joint torques and the simulation environment, a loss function can be constructed based on the state and reward obtained from the interaction of joint torques in the simulation environment. Then, the parameters of the control policy network are updated based on the loss function.

[0213] Specifically, in the simulation environment, based on indicators such as the bipedal robot's posture stability, positional deviation, and zero-speed holding effect within the current control cycle, corresponding reward values ​​are generated to characterize the performance of the current control strategy in achieving the dwelling target. Then, based on the difference between the reward value and the expected control target, a loss function is calculated to measure the degree of deviation between the current control strategy output and the ideal dwelling state.

[0214] Optionally, the loss function may also include a gradient penalty term, such as:

[0215] (3)

[0216] in, For loss function, To control the parameters of the policy network, To reinforce the learning loss term, the state... s ,action a With rewards r Build, The gradient penalty coefficient is... To control the policy network in the state s The following action output, The output of the control policy network is relative to the parameters gradient, It is the square of the gradient norm, i.e., the regularization term.

[0217] Based on the loss function shown in Equation (3), the parameters of the control policy network can be iteratively updated using other optimization algorithms such as stochastic gradient descent or Adam (Adaptive Moment Estimation) optimization. During the optimization process, the network parameters are fine-tuned according to the gradient information of the loss function in each control cycle to gradually reduce the loss value and achieve convergence of the control policy network from the initial state to the optimal dwell control policy.

[0218] Through multiple rounds of simulation training and parameter updates, the control strategy network can gradually learn to output better control torques under different postures and external disturbances, thereby improving the posture stability of the bipedal robot in zero-speed stationary tasks.

[0219] In some example implementations, after merging the joint torques, the joint torques can be constrained and scaled to update the parameters of the control strategy network based on the interaction results between the processed joint torques and the simulation environment.

[0220] Specifically, the control torque of each joint can be constrained based on a preset torque threshold, cutting off control torques exceeding the threshold to a safe range to prevent excessive output from causing instability or actuator overload. After constraint processing, a uniform scaling factor can be applied to the overall control torque to ensure that the output torque matches the characteristics of the joint actuator and the simulation environment, thereby guaranteeing the stability and consistency of the control process.

[0221] The joint torques, after constraint and scaling processing, serve as the execution signals for the current control cycle, driving the coordinated movements of each joint to maintain posture and achieve dynamic balance during the stationary control process of the bipedal robot. During the training phase, the parameters of the control strategy network can be updated based on the deviation between the state changes caused by the joint torques and the zero-speed stationary target, allowing the control strategy network to gradually learn the optimal joint torque distribution rules and posture control strategies.

[0222] This disclosure also provides an exemplary implementation method for training a dwell control model for a bipedal robot based on deep reinforcement learning. (See references) Figure 5 As shown, the method may include the following steps S501 to S504:

[0223] Step S501: Obtain the current position value of the wheel joints of the bipedal robot.

[0224] For example, the current position value of the wheel joint can be obtained by collecting joint angle information in real time through angle encoders set at each wheel joint of the bipedal robot.

[0225] Step S502: Calculate the second difference in joint position between the current position value and the dwell position value of the wheel joint.

[0226] The dwell position value in this step is the ideal wheel joint angle when the bipedal robot achieves a zero-speed dwell state. By comparing the current position value of the wheel joint with the dwell position value, a second difference is obtained, which reflects the degree of deviation between the current wheel joint posture and the target dwell posture.

[0227] Understandably, the second difference can characterize the posture deviation trend of the bipedal robot during the stationary process, providing accurate error information input for the control strategy network.

[0228] Step S503: Under the condition that the control objective is to stop at zero speed, the second difference is used as an observation to construct input data for training the control policy network.

[0229] The control strategy network uses the second difference as the main observation input, and can also acquire and integrate other state information such as base angular velocity, gravity projection, and joint velocity to construct an observation vector that reflects the overall state of the bipedal robot.

[0230] After the observation vector is input into the control strategy network, the network learns how to generate corresponding control action outputs, including target position values ​​for the wheel joints, through interactive training with the simulation environment. Through continuous iterative training, the control strategy network optimizes the action generation mechanism, enabling the bipedal robot to automatically adjust wheel support force and leg posture during stationary maneuvers, thereby reducing posture deviations and achieving stable zero-speed stationary control.

[0231] Step S504: Compare the target position value of the wheel joint output by the control strategy network with the current position value, calculate the control torque of the wheel joint, and update the parameters of the control strategy network by combining the control torque of the wheel joint with the interaction results of the simulation environment.

[0232] Specifically, the position error can be obtained by comparing the target position value with the actual position value of the wheel joint.

[0233] Based on this positional error, proportional-derivative control can be used to calculate the control torque of the wheel-foot joint, which is then used to drive the wheel-foot joint to perform posture adjustment movements. Simultaneously, the leg joints can also be synchronously controlled, as shown in the reference... Figure 4 The process shown calculates the control torque for each leg joint, which is used to maintain the stability of the upper body's center of gravity and posture.

[0234] Similarly, during the training phase, the calculated control torques of the wheel-leg joints can be input as execution signals to the simulation environment to drive the bipedal robot to generate interactive behaviors. The simulation environment outputs corresponding environmental states and reward signals based on the control torques, forming the interactive results of the simulation environment. The control strategy network receives the environmental states and reward signals, evaluates the current output control effect, and updates the parameters of the control strategy network based on the evaluation results.

[0235] In some exemplary implementations, the control torques calculated from the wheel joints and leg joints can be fused to form a joint control output. This joint control output, combined with the interaction results with the simulation environment, updates the parameters of the control strategy network. This joint control output reflects the synergistic relationship between wheel support and leg posture adjustment in a stationary state. By introducing the synergistic control results of the two types of joint torques during training, the control strategy network can simultaneously learn the coordination patterns of the wheel and leg joints in zero-speed stationary tasks, achieving dynamic balance and stability maintenance of the whole-body posture, thereby improving the overall stationary control performance of the biwheeled robot under complex terrain or external disturbance conditions.

[0236] Depend on Figure 2 As shown in the training method for the dwell control model of the bipedal robot, the training process uses the joint position difference between adjacent control cycles as the main observation input, i.e., it uses a relative error approach to train the control strategy network. This training method emphasizes capturing subtle angular changes in the wheel-leg joints within continuous control cycles, enabling the control strategy network to learn the dynamic response characteristics of the joints. During dwell control, based on the relative error input, posture deviations can be quickly identified and corrected, achieving a smooth response to instantaneous disturbances and subtle posture adjustments.

[0237] Compared to Figure 2 The training method for the dwell control model of the bipedal robot shown is as follows: Figure 5 The training method for the dwell control model of the bipedal robot in this paper adopts absolute position error, that is, the second difference between the current joint position and the target dwell position is used as the observation input of the control strategy network. By directly comparing the deviation between the current posture and the ideal dwell posture, the control strategy network can learn posture balance and steady-state control. During the training process, it gradually develops the ability to recognize and correct absolute posture, thereby obtaining a stronger steady-state maintenance ability in dwell control tasks.

[0238] The two training methods can be used separately or in combination depending on different task scenarios, which can improve overall stability while taking into account response speed, and achieve a more reliable dwell control effect.

[0239] This disclosure also provides an exemplary embodiment of a method for controlling the stationary position of a bipedal robot. (See reference...) Figure 6 As shown, the method may include the following steps S601 to S605:

[0240] Step S601: In each control cycle, obtain the current position value of the wheel joints of the bipedal robot.

[0241] During actual operation, the current position values ​​of the wheel joints of the bipedal robot can be measured in real time using angle encoders, inertial measurement units, or other angle detection sensors installed at each joint. The main controller reads the current joint angle data at the beginning of each control cycle to obtain the corresponding current position value of the wheel joint.

[0242] Step S602: Calculate the joint position difference between the wheel-foot joint in the current control cycle and the previous control cycle.

[0243] The wheel joint position values ​​of the current control cycle are compared with those of the previous control cycle to calculate the difference. This difference reflects the change in wheel joint angle within adjacent control cycles and serves as the basic input data for the control strategy network.

[0244] Step S603: Input the joint position difference into the pre-trained control strategy network to obtain the target position difference of the wheel joint in the current control cycle.

[0245] The target position difference refers to the expected angle change output by the control strategy network in the current control cycle to guide the movement of the wheel joint. It represents the angle increment that the wheel joint should produce relative to the previous control cycle and describes the expected rotation amplitude and direction of the wheel joint between adjacent control cycles.

[0246] In its implementation, the control strategy network takes multiple state information as inputs, including the base angular velocity, gravity direction projection, current position and velocity of the leg joints, joint position difference of the wheel-leg joints, current velocity of the wheel-leg joints, and control actions from the previous control cycle. These inputs can be scaled or normalized according to a preset scaling factor to balance the numerical scales of different physical quantities. Based on this, the control strategy network can comprehensively determine the current posture state and dynamic change trend of the bipedal robot and output the corresponding target position difference.

[0247] By dynamically calculating the difference in target position, the wheel joints can achieve smooth transitions and fine adjustments between adjacent control cycles, thereby maintaining the stable posture of the bipedal robot during the stationary process.

[0248] Step S604: Calculate the control torque of the wheel-foot joint based on the current position value of the wheel-foot joint in the current control cycle and the difference between the target position and the current position value.

[0249] Optional, see reference Figure 7 As shown, step S604 may further include steps S701 and S702:

[0250] Step S701: Calculate the target position value of the wheel-foot joint in the current control cycle based on the difference between the target positions of the wheel-foot joint in the current control cycle and the target position value in the previous control cycle.

[0251] During the control execution phase, to ensure that the target position difference output by the control strategy network is consistent with the dimensions of the actual joint angle, and to ensure that the range of motion conforms to the physical limitations of the mechanical structure, the target position difference can be scaled according to a preset position scaling factor. This position scaling factor is used to proportionally adjust the target position difference so that its amplitude matches the actual movement capability of the wheel-foot joint, avoiding situations where the movement is too large or the response is too small.

[0252] It should be noted that the position scaling factor can be determined based on the actual control accuracy requirements and the maximum allowable angle variation range of the wheel joint. In practical implementation, a suitable position scaling factor can be obtained through simulation debugging or experimental calibration. For example, based on the maximum angle variation range of the wheel joint, the output range of the control strategy network can be linearly mapped to the angle range that the joint can execute, ensuring that the action commands output by the network have sufficient response sensitivity without exceeding mechanical safety limits. By reasonably setting the position scaling factor, the stability and executability of the control strategy in actual operation can be improved while maintaining control accuracy.

[0253] Furthermore, the target position value is calculated based on the scaled target position difference and the target position value from the previous control cycle. For example, the scaled target position difference is added to the target position value from the previous control cycle to obtain the target position value of the wheel-leg joint in the current control cycle. This target position value reflects the control strategy network's judgment of the current posture change trend, and is used to achieve continuous and smooth adjustment of the wheel-leg joint, thereby supporting stable posture control of the bipedal robot during stationary maneuvers.

[0254] For example, the target position value of the wheel joint in the current control cycle can be calculated according to formula (4). :

[0255] (4)

[0256] in, This represents the target position value of the wheel joint in the previous control cycle. This represents the difference in target position of the wheel joint during the current control cycle. This is the position scaling factor.

[0257] Step S702: Calculate the control torque of the wheel joint based on the target position value, current position value, and current velocity of the wheel joint in the current control cycle.

[0258] For example, by comparing the target position value of the wheel joint in the current control cycle with the current position value, the position error of the wheel joint is obtained. This position error reflects the deviation between the target angle expected to be output by the control strategy network and the current actual angle.

[0259] Next, based on the current speed of the wheel joint and with zero speed as the target speed, the speed error of the wheel joint is obtained. This speed error is used to describe the dynamic change trend of the wheel joint within the current control cycle, and can reflect the slight slippage of the wheel due to inertia or external force during the stationary process.

[0260] After obtaining the position error and velocity error, proportional-derivative control can be used to calculate the control torque of the wheel joint using the position error and velocity error.

[0261] For example, the control torque of the wheel-foot joint can be calculated according to formula (5). :

[0262] (5)

[0263] in, This refers to the positional error of the wheel joint. This represents the target position value of the wheel joint. This represents the current position value of the wheel joint. For the speed error of the wheel joint, This represents the current velocity of the wheel joint. For the proportional gain of the wheel-foot joint. The differential gain of the wheel joint.

[0264] It should be noted that the parameters in formula (5) are used to describe the variables in the control execution phase, while the parameters in formula (1) correspond to the variables in the control policy network training phase. To distinguish the meaning of parameters with the same name in the two phases, different symbols are used to represent the relevant parameters.

[0265] The calculated control torque is used to drive the wheel joints to perform corresponding actions, so as to achieve posture correction and balance maintenance, and provide the execution basis for the stable control of the bipedal robot during zero-speed dwell.

[0266] Step S605: Send the control torque to the joint actuator of the bipedal robot so that the joint actuator can control the stopping of the wheel joints according to the control torque.

[0267] In addition to calculating the control torque of the wheel-foot joints, the control torque of the leg joints can also be calculated. For example, by combining the current position and velocity values ​​of the leg joints, the control torque of the leg joints can be calculated using proportional-derivative control. The control torque of the leg joints is used to maintain the posture stability of the leg joints and works in conjunction with the control torque of the wheel-foot joints generated by the control strategy network.

[0268] Subsequently, the main controller integrates the control torque of the wheel joint and the control torque of the leg joint to form a joint control command set, which is then sent to the corresponding joint actuators to achieve coordinated control and posture maintenance during the stationary process.

[0269] Specifically, each joint actuator adjusts the output torque of the corresponding joint based on the received control torque, thereby driving the wheel-foot joints and leg joints to perform corresponding compensating actions. During the dwell control process, the joint actuators make subtle posture adjustments to the wheel-foot joints and leg joints based on the control torque output by the main controller, enabling the bipedal robot to maintain balance at zero speed and resist posture deviations caused by external disturbances.

[0270] It should be noted that the control policy network in this example implementation is based on another example implementation, such as... Figure 2 The training method shown is for the dwell control model of a bipedal robot based on deep reinforcement learning.

[0271] This disclosure also provides an exemplary implementation of another method for controlling the stationary position of a bipedal robot. (See reference...) Figure 8 As shown, the method may include the following steps S801 to S805:

[0272] Step S801: Obtain the current position value of the wheel joints of the bipedal robot.

[0273] During actual operation, the current position value of the wheel-foot joint can be acquired in real time through angle encoders, inertial measurement units, or other angle detection sensors installed at each joint. The main controller reads the current joint angle data during operation to obtain the corresponding current position value of the wheel-foot joint.

[0274] Step S802: Calculate the third difference between the current position value and the dwell position value of the wheel joint.

[0275] The dwell position value in this step refers to the ideal wheel-leg joint angle set to achieve body posture stability and center of gravity balance under the zero-speed dwell target. The third difference is used to characterize the degree of deviation between the current actual posture of the wheel-leg joint and the ideal dwell posture.

[0276] By continuously calculating the third difference, the trend of joint posture changes can be grasped in real time, providing real-time status basis for the input of the subsequent control strategy network.

[0277] Step S803: Input the third difference into the pre-trained control strategy network to obtain the target position value of the wheel joint.

[0278] The third difference, along with state information such as base angular velocity, gravity projection, joint velocity, and the action output from the previous control cycle, are input into the pre-trained control strategy network to obtain the target position value of the wheel-foot joint. The target position value is the dynamically generated output of the pre-trained control strategy network after receiving real-time state information such as the third difference, base angular velocity, gravity projection, and joint velocity, representing the desired angle that the wheel-foot joint should adjust to in the current state.

[0279] Step S804: Calculate the control torque of the wheel joint based on the current position value and the target position value of the wheel joint.

[0280] Specifically, the target position value of the wheel joint is compared with the current position value, and the difference between the two is calculated to obtain the position error. Simultaneously, based on the current velocity of the wheel joint and using zero velocity as the target velocity, the velocity error is calculated. Furthermore, proportional-derivative control can be used to calculate the control torque of the wheel joint based on the position error and velocity error.

[0281] Step S805: Send the control torque to the joint actuator of the bipedal robot so that the joint actuator can control the stopping of the wheel joints according to the control torque.

[0282] After obtaining the control torque of the wheel-foot joint, the main controller integrates the control torque of the leg joint and the control torque of the wheel-foot joint to form a joint control command set, which is then sent to the corresponding joint actuators to achieve coordinated control and posture maintenance during the stationary process.

[0283] It should be noted that the control policy network in this example implementation is based on another example implementation, such as... Figure 5 The training method shown is for the dwell control model of a bipedal robot based on deep reinforcement learning.

[0284] In an exemplary embodiment of this disclosure, a training device for a stationary control model of a bipedal robot based on deep reinforcement learning is also provided. (See reference...) Figure 9 As shown, the first training device 900 for a bipedal robot dwell control model based on deep reinforcement learning includes a first position acquisition module 901, a first difference determination module 902, a first input construction module 903, and a first network training module 904, wherein:

[0285] The first position acquisition module 901 is used to acquire the current position value of the wheel joints of the bi-wheeled robot in each control cycle.

[0286] The first difference determination module 902 is used to calculate the joint position difference between the wheel-foot joint in the current control cycle and the previous control cycle.

[0287] The first input construction module 903 is used to construct input data for training the control strategy network by using the joint position difference as an observation under the condition that the control target is zero-speed dwell.

[0288] The first network training module 904 is used to compare the target position difference of the wheel-foot joint output by the control strategy network with the actual position difference, calculate the control torque of the wheel-foot joint, and update the parameters of the control strategy network by combining the interaction results of the control torque of the wheel-foot joint with the simulation environment.

[0289] The specific details of each module in the training device for the dwell control model of the bipedal robot based on deep reinforcement learning have been described in detail in the corresponding training method for the dwell control model of the bipedal robot based on deep reinforcement learning, so they will not be repeated here.

[0290] In an exemplary embodiment of this disclosure, another training device for a bipedal robot stationary control model based on deep reinforcement learning is also provided. (See reference...) Figure 10 As shown, the second type of training device 1000 for a bipedal robot dwell control model based on deep reinforcement learning includes a second position acquisition module 1001, a second difference determination module 1002, a second input construction module 1003, and a second network training module 1004, wherein:

[0291] The second position acquisition module 1001 is used to acquire the current position value of the wheel joints of the bipedal robot;

[0292] The second difference determination module 1002 is used to calculate the second difference between the current position value and the dwell position value of the wheel joint;

[0293] The second input construction module 1003 is used to construct input data for training the control policy network by using the second difference as an observation under the condition that the control target is zero-speed dwell.

[0294] The second network training module 1004 is used to compare the target position value of the wheel joint output by the control strategy network with the current position value, calculate the control torque of the wheel joint, and update the parameters of the control strategy network by combining the interaction results of the control torque of the wheel joint and the simulation environment.

[0295] The specific details of each module in the training device for the dwell control model of the bipedal robot based on deep reinforcement learning have been described in detail in the corresponding training method for the dwell control model of the bipedal robot based on deep reinforcement learning, so they will not be repeated here.

[0296] In an exemplary embodiment of this disclosure, a dwell control device for a bipedal robot is also provided. (See reference...) Figure 11 As shown, the first type of bipedal robot dwell control device 1100 includes a third position acquisition module 1101, a third difference determination module 1102, a relative position determination module 1103, a first torque determination module 1104, and a first dwell control module 1105, wherein:

[0297] The third position acquisition module 1101 is used to acquire the current position value of the wheel joints of the bipedal robot in each control cycle.

[0298] The third difference determination module 1102 is used to calculate the joint position difference of the wheel-foot joint between the current control cycle and the previous control cycle.

[0299] The relative position determination module 1103 is used to input the joint position difference into the pre-trained control strategy network to obtain the target position difference of the wheel joint in the current control cycle.

[0300] The first torque determination module 1104 is used to calculate the control torque of the wheel joint based on the current position value of the wheel joint in the current control cycle and the target position difference.

[0301] The first dwell control module 1105 is used to send control torque to the joint actuator of the bipedal robot so that the joint actuator can control the dwell of the wheel joint according to the control torque;

[0302] The control policy network is based on the embodiments of this disclosure as follows: Figure 2 The training method shown is for the dwell control model of a bipedal robot based on deep reinforcement learning.

[0303] The specific details of each module in the above-mentioned bipedal robot dwell control device have been described in detail in the corresponding bipedal robot dwell control method, so they will not be repeated here.

[0304] In an exemplary embodiment of this disclosure, a dwell control device for a bipedal robot is also provided. (See reference...) Figure 12 As shown, the second type of bipedal robot dwell control device 1200 includes a fourth position acquisition module 1201, a fourth difference determination module 1202, an absolute position determination module 1203, a second torque determination module 1204, and a second dwell control module 1205, wherein:

[0305] The fourth position acquisition module 1201 is used to acquire the current position value of the wheel joints of the bipedal robot;

[0306] The fourth difference determination module 1202 is used to calculate the third difference between the current position value and the dwell position value of the wheel joint;

[0307] The absolute position determination module 1203 is used to input the third difference into the pre-trained control strategy network to obtain the target position value of the wheel joint;

[0308] The second torque determination module 1204 is used to calculate the control torque of the wheel joint based on the current position value and the target position value of the wheel joint in the current control cycle.

[0309] The second dwell control module 1205 is used to send control torque to the joint actuator of the bipedal robot so that the joint actuator can control the dwell of the wheel joint according to the control torque;

[0310] The control policy network is based on the embodiments of this disclosure as follows: Figure 5 The training method shown is for the dwell control model of a bipedal robot based on deep reinforcement learning.

[0311] The specific details of each module in the above-mentioned bipedal robot dwell control device have been described in detail in the corresponding bipedal robot dwell control method, so they will not be repeated here.

[0312] In an exemplary embodiment of this disclosure, a biwheeled robot is also provided. This biwheeled robot includes a processor and a memory. The memory stores computer-readable instructions, which, when executed by the processor, implement the described method. (Reference) Figure 13 The diagram shows a schematic of a bipedal robot, including a wheel joint 1300.

[0313] refer to Figure 14 As shown, an electronic device capable of implementing the above method is also provided. The electronic device 1400 includes a processor 1401 and a memory 1402. The memory 1402 stores computer-readable instructions, which, when executed by the processor 1401, implement the method of this disclosure.

[0314] In an exemplary embodiment of this disclosure, a computer-readable storage medium is also provided, having stored thereon computer program code instructions that, when invoked by a robot's processor, cause the robot to perform the method described in the embodiments.

[0315] refer to Figure 15As shown, a program product 1500 for implementing the above-described method according to an embodiment of the present disclosure is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0316] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0317] Finally, the above preferred embodiments are only used to illustrate the technical solutions of this application and are not restrictive. Although this application has been described in detail, those skilled in the art should understand that changes in form and detail can be made without departing from the scope defined by the claims of this application. The dimensions in the drawings are not related to the specific physical object, and the physical object dimensions can be arbitrarily changed.

Claims

1. A training method for a dwell control model of a bipedal robot based on deep reinforcement learning, characterized in that, The method comprises: In each control cycle, the current position value of the wheel-foot joint of the double-wheel-foot robot is obtained; The joint position difference value of the wheel-foot joint between the current control cycle and the previous control cycle is calculated; Under the condition that the control target is zero-speed stay, the joint position difference value is taken as an observation quantity to construct input data for training a control strategy network; The target position difference value of the wheel-foot joint output by the control strategy network is compared with the actual position difference value, the control torque of the wheel-foot joint is calculated, and the parameters of the control strategy network are updated in combination with the interaction result of the control torque of the wheel-foot joint and the simulation environment.

2. The deep reinforcement learning-based biped robot standing control model training method according to claim 1, wherein, The comparison of the target position difference value of the wheel-foot joint output by the control strategy network with the actual position difference value and the calculation of the control torque of the wheel-foot joint comprise: The target position difference value of the wheel-foot joint output by the control strategy network is compared with the actual position difference value to obtain the position difference error of the wheel-foot joint; According to the current speed of the wheel-foot joint and taking zero speed as the target speed, the speed error of the wheel-foot joint is calculated; Based on the position difference error and the speed error, the control torque of the wheel-foot joint is calculated.

3. The method of claim 2, wherein the method further comprises: determining a reward value based on the state of the biped robot and the action of the biped robot; and updating the policy network based on the reward value. The calculation of the control torque of the wheel-foot joint based on the position difference error and the speed error comprises: The control torque of the wheel-foot joint is calculated based on proportional-differential control using the position difference error and the speed error.

4. The method of claim 3, wherein the method further comprises: The calculation of the control torque of the wheel-foot joint based on proportional-differential control using the position difference error and the speed error comprises: wherein, is a control torque for the wheel-leg joint, is a position difference error, is a velocity error, is a target position difference for the wheel-leg joint, is an actual position difference for the wheel-leg joint, is a current velocity for the wheel-leg joint, is a proportional gain for the wheel-leg joint, is a derivative gain for the wheel-leg joint.

5. The method of claim 1, wherein the method is performed by a processor. The calculation of the joint position difference value of the wheel-foot joint between the current control cycle and the previous control cycle comprises: Noise is added to the current position value of the wheel-foot joint in the current control cycle and the actual position value of the wheel-foot joint in the previous control cycle respectively to obtain the current position value with noise and the actual position value with noise; The joint position difference value is calculated according to the current position value with noise and the actual position value with noise.

6. The method of claim 5, wherein the method further comprises: The same noise is added to the current position value of the wheel-foot joint in the current control cycle and the actual position value of the wheel-foot joint in the previous control cycle.

7. The method of claim 5, wherein the method further comprises: determining a reward value based on the state of the biped robot and the action; and updating the policy network based on the reward value. The method further comprises: The intensity of the noise is adjusted according to a preset noise scaling factor.

8. The deep reinforcement learning-based biped robot standing control model training method according to claim 1, wherein, The method further comprises: The current position value of each leg joint of the double-wheel-foot robot is obtained; The first difference value between the current position value of each leg joint and the stay position value is calculated; Under the condition that the control target is zero-speed stay, the first difference value is taken as an observation quantity to construct input data for training a control strategy network; The target position value of the leg joint output by the control strategy network is compared with the current position value to calculate the control torque of each leg joint.

9. The deep reinforcement learning-based biped robot standing control model training method according to claim 8, characterized in that, The comparison of the target position value of the leg joint output by the control strategy network with the current position value and the calculation of the control torque of each leg joint comprise: The target position value of the leg joint output by the control strategy network is compared with the current position value to obtain the position error of each leg joint; According to the current speed of the leg joint and taking zero speed as the target speed, the speed error of the leg joint is calculated; Based on the position error and the speed error, the control torque of the leg joint is calculated. calculating a velocity error of each leg joint according to a current velocity of the leg joint and taking zero velocity as a target velocity; calculating a control torque of each leg joint based on proportional-differential control and using the position error and the velocity error.

10. The deep reinforcement learning-based biped robot standing control model training method according to claim 9, wherein, The calculating a control torque of each leg joint based on proportional-differential control and using the position error and the velocity error comprises: wherein, is a control torque for a leg joint, is a position error for a leg joint, is a target position value for a leg joint, is a current position value for a leg joint, is a velocity error for a leg joint, is a current velocity for a leg joint, is a proportional gain for a leg joint, is a differential gain for a leg joint.

11. The deep reinforcement learning-based biped robot standing control model training method according to claim 8, characterized in that, After the control torque of the wheel-foot joint is calculated, the method further comprises: combining the control torque of the wheel-foot joint and the control torque of each leg joint to obtain a joint torque; updating parameters of the control policy network according to an interaction result of the joint torque and the simulation environment.

12. The deep reinforcement learning-based biped robot standing control model training method according to claim 11, characterized in that, The method further comprises: performing constraint processing and scaling processing on the joint torque to update the parameters of the control policy network according to an interaction result of the processed joint torque and the simulation environment.

13. The deep reinforcement learning-based biped robot standing control model training method according to claim 11, wherein, The interaction result comprises a state and a reward; The updating the parameters of the control policy network according to the interaction result of the joint torque and the simulation environment comprises: constructing a loss function based on the state and the reward of the joint torque obtained in the interaction with the simulation environment; updating the parameters of the control policy network based on the loss function.

14. The deep reinforcement learning-based biped robot standing control model training method according to claim 13, characterized in that, The loss function further comprises a gradient penalty term; The loss function is: wherein is a loss function, is a parameter of the control policy network, is a reinforcement learning loss term constructed from state s , action a and reward r , is a gradient penalty coefficient, is the action output of the control policy network in state s , is the gradient of the output of the control policy network with respect to the parameter , is the square of the gradient norm, i.e. the regularization term.

15. The method of claim 1-10, wherein, The constructing input data for training the control policy network comprises: obtaining a base angular velocity and a gravity projection of the biped robot in a current control period; obtaining joint action data of the biped robot in a previous control period output by the control policy network, wherein the joint action data comprises a target position value of each leg joint and a target position difference value of the wheel-foot joint; constructing the input data according to the base angular velocity, the gravity projection, the joint action data, a current velocity of each leg joint, a current velocity of the wheel-foot joint, and a calculated joint position value of each leg joint and a joint position difference value of the wheel-foot joint.

16. A method for training a deep reinforcement learning-based biorobot stopping control model, characterized in that, comprises: obtaining a current position value of a wheel-foot joint of the biped robot; calculating a second difference value of the joint position between the current position value and a resting position value of the wheel-foot joint; under the condition that the control target is zero-speed resting, taking the second difference value as an observation quantity to construct input data for training the control policy network; comparing the target position value of the wheel-foot joint output by the control policy network with the current position value, calculating a control torque of the wheel-foot joint, and updating the parameters of the control policy network in combination with an interaction result of the control torque and the simulation environment.

17. A bioradial robot stop control method characterized by comprising: comprises: in each control period, obtaining a current position value of a wheel-foot joint of the biped robot; calculating a joint position difference value of the wheel-foot joint between a current control period and a previous control period; inputting the joint position difference value into a pre-trained control policy network to obtain a target position difference value of the wheel-foot joint in the current control period; calculating a control torque of the wheel-legged joint based on the current position value of the wheel-legged joint in the current control period and the target position difference value; sending the control torque to a joint driver of the dual-wheel-legged robot to enable the joint driver to perform the stay control on the wheel-legged joint according to the control torque; The control strategy network is obtained according to the dual-wheel-legged robot stay control model training method based on deep reinforcement learning in any one of claims 1-15.

18. The bioradial robot stopping control method according to claim 17, characterized by, The control torque of the wheel-legged joint is calculated based on the current position value of the wheel-legged joint in the current control period and the target position difference value. The target position value of the wheel-legged joint in the current control period is calculated according to the target position difference value of the wheel-legged joint in the current control period and the target position value in the previous control period. The control torque of the wheel-legged joint is calculated based on the target position value, the current position value and the current speed of the wheel-legged joint in the current control period.

19. The bioradial robot stopping control method according to claim 18, characterized by, The target position value of the wheel-legged joint in the current control period is calculated according to the target position difference value and the target position value in the previous control period. The target position difference value is scaled according to a preset position scaling factor to obtain a scaled target position difference value. The target position value is calculated according to the scaled target position difference value and the target position value in the previous control period.

20. The bioradial robot stay control method according to claim 18, characterized by, The control torque of the wheel-legged joint is calculated based on the target position value, the current position value and the current speed of the wheel-legged joint in the current control period. The position error of the wheel-legged joint is obtained by comparing the target position value of the wheel-legged joint in the current control period with the current position value. The current speed of the wheel-legged joint is obtained, and the zero speed is taken as the target speed to obtain the speed error of the wheel-legged joint. The control torque of the wheel-legged joint is calculated based on proportional-differential control and the position error and the speed error.

21. A bioradial robot stop control method characterized by comprising: It includes: obtaining the current position value of the wheel-legged joint of the dual-wheel-legged robot; calculating a third difference value between the current position value of the wheel-legged joint and a stay position value; inputting the third difference value into a pre-trained control strategy network to obtain a target position value of the wheel-legged joint; calculating a control torque of the wheel-legged joint based on the current position value of the wheel-legged joint and the target position value; sending the control torque to a joint driver of the dual-wheel-legged robot to enable the joint driver to perform the stay control on the wheel-legged joint according to the control torque; The control strategy network is obtained according to the dual-wheel-legged robot stay control model training method based on deep reinforcement learning in any one of claims 1-15.

22. A deep reinforcement learning based biped robot standing control model training device, characterized by, It includes: The first position acquisition module is configured to obtain the current position value of the wheel-legged joint of the dual-wheel-legged robot in each control period. The first difference value determination module is configured to calculate the joint position difference value of the wheel-legged joint between the current control period and the previous control period. The first input construction module is configured to, under the condition that the control target is zero-speed stay, construct input data for training the control policy network by taking the joint position difference as an observation value. The first network training module is configured to compare the target position difference of the wheel-foot joint output by the control policy network with the actual position difference, calculate the control torque of the wheel-foot joint, and update the parameters of the control policy network in combination with the interaction result of the control torque of the wheel-foot joint and the simulation environment.

23. A deep reinforcement learning based biped robot standing control model training device, characterized by, The method comprises the following steps: The second position acquisition module is configured to acquire the current position value of the wheel-foot joint of the double-wheel-foot robot. The second difference determination module is configured to calculate the second difference of the joint position between the current position value and the stay position value of the wheel-foot joint. The second input construction module is configured to, under the condition that the control target is zero-speed stay, construct input data for training the control policy network by taking the second difference as an observation value. The second network training module is configured to compare the target position value of the wheel-foot joint output by the control policy network with the current position value, calculate the control torque of the wheel-foot joint, and update the parameters of the control policy network in combination with the interaction result of the control torque of the wheel-foot joint and the simulation environment.

24. A biorobot stopping control device, characterized by comprising: The method comprises the following steps: The third position acquisition module is configured to acquire the current position value of the wheel-foot joint of the double-wheel-foot robot in each control period. The third difference determination module is configured to calculate the joint position difference of the wheel-foot joint between the current control period and the previous control period. The relative position determination module is configured to input the joint position difference into the pre-trained control policy network to obtain the target position difference of the wheel-foot joint in the current control period. The first torque determination module is configured to calculate the control torque of the wheel-foot joint based on the current position value of the wheel-foot joint in the current control period and the target position difference. The first stay control module is configured to send the control torque to the joint drive of the double-wheel-foot robot, so that the joint drive controls the wheel-foot joint according to the control torque. The control policy network is obtained according to the double-wheel-foot robot stay control model training method based on deep reinforcement learning in any one of claims 1-15.

25. A biorobot stopping control device, characterized by comprising: The method comprises the following steps: The fourth position acquisition module is configured to acquire the current position value of the wheel-foot joint of the double-wheel-foot robot. The fourth difference determination module is configured to calculate the third difference between the current position value and the stay position value of the wheel-foot joint. The absolute position determination module is configured to input the third difference into the pre-trained control policy network to obtain the target position value of the wheel-foot joint. The second torque determination module is configured to calculate the control torque of the wheel-foot joint based on the current position value of the wheel-foot joint and the target position value. The second stay control module is configured to send the control torque to the joint drive of the double-wheel-foot robot, so that the joint drive controls the wheel-foot joint according to the control torque. The control strategy network is obtained according to the deep reinforcement learning-based biped robot staying control model training method in claim 16.

26. An electronic device, comprising: Comprise: a processor; and a memory, the memory having stored thereon computer readable instructions which, when executed by the processor, implement the method of any one of claims 1-21.

27. A dual-wheel foot robot, characterized by Comprise: a processor; and a memory, the memory having stored thereon computer readable instructions which, when executed by the processor, implement the method of any one of claims 1-21.

28. A computer-readable storage medium, characterized in that, The computer readable storage medium has stored thereon computer program code instructions which, when invoked by a processor of a biped robot, cause the robot to perform the method of any one of claims 1-21.

Images