Double-wheel-foot robot stay control model training method and device based on deep reinforcement learning

Abstract

The invention provides a double-wheel-foot robot staying control model training method and device based on deep reinforcement learning, and relates to the technical field of sensors and robots. The method comprises the following steps: acquiring a current position value of a wheel-foot joint of the double-wheel-foot robot in each control period; calculating a joint position difference value of the wheel foot joint between the current control period and the previous control period; under the condition that the control target is zero-speed staying, the wheel-foot joint position difference value serves as an observed quantity, and input data used for training a control strategy network is constructed; and comparing the target position difference value of the wheel-foot joint output by the control strategy network with the actual position difference value, calculating the control torque of the wheel-foot joint, and updating the parameters of the control strategy network in combination with the interaction result of the control torque and the simulation environment. According to the invention, wheel slipping accumulative errors can be effectively avoided, and stable control of zero-speed stay of the double-wheel-foot robot is realized.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of sensors and robotics, and relates to a double-wheel-foot robot stay control model training method and device based on deep reinforcement learning. BACKGROUND

[0002] With the development of robotics, double-wheel-foot robots gradually become an important research direction in the related field due to the advantages of both wheeled and foot structures. Specifically, double-wheel-foot robots can achieve high-speed and low-energy translational motion through wheels, and can also complete complex terrain adaptation and posture stability control through leg joints. However, due to the characteristics of continuous rotation joints and multi-degree-of-freedom joints in the wheel-foot structure, the control of the double-wheel-foot robot in the stay state is extremely complex.

[0003] In related technologies, the double-wheel-foot robot control system can be based on reinforcement learning (RL) and use two modes of speed control or absolute position control to develop policy network training, and use the trained policy network to adjust the wheel motion in the running stage. In the speed control mode, the speed returned by the encoder of the double-wheel-foot robot wheel is usually calculated by the difference of position, which is more susceptible to noise interference than position feedback. In order to avoid noise amplification, the control parameters used for speed adjustment are generally set small, resulting in insufficient wheel driving force, and the double-wheel-foot robot is prone to continuous forward or backward drift under external disturbance. In contrast, the absolute position control mode controls the motion of the double-wheel-foot robot by setting absolute position coordinates, which can reduce the forward or backward drift during the stay process to some extent. However, due to the slippage phenomenon often accompanied by the wheel structure in actual motion, the cumulative error is easily caused with the increase of running time, so that the stay position output by the policy network still has a large deviation from the target position, increasing the risk of control failure.

[0004] Therefore, it is urgent to provide a new double-wheel-foot robot control scheme to realize stable control of the double-wheel-foot robot zero-speed stay.

[0005] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0006] The present disclosure provides a double-wheel-foot robot stay control model training method and device based on deep reinforcement learning to overcome the problem in related technologies that the cumulative error caused by wheel slippage makes the stay position output by the policy network still have a large deviation from the target position, thereby affecting the stability of the double-wheel-foot robot zero-speed stay.

[0007] Additional aspects and advantages of the present disclosure will be set forth in part in the description that follows, and in part will be apparent from the description, or can be learned by practice of the present disclosure.

[0008] According to a first aspect of the present disclosure, a deep reinforcement learning-based double-wheel-foot robot stay control model training method is provided, comprising: In each control cycle, the current position value of the wheel-foot joint of the double-wheel-foot robot is obtained; The joint position difference value of the wheel-foot joint between the current control cycle and the previous control cycle is calculated; Under the condition that the control target is zero-speed stay, the joint position difference value is taken as an observation quantity, and input data for training the control policy network is constructed; The target position difference value and the actual position difference value of the wheel-foot joint output by the control policy network are compared, the control torque of the wheel-foot joint is calculated, and the parameters of the control policy network are updated in combination with the interaction result of the control torque of the wheel-foot joint and the simulation environment.

[0009] In an exemplary embodiment of the present disclosure, comparing the target position difference value and the actual position difference value of the wheel-foot joint output by the control policy network to calculate the control torque of the wheel-foot joint comprises: The position difference error of the wheel-foot joint is obtained by comparing the target position difference value and the actual position difference value of the wheel-foot joint output by the control policy network; According to the current speed of the wheel-foot joint and taking zero speed as the target speed, the speed error of the wheel-foot joint is calculated; Based on the position difference error and the speed error, the control torque of the wheel-foot joint is calculated.

[0010] In an exemplary embodiment of the present disclosure, based on the position difference error and the speed error, the control torque of the wheel-foot joint is calculated, comprising: Based on proportional-differential control, the control torque of the wheel-foot joint is calculated using the position difference error and the speed error.

[0011] In an exemplary embodiment of the present disclosure, based on proportional-differential control, the control torque of the wheel-foot joint is calculated using the position difference error and the speed error, comprising: wherein, is the control torque of the wheel-foot joint, is the position difference error, is the speed error, is the target position difference value of the wheel-foot joint, is the actual position difference value of the wheel-foot joint, is the current speed of the wheel-foot joint, a proportional gain for the wheel-foot joint, a differential gain for the wheel-foot joint.

[0012] In an example embodiment of the present disclosure, the joint position difference value of the wheel-foot joint between the current control period and the last control period is calculated, including: adding noise to the current position value of the wheel-foot joint in the current control period and the actual position value in the last control period respectively to obtain a noisy current position value and a noisy actual position value; calculating the joint position difference value according to the noisy current position value and the noisy actual position value.

[0013] In an example embodiment of the present disclosure, the same noise is added to the current position value of the wheel-foot joint in the current control period and the actual position value in the last control period.

[0014] In an example embodiment of the present disclosure, the method further includes: adjusting the intensity of the noise according to a preset noise scaling factor.

[0015] In an example embodiment of the present disclosure, the method further includes: obtaining a current position value of each leg joint of the double-wheel-foot robot; calculating a first difference value between the current position value and a resting position value of each leg joint; under the condition that the control target is zero-speed resting, taking the first difference value as an observation quantity, and constructing input data for training the control policy network; comparing the target position value of the leg joint output by the control policy network with the current position value, and calculating the control torque of each leg joint.

[0016] In an example embodiment of the present disclosure, comparing the target position value of the leg joint output by the control policy network with the current position value to calculate the control torque of each leg joint includes: comparing the target position value of the leg joint output by the control policy network with the current position value to obtain a position error of each leg joint; calculating a speed error of each leg joint according to the current speed of each leg joint and taking zero speed as a target speed; based on proportional-differential control, using the position error and the speed error to calculate the control torque of each leg joint.

[0017] In an example embodiment of the present disclosure, based on proportional-differential control, using the position error and the speed error to calculate the control torque of each leg joint includes: wherein, a control torque for a leg joint, a position error for a leg joint, a target position value for a leg joint, a current position value for a leg joint, a velocity error for a leg joint, a current velocity for a leg joint, a proportional gain for a leg joint, a differential gain for a leg joint.

[0018] In an example embodiment of the present disclosure, after the control torque of the wheel-foot joint is calculated, the method further comprises: combining the control torque of the wheel-foot joint and the control torque of each leg joint to obtain a joint torque; updating the parameters of the control policy network according to an interaction result of the joint torque and the simulation environment.

[0019] In an example embodiment of the present disclosure, updating the parameters of the control policy network according to the combined control torque further comprises: performing constraint processing and scaling processing on the joint torque, so as to update the parameters of the control policy network according to an interaction result of the processed joint torque and the simulation environment.

[0020] In an example embodiment of the present disclosure, the interaction result comprises a state and a reward; updating the parameters of the control policy network according to an interaction result of the joint torque and the simulation environment comprises: constructing a loss function based on the state and the reward obtained in the interaction of the joint torque in the simulation environment; updating the parameters of the control policy network based on the loss function.

[0021] In an example embodiment of the present disclosure, the loss function further comprises a gradient penalty term; the loss function is: wherein, is a loss function, is a parameter of the control policy network, is a reinforcement learning loss term, constructed by a state s , an action a and a reward r , is a gradient penalty coefficient, is an action output of the control policy network in the state s , is a gradient of the output of the control policy network with respect to the parameter , is the square of the gradient norm, i.e., the regularization term.

[0022] In an exemplary embodiment of the present disclosure, input data for training a control policy network is constructed, comprising: obtaining base angular velocity and gravity projection of the double-wheel-legged robot in a current control period; obtaining joint action data of the double-wheel-legged robot in a previous control period output by the control policy network; wherein the joint action data comprises target position values of each leg joint and target position difference values of the wheel-legged joint; According to the base angular velocity, the gravity projection, the joint action data, the obtained current speed of each leg joint and the current speed of the wheel-legged joint, and the calculated joint position values of each leg joint and the joint position difference values of the wheel-legged joint, the input data is constructed.

[0023] According to a second aspect of the present disclosure, a double-wheel-legged robot stay control model training method based on deep reinforcement learning is provided, comprising: obtaining current position values of the wheel-legged joint of the double-wheel-legged robot; calculating a second difference value of the joint position between the current position values and the stay position values of the wheel-legged joint; under the condition that the control target is zero-speed stay, taking the second difference value as an observation, constructing input data for training a control policy network; comparing the target position values of the wheel-legged joint output by the control policy network with the current position values, calculating the control torque of the wheel-legged joint, and updating the parameters of the control policy network in combination with the interaction results of the control torque of the wheel-legged joint and the simulation environment.

[0024] According to a third aspect of the present disclosure, a double-wheel-legged robot stay control method is provided, comprising: in each control period, obtaining current position values of the wheel-legged joint of the double-wheel-legged robot; calculating a joint position difference value of the wheel-legged joint between the current control period and the previous control period; inputting the joint position difference value into the pre-trained control policy network to obtain target position difference values of the wheel-legged joint in the current control period; based on the current position values and the target position difference values of the wheel-legged joint in the current control period, calculating the control torque of the wheel-legged joint; sending the control torque to the joint driver of the double-wheel-legged robot to enable the joint driver to perform stay control on the wheel-legged joint according to the control torque; wherein the control policy network is obtained by the double-wheel-legged robot stay control model training method based on deep reinforcement learning in the first aspect of the present disclosure.

[0025] In an example embodiment of the present disclosure, the control torque of the leg joint is calculated based on the current position value of the leg joint in the current control period and the target position difference value, including: The target position value of the leg joint in the current control period is calculated according to the target position difference value of the leg joint in the current control period and the target position value in the previous control period. The control torque of the leg joint is calculated based on the target position value, the current position value and the current speed of the leg joint in the current control period.

[0026] In an example embodiment of the present disclosure, the target position value of the leg joint in the current control period is calculated according to the target position difference value of the leg joint in the current control period and the target position value in the previous control period, including: The target position difference value is scaled according to a preset position scaling factor to obtain a scaled target position difference value. The target position value is calculated according to the scaled target position difference value and the target position value in the previous control period.

[0027] In an example embodiment of the present disclosure, the control torque of the leg joint is calculated based on the target position value, the current position value and the current speed of the leg joint in the current control period, including: The position error of the leg joint is obtained by comparing the target position value with the current position value of the leg joint in the current control period. The current speed of the leg joint is obtained, and the zero speed is taken as the target speed to obtain the speed error of the leg joint. The control torque of the leg joint is calculated using the position error and the speed error based on proportional-differential control.

[0028] According to a fourth aspect of the present disclosure, a double-wheel-legged robot staying control method is provided, including: The current position value of the leg joint of the double-wheel-legged robot is obtained. The difference value of the joint position between the current position value and the staying position value of the leg joint is calculated. The difference value of the joint position is input into a pre-trained control strategy network to obtain the target position value of the leg joint. The control torque of the leg joint is calculated based on the current position value and the target position value of the leg joint. The control torque is sent to the joint driver of the double-wheel-legged robot to enable the joint driver to control the leg joint according to the control torque. The control strategy network is obtained according to the double-wheel-legged robot staying control model training method based on deep reinforcement learning in the second aspect of the present disclosure.

[0029] According to a fifth aspect of the present disclosure, a deep reinforcement learning-based biped robot stay control model training apparatus is provided, comprising: a first position acquisition module configured to acquire a current position value of a wheel-foot joint of the biped robot in each control cycle; a first difference determination module configured to calculate a joint position difference between the wheel-foot joint in a current control cycle and a previous control cycle; a first input construction module configured to, under the condition that the control target is zero-speed stay, construct input data for training a control policy network by taking the joint position difference as an observation; a first network training module configured to compare a target position difference of the wheel-foot joint output by the control policy network with an actual position difference, calculate a control torque of the wheel-foot joint, and update parameters of the control policy network in combination with an interaction result of the control torque of the wheel-foot joint and a simulation environment.

[0030] According to a sixth aspect of the present disclosure, a deep reinforcement learning-based biped robot stay control model training apparatus is provided, comprising: a second position acquisition module configured to acquire a current position value of a wheel-foot joint of the biped robot; a second difference determination module configured to calculate a second difference of a joint position between the current position value of the wheel-foot joint and a stay position value; a second input construction module configured to, under the condition that the control target is zero-speed stay, construct input data for training a control policy network by taking the second difference of the joint position as an observation; a second network training module configured to compare a target position value of the wheel-foot joint output by the control policy network with the current position value, calculate a control torque of the wheel-foot joint, and update parameters of the control policy network in combination with an interaction result of the control torque of the wheel-foot joint and a simulation environment.

[0031] According to a seventh aspect of the present disclosure, a biped robot stay control apparatus is provided, comprising: a third position acquisition module configured to acquire a current position value of a wheel-foot joint of the biped robot in each control cycle; a third difference determination module configured to calculate a joint position difference between the wheel-foot joint in a current control cycle and a previous control cycle; a relative position determination module configured to input the joint position difference to a pre-trained control policy network to obtain a target position difference of the wheel-foot joint in the current control cycle; a first torque determination module configured to calculate a control torque of the wheel-foot joint based on the current position value and the target position difference of the wheel-foot joint in the current control cycle; The first stop control module is configured to send a control torque to a joint driver of the biped robot, so that the joint driver controls the biped joint to stop according to the control torque. The control strategy network is obtained according to the biped robot stop control model training method based on deep reinforcement learning of the first aspect of the present disclosure.

[0032] According to the eighth aspect of the present disclosure, a biped robot stop control device is provided, comprising: The fourth position acquisition module is configured to acquire a current position value of a biped joint of the biped robot. The fourth difference determination module is configured to calculate a third difference between the current position value of the biped joint and the stop position value. The absolute position determination module is configured to input the third difference into the pre-trained control strategy network to obtain a target position value of the biped joint. The second torque determination module is configured to calculate a control torque of the biped joint based on the current position value and the target position value of the biped joint. The second stop control module is configured to send the control torque to the joint driver of the biped robot, so that the joint driver controls the biped joint to stop according to the control torque. The control strategy network is obtained according to the biped robot stop control model training method based on deep reinforcement learning of the second aspect of the present disclosure.

[0033] According to the ninth aspect of the present disclosure, an electronic device is provided, comprising: A processor; and The memory stores computer readable instructions, and the computer readable instructions are executed by the processor to implement the method of the above-mentioned embodiment.

[0034] According to the tenth aspect of the present disclosure, a biped robot is provided, comprising: A processor; and The memory stores computer readable instructions, and the computer readable instructions are executed by the processor to implement the method of the above-mentioned embodiment.

[0035] According to the eleventh aspect of the present disclosure, a computer readable storage medium is provided, and the computer readable storage medium stores computer program code instructions, when the computer program code instructions are called by the processor of the biped robot, the robot executes the method of the above-mentioned embodiment.

[0036] According to the above technical solution, the present disclosure has at least one of the following advantages and positive effects: The present disclosure adopts a relative position difference value, i.e., a joint position difference value, as a control quantity in the training process, so that the control strategy can be adjusted according to the actual change of the wheel-foot joint in the adjacent control period, more accurately reflects the motion state in the staying process, and thus avoids the accumulated error caused by wheel slipping in absolute position control. Therefore, the method can reduce the deviation of staying positioning caused by error accumulation, thereby improving the stability of the double-wheel-foot robot in complex terrain and long-time running conditions to maintain zero-speed staying. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor.

[0038] Figure 1 A system architecture diagram of the double-wheel-foot robot staying control model training method and the double-wheel-foot robot staying control method based on deep reinforcement learning in the embodiments of the present disclosure is shown.

[0039] Figure 2 A flowchart of a double-wheel-foot robot staying control model training method based on deep reinforcement learning in the embodiments of the present disclosure is shown.

[0040] Figure 3 A flowchart of calculating the control torque of the wheel-foot joint in the embodiments of the present disclosure is shown.

[0041] Figure 4 A flowchart of calculating the control torque of each leg joint in the embodiments of the present disclosure is shown.

[0042] Figure 5 A flowchart of another double-wheel-foot robot staying control model training method based on deep reinforcement learning in the embodiments of the present disclosure is shown.

[0043] Figure 6 A flowchart of a double-wheel-foot robot staying control method in the embodiments of the present disclosure is shown.

[0044] Figure 7 A flowchart of another calculating the control torque of the wheel-foot joint in the embodiments of the present disclosure is shown.

[0045] Figure 8 A flowchart of another double-wheel-foot robot staying control method in the embodiments of the present disclosure is shown.

[0046] Figure 9A block diagram of a deep reinforcement learning-based biped robot staying control model training device in an embodiment of the present disclosure is shown.

[0047] Figure 10 A block diagram of another deep reinforcement learning-based biped robot staying control model training device in an embodiment of the present disclosure is shown.

[0048] Figure 11 A block diagram of a biped robot staying control device in an embodiment of the present disclosure is shown.

[0049] Figure 12 A block diagram of another biped robot staying control device in an embodiment of the present disclosure is shown.

[0050] Figure 13 A schematic diagram of a biped robot in an embodiment of the present disclosure is shown.

[0051] Figure 14 A structural schematic diagram of a computer system of an electronic device suitable for implementing an embodiment of the present disclosure is shown.

[0052] Figure 15 A schematic diagram of a computer-readable storage medium in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0053] In the present disclosure description, the terms "first" and "second" are used only for description, and do not indicate relative importance or imply the number of technical features. Therefore, the "first" and "second" features can explicitly or implicitly include at least one of the features. The meaning of "a plurality of" is at least two, unless otherwise explicitly limited.

[0054] First, the related terms involved in the example embodiments of the present disclosure are explained and described: Biped joint position: refers to the joint angle between the wheel and the leg link in the biped robot, used to represent the posture state of the biped joint at the current time.

[0055] Zero-speed staying: in the control process of the biped robot, the overall linear speed and angular speed tend to zero, and the stable state is reached. In this state, the biped robot can maintain balance near the target staying position and does not move or fall significantly.

[0056] Policy network: a control model trained under the reinforcement learning framework, which is used by the present disclosure to map the state information collected by the biped robot in each control period to the control output. The policy network is usually implemented by a deep neural network, which continuously updates the model parameters based on the control objective of zero-speed staying by interacting with the simulation environment, so as to realize the stable staying control of the biped robot.

[0057] Control torque: The torque signal output by the main controller of the dual-wheel robot, acting on the joint driver, used to adjust the rotation of the wheel-foot joint, and to realize the dwell control of the dual-wheel robot.

[0058] Proportional-derivative control (PD control): A commonly used closed-loop control method, which generates a control signal by calculating the proportional term of the current error and the derivative term of the error change rate. The proportional term is used to quickly respond to the deviation, and the derivative term is used to suppress dynamic oscillation, so as to achieve smooth and stable control effect.

[0059] Control period: Refers to the time interval for completing state acquisition and control output in a complete control cycle. Among them, the control period of the training stage corresponds to the time step of the training environment, which is used to describe the time interval for completing joint position acquisition, joint position difference calculation, strategy network output and parameter updating in the training process; The control period of the deployment stage corresponds to the refresh period of the main controller, which refers to the time interval for the main controller to collect joint data, calculate control torque and output to the joint driver. The control periods of the two stages are consistent in time scale to ensure the dynamic response consistency of the model in the training and actual running process.

[0060] Figure 1 The system architecture diagram of the dual-wheel robot dwell control model training method and the dual-wheel robot dwell control method based on deep reinforcement learning in the embodiments of the present disclosure is shown. As shown in Figure 1 The system architecture 100 can include a terminal device 101, a dual-wheel robot 102, a network 103 and a server 104.

[0061] The terminal device 101 includes but is not limited to desktop computers, portable computers, smartphones and tablet computers, etc. The terminal device 101 is configured with a graphical user interface, which can provide visualization function to display the real-time attitude information of the dual-wheel robot 102, the wheel-foot joint motion state and the control strategy output result in the training process. In addition, the terminal device 101 can also provide parameter configuration function, allowing users to set the training parameters of the control strategy network, the control period and the zero-speed dwell threshold, etc., and cooperate with the server 104 to complete the visualization analysis and model management of the training data.

[0062] The bipedal robot 102 is equipped with various sensors, such as inertial measurement units and joint angle encoders, to collect real-time operational status information, including the current position values ​​of the wheel and leg joints. It is important to note that the bipedal robot 102 includes at least a main controller and joint actuators. For example, the main controller can be used to run a deployed control strategy network. In each control cycle, it receives sensor sampling data, calculates the joint position difference between the current and previous control cycles, and uses this joint position difference as an observation input to the control strategy network to predict the target position difference of the wheel and leg joints. Then, based on the deviation between the predicted target position difference and the actual position difference, it calculates the control torque of the wheel and leg joints and sends the control torque to the joint actuators. The joint actuators drive the wheel and leg joints to perform corresponding actions based on the control torque to achieve dwell control.

[0063] Server 104 is equipped with a training module for a control strategy network. This module receives sensor data, such as joint position values, uploaded by the bipedal robot 102. After preprocessing the joint position values, it trains the control strategy network based on a deep reinforcement learning framework. During training, server 104 calculates the control torque based on the zero-speed stop control objective and updates the parameters of the control strategy network by combining the interaction results of the control torque and the simulation environment. After training, server 104 can distribute the trained control strategy network to the bipedal robot 102 for use in executing stop control during operation.

[0064] Network 103 serves as the medium for providing a communication link between terminal device 101, bipedal robot 102, and server 104. Network 103 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc. It should be understood that... Figure 1 The number and type of terminal devices, bipedal robots, networks, and servers shown are merely illustrative. Depending on implementation needs, there can be any number and type of terminal devices, bipedal robots, networks, and servers; for example, the servers could be cloud servers.

[0065] Through the coordinated operation of various parts in the system architecture 100, a complete process can be realized, from wheel joint position acquisition, joint position difference calculation, control strategy network training to actual dwell control execution, thereby improving the control accuracy and stability of the bipedal robot in zero-speed dwell tasks.

[0066] This disclosure provides an example implementation of a method for training a dwell control model for a bipedal robot based on deep reinforcement learning. (See reference...) Figure 2 As shown, the method may include the following steps S201 to S204: Step S201: In each control cycle, obtain the current position value of the wheel joints of the bipedal robot; Step S202, calculate the joint position difference value of the wheel-foot joint between the current control period and the last control period; Step S203, under the condition that the control target is zero-speed stay, take the joint position difference value as an observation, and construct input data for training the control strategy network; Step S204, compare the target position difference value of the wheel-foot joint output by the control strategy network with the actual position difference value, calculate the control torque of the wheel-foot joint, and update the parameters of the control strategy network in combination with the interaction result of the wheel-foot joint control torque and the simulation environment.

[0067] The training method of the double-wheel-foot robot stay control model based on deep reinforcement learning provided by the example embodiment of the present disclosure uses the relative position difference value, i.e., the joint position difference value, as the control quantity in the training process, so that the control strategy can be adjusted according to the actual change of the wheel-foot joint in the adjacent control period, more accurately reflect the motion state in the stay process, and thus avoid the cumulative error caused by wheel slipping in absolute position control. Therefore, the method can reduce the stay positioning deviation caused by error accumulation, thereby improving the stability of the double-wheel-foot robot in maintaining zero-speed stay under complex terrain and long-time running conditions.

[0068] In the following, the training method of the double-wheel-foot robot stay control model based on deep reinforcement learning in the example embodiment will be described in detail.

[0069] In step S201, in each control period, the current position value of the wheel-foot joint of the double-wheel-foot robot is obtained.

[0070] In the training environment, the current position value of the wheel-foot joint of the double-wheel-foot robot can be measured in real time by the angle encoder, inertial measurement unit or other angle detection sensor installed at each joint. For example, at the beginning of each control period, the current joint angle data is collected by the sensor to obtain the current position value of the corresponding wheel-foot joint. The current position value of the wheel-foot joint can reflect the relative motion relationship between the wheel and the leg linkage, and is an important basic data for calculating the joint position difference value.

[0071] By obtaining the current position value of the wheel-foot joint in each control period in real time, the change of the wheel-foot joint position can be determined, providing basic data support for subsequent calculation of the joint position difference value between adjacent control periods.

[0072] The control period can be flexibly set according to system control requirements, for example, in the range of 10 milliseconds to 20 milliseconds, to ensure that the sensor sampling has sufficient time resolution, thereby improving the accuracy and response performance of subsequent difference calculation and control output.

[0073] In step S202, a joint position difference value of the wheel-foot joint between the current control period and the previous control period is calculated.

[0074] Specifically, the joint position value of the wheel-foot joint in the current control period is compared with the joint position value of the wheel-foot joint in the previous control period to calculate the difference value therebetween. The joint position difference value is used to represent the relative motion change of the wheel-foot joint in adjacent control periods and can reflect the posture adjustment trend of the biped robot during the stay process and the dynamic response of the wheel-foot structure.

[0075] By calculating the joint position difference value between adjacent control periods, the slight displacement change of the wheel-foot joint can be effectively captured to provide input information reflecting the instantaneous posture change for the subsequent control strategy network, thereby realizing dynamic adjustment of the stay process.

[0076] In some example embodiments, to enhance the stability and robustness of the control strategy network under different operating conditions, a disturbance processing can be introduced to the collected wheel-foot joint position data before calculating the joint position difference value.

[0077] Specifically, noise can be added to the current position value of the wheel-foot joint in the current control period and the actual position value of the wheel-foot joint in the previous control period, respectively, to obtain a noisy current position value and a noisy actual position value. The noise can be random noise or a disturbance signal generated based on a preset distribution (such as uniform distribution, normal distribution, etc.), which is used to simulate sensor measurement error, mechanical jitter or external interference, etc., so as to introduce certain uncertainty in the training process.

[0078] Subsequently, the joint position difference value is calculated according to the noisy current position value and the noisy actual position value. In this way, the training data can be closer to the actual operating conditions, and the control strategy network can still maintain stable response under the condition of measurement noise or external interference, thereby improving the generalization performance and control stability of the control strategy network in complex terrain and long-term operation scenarios.

[0079] It should be noted that in this example embodiment, the same noise is added to the current position value of the wheel-foot joint in the current control period and the actual position value of the wheel-foot joint in the previous control period. By introducing the same noise into the joint position values of the two control periods, it can be ensured that the noise terms cancel each other out when calculating the difference value, thereby avoiding fluctuations in the joint position difference value caused by differences in random disturbances and maintaining the stability and consistency of the joint position difference value calculation.

[0080] In addition, in order to simulate different degrees of measurement errors or external disturbances, the intensity of the noise can also be adjusted according to a preset noise scaling factor. The noise scaling factor can be dynamically set according to the task requirements or environmental complexity in the training stage, and is used to control the intensity range of the noise signal, so that the control policy network can gradually adapt to different training conditions from low noise to high noise, thereby improving the robustness and convergence stability of the control policy network in complex running scenarios.

[0081] Taking random noise as an example, a random noise signal with a value range of [-1, 1] can be generated in each control period, and the dimension is the same as the position data of the leg joint. Before calculating the joint position difference, the same random noise term is added to the current position value of the leg joint in the current control period and the actual position value in the previous control period, respectively, to obtain the current position value with noise and the actual position value with noise.

[0082] The intensity of the random noise can be weighted and adjusted according to a preset noise scaling vector to control the disturbance amplitude of different leg joints. It should be noted that the noise scaling vector can be composed of a plurality of noise scaling factors, each of which corresponds to a leg joint and is used to determine the noise weight of the leg joint.

[0083] In addition, a global amplification coefficient can also be set to uniformly scale the overall noise signal, which is used to control the overall intensity of the disturbance signal in the training process.

[0084] Through the combination of the noise scaling factor and the global amplification coefficient, fine control of the noise intensity of different leg joints can be realized while keeping the overall disturbance range consistent, thereby improving the stability and generalization performance of the training process.

[0085] Subsequently, the joint position difference between adjacent control periods is calculated according to the current position value with noise and the actual position value with noise, that is, the difference between the current position value with noise and the actual position value with noise is calculated to obtain the joint position difference.

[0086] In this way, simulated measurement errors and environmental disturbances can be introduced in the training stage, so that the distribution of the training data is closer to the actual running conditions, thereby the control policy network can maintain stable posture adjustment ability and high stay control precision even if there are sensor noise or external disturbances such as uneven terrain after the model is deployed.

[0087] In step S203, under the condition that the control target is zero-speed stay, the joint position difference is taken as an observation to construct input data for training the control policy network.

[0088] By taking the joint position difference as the observation, the dynamic balance process of the biped robot in the zero-speed stay state can be accurately reflected, thereby providing continuous and measurable state input for the control strategy network.

[0089] In constructing the training input, the joint position difference can be taken as the observation, or the joint position difference and other state information can be combined to form a complete observation vector for representing the overall attitude state of the biped robot in the current control period. Subsequently, the observation or the observation vector is input to the control strategy network to predict the target joint position difference of each biped joint in the next control period.

[0090] By training under the zero-speed stay condition with the joint position difference as the main observation, the control strategy network can learn the attitude adjustment rule under different attitude disturbances, thereby improving the balance maintaining ability and stability of the biped robot in the static state.

[0091] In some example embodiments, when constructing the input data for training the control strategy network, the base angular velocity and the gravity projection of the biped robot in the current control period can be obtained. The base angular velocity is used to represent the rotation rate of the biped robot as a whole, and the gravity projection is used to describe the components of the gravity in different directions in the robot body coordinate system, which reflects the degree of deviation of the robot body attitude from the gravity direction. Both of them together reflect the attitude change of the biped robot in space, which helps to evaluate its stability and response ability to external disturbances.

[0092] The joint action data output by the control strategy network in the previous control period can also be obtained, including the target position value of each leg joint and the target position difference of the biped joint. The target position value of each leg joint represents the ideal angular position generated by the control strategy network for the leg joint, which is used to guide the biped robot to maintain or adjust to the desired attitude during control. The target position difference of the biped joint represents the angular change amount of the biped joint relative to the previous control period, which is used to fine-tune the attitude of the biped robot during the stay process to maintain a stable stay state.

[0093] At the same time, the current speed of each leg joint and the current speed of the biped joint are obtained, and the joint position value of each leg joint and the joint position difference of the biped joint are calculated. The joint speed and the joint position together represent the motion state of the biped robot in the current control period, and the joint position difference of the biped joint reflects the small angular change of the biped joint between adjacent control periods, which is an important basis for fine adjustment of the stay control.

[0094] Finally, the obtained base angular velocity, gravity projection, joint action data, joint velocity and joint position and other information are integrated into a set of multi-dimensional input data for the control policy network to train. For example, the various types of state information can be scaled or standardized to ensure that the data of different dimensions and value ranges are in similar numerical intervals, thereby ensuring that the influence of various types of state information on network parameter updates during training remains balanced.

[0095] In this example, the constructed input data can provide comprehensive state information of the biped robot in each control cycle, providing high-quality training samples for the control policy network, which can help the control policy network learn how to respond to different posture adjustment tasks and optimize the balance control of the biped robot during zero-speed stay.

[0096] In step S204, the target position difference value of the biped joint output by the control policy network is compared with the actual position difference value, the control torque of the biped joint is calculated, and the parameters of the control policy network are updated in combination with the interaction results of the simulation environment.

[0097] Specifically, in each control cycle, the target position difference value of the biped joint output by the control policy network is compared with the actual position difference value, and a control error signal is calculated according to the deviation between the two. The actual position difference value refers to the actual angular change of the biped joint between two adjacent control cycles, which is used to represent the angular change amplitude of the biped joint in actual motion and reflects the posture adjustment of the biped robot during stay.

[0098] Further, according to the control error signal, in combination with proportional-differential control or nonlinear damping control, adaptive gain control, fuzzy PD control, neural network approximation control and other control laws, the corresponding control torque is calculated to drive the biped joint to perform corresponding compensation actions, thereby realizing real-time adjustment and stable control of the posture of the biped robot.

[0099] During the training phase, the calculated control torque of the biped joint can be input as an execution signal to the simulation environment to drive the biped robot to produce interactive behavior. The simulation environment outputs corresponding environment states and reward signals according to the control torque, forming the interaction results of the simulation environment. The control policy network receives the environment states and reward signals to evaluate the control effect of the current output, and updates the parameters of the control policy network based on the evaluation results.

[0100] Specifically, the effectiveness of the control policy network output is determined by comparing the deviation between the state change caused by the current control output and the zero-speed stay target. If the deviation is large, the network parameters are modified according to the gradient of the reward signal to guide the policy network to generate control torques that are more consistent with the stay target in subsequent training. Through continuous iteration of the above process, the control policy network can gradually learn the relationship between the wheel-foot joint control torque and the simulation environment state feedback, so as to adaptively output stable control torques under different disturbance conditions, and realize the steady-state control and posture keeping of the double-wheel-foot robot in the zero-speed state.

[0101] In some example embodiments, with reference to Figure 3 As shown in FIG. 3, the control torque of the wheel-foot joint can be calculated according to the following steps S301 to S303: Step S301, compare the target position difference value of the wheel-foot joint output by the control policy network with the actual position difference value, to obtain the position difference error of the wheel-foot joint. The position difference error is used to reflect the deviation between the actual angle change of the wheel-foot joint and the expected output of the control policy network.

[0102] Step S302, calculate the speed error of the wheel-foot joint according to the current speed of the wheel-foot joint and taking the zero speed as the target speed. The speed error is used to represent the dynamic deviation of the wheel-foot joint from the static state during the stay control process, and can be used to offset the posture change caused by the slight sliding of the wheel or external force disturbance, to provide dynamic compensation basis for the control torque.

[0103] Step S303, calculate the control torque of the wheel-foot joint based on the position difference error and the speed error.

[0104] Exemplarily, the position difference error and the speed error can be combined by weighting to obtain the control torque for driving the wheel-foot joint. For example, based on proportional-differential control, the control torque of the wheel-foot joint is calculated using the position difference error and the speed error, that is: (1) wherein, is the control torque of the wheel-foot joint, is the position difference error, is the speed error, is the target position difference value of the wheel-foot joint, is the actual position difference value of the wheel-foot joint, is the current speed of the wheel-foot joint, is the proportional gain of the wheel-foot joint, is the differential gain of the wheel-foot joint.

[0105] In formula (1), the proportional term is used to correct the error between the target position difference and the actual position difference of the wheel-foot joint, so as to realize the rapid tracking of the target posture of the wheel-foot joint. The differential term is used to suppress the instantaneous fluctuation caused by the change of the speed, and to smooth the control output, so that stable differential following can be realized when the wheel slips.

[0106] In some example embodiments, when the stay control of the double-wheel-foot robot is performed, the balance of the double-wheel-foot robot depends not only on the adjustment of the ground support force of the wheel-foot joint, but also on the coordination of the leg joint to offset the center of gravity offset caused by the posture change of the upper body. Therefore, the control link of the leg joint can be further introduced, and the overall stability and anti-disturbance ability in the stay control process can be enhanced by fine adjustment of the leg posture.

[0107] Reference Figure 4 As shown, the control torque of each leg joint can be calculated according to the following steps S401 to S404 to establish the coordinated control between the leg joint and the wheel-foot joint in the stay control training process, so that the control strategy network can learn the coordination relationship between the leg support and the wheel-foot posture adjustment at the same time: Step S401, obtaining the current position value of each leg joint of the double-wheel-foot robot.

[0108] It can be understood that the current position value of the leg joint is the angle value of the leg joint, which can be collected in real time by the angle encoder installed at each leg joint, and is used to represent the posture state of the leg structure of the double-wheel-foot robot in the current control period.

[0109] Step S402, calculating the first difference value between the current position value and the stay position value of each leg joint.

[0110] The stay position value in this step refers to the target angle value of each leg joint of the double-wheel-foot robot in the zero-speed stay state, which is the ideal posture reference position set for maintaining the overall posture stability and the center of gravity balance. The stay position value can be obtained by static posture planning or simulation calibration, and is used to represent the standard joint posture that the double-wheel-foot robot should maintain in the stay state. The first difference value is used to reflect the angle deviation between the current posture and the target stay posture of the leg joint, and is an important input parameter for control strategy network training and control calculation, which is used to describe the posture error and offset degree of the double-wheel-foot robot in the stay process.

[0111] Step S403, under the condition that the control target is zero-speed stay, the first difference value is taken as an observation quantity to construct the input data for training the control strategy network.

[0112] Optionally, in constructing the input data, the first difference value of the leg joint and the joint position difference value of the wheel-foot joint, the base angular velocity, and the gravity projection information can be used as inputs to guide the control strategy network to learn how to coordinate the linkage control of the leg joint and the wheel-foot joint, so as to realize the overall balance maintenance and posture stability of the double-wheel-foot robot.

[0113] In step S404, the target position value of the leg joint output by the control strategy network is compared with the current position value, and the control torque of each leg joint is calculated.

[0114] For example, first, the target position value of the leg joint output by the control strategy network is compared with the current position value to obtain the position error of each leg joint. The target position value is the expected angle dynamically calculated by the control strategy network based on the current state, which is used to guide the leg joint movement to approach the resting position value. Then, according to the current speed of each leg joint and taking zero speed as the target speed, the speed error of each leg joint is calculated. Finally, based on proportional-differential control, the position error and the speed error are used to calculate the control torque of each leg joint.

[0115] For example, when calculating the control torque of each leg joint, the following formula can be used: (2) wherein, is the control torque of the leg joint, is the position error of the leg joint, is the target position value of the leg joint, is the current position value of the leg joint, is the speed error of the leg joint, is the current speed of the leg joint, is the proportional gain of the leg joint, is the differential gain of the leg joint.

[0116] In formula (2), the proportional term is used to correct the error between the target position of the leg joint and the actual position, to realize fast response and accurate tracking of the target posture of the leg joint. The differential term is used to suppress the speed deviation of the leg joint, to reduce the oscillation caused by inertia or disturbance in the dynamic process, and to ensure that the leg joint reaches the target position smoothly.

[0117] The control torque of each leg joint calculated by the calculation can realize real-time correction of the leg posture of the double-wheel-foot robot, coordinate with the wheel-foot joint movement, and thus improve the posture stability and disturbance resistance of the double-wheel-foot robot in the resting task.

[0118] Further, after the control torque of the wheel-foot joint and the control torque of each leg joint are calculated, the control torque of the wheel-foot joint and the control torque of each leg joint can be combined to obtain a joint torque, that is, a complete control vector covering the joints related to the staying control is formed. The control vector is output as the current control period, used to drive the dual-wheel-foot robot to coordinate the action in the staying state, and realize the stable holding of the overall posture.

[0119] In some example embodiments, when updating the parameters of the control policy network according to the interaction result of the joint torque in the simulation environment, a loss function can be constructed based on the state and the reward obtained by the joint torque in the simulation environment interaction. Then, the parameters of the control policy network are updated based on the loss function.

[0120] Specifically, in the simulation environment, according to the posture stability, position offset and zero-speed keeping effect of the dual-wheel-foot robot in the current control period, a corresponding reward value is generated, which is used to represent the performance of the current control policy in achieving the staying target. Then, according to the difference between the reward value and the expected control target, a loss function is calculated to measure the deviation between the current control policy output and the ideal staying state.

[0121] Optionally, the loss function also includes a gradient penalty term, such as: (3) wherein, is the loss function, is the parameter of the control policy network, is the reinforcement learning loss term, which is constructed by the state s , the action a and the reward r , is the gradient penalty coefficient, is the action output of the control policy network in the state s , is the gradient of the output of the control policy network with respect to the parameter , is the square of the gradient norm, that is, the regularization term.

[0122] Based on the loss function shown in formula (3), the parameters of the control policy network can be iteratively updated by a stochastic gradient descent algorithm or an Adam (Adaptive Moment Estimation) optimization algorithm or other optimization algorithms. In the optimization process, the network parameters are fine-tuned according to the gradient information of the loss function in each control period, so as to gradually reduce the loss value, and realize the convergence of the control policy network from the initial state to the optimal staying control policy.

[0123] Through multiple rounds of simulation training and parameter updating, the control strategy network can gradually learn to output more optimal control torques under different postures and external disturbance conditions, thereby improving the posture stability of the double-wheel-foot robot in the zero-speed stay task.

[0124] In some example embodiments, after the joint torques are obtained, the joint torques can also be subjected to constraint processing and scaling processing, so as to update the parameters of the control strategy network according to the interaction results of the processed joint torques and the simulation environment.

[0125] Specifically, the control torques of the joints can be subjected to amplitude constraint according to a preset torque threshold, and the control torques exceeding the torque threshold can be truncated to a safe range, so as to prevent instability or actuator overload caused by excessively large output. After the constraint processing is completed, a uniform scaling coefficient can also be applied to the overall control torque, so that the output torque is matched with the joint driver characteristics and the simulation environment, thereby ensuring the stability and consistency of the control process.

[0126] The joint torques subjected to the constraint processing and the scaling processing serve as the execution signals of the current control period, and are used to drive the coordinated action of the joints, so as to realize the posture keeping and dynamic balance of the double-wheel-foot robot in the stay control process. In the training stage, the parameters of the control strategy network can be updated according to the deviation between the state changes caused by the joint torques and the zero-speed stay target, so that the control strategy network gradually learns the optimal joint torque distribution rule and posture control strategy.

[0127] The example embodiments of the present disclosure also provide another double-wheel-foot robot stay control model training method based on deep reinforcement learning. Referring to Figure 5 The method can include the following steps S501 to S504: Step S501: obtaining a current position value of a wheel-foot joint of the double-wheel-foot robot.

[0128] For example, the joint angle information can be collected in real time by an angle encoder arranged at each wheel-foot joint of the double-wheel-foot robot, so as to obtain the current position value of the wheel-foot joint.

[0129] Step S502: calculating a second difference value of the joint position between the current position value and a stay position value of the wheel-foot joint.

[0130] The stay position value in this step is an ideal wheel-foot joint angle when the double-wheel-foot robot realizes the zero-speed stay state. By comparing the current position value and the stay position value of the wheel-foot joint, the second difference value is obtained, which is used to reflect the deviation degree between the current wheel-foot joint posture and the target stay posture.

[0131] It can be understood that the second difference value can represent the posture deviation trend of the double-wheel-foot robot in the stay process, and provide accurate error information input for the control strategy network.

[0132] Step S503, under the condition that the control target is zero-speed stay, the second difference value is taken as an observation quantity, and input data for training the control strategy network is constructed.

[0133] The control strategy network takes the second difference value as the main observation quantity input, and can also acquire and integrate other state information such as base angular velocity, gravity projection, joint speed, etc., to construct an observation vector reflecting the overall state of the double-wheel-foot robot.

[0134] After inputting the observation vector into the control strategy network, the control strategy network is trained through interaction with the simulation environment, and gradually learns how to generate corresponding control action output, including the target position value of the wheel-foot joint. Through continuous iterative training, the control strategy network can optimize the action generation mechanism, so that the double-wheel-foot robot automatically adjusts the wheel-foot support force and leg posture during the stay process, thereby reducing the posture deviation and achieving stable zero-speed stay control.

[0135] Step S504, comparing the target position value of the wheel-foot joint output by the control strategy network with the current position value, calculating the control torque of the wheel-foot joint, and updating the parameters of the control strategy network in combination with the interaction result of the simulation environment.

[0136] Specifically, by comparing the target position value of the wheel-foot joint with the actual position value, a position error can be obtained.

[0137] Based on the position error, the proportional-differential control can be used to calculate the control torque of the wheel-foot joint for driving the wheel-foot joint to perform posture adjustment actions. At the same time, synchronous control can also be performed on the leg joints, such as the flow shown in reference Figure 4 As shown, the control torque of each leg joint is calculated for maintaining the upper body center of gravity and posture stability.

[0138] Similarly, in the training phase, the calculated control torque of the wheel-foot joint can be input to the simulation environment as an execution signal to drive the double-wheel-foot robot to produce interactive behavior. The simulation environment outputs corresponding environment state and reward signal according to the control torque, forming the interaction result of the simulation environment. The control strategy network receives the environment state and reward signal to evaluate the control effect of the current output, and updates the parameters of the control strategy network based on the evaluation result.

[0139] In some example embodiments, the control torques of the leg joints and the wheel-foot joints can be fused to form a joint control output, and the joint control output can be used to update the parameters of the control policy network in combination with the interaction results of the joint control output and the simulation environment. The joint control output can reflect the coordination relationship between the wheel-foot support and the leg posture adjustment of the biped robot in the stationary state. By introducing the coordinated control results of the two types of joint torques during the training process, the control policy network can simultaneously learn the coordination rules of the wheel-foot and the leg in the zero-speed stationary task, achieve dynamic balance and stable maintenance of the overall posture, and thus improve the overall stationary control performance of the biped robot in complex terrain or external disturbance conditions.

[0140] As can be seen from the biped robot stationary control model training method shown in Figure 2 , the training process takes the joint position difference between adjacent control periods as the main observation input, that is, the control policy network is trained in the relative error mode. This training method emphasizes capturing the subtle angle changes of the wheel-foot joints in the continuous control period, so that the control policy network can learn the dynamic response characteristics of the joints. During the stationary control process, the relative error input can be used to quickly identify and correct the posture deviation, and smooth response to instantaneous disturbances and subtle posture adjustment can be achieved.

[0141] Compared with the biped robot stationary control model training method shown in Figure 2 , the biped robot stationary control model training method in Figure 5 uses absolute position error, that is, the second difference between the current joint position and the target stationary position is used as the observation input of the control policy network. By directly comparing the deviation between the current posture and the ideal stationary posture, the control policy network can learn the posture balance and steady-state control, and gradually form the ability to identify and correct the absolute posture during the training process, thereby achieving stronger steady-state maintenance capability in the stationary control task.

[0142] The two training methods can be used separately or in combination according to different task scenarios, taking into account the response speed while improving overall stability, and achieving more reliable stationary control effect.

[0143] The example embodiments of the present disclosure also provide a biped robot stationary control method. Referring to Figure 6 , the method can include the following steps S601 to S605: Step S601, in each control period, the current position value of the wheel-foot joint of the biped robot is obtained.

[0144] During actual operation, the current position values ​​of the wheel joints of the bipedal robot can be measured in real time using angle encoders, inertial measurement units, or other angle detection sensors installed at each joint. The main controller reads the current joint angle data at the beginning of each control cycle to obtain the corresponding current position value of the wheel joint.

[0145] Step S602: Calculate the joint position difference between the wheel-foot joint in the current control cycle and the previous control cycle.

[0146] The wheel joint position values ​​of the current control cycle are compared with those of the previous control cycle to calculate the difference. This difference reflects the change in wheel joint angle within adjacent control cycles and serves as the basic input data for the control strategy network.

[0147] Step S603: Input the joint position difference into the pre-trained control strategy network to obtain the target position difference of the wheel joint in the current control cycle.

[0148] The target position difference refers to the expected angle change output by the control strategy network in the current control cycle to guide the movement of the wheel joint. It represents the angle increment that the wheel joint should produce relative to the previous control cycle and describes the expected rotation amplitude and direction of the wheel joint between adjacent control cycles.

[0149] In its implementation, the control strategy network takes multiple state information as inputs, including the base angular velocity, gravity direction projection, current position and velocity of the leg joints, joint position difference of the wheel-leg joints, current velocity of the wheel-leg joints, and control actions from the previous control cycle. These inputs can be scaled or normalized according to a preset scaling factor to balance the numerical scales of different physical quantities. Based on this, the control strategy network can comprehensively determine the current posture state and dynamic change trend of the bipedal robot and output the corresponding target position difference.

[0150] By dynamically calculating the difference in target position, the wheel joints can achieve smooth transitions and fine adjustments between adjacent control cycles, thereby maintaining the stable posture of the bipedal robot during the stationary process.

[0151] Step S604: Calculate the control torque of the wheel-foot joint based on the current position value of the wheel-foot joint in the current control cycle and the difference between the target position and the current position value.

[0152] Optional, see reference Figure 7 As shown, step S604 may further include steps S701 and S702: Step S701: Calculate the target position value of the wheel-foot joint in the current control cycle based on the difference between the target positions of the wheel-foot joint in the current control cycle and the target position value in the previous control cycle.

[0153] In the control execution stage, in order to keep the dimension of the target position difference value output by the control policy network consistent with the actual joint angle and ensure that the action amplitude conforms to the physical limit of the mechanical structure, the target position difference value can be scaled according to a preset position scaling factor. The position scaling factor is used to proportionally adjust the target position difference value, so that its amplitude matches the actual movement ability of the leg joint, avoiding the situation of overlarge action or oversmall reaction.

[0154] It should be noted that the selection of the position scaling factor can be determined according to the actual control accuracy requirement and the maximum allowable angle change range of the leg joint. In actual implementation, a suitable position scaling factor can be obtained through simulation debugging or experimental calibration. For example, according to the maximum angle change range of the leg joint, the output range of the control policy network can be linearly mapped to the angle range that can be executed by the joint, so as to ensure that the action instruction output by the network has sufficient response sensitivity and will not exceed the mechanical safety limit. By reasonably setting the position scaling factor, the stability and executability of the control policy in actual operation can be improved while maintaining the control accuracy.

[0155] Further, the target position value is calculated according to the scaled target position difference value and the target position value of the last control cycle. For example, the scaled target position difference value and the target position value of the last control cycle are added to obtain the target position value of the leg joint in the current control cycle. The target position value can reflect the judgment result of the control policy network on the current posture change trend, and is used to realize continuous and smooth adjustment of the leg joint, thereby supporting stable posture control of the biped robot during the stay process.

[0156] For example, the target position value of the leg joint in the current control cycle can be calculated according to formula (4) : (4) wherein, is the target position value of the leg joint in the last control cycle, is the target position difference value of the leg joint in the current control cycle, is the position scaling factor.

[0157] In step S702, the control torque of the leg joint is calculated based on the target position value of the leg joint in the current control cycle, the current position value and the current speed.

[0158] For example, the target position value of the leg joint in the current control cycle is compared with the current position value to obtain the position error of the leg joint. The position error reflects the deviation between the target angle expected to be output by the control policy network and the current actual angle.

[0159] Then, according to the current speed of the wheel-foot joint and taking zero speed as the target speed, the speed error of the wheel-foot joint is obtained. The speed error is used to describe the dynamic change trend of the wheel-foot joint in the current control period, and can reflect the slight sliding of the wheel caused by inertia or external force during the stay.

[0160] After obtaining the position error and the speed error, proportional-differential control can be used to calculate the control torque of the wheel-foot joint using the position error and the speed error.

[0161] For example, the control torque of the wheel-foot joint can be calculated according to formula (5) : (5) wherein, is the position error of the wheel-foot joint, is the target position value of the wheel-foot joint, is the current position value of the wheel-foot joint, is the speed error of the wheel-foot joint, is the current speed of the wheel-foot joint, is the proportional gain of the wheel-foot joint, is the differential gain of the wheel-foot joint.

[0162] It should be noted that the parameters in formula (5) are used to describe the variables in the control execution stage, and the parameters in formula (1) correspond to the variables in the control strategy network training stage. In order to distinguish the meanings of the parameters with the same name in the two stages, different symbols are used for the related parameters.

[0163] The calculated control torque is used to drive the wheel-foot joint to perform the corresponding action to realize posture correction and balance maintenance, and provides an execution basis for stable control of the dual-wheel-foot robot during zero-speed stay.

[0164] In step S605, the control torque is sent to the joint driver of the dual-wheel-foot robot, so that the joint driver controls the wheel-foot joint according to the control torque.

[0165] In addition to calculating the control torque of the wheel-foot joint, the control torque of the leg joint can also be calculated, such as using proportional-differential control to calculate the control torque of the leg joint by combining the current position value and the current speed value of the leg joint. The control torque of the leg joint is used to maintain the posture stability of the leg joint, and cooperates with the control torque of the wheel-foot joint generated by the control strategy network.

[0166] Subsequently, the main controller integrates the control torque of the wheel-foot joint and the control torque of the leg joint to form a joint control instruction set, and sends it to the corresponding joint driver to realize cooperative control and posture maintenance during the stay.

[0167] Specifically, each joint driver adjusts the output torque of the corresponding joint according to the received control torque, so as to drive the wheel-foot joint and the leg joint to perform a corresponding compensation action. In the stay control process, the joint driver adjusts the fine posture of the wheel-foot joint and the leg joint according to the control torque output by the main controller, so that the double-wheel-foot robot maintains balance and resists posture deviation caused by external disturbances in the zero-speed state.

[0168] It should be noted that the control strategy network in this example embodiment is obtained according to another example embodiment, i.e., a deep reinforcement learning-based double-wheel-foot robot stay control model training method as shown in Figure 2

[0169] The example embodiments of the present disclosure also provide another double-wheel-foot robot stay control method. Referring to Figure 8 Step S801: Obtain the current position value of the wheel-foot joint of the double-wheel-foot robot.

[0170] In the actual running phase, the current position value of the wheel-foot joint can be collected in real time by an angle encoder, an inertial measurement unit or other angle detection sensor installed at each joint. The main controller reads the current joint angle data during the running process to obtain the current position value of the corresponding wheel-foot joint.

[0171] Step S802: Calculate a third difference value between the current position value of the wheel-foot joint and a stay position value.

[0172] The stay position value in this step refers to the ideal wheel-foot joint angle set to achieve body posture stability and center of gravity balance under the zero-speed stay target. The third difference value is used to represent the deviation between the actual posture of the current wheel-foot joint and the ideal stay posture.

[0173] By continuously calculating the third difference value, the change trend of the joint posture can be grasped in real time, providing real-time state basis for the input of the subsequent control strategy network.

[0174] Step S803: Input the third difference value into the pre-trained control strategy network to obtain a target position value of the wheel-foot joint.

[0175] The third difference value, the base angular velocity, the gravity projection, the joint speed and the action output of the previous control cycle are input into the pre-trained control strategy network to obtain the target position value of the wheel-foot joint. The target position value is the output result dynamically generated by the pre-trained control strategy network after receiving the real-time state information such as the third difference value, the base angular velocity, the gravity projection and the joint speed, and represents the expected angle to which the wheel-foot joint should be adjusted under the current state. ​​

[0176] Step S804, based on the current position value of the wheel-foot joint and the target position value, the control torque of the wheel-foot joint is calculated.

[0177] Specifically, the target position value of the wheel-foot joint is compared with the current position value, and the difference between the two is calculated to obtain the position error. At the same time, according to the current speed of the wheel-foot joint, and taking zero speed as the target speed, the speed error is calculated. Further, the proportional-differential control can be used to calculate the control torque of the wheel-foot joint according to the position error and the speed error.

[0178] Step S805, the control torque is sent to the joint driver of the biped robot, so that the joint driver controls the wheel-foot joint according to the control torque.

[0179] After obtaining the control torque of the wheel-foot joint, the main controller integrates the control torque of the leg joint and the control torque of the wheel-foot joint to form a joint control instruction set, which is sent to the corresponding joint driver to realize the cooperative control and posture keeping in the stay process.

[0180] It should be noted that the control strategy network in this example embodiment is obtained according to another example embodiment, i.e., the biped robot stay control model training method based on deep reinforcement learning as shown in Figure 5

[0181] In the example embodiments of the present disclosure, a biped robot stay control model training device based on deep reinforcement learning is also provided. Referring to Figure 9 The first biped robot stay control model training device 900 based on deep reinforcement learning includes a first position acquisition module 901, a first difference determination module 902, a first input construction module 903, and a first network training module 904, wherein: The first position acquisition module 901 is configured to acquire the current position value of the wheel-foot joint of the biped robot in each control period. The first difference determination module 902 is configured to calculate the joint position difference of the wheel-foot joint between the current control period and the previous control period. The first input construction module 903 is configured to, under the condition that the control target is zero-speed stay, take the joint position difference as an observation, and construct input data for training the control strategy network. The first network training module 904 is configured to compare the target position difference of the wheel-foot joint output by the control strategy network with the actual position difference, calculate the control torque of the wheel-foot joint, and update the parameters of the control strategy network in combination with the interaction result of the wheel-foot joint and the simulation environment.

[0182] ​The specific details of the modules in the above deep reinforcement learning-based biped robot staying control model training device have been described in detail in the corresponding deep reinforcement learning-based biped robot staying control model training method, and thus will not be described here again.

[0183] In the example embodiments of the present disclosure, another deep reinforcement learning-based biped robot staying control model training device is also provided. Referring to Figure 10 As shown in the figure, the second deep reinforcement learning-based biped robot staying control model training device 1000 includes a second position acquisition module 1001, a second difference determination module 1002, a second input construction module 1003, and a second network training module 1004, wherein: The second position acquisition module 1001 is configured to acquire the current position value of the wheel-foot joint of the biped robot; The second difference determination module 1002 is configured to calculate the second difference of the joint position between the current position value and the staying position value of the wheel-foot joint; The second input construction module 1003 is configured to, under the condition that the control target is zero-speed staying, take the second difference as an observation, and construct the input data for training the control policy network; The second network training module 1004 is configured to compare the target position value and the current position value of the wheel-foot joint output by the control policy network, calculate the control torque of the wheel-foot joint, and update the parameters of the control policy network in combination with the interaction result of the control torque of the wheel-foot joint and the simulation environment.

[0184] The specific details of the modules in the above deep reinforcement learning-based biped robot staying control model training device have been described in detail in the corresponding deep reinforcement learning-based biped robot staying control model training method, and thus will not be described here again.

[0185] In the example embodiments of the present disclosure, a biped robot staying control device is also provided. Referring to Figure 11 As shown in the figure, the first biped robot staying control device 1100 includes a third position acquisition module 1101, a third difference determination module 1102, a relative position determination module 1103, a first torque determination module 1104, and a first staying control module 1105, wherein: The third position acquisition module 1101 is configured to acquire the current position value of the wheel-foot joint of the biped robot in each control period; The third difference determination module 1102 is configured to calculate the difference of the joint position of the wheel-foot joint between the current control period and the previous control period; The relative position determination module 1103 is configured to input the joint position difference value into a pre-trained control strategy network to obtain a target position difference value of the leg joint in a current control period. The first torque determination module 1104 is configured to calculate a control torque of the leg joint based on the current position value and the target position difference value of the leg joint in the current control period. The first stay control module 1105 is configured to send the control torque to the joint driver of the biped robot to enable the joint driver to perform stay control on the leg joint according to the control torque. The control strategy network is obtained according to a biped robot stay control model training method based on deep reinforcement learning, as shown in Figure 2 The biped robot stay control method.

[0186] The specific details of the modules in the biped robot stay control device have been described in detail in the corresponding biped robot stay control method, and thus will not be described here again.

[0187] In the example embodiments of the present disclosure, a biped robot stay control device is also provided. As shown in Figure 12 The second biped robot stay control device 1200 includes a fourth position acquisition module 1201, a fourth difference determination module 1202, an absolute position determination module 1203, a second torque determination module 1204, and a second stay control module 1205, wherein: The fourth position acquisition module 1201 is configured to acquire a current position value of a leg joint of a biped robot. The fourth difference determination module 1202 is configured to calculate a third difference value between the current position value and a stay position value of the leg joint. The absolute position determination module 1203 is configured to input the third difference value into a pre-trained control strategy network to obtain a target position value of the leg joint. The second torque determination module 1204 is configured to calculate a control torque of the leg joint based on the current position value and the target position value of the leg joint in a current control period. The second stay control module 1205 is configured to send the control torque to the joint driver of the biped robot to enable the joint driver to perform stay control on the leg joint according to the control torque. The control strategy network is obtained according to a biped robot stay control model training method based on deep reinforcement learning, as shown in Figure 5 The biped robot stay control method.

[0188] The specific details of the modules in the biped robot stay control device have been described in detail in the corresponding biped robot stay control method, and thus will not be described here again.

[0189] In the example embodiments of the present disclosure, a double-wheel foot robot is also provided, which comprises a processor and a memory, and the memory stores computer readable instructions which are executed by the processor to implement the above method. Figure 13 As shown in FIG. 13, a schematic diagram of a double-wheel foot robot is shown, which comprises a wheel-foot joint 1300.

[0190] In the example embodiments of the present disclosure, a double-wheel foot robot is also provided, which comprises a processor and a memory, and the memory stores computer readable instructions which are executed by the processor to implement the above method. Figure 14 As shown in FIG. 14, an electronic device capable of implementing the above method is also provided. The electronic device 1400 comprises a processor 1401 and a memory 1402, and the memory 1402 stores computer readable instructions which are executed by the processor 1401 to implement the method in the example embodiments of the present disclosure.

[0191] In the example embodiments of the present disclosure, a computer readable storage medium is also provided, which stores computer program code instructions, and when the computer program code instructions are called by the processor of the robot, the robot executes the method as described in the embodiments.

[0192] In the example embodiments of the present disclosure, a computer readable storage medium is also provided, which stores computer program code instructions, and when the computer program code instructions are called by the processor of the robot, the robot executes the method as described in the embodiments. Figure 15 As shown in FIG. 15, a program product 1500 for implementing the above method according to the embodiments of the present disclosure is described, which can adopt a portable compact disc read-only memory (CD-ROM) and comprises program code, and can run on a terminal device such as a personal computer. However, the program product of the present disclosure is not limited to this, and in this document, the readable storage medium can be any tangible medium containing or storing a program which can be used by or in conjunction with an instruction execution system, device or apparatus.

[0193] Through the above description of the embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the example embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or a network, and includes a number of instructions to make a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) execute the method according to the example embodiments of the present disclosure.

[0194] Finally, the above preferred embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail, those skilled in the art should understand that changes can be made in form and details without departing from the scope defined by the claims of the present application. The sizes of the drawings are not related to the actual size, and the actual size can be changed arbitrarily.

Claims

1. A training method for a dwell control model of a bipedal robot based on deep reinforcement learning, characterized in that, include: Within each control cycle, acquire the current position value of the wheel joints of the bipedal robot; Calculate the joint position difference of the wheel-foot joint between the current control cycle and the previous control cycle; Under the condition that the control objective is to stop at zero speed, the joint position difference is used as an observation to construct input data for training the control strategy network; The target position difference of the wheel-foot joint output by the control strategy network is compared with the actual position difference. The control torque of the wheel-foot joint is calculated, and the parameters of the control strategy network are updated based on the interaction results of the control torque of the wheel-foot joint and the simulation environment.

2. The training method for a dwell control model of a bipedal robot based on deep reinforcement learning according to claim 1, characterized in that, The step of comparing the target position difference of the wheel-foot joint output by the control strategy network with the actual position difference, and calculating the control torque of the wheel-foot joint, includes: The target position difference of the wheel-foot joint output by the control strategy network is compared with the actual position difference to obtain the position difference error of the wheel-foot joint. Based on the current speed of the wheel joint, and with zero speed as the target speed, calculate the speed error of the wheel joint; The control torque of the wheel joint is calculated based on the position difference error and the speed error.

3. The training method for the dwell control model of a bipedal robot based on deep reinforcement learning according to claim 2, characterized in that, The calculation of the control torque of the wheel joint based on the position difference error and the velocity error includes: Based on proportional-derivative control, the control torque of the wheel joint is calculated using the position difference error and the velocity error.

4. The training method for the dwell control model of a bipedal robot based on deep reinforcement learning according to claim 3, characterized in that, The proportional-derivative control method, which calculates the control torque of the wheel joint using the position difference error and the velocity error, includes: in, For the control torque of the wheel joint, This is the position difference error. For speed error, This represents the target position difference of the wheel joint. This represents the actual position difference of the wheel joint. This represents the current velocity of the wheel joint. For the proportional gain of the wheel-foot joint. The differential gain of the wheel joint.

5. The training method for a dwell control model of a bipedal robot based on deep reinforcement learning according to claim 1, characterized in that, The calculation of the joint position difference of the wheel-foot joint between the current control cycle and the previous control cycle includes: Noise is added to the current position value of the wheel joint in the current control cycle and the actual position value in the previous control cycle to obtain the current position value with noise and the actual position value with noise. The joint position difference is calculated based on the current position value with noise and the actual position value with noise.

6. The training method for a dwell control model of a bipedal robot based on deep reinforcement learning according to claim 5, characterized in that, The same noise is added to the current position value of the wheel joint in the current control cycle and the actual position value in the previous control cycle.

7. The training method for a dwell control model of a bipedal robot based on deep reinforcement learning according to claim 5, characterized in that, The method further includes: The intensity of the noise is adjusted according to a preset noise scaling factor.

8. The training method for a dwell control model of a bipedal robot based on deep reinforcement learning according to claim 1, characterized in that, The method further includes: Obtain the current position value of each leg joint of the bipedal robot; Calculate the first difference between the current position value and the dwell position value of each leg joint; Under the condition that the control objective is to stop at zero speed, the first difference is used as an observation to construct input data for training the control policy network; The target position value of the leg joint output by the control strategy network is compared with the current position value, and the control torque of each leg joint is calculated.

9. The training method for a dwell control model of a bipedal robot based on deep reinforcement learning according to claim 8, characterized in that, The step of comparing the target position value of the leg joint output by the control strategy network with the current position value, and calculating the control torque of each leg joint, includes: The target position value of the leg joint output by the control strategy network is compared with the current position value to obtain the position error of each leg joint; Based on the current velocity of each leg joint, and with zero velocity as the target velocity, calculate the velocity error of each leg joint; Based on proportional-derivative control, the control torque of each leg joint is calculated using the position error and the velocity error.

10. The training method for a dwell control model of a bipedal robot based on deep reinforcement learning according to claim 9, characterized in that, The proportional-derivative control method, which calculates the control torque for each leg joint using the position error and the velocity error, includes: in, For the control torque of the leg joint, This refers to the positional error of the leg joints. This represents the target position value for the leg joint. This represents the current position value of the leg joint. For the speed error of the leg joint, This represents the current velocity of the leg joint. The proportional gain for the leg joints. The differential gain is for the leg joint.

11. The training method for a dwell control model of a bipedal robot based on deep reinforcement learning according to claim 8, characterized in that, After calculating the control torque of the wheel joint, the method further includes: The control torque of the wheel joint and the control torque of each leg joint are combined to obtain the joint torque; The parameters of the control strategy network are updated based on the interaction results between the joint torque and the simulation environment.

12. The training method for a dwell control model of a bipedal robot based on deep reinforcement learning according to claim 11, characterized in that, The method further includes: The joint torques are constrained and scaled to update the parameters of the control strategy network based on the interaction results between the processed joint torques and the simulation environment.

13. The training method for a dwell control model of a bipedal robot based on deep reinforcement learning according to claim 11, characterized in that, The interaction results include status and reward; The step of updating the parameters of the control strategy network based on the interaction results between the joint torque and the simulation environment includes: Based on the state and reward obtained from the joint torque in the simulation environment interaction, a loss function is constructed; The parameters of the control policy network are updated based on the loss function.

14. The training method for a dwell control model of a bipedal robot based on deep reinforcement learning according to claim 13, characterized in that, The loss function also includes a gradient penalty term; The loss function is: in, For loss function, To control the parameters of the policy network, To reinforce the learning loss term, the state... s ,action a With rewards r Build, The gradient penalty coefficient is... To control the policy network in the state s The following action output, The output of the control policy network is relative to the parameters gradient, It is the square of the gradient norm, i.e., the regularization term.

15. The training method for a dwell control model of a bipedal robot based on deep reinforcement learning according to any one of claims 1 to 10, characterized in that, The input data used to construct the control policy network includes: Obtain the base angular velocity and gravity projection of the bipedal robot in the current control cycle; The joint motion data of the bipedal robot output by the control strategy network in the previous control cycle is obtained; wherein, the joint motion data includes the target position value of each leg joint and the target position difference of the wheel joints; The input data is constructed based on the base angular velocity, the gravity projection, the joint motion data, the current velocity of each leg joint, the current velocity of the wheel joint, the calculated joint position value of each leg joint, and the joint position difference of the wheel joint.

16. A training method for a dwell control model of a bipedal robot based on deep reinforcement learning, characterized in that, include: Obtain the current position values ​​of the wheel joints of the bipedal robot; Calculate the second difference in joint position between the current position value and the dwell position value of the wheel joint; Under the condition that the control objective is to stop at zero speed, the second difference is used as an observation to construct the input data for training the control policy network; The target position value of the wheel joint output by the control strategy network is compared with the current position value. The control torque of the wheel joint is calculated, and the parameters of the control strategy network are updated based on the interaction results of the control torque of the wheel joint and the simulation environment.

17. A method for controlling the stationary position of a bipedal robot, characterized in that, include: Within each control cycle, acquire the current position value of the wheel joints of the bipedal robot; Calculate the joint position difference of the wheel-foot joint between the current control cycle and the previous control cycle; The joint position difference is input into a pre-trained control strategy network to obtain the target position difference of the wheel joint in the current control cycle. Based on the current position value of the wheel joint in the current control cycle and the difference between the target position and the current position value, the control torque of the wheel joint is calculated; The control torque is sent to the joint actuator of the bipedal robot so that the joint actuator controls the dwell of the wheel joint according to the control torque; The control strategy network is obtained using the deep reinforcement learning-based method for training a bipedal robot stationary control model according to any one of claims 1 to 15.

18. The method for controlling the dwelling of a bipedal robot according to claim 17, characterized in that, The step of calculating the control torque of the wheel-foot joint based on the current position value of the wheel-foot joint in the current control cycle and the difference between the target position and the current position includes: The target position value of the wheel joint in the current control cycle is calculated based on the difference between the target position of the wheel joint in the current control cycle and the target position value in the previous control cycle. The control torque of the wheel joint is calculated based on the target position value, current position value, and current speed of the wheel joint in the current control cycle.

19. The method for controlling the dwelling of a bipedal robot according to claim 18, characterized in that, The step of calculating the target position value of the wheel joint in the current control cycle based on the target position difference of the wheel joint in the current control cycle and the target position value in the previous control cycle includes: The target position difference is scaled according to a preset position scaling factor to obtain the scaled target position difference. The target position value is calculated based on the scaled target position difference and the target position value of the previous control cycle.

20. The method for controlling the dwelling of a bipedal robot according to claim 18, characterized in that, The calculation of the control torque of the wheel-foot joint based on the target position value, current position value, and current velocity of the wheel-foot joint in the current control cycle includes: The target position value of the wheel joint in the current control cycle is compared with the current position value to obtain the position error of the wheel joint; The current speed of the wheel joint is obtained, and zero speed is used as the target speed to obtain the speed error of the wheel joint; Based on proportional-derivative control, the control torque of the wheel joint is calculated using the position error and the velocity error.

21. A method for controlling the stationary position of a bipedal robot, characterized in that, include: Obtain the current position values ​​of the wheel joints of the bipedal robot; Calculate the third difference between the current position value and the dwell position value of the wheel joint; The third difference is input into a pre-trained control strategy network to obtain the target position value of the wheel joint; Based on the current position value of the wheel-foot joint and the target position value, calculate the control torque of the wheel-foot joint; The control torque is sent to the joint actuator of the bipedal robot so that the joint actuator controls the dwell of the wheel joint according to the control torque; The control strategy network is obtained using the deep reinforcement learning-based stationary control model training method for bipedal robots as described in claim 16.

22. A training device for a stationary control model of a bipedal robot based on deep reinforcement learning, characterized in that, include: The first position acquisition module is used to acquire the current position value of the wheel joints of the bipedal robot in each control cycle. The first difference determination module is used to calculate the joint position difference of the wheel joint between the current control cycle and the previous control cycle. The first input construction module is used to construct input data for training the control strategy network by using the joint position difference as an observation under the condition that the control target is zero-speed stop. The first network training module is used to compare the target position difference of the wheel-foot joint output by the control strategy network with the actual position difference, calculate the control torque of the wheel-foot joint, and update the parameters of the control strategy network by combining the interaction results of the control torque of the wheel-foot joint with the simulation environment.

23. A training device for a stationary control model of a bipedal robot based on deep reinforcement learning, characterized in that, include: The second position acquisition module is used to acquire the current position value of the wheel joints of the bipedal robot; The second difference determination module is used to calculate the second difference between the current position value and the dwell position value of the wheel joint; The second input construction module is used to construct input data for training the control policy network by using the second difference as an observation under the condition that the control target is zero-speed dwell. The second network training module is used to compare the target position value of the wheel joint output by the control strategy network with the current position value, calculate the control torque of the wheel joint, and update the parameters of the control strategy network by combining the interaction results of the control torque of the wheel joint and the simulation environment.

24. A dwell control device for a bipedal robot, characterized in that, include: The third position acquisition module is used to acquire the current position value of the wheel joints of the bipedal robot in each control cycle. The third difference determination module is used to calculate the joint position difference of the wheel joint between the current control cycle and the previous control cycle. The relative position determination module is used to input the joint position difference to a pre-trained control strategy network to obtain the target position difference of the wheel joint in the current control cycle. The first torque determination module is used to calculate the control torque of the wheel joint based on the current position value of the wheel joint in the current control cycle and the target position difference. The first dwell control module is used to send the control torque to the joint actuator of the bipedal robot, so that the joint actuator controls the dwell of the wheel joint according to the control torque; The control strategy network is obtained using the deep reinforcement learning-based method for training a bipedal robot stationary control model according to any one of claims 1 to 15.

25. A dwelling control device for a bipedal robot, characterized in that, include: The fourth position acquisition module is used to acquire the current position value of the wheel joints of the bipedal robot; The fourth difference determination module is used to calculate the third difference between the current position value and the dwell position value of the wheel joint; An absolute position determination module is used to input the third difference into a pre-trained control strategy network to obtain the target position value of the wheel joint; The second torque determination module is used to calculate the control torque of the wheel joint based on the current position value of the wheel joint and the target position value; The second dwell control module is used to send the control torque to the joint actuator of the bipedal robot, so that the joint actuator controls the dwell of the wheel joint according to the control torque; The control strategy network is obtained using the deep reinforcement learning-based stationary control model training method for bipedal robots as described in claim 16.

26. An electronic device, characterized in that, include: processor; as well as A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 21.

27. A bipedal robot, characterized in that, include: processor; as well as A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 21.

28. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program code instructions that, when invoked by the processor of the bipedal robot, cause the robot to perform the method as described in any one of claims 1 to 21.

Images