A robot standing control method, device, equipment and storage medium
By training a robot's standing control strategy using reinforcement learning algorithms in a physical simulation environment, and combining the stability relationship between the center of mass position and the supporting polygon, the problem of stable standing of the robot when the chassis moves or is disturbed is solved, and stable standing control in complex environments is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- DIGITAL HUAXIA (SHENZHEN) TECHNOLOGY CO LTD
- Filing Date
- 2026-02-11
- Publication Date
- 2026-06-02
AI Technical Summary
Existing technologies struggle to maintain stable standing when the robot chassis undergoes arbitrary movement or disturbance, especially under complex motion conditions. The increased uncertainty in traditional methods leads to robot instability or falls.
In a pre-defined physical simulation environment, a robot standing control strategy is trained using a reinforcement learning algorithm. A reward function is constructed by utilizing the stable relationship between the center of mass position and the supporting polygon. Standing control commands are acquired and generated in real time to adjust the robot's posture and center of gravity distribution.
It enables robots to stand stably under complex motion and disturbance conditions, improves robustness and adaptability, and ensures safety and reliability in various robot platforms and chassis.
Smart Images
Figure CN122131655A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robot control technology, and in particular to a robot standing control method, device, equipment, and storage medium. Background Technology
[0002] As humanoid robots and similar robots are increasingly used in industry, public services, and complex environments, they often need to stand on movable chassis, such as wheeled chassis, tracked chassis, or other mobile platforms. When the chassis is moving, turning, accelerating, decelerating, or subjected to external disturbances, the robot system as a whole will be affected by continuously changing non-stationary motion, resulting in a significant decrease in standing stability.
[0003] In existing technologies, robot standing control on a chassis typically relies on feedback control methods based on dynamic models, such as zero-torque points, virtual constraints, or linearized model control. These methods usually assume that the chassis motion can be accurately modeled or are designed only for specific working conditions. When the chassis motion is complex or there are unknown disturbances, the model uncertainty increases significantly, control performance is prone to deterioration, and may even lead to robot instability or falls.
[0004] On the other hand, reinforcement learning methods have made progress in robot motion control in recent years, but existing work is mostly focused on flat walking or gait generation tasks, with little systematic research on the "stability of robots standing on a moving chassis". In particular, there is a lack of control schemes that combine traditional stability criteria, such as the relationship between the center of gravity and support, and the idea of virtual constraints, with reinforcement learning methods.
[0005] It should be noted that in practical applications, the movable chassis usually has high mobility and diverse motion forms, and its motion state has a direct impact on the robot's standing stability. Traditional standing control methods that are only for stationary or quasi-stationary ground are difficult to apply.
[0006] In conclusion, how to control a robot to maintain a stable standing position when the chassis undergoes arbitrary movement or disturbance is a technical problem that urgently needs to be solved. Summary of the Invention
[0007] In view of this, the purpose of this invention is to provide a robot standing control method, device, equipment, and storage medium, capable of controlling the robot to maintain stable standing when the chassis undergoes arbitrary movement or disturbance. The specific solution is as follows: In a first aspect, this application provides a robot standing control method, including: The target robot is trained to stand and control in a preset physical simulation environment, and the first state information of the target robot is acquired in real time during the training process. The target robot is trained to stand upright using a reinforcement learning algorithm based on the first state information and a preset reward function, thereby obtaining a corresponding target standing control strategy; wherein the preset reward function is constructed based at least on the stable relationship between the centroid position of the target robot and the supporting polygon; The target standing control strategy is deployed to the target robot, and the second state information of the target robot is acquired in real time during the application of the target robot; wherein, the target robot stands on the target movable chassis; Using the target standing control strategy, a corresponding target standing control command is generated based on the second state information, and the target robot is controlled to execute the target standing control command; the target standing control command is used to control any one or a combination of several of the torque, position, and speed of the target robot; wherein, the target standing control strategy is used to adjust the posture and center of gravity distribution of the target robot during the movement of the target movable chassis.
[0008] Optionally, the preset physical simulation environment includes a preset movable chassis, and the target robot stands on the preset movable chassis.
[0009] Optionally, before acquiring the first state information of the target robot in real time, the method further includes: In the preset physical simulation environment, the preset movable chassis is controlled to move based on a preset motion pattern, and external force disturbances are applied to the preset movable chassis and / or the target robot in order to obtain the first state information of the target robot on the preset movable chassis in real time; wherein, the preset motion pattern includes any one or a combination of several of translational motion, rotational motion, acceleration and deceleration motion and periodic motion; Accordingly, the real-time acquisition of the first state information of the target robot includes: The system acquires in real time the target robot's velocity, center of mass position, torso posture, angular velocity, foot contact status, and support polygon information.
[0010] Optionally, after acquiring the first state information of the target robot in real time, the method further includes: The first motion state information of the preset movable chassis is acquired in real time so as to utilize the reinforcement learning algorithm and train the target robot to stand up based on the first state information, the first motion state information and the preset reward function to obtain the target standing control strategy. Accordingly, the real-time acquisition of the second state information of the target robot includes: The second state information of the target robot on the target movable chassis is acquired in real time, and the second motion state information of the target movable chassis is acquired, so as to generate the target standing control command based on the second state information and the second motion state information using the target standing control strategy.
[0011] Optionally, before training the target robot to stand control using a reinforcement learning algorithm based on the first state information and a preset reward function, the method further includes: Determine preset joint relationships and preset geometric relationships; the preset joint relationships are the joint relationships corresponding to when the target robot maintains a stable standing position, and the preset geometric relationships are the geometric relationships between the limbs when the target robot maintains a stable standing position. Generate constraint reward terms corresponding to the preset joint relationships and the preset geometric relationships, and construct the preset reward function based on the constraint reward terms.
[0012] Optionally, the process of training the target robot to stand with control using a reinforcement learning algorithm based on the first state information and a preset reward function includes: Determine the projection range corresponding to the supporting polygon information; If the centroid is located within the projection range, and the distance between the centroid and the boundary of the projection range is greater than or equal to a preset distance threshold, then the preset reward function is used to generate a corresponding first positive reward. If the centroid is located within the projection range, and the distance between the centroid and the boundary of the projection range is less than the preset distance threshold, then the preset reward function is used to generate a corresponding first negative reward. If the centroid position is not within the projection range, then the preset reward function is used to generate a corresponding second negative reward; The target robot is trained to stand up based on the first positive reward, the first negative reward, and the second negative reward.
[0013] Optionally, the process of training the target robot to stand with control using a reinforcement learning algorithm based on the first state information and a preset reward function includes: If the torso posture is a preset upright posture, then the preset reward function is used to generate a corresponding second positive reward; If the joint output torque of the target robot exceeds the preset torque threshold, a corresponding third negative reward is generated using the preset reward function. If the joint change angle of the target robot exceeds a preset angle threshold, a corresponding fourth negative reward is generated using the preset reward function. If the target robot falls, the standing control training for the current round is stopped, and the corresponding fifth negative reward is generated using the preset reward function; The target robot is trained to stand upright based on the second positive reward, the third negative reward, the fourth negative reward, and the fifth negative reward.
[0014] Secondly, this application provides a robot standing control device, comprising: The first state information acquisition module is used to train the target robot to stand and control in a preset physical simulation environment, and to acquire the first state information of the target robot in real time during the training process. A standing control strategy determination module is used to train the target robot to stand using a reinforcement learning algorithm based on the first state information and a preset reward function, so as to obtain a corresponding target standing control strategy; wherein, the preset reward function is constructed based at least on the stable relationship between the centroid position of the target robot and the supporting polygon; The second state information acquisition module is used to deploy the target standing control strategy to the target robot and acquire the second state information of the target robot in real time during the application of the target robot; wherein the target robot stands on the target movable chassis; The standing control command generation module is used to generate corresponding target standing control commands based on the second state information using the target standing control strategy, and to control the target robot to execute the target standing control commands; the target standing control commands are used to control any one or a combination of several of the torque, position, and speed of the target robot; wherein, the target standing control strategy is used to adjust the posture and center of gravity distribution of the target robot during the movement of the target movable chassis.
[0015] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor is used to execute the computer program to implement the aforementioned robot standing control method.
[0016] Fourthly, this application provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned robot standing control method.
[0017] In this application, the target robot is first trained to stand in a preset physical simulation environment, and its first state information is acquired in real time during the training process. A reinforcement learning algorithm is then used to train the target robot to stand based on the first state information and a preset reward function to obtain a corresponding target standing control strategy. The preset reward function is constructed based at least on the stable relationship between the target robot's center of mass and the supporting polygon. The target standing control strategy is then deployed to the target robot, and its second state information is acquired in real time during the application process. The target robot stands on a target movable chassis. Finally, the target standing control strategy is used to generate corresponding target standing control commands based on the second state information, and the target robot is controlled to execute these commands. The target standing control commands are used to control any one or a combination of torque, position, and velocity of the target robot. The target standing control strategy is used to adjust the target robot's posture and center of gravity distribution during the movement of the target movable chassis. As can be seen from the above, this application first trains the target robot's standing control in a preset physical simulation environment. During the training process, the robot's first state information is collected in real time. Then, based on a reinforcement learning algorithm, the collected first state information and a preset reward function are combined to train a suitable target standing control strategy. Afterward, the target standing control strategy is deployed to the target robot. During the actual application phase, the second state information is acquired in real time. Then, based on the deployed target standing control strategy and the second state information, corresponding target standing control commands are generated to control the robot to execute the target standing control commands and maintain a stable standing position. In this way, from simulation training to actual deployment, this application allows the robot to utilize the trained target standing control strategy to dynamically output suitable control commands based on real-time state information, precisely adjust joint-related torques, positions, or speeds, effectively cope with various working conditions during movement, and achieve stable standing of the robot in complex scenarios such as chassis movement and external disturbances, significantly improving the robustness and adaptability of the robot's standing control. Meanwhile, this application does not require precise modeling of chassis dynamics, and the target standing control strategy is more adaptable, which can play a role in various robot platforms and chassis forms, ensuring the safety and reliability of the robot standing in real complex environments. Attached Figure Description
[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0019] Figure 1 A flowchart of a robot standing control method provided in this application; Figure 2 This application provides a schematic diagram of the structure of a robot standing control device; Figure 3 This application provides a structural diagram of an electronic device. Detailed Implementation
[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0021] With the increasing application of humanoid and humanoid robots in industry, public services, and complex environments, robots often need to stand on movable chassis, such as wheeled chassis, tracked chassis, or other mobile platforms. When the chassis moves, turns, accelerates, decelerates, or is subjected to external disturbances, the overall robot system is affected by continuously changing non-stationary motion, leading to a significant decrease in standing stability. In existing technologies, robot standing control on chassis typically relies on feedback control methods based on dynamic models, such as zero-torque points, virtual constraints, or linearized model control. These methods usually assume that the chassis motion can be accurately modeled or are designed only for specific working conditions. When chassis motion is complex or there are unknown disturbances, model uncertainty increases significantly, control performance easily deteriorates, and may even lead to robot instability or falls. On the other hand, reinforcement learning methods have made progress in robot motion control in recent years, but existing work mainly focuses on flat ground walking or gait generation tasks, with less systematic research on the "stable standing problem of robots on moving chassis," especially lacking control schemes that combine traditional stability criteria, such as the relationship between center of gravity and support, and the concept of virtual constraints, with reinforcement learning methods. Therefore, this application provides a robot standing control scheme that can control the robot to maintain a stable standing position when the chassis undergoes arbitrary movement or disturbance.
[0022] See Figure 1 As shown in the figure, an embodiment of the present invention discloses a robot standing control method, which may include: Step S11: Train the target robot to stand in a preset physical simulation environment, and acquire the first state information of the target robot in real time during the training process.
[0023] In this embodiment, during the reinforcement learning training phase, a training environment is constructed in a physical simulation platform, in which the target robot stands on a preset movable chassis, including: 1. Robot Model It includes complete joint structure, mass distribution and actuator constraints to realistically reflect the standing dynamics of a robot.
[0024] 2. Chassis motion modeling
[0025] Apply translation, rotation, acceleration / deceleration, or periodic motion to a pre-set movable chassis to continuously subject the robot to the effects of non-steady base motion.
[0026] 3. Disturbance Injection Mechanism
[0027] During training, external forces are randomly applied to the robot or a pre-set movable chassis to simulate uncertainties such as pushing and bumping in a real environment.
[0028] It is understood that before acquiring the first state information of the target robot in real time, the process may include: controlling the preset movable chassis to move based on a preset motion pattern in the preset physical simulation environment, and applying external force disturbance to the preset movable chassis and / or the target robot, so as to acquire the first state information of the target robot on the preset movable chassis in real time; wherein, the preset motion pattern includes any one or a combination of several of translational motion, rotational motion, acceleration / deceleration motion, and periodic motion; correspondingly, acquiring the first state information of the target robot in real time may specifically include: acquiring the target robot's velocity information, center of mass position, torso posture, angular velocity information, contact state information of both feet, and support polygon information in real time. Specifically, the first state information of the target robot includes, but is not limited to, the robot's center of mass position, velocity information, torso posture, angular velocity information, and support state information of both feet.
[0029] It should be noted that after acquiring the first state information of the target robot in real time, the algorithm may further include: acquiring the first motion state information of the preset movable chassis in real time. Specifically, the input to the reinforcement learning algorithm includes not only the first state information of the target robot, but also the current first motion state information of the preset movable chassis. The first motion state information includes, but is not limited to, translational motion, rotational motion, acceleration / deceleration motion, and periodic motion.
[0030] Step S12: Using a reinforcement learning algorithm, and based on the first state information and a preset reward function, train the target robot to stand up and control, so as to obtain a corresponding target standing control strategy; wherein, the preset reward function is constructed based at least on the stable relationship between the centroid position of the target robot and the supporting polygon.
[0031] In this embodiment, the desired joint or geometric relationship described by traditional virtual constraints is introduced as a soft constraint into the reward function to guide the robot to form a stable posture that conforms to physical intuition during reinforcement learning. It should be noted that the virtual constraint is not applied as a rigid control condition, but is integrated into the reinforcement learning process in the form of reward guidance, enabling the robot to autonomously form a posture and joint coordination relationship that meets stability requirements during the learning process. The specific process may include: first, determining preset joint relationships and preset geometric relationships; the preset joint relationships are the joint relationships corresponding to when the target robot maintains a stable standing position, and the preset geometric relationships are the geometric relationships between the limbs when the target robot maintains a stable standing position; then, generating constraint reward terms corresponding to the preset joint relationships and preset geometric relationships, and constructing a preset reward function based on the constraint reward terms.
[0032] It should be noted that the process of training the target robot to stand using a reinforcement learning algorithm and based on the first state information and a preset reward function may include: determining the projection range corresponding to the supporting polygon information; if the centroid position is within the projection range and the distance between the centroid position and the boundary of the projection range is greater than or equal to a preset distance threshold, then generating a corresponding first positive reward using the preset reward function; if the centroid position is within the projection range and the distance between the centroid position and the boundary of the projection range is less than the preset distance threshold, then generating a corresponding first negative reward using the preset reward function; if the centroid position is not within the projection range, then generating a corresponding second negative reward using the preset reward function; and training the target robot to stand based on the first positive reward, the first negative reward, and the second negative reward. Specifically, in this embodiment, a centroid stability reward is introduced into the preset reward function to provide a positive reward when the robot's centroid position is within the projection range corresponding to the supporting polygon, and to penalize when the centroid approaches or exceeds the boundary of the projection range.
[0033] It should be noted that the process of training the target robot to stand using a reinforcement learning algorithm and based on the first state information and a preset reward function may further include: if the torso posture is a preset upright posture, then a corresponding second positive reward is generated using the preset reward function; if the joint output torque of the target robot exceeds a preset torque threshold, then a corresponding third negative reward is generated using the preset reward function; if the joint change angle of the target robot exceeds a preset angle threshold, then a corresponding fourth negative reward is generated using the preset reward function; if the target robot falls, then the current round of standing control training is stopped, and a corresponding fifth negative reward is generated using the preset reward function; standing control training of the target robot is performed based on the second positive reward, the third negative reward, the fourth negative reward, and the fifth negative reward. Specifically, in this embodiment, energy consumption and smoothness penalties are introduced into the preset reward function to penalize excessive joint output or drastic changes, thereby improving control smoothness. In addition, in this embodiment, a failure penalty mechanism is also introduced into the preset reward function to immediately terminate the current training round and impose a penalty when the robot falls or becomes significantly unstable. Meanwhile, the embodiment also introduces a posture maintenance reward into the preset reward function to encourage the robot's torso to maintain a near-preset upright posture, thereby suppressing excessive pitch and roll angle changes.
[0034] In this embodiment, a reinforcement learning algorithm is used to train the target robot to stand upright, thereby obtaining the target robot's standing upright control strategy under chassis motion conditions.
[0035] Step S13: Deploy the target standing control strategy to the target robot, and acquire the second state information of the target robot in real time during the application of the target robot; wherein the target robot stands on the target movable chassis.
[0036] In this embodiment, the target standing control strategy obtained through reinforcement learning training can be directly deployed in a real robot system. During the application of the target robot, the second state information of the target robot is acquired in real time. The specific process may include: acquiring the second state information of the target robot on the target movable chassis in real time, and acquiring the second motion state information of the target movable chassis, so as to use the target standing control strategy to generate the target standing control command based on the second state information and the second motion state information.
[0037] Specifically, during the application of the target robot, the second state information of the target robot on the target movable chassis can be acquired in real time, including but not limited to the target robot's velocity information, center of mass position, torso posture, angular velocity information, contact state information of both feet, and support polygon information. At the same time, the second motion state information of the target movable chassis can be acquired, including but not limited to translational motion, rotational motion, acceleration / deceleration motion, and periodic motion.
[0038] Step S14: Using the target standing control strategy, generate corresponding target standing control instructions based on the second state information, and control the target robot to execute the target standing control instructions; the target standing control instructions are used to control any one or a combination of several of the torque, position, and speed of the target robot; wherein, the target standing control strategy is used to adjust the posture and center of gravity distribution of the target robot during the movement of the target movable chassis.
[0039] In this embodiment, during the application of the target robot, the target standing control strategy can output target standing control commands in real time based on the second state information of the target robot and the second motion state information of the target movable chassis, and control the target robot to execute the target standing control commands, thereby achieving stable standing control. It should be noted that the target standing control command is at least one of torque commands, position commands, or velocity commands. Therefore, in this embodiment, stable standing control of the robot can be achieved through state observation → standing control strategy decision → execution control → physical feedback. Furthermore, standing control can be extended to integrated standing-transition-walking control. During chassis movement, the standing control strategy adjusts the robot's torso posture and center of mass distribution in real time, enabling the robot to maintain stable standing under unstable base conditions.
[0040] As can be seen from the above, in this embodiment, the target robot is first trained to stand in a preset physical simulation environment, and the first state information of the target robot is acquired in real time during the training process. A reinforcement learning algorithm is then used to train the target robot to stand based on the first state information and a preset reward function to obtain a corresponding target standing control strategy. The preset reward function is constructed at least based on the stable relationship between the target robot's center of mass position and the supporting polygon. The target standing control strategy is then deployed to the target robot, and the second state information of the target robot is acquired in real time during its application. The target robot stands on a target movable chassis. Finally, the target standing control strategy is used to generate corresponding target standing control commands based on the second state information, and the target robot is controlled to execute these commands. The target standing control commands are used to control any one or a combination of torque, position, and velocity of the target robot. The target standing control strategy is used to adjust the posture and center of gravity distribution of the target robot during the movement of the target movable chassis. As can be seen from the above, in this embodiment, the target robot's standing control is first trained in a preset physical simulation environment. During the training process, the robot's first state information is collected in real time. Then, based on a reinforcement learning algorithm, the collected first state information and a preset reward function are combined to train a suitable target standing control strategy. Afterward, the target standing control strategy is deployed to the target robot. During the actual application phase, the second state information is acquired in real time. Then, based on the second state information, the deployed target standing control strategy generates corresponding target standing control commands to control the robot to maintain a stable standing position. In this way, from simulation training to actual deployment, this embodiment allows the robot to utilize the trained target standing control strategy to dynamically output suitable control commands based on real-time state information, precisely adjust joint-related torques, positions, or speeds, effectively cope with various working conditions during movement, and achieve stable standing of the robot in complex scenarios such as chassis movement and external disturbances, significantly improving the robustness and adaptability of the robot's standing control. Meanwhile, this embodiment does not require precise modeling of chassis dynamics, resulting in greater adaptability of the target standing control strategy. It can function across various robot platforms and chassis types, ensuring the safety and reliability of the robot's standing in real, complex environments. In some implementations, the standing control method can serve as a basic control module for the robot to perform actions such as mounting and dismounting from the chassis, switching between walking and standing, or recovering from falls. It can work in conjunction with other motion control strategies to achieve continuous and stable control of the robot in complex task scenarios.
[0041] Accordingly, see Figure 2As shown in the illustration, this application also provides a robot standing control device, which may include: The first state information acquisition module 11 is used to train the target robot to stand in a preset physical simulation environment and to acquire the first state information of the target robot in real time during the training process. The standing control strategy determination module 12 is used to train the target robot to stand using a reinforcement learning algorithm based on the first state information and a preset reward function, so as to obtain a corresponding target standing control strategy; wherein, the preset reward function is constructed based at least on the stable relationship between the centroid position of the target robot and the supporting polygon; The second state information acquisition module 13 is used to deploy the target standing control strategy to the target robot and acquire the second state information of the target robot in real time during the application of the target robot; wherein the target robot stands on the target movable chassis; The standing control command generation module 14 is used to generate corresponding target standing control commands based on the second state information using the target standing control strategy, and to control the target robot to execute the target standing control commands; the target standing control commands are used to control any one or a combination of several of the torque, position, and speed of the target robot; wherein, the target standing control strategy is used to adjust the posture and center of gravity distribution of the target robot during the movement of the target movable chassis.
[0042] In some specific embodiments, the preset physical simulation environment includes a preset movable chassis, and the target robot stands on the preset movable chassis.
[0043] In some specific embodiments, the robot standing control device may further include: The disturbance application module is used to control the preset movable chassis to move based on a preset motion pattern in the preset physical simulation environment, and to apply external force disturbance to the preset movable chassis and / or the target robot, so as to obtain the first state information of the target robot on the preset movable chassis in real time; wherein, the preset motion pattern includes any one or a combination of several of translational motion, rotational motion, acceleration and deceleration motion and periodic motion; Accordingly, the first status information acquisition module 11 may include: The first state information acquisition unit is used to acquire in real time the target robot's speed information, center of mass position, torso posture, angular velocity information, contact state information of both feet, and supporting polygon information.
[0044] In some specific embodiments, the robot standing control device may further include: The first motion state information acquisition module is used to acquire the first motion state information of the preset movable chassis in real time, so as to use the reinforcement learning algorithm and perform standing control training on the target robot based on the first state information, the first motion state information and the preset reward function to obtain the target standing control strategy. Correspondingly, the second status information acquisition module 13 may include: The second state information acquisition unit is used to acquire the second state information of the target robot on the target movable chassis in real time, and to acquire the second motion state information of the target movable chassis, so as to generate the target standing control command based on the second state information and the second motion state information using the target standing control strategy.
[0045] In some specific embodiments, the robot standing control device may further include: The joint relationship determination module is used to determine preset joint relationships and preset geometric relationships; the preset joint relationships are the joint relationships corresponding to when the target robot maintains a stable standing position, and the preset geometric relationships are the geometric relationships between the limbs when the target robot maintains a stable standing position. The preset reward function construction module is used to generate constraint reward terms corresponding to the preset joint relationship and the preset geometric relationship, and to construct the preset reward function based on the constraint reward terms.
[0046] In some specific embodiments, the standing control strategy determination module 12 may include: The projection range determination unit is used to determine the projection range corresponding to the supporting polygon information; The first positive reward generation unit is used to generate a corresponding first positive reward using the preset reward function if the centroid position is located within the projection range and the distance between the centroid position and the boundary of the projection range is greater than or equal to a preset distance threshold. The first negative reward generation unit is used to generate a corresponding first negative reward using the preset reward function if the centroid position is located within the projection range and the distance between the centroid position and the boundary of the projection range is less than the preset distance threshold. The second negative reward generation unit is used to generate a corresponding second negative reward using the preset reward function if the centroid position is not within the projection range. The first standing control training unit is used to train the target robot to stand control based on the first positive reward, the first negative reward, and the second negative reward.
[0047] In some specific embodiments, the standing control strategy determination module 12 may include: The second positive reward generation unit is used to generate a corresponding second positive reward using the preset reward function if the torso posture is a preset upright posture. The third negative reward generation unit is used to generate a corresponding third negative reward using the preset reward function if the joint output torque of the target robot exceeds a preset torque threshold. The fourth negative reward generation unit is used to generate a corresponding fourth negative reward using the preset reward function if the joint change angle of the target robot exceeds a preset angle threshold. The fifth negative reward generation unit is used to stop the current round of standing control training if the target robot falls, and to generate a corresponding fifth negative reward using the preset reward function. The second standing control training unit is used to train the target robot to stand control based on the second positive reward, the third negative reward, the fourth negative reward, and the fifth negative reward.
[0048] Furthermore, embodiments of this application also disclose an electronic device, Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the robot standing control method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.
[0049] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.
[0050] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.
[0051] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the robot standing control method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.
[0052] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed robot standing control method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.
[0053] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.
[0054] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0055] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0056] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0057] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A robot standing control method, characterized in that, include: The target robot is trained to stand and control in a preset physical simulation environment, and the first state information of the target robot is acquired in real time during the training process. The target robot is trained to stand upright using a reinforcement learning algorithm based on the first state information and a preset reward function, thereby obtaining a corresponding target standing control strategy; wherein the preset reward function is constructed based at least on the stable relationship between the centroid position of the target robot and the supporting polygon; The target standing control strategy is deployed to the target robot, and the second state information of the target robot is acquired in real time during the application of the target robot; wherein, the target robot stands on the target movable chassis; Using the target standing control strategy, a corresponding target standing control command is generated based on the second state information, and the target robot is controlled to execute the target standing control command; the target standing control command is used to control any one or a combination of several of the torque, position, and speed of the target robot; wherein, the target standing control strategy is used to adjust the posture and center of gravity distribution of the target robot during the movement of the target movable chassis.
2. The robot standing control method according to claim 1, characterized in that, The preset physical simulation environment includes a preset movable chassis, and the target robot stands on the preset movable chassis.
3. The robot standing control method according to claim 2, characterized in that, Before acquiring the first state information of the target robot in real time, the method further includes: In the preset physical simulation environment, the preset movable chassis is controlled to move based on a preset motion pattern, and external force disturbances are applied to the preset movable chassis and / or the target robot in order to obtain the first state information of the target robot on the preset movable chassis in real time; wherein, the preset motion pattern includes any one or a combination of several of translational motion, rotational motion, acceleration and deceleration motion and periodic motion; Accordingly, the real-time acquisition of the first state information of the target robot includes: The system acquires in real time the target robot's velocity, center of mass position, torso posture, angular velocity, foot contact status, and support polygon information.
4. The robot standing control method according to claim 3, characterized in that, After acquiring the first state information of the target robot in real time, the method further includes: The first motion state information of the preset movable chassis is acquired in real time so as to utilize the reinforcement learning algorithm and train the target robot to stand up based on the first state information, the first motion state information and the preset reward function to obtain the target standing control strategy. Accordingly, the real-time acquisition of the second state information of the target robot includes: The second state information of the target robot on the target movable chassis is acquired in real time, and the second motion state information of the target movable chassis is acquired, so as to generate the target standing control command based on the second state information and the second motion state information using the target standing control strategy.
5. The robot standing control method according to claim 1, characterized in that, Before training the target robot to stand control using a reinforcement learning algorithm based on the first state information and a preset reward function, the method further includes: Determine preset joint relationships and preset geometric relationships; the preset joint relationships are the joint relationships corresponding to when the target robot maintains a stable standing position, and the preset geometric relationships are the geometric relationships between the limbs when the target robot maintains a stable standing position. Generate constraint reward terms corresponding to the preset joint relationships and the preset geometric relationships, and construct the preset reward function based on the constraint reward terms.
6. The robot standing control method according to claim 3, characterized in that, The process of training the target robot to stand with control using a reinforcement learning algorithm based on the first state information and a preset reward function includes: Determine the projection range corresponding to the supporting polygon information; If the centroid is located within the projection range, and the distance between the centroid and the boundary of the projection range is greater than or equal to a preset distance threshold, then the preset reward function is used to generate a corresponding first positive reward. If the centroid is located within the projection range, and the distance between the centroid and the boundary of the projection range is less than the preset distance threshold, then the preset reward function is used to generate a corresponding first negative reward. If the centroid position is not within the projection range, then the preset reward function is used to generate a corresponding second negative reward; The target robot is trained to stand up based on the first positive reward, the first negative reward, and the second negative reward.
7. The robot standing control method according to claim 3 or 6, characterized in that, The process of training the target robot to stand with control using a reinforcement learning algorithm based on the first state information and a preset reward function includes: If the torso posture is a preset upright posture, then the preset reward function is used to generate a corresponding second positive reward; If the joint output torque of the target robot exceeds the preset torque threshold, a corresponding third negative reward is generated using the preset reward function. If the joint change angle of the target robot exceeds a preset angle threshold, a corresponding fourth negative reward is generated using the preset reward function. If the target robot falls, the standing control training for the current round is stopped, and the corresponding fifth negative reward is generated using the preset reward function; The target robot is trained to stand upright based on the second positive reward, the third negative reward, the fourth negative reward, and the fifth negative reward.
8. A robot standing control device, characterized in that, include: The first state information acquisition module is used to train the target robot to stand and control in a preset physical simulation environment, and to acquire the first state information of the target robot in real time during the training process. A standing control strategy determination module is used to train the target robot to stand using a reinforcement learning algorithm based on the first state information and a preset reward function, so as to obtain a corresponding target standing control strategy; wherein, the preset reward function is constructed based at least on the stable relationship between the centroid position of the target robot and the supporting polygon; The second state information acquisition module is used to deploy the target standing control strategy to the target robot and acquire the second state information of the target robot in real time during the application of the target robot; wherein the target robot stands on the target movable chassis; The standing control command generation module is used to generate corresponding target standing control commands based on the second state information using the target standing control strategy, and to control the target robot to execute the target standing control commands; the target standing control commands are used to control any one or a combination of several of the torque, position, and speed of the target robot; wherein, the target standing control strategy is used to adjust the posture and center of gravity distribution of the target robot during the movement of the target movable chassis.
9. An electronic device, characterized in that, The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the robot standing control method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the robot standing control method as described in any one of claims 1 to 7.