A quadruped single-arm robot motion control method, system and readable storage medium

Abstract

The present invention provides a quadruped single-arm robot motion control method, system and readable storage medium, which relate to the technical field of quadruped single-arm robot motion control, wherein the method comprises the following steps: obtaining command instruction information data and self-state information data of the quadruped single-arm robot during the motion process; constructing a motion control strategy, inputting the command instruction information data and the self-state information data into the motion control strategy to obtain corresponding target joint angle data; constructing a value evaluation strategy, inputting the command instruction information data and the self-state information data into the value evaluation strategy to obtain corresponding cumulative reward expected value data, and the like; the present invention adopts an integrated motion control strategy to explore the foot-arm coordination control capability of the quadruped single-arm robot during the motion process, and finally verifies the effectiveness of the control strategy in a simulation environment.

Description

Technical Field

[0001] The present invention relates to the technical field of motion control of a quadruped single-arm robot, and more particularly to a motion control method, system and readable storage medium of a quadruped single-arm robot. Background Art

[0002] Quadruped single-arm robots, characterized by static stability, high payload capacity, and high robustness, have long been a research hotspot in the field. However, the functions that quadruped single-arm robots, which rely solely on legs, can perform are significantly limited. Even the most basic tasks, beyond visual inspection, require some form of manipulation, which has led to extensive research on quadruped single-arm robots. By adding a robotic arm to their back, quadruped single-arm robots can possess even more powerful manipulation capabilities. In daily production and development, quadruped single-arm robots can use their robotic arms to grasp and assemble parts, improving production efficiency; they can also use them for search and rescue operations; and they can use their arms to transfer objects to other quadruped single-arm robots or humans, enabling division of labor and collaboration. Combining the stability of quadruped single-arm robots with the flexibility of robotic arms, quadruped single-arm robots can adapt to various complex environments and accomplish a variety of tasks.

[0003] Currently, when quadrupedal, single-arm robots move, the robotic arm is often treated as a burden, fixed to the robot's back in a certain posture to avoid interfering with the quadruped's movement. However, in nature, the tail often serves as an auxiliary weapon for movement. For example, when a cat walks between steep rooftops, it often swings its tail from side to side to balance its body, and cheetahs often use their tails to complete sharp turns at high speeds. Therefore, it is necessary to explore the coordination between the foot and arm of quadrupedal, single-arm robots during movement.

[0004] With the development of machine learning technology, reinforcement learning has achieved remarkable results in various fields. Applying reinforcement learning to the motion control of quadruped single-arm robots, through trial-and-error and feedback mechanisms, can achieve coordinated control between the legs and the robotic arm, achieving low-energy, highly dynamic, and robust control, and making adaptive decisions even in the face of unknown situations. However, current reinforcement learning-based motion control methods for quadruped single-arm robots mostly employ "semi-coupling" or gradually increasing controllable degrees of freedom for training, which does not fully leverage the advantages of reinforcement learning and can easily lead to a lack of coordination between the legs and the robotic arm, as well as slow, jerky, and unnatural movements. Coordinated motion control of the legs and arms is crucial for the movement of quadruped single-arm robots.

[0005] Therefore, how to provide a quadruped single-arm robot motion control method that can solve the above problems is an issue that technicians in this field urgently need to solve. Summary of the Invention

[0006] In view of this, the present invention provides a motion control method, system and readable storage medium for a quadruped single-arm robot. It adopts an integrated motion control strategy to explore the leg-arm coordination control capability of the quadruped single-arm robot during movement, and finally verifies the effectiveness of the control strategy in a simulation environment.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A motion control method for a quadruped single-arm robot comprises the following steps:

[0009] Obtain command information data and self-status information data of the quadruped single-arm robot during its movement;

[0010] Constructing a motion control strategy, inputting the command instruction information data and the self-state information data into the motion control strategy to obtain corresponding target joint angle data;

[0011] Constructing a value evaluation strategy, inputting the command instruction information data and the self-state information data into the value evaluation strategy to obtain corresponding cumulative reward expected value data;

[0012] Inputting the target joint angle data into the quadruped single-arm robot for motion control and calculating the instant reward value data;

[0013] Adjust and optimize the motion control strategy and value evaluation strategy parameters based on the accumulated reward expectation value data and immediate reward value data;

[0014] Repeat the above process until the quadruped single-arm robot reaches the ideal operating state in the simulation environment;

[0015] A state assessment strategy is constructed to evaluate the linear velocity of the quadruped single-arm robot, and the state assessment strategy is trained using supervised learning in a simulation environment.

[0016] The motion of the quadruped single-arm robot is controlled using optimized motion control strategy and state evaluation strategy.

[0017] Preferably, the specific process of calculating the instant reward value data includes:

[0018] Inputting the target joint angle data into the quadruped single-arm robot for motion control, and obtaining the current state data of the quadruped single-arm robot according to the interaction between the quadruped single-arm robot and the simulation environment;

[0019] Construct a reward function, input the target joint angle data and the current state data of the quadruped single-arm robot into the reward function for calculation, and obtain the corresponding immediate action reward value.

[0020] Preferably, the specific expression of the reward function is:

[0021] r l =r vel -k c (r clear +r slip +r base +r vz +r joint +r smooth )

[0022] Where r l represents the reward function, r vel represents the speed following term, r clear represents the leg-lifting height term, r slip Represents the leg sliding term, r base represents the body orientation term, r vz represents the body velocity term, r joint represents the joint angle term, r smooth represents the motion smoothing term, k c Represents the course factor.

[0023] Preferably, the training process of the motion control strategy and the value evaluation strategy includes:

[0024] Using a proximal strategy optimization algorithm to train the motion control strategy and the value evaluation strategy;

[0025] The trained motion control strategy is used to collect simulated motion data of the quadruped single-arm robot in a simulation environment, and the state evaluation strategy is trained using a supervised learning method.

[0026] Preferably, the motion control strategy, the value evaluation strategy and the state assessment strategy are all neural networks.

[0027] The present invention also provides a quadruped single-arm robot motion control system, comprising:

[0028] An acquisition module is used to obtain command information data and its own status information data during the movement of the quadruped single-arm robot;

[0029] A first calculation module is used to construct a motion control strategy, input the command instruction information data and the self-state information data into the motion control strategy to obtain corresponding target joint angle data;

[0030] Constructing a value evaluation strategy, inputting the command instruction information data and the self-state information data into the value evaluation strategy to obtain corresponding cumulative reward expected value data;

[0031] A second calculation module is used to input the target joint angle data into the quadruped single-arm robot for motion control and calculate the immediate reward value data;

[0032] An optimization module is used to adjust and optimize the motion control strategy and value evaluation strategy parameters based on the obtained cumulative reward expectation value data and instant reward value data;

[0033] An adjustment module is used to repeat the above process until the quadruped single-arm robot reaches an ideal operating state in the simulation environment;

[0034] The training module is used to build a state assessment strategy to evaluate the linear velocity of the quadruped single-arm robot and train the state assessment strategy using supervised learning in a simulation environment;

[0035] The control module is used to control the motion of the quadruped single-arm robot using an optimized motion control strategy and a state evaluation strategy.

[0036] The present invention also provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the quadruped single-arm robot motion control method as described above is implemented.

[0037] The above technical solution demonstrates that, compared to existing technologies, the present invention provides a motion control method, system, and readable storage medium for a quadruped, single-arm robot. This method utilizes an integrated motion control strategy to explore the role of the robotic arm in promoting the movement of the quadruped, single-arm robot. This integrated motion control strategy is trained using reinforcement learning. From the outset, the robotic arm is trained dynamically, without requiring any prior knowledge. The algorithm's reliability is ultimately verified in a simulation environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0039] Figure 1 This is an overall flow chart of a motion control method for a quadruped single-arm robot provided by the present invention;

[0040] Figure 2 A schematic diagram of the integrated motion control framework provided by the present invention;

[0041] Figure 3 This is a structural principle block diagram of the motion control of a quadruped single-arm robot provided by the present invention. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0043] See also Figure 1-2 As shown, an embodiment of the present invention provides a motion control method for a quadruped single-arm robot, comprising the following steps:

[0044] Obtain the command information data and its own state information data s of the quadruped single-arm robot at time t during its movement t , wherein the self-state information data may include: body state information data and joint state information data, and the body state information data s t Can include: quadruped single-arm robot posture Angular velocity of the quadruped single-arm robot And the linear speed of the quadruped single-arm robot The posture and angular velocity of the quadruped single-arm robot can be directly measured by the inertial measurement unit (IMU). The joint state information data can include: the current joint angle q t ∈R 18 , joint speed and target joint angle error The joint angle and speed information can be directly obtained from the joint encoder. The command information data is randomly generated during the training process, mainly including the forward speed of the quadruped single-arm robot. lateral speed and yaw rate

[0045] Construct a motion control strategy, input command information data and its own state information data into the motion control strategy to obtain the corresponding target joint angle data;

[0046] Construct a value evaluation strategy, input command information data and its own status information data into the value evaluation strategy to obtain the corresponding cumulative reward expected value data;

[0047] The target joint angle data is input into the quadruped single-arm robot for motion control and the immediate reward value data is calculated;

[0048] Adjust and optimize the motion control strategy and value evaluation strategy parameters based on the accumulated reward expectation value data and immediate reward value data;

[0049] Repeat the above process until the quadruped single-arm robot reaches the ideal operating state in the simulation environment;

[0050] A state assessment strategy is constructed to evaluate the linear velocity of the quadruped single-arm robot, and the state assessment strategy is trained using supervised learning in a simulation environment.

[0051] The motion of the quadruped single-arm robot is controlled using optimized motion control strategy and state evaluation strategy.

[0052] Specifically, the specific process of obtaining the target joint angle data may include:

[0053] With the body status information data s t As the input of the motion control strategy π, the output of each joint relative to the initial joint angle q init The displacement Δq t , thereby obtaining the target joint angle data The specific expression is:

[0054] π(s t )=Δq t

[0055]

[0056] Where λ q It can be taken as 0.3 to scale the output of the neural network, which is conducive to stabilizing the state of the quadruped single-arm robot and accelerating the convergence of the strategy.

[0057] In a specific embodiment, the specific process of calculating the instant reward value data includes:

[0058] The target joint angle data is input into the quadruped single-arm robot for motion control, and the current state data of the quadruped single-arm robot is obtained based on the interaction between the quadruped single-arm robot and the simulation environment;

[0059] Construct a reward function, input the target joint angle data and the current state data of the quadruped single-arm robot into the reward function for calculation, and obtain the corresponding immediate action reward value.

[0060] In a specific embodiment, the specific expression of the reward function is:

[0061] r l=r vel -k c (r clear +r slip +r base +r vz +r joint +r smooth )

[0062] Where r l represents the reward function, r vel represents the speed following term, r clear represents the leg-lifting height term, r slip Represents the leg sliding term, r base represents the body orientation term, r vz represents the body velocity term, r joint represents the joint angle term, r smooth represents the motion smoothing term, k c Represents the course factor.

[0063] Specifically, the embodiment of the present invention uses reinforcement learning to maximize the reward expectation To train the motion control strategy π, T represents the end time of each training round, r t represents the reward at time t, which is a discount coefficient and is usually less than 1. The form of the reward function often determines the type of task. The purpose of the reward function designed in this embodiment of the present invention is to encourage the quadruped single-arm robot to follow the command instructions and punish unnatural and unstable movements. The linear velocity V is defined as x,y,z 、Body angular velocity ω x,y,z 、Body angle θ x,y,z , the height p of the end of the i-th leg z,i , the linear velocity of the end of the i-th leg in the x and y directions Joint angle q, initial joint angle q init and the target joint angle At time t, the reward function defined in this embodiment of the present invention is:

[0064] r l =r vel -(r clear +r slip +r base +r vz +r joint +r smooth )

[0065]

[0066]

[0067]

[0068] r base =c orn (||θ x,y || 2 +c z θ z 2 )

[0069]

[0070] r joint =c q ||q t -q init || 2

[0071]

[0072] where c z It is an indicator of whether the body is deflected. When the body is deflected, c z =0, otherwise c z =1,c i is the indicator of the contact between the i-th leg and the ground. When the end of the foot touches the ground, c i =1, otherwise c i =0, is the maximum leg lift height at the foot end, set it to 0.07, c vel 、c clear 、c slip 、c orn 、c vz 、c q 、c ca1 and c ca2 The parameter values ​​are 9, 100, 0.07, 3, 1.2, 0.5, 2.5 and 1.5 respectively.

[0073] If the above reward function is used to train a quadruped single-arm robot directly at the beginning of training, the penalty term in the reward function will penalize the movement of the joints, and the quadruped single-arm robot will stay still. Therefore, the concept of curriculum is introduced. The present invention multiplies the penalty term in the reward function by a curriculum factor k c ∈[0,1], the course factor is a relatively small value at the beginning of training and increases linearly as training progresses, so our reward function becomes:

[0074] r l =r vel -k c (r clear +r slip +r base +r vz +r joint +r smooth)

[0075] In the early stage of training, the movement of the quadruped single-arm robot is encouraged. As the training progresses, the penalty term is used to adjust and optimize the movement of the quadruped single-arm robot.

[0076] The goal of the embodiment of the present invention is to train a quadruped single-arm robot motion control strategy, which can control the quadruped single-arm robot to move according to the command instructions. The sampling range of the command instructions is as well as At this point, the course factor k can be introduced g ∈[0,1] controls the sampling range of command instructions. In the early stage of training, fixed command instructions are trained to first let the quadruped single-arm robot learn to move without deflection at a lower speed, and then control k g Gradually increase the sampling range of command instructions to allow the quadruped single-arm robot to learn more actions. If the quadruped single-arm robot command instructions are trained with a complete range sampling at the beginning of training, the quadruped single-arm robot will not learn any actions due to the diversity of tasks.

[0077] Deploying the strategy trained in a simulation environment to a physical quadruped single-arm robot often results in unsatisfactory deployment results due to the gap between simulation and reality. To solve this problem, the embodiment of the present invention randomizes the dynamic parameters of the quadruped single-arm robot during training. The dynamic parameters mainly include the ground friction coefficient, the mass of each part of the quadruped single-arm robot, and the maximum torque of the joints. At this time, a new curriculum factor k can be introduced noise ∈[0,1] controls the sampling range of dynamic parameters, and the operation is related to k g Similarly, the dynamic parameters of the quadruped single-arm robot are fixed at the beginning of training, and the quadruped single-arm robot is first trained to move with fixed parameters, and then the k noise Gradually increase the sampling range of dynamic parameters to increase the robustness of the resulting strategy and narrow the gap between simulation and reality.

[0078] Also introduce the course factor k terrain ∈[0,1] controls the flatness of the terrain in the simulation environment, so that the strategy can be learned in more complex terrains. In the early stage of training, the quadruped single-arm robot is trained on a flat ground. As the training progresses, the k terrain Gradually increasing the steepness of the terrain in the simulation environment allows the quadruped single-arm robot to learn through more complex interactions to cope with complex terrain in the real environment.

[0079] In a specific embodiment, the training process of the motion control strategy and the value evaluation strategy includes:

[0080] Use proximal strategy optimization algorithm to train motion control strategy and value evaluation strategy;

[0081] The trained motion control strategy is used to collect simulated motion data of a quadruped single-arm robot in a simulation environment, and a supervised learning method is used to train the state evaluation strategy.

[0082] In a specific embodiment, the motion control strategy, the value evaluation strategy, and the state assessment strategy are all neural networks.

[0083] Specifically, the motion control strategy and the value evaluation strategy have similar structures, both of which are multi-layer perceptrons (MLPs) with 3 hidden layers. The dimensions of each layer are 256, 128, and 32 respectively, and the activation function is the ReLu function.

[0084] The state evaluation strategy is designed as a multi-layer perceptron (MLP) with two hidden layers. The dimensions of each layer are 256 and 128 respectively. The activation function is also the ReLu function. The output is the linear velocity of the quadruped single-arm robot in three directions. It is trained using supervised learning.

[0085] Training the aforementioned strategies in a simulation environment consists of two main phases. First, a proximal policy optimization algorithm is used to train the motion control and value evaluation strategies. The linear velocity of the robot, input to the neural network, is derived from the real-world values ​​obtained in the simulation environment. After training, the trained motion control strategies are used to collect data in the simulation environment. A supervised learning approach is then used to train the state evaluation strategy, enabling it to estimate the linear velocity of the robot based on the robot's latest state. During deployment of the physical quadruped robot, the robot's latest state and the output of the state evaluation strategy are used as input to the motion control strategy, enabling it to drive the robot to complete the desired motion.

[0086] See also Figure 3 As shown, an embodiment of the present invention further provides a control system using a quadruped single-arm robot motion control method according to any one of the above embodiments, comprising:

[0087] An acquisition module is used to obtain command information data and its own status information data during the movement of the quadruped single-arm robot;

[0088] The first calculation module is used to construct a motion control strategy, input command instruction information data and its own state information data into the motion control strategy to obtain corresponding target joint angle data;

[0089] Construct a value evaluation strategy, input command information data and its own status information data into the value evaluation strategy to obtain the corresponding cumulative reward expected value data;

[0090] The second calculation module is used to input the target joint angle data into the quadruped single-arm robot for motion control and calculate the immediate reward value data;

[0091] An optimization module is used to adjust and optimize the motion control strategy and value evaluation strategy parameters based on the obtained cumulative reward expectation value data and instant reward value data;

[0092] An adjustment module is used to repeat the above process until the quadruped single-arm robot reaches an ideal operating state in the simulation environment;

[0093] The training module is used to build a state assessment strategy to evaluate the linear velocity of the quadruped single-arm robot and train the state assessment strategy using supervised learning in a simulation environment;

[0094] The control module is used to control the motion of the quadruped single-arm robot using an optimized motion control strategy and a state evaluation strategy.

[0095] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0096] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A motion control method for a quadruped single-arm robot, characterized in that: The following steps are involved: Obtain command information data and self-status information data of the quadruped single-arm robot during its movement; Constructing a motion control strategy, inputting the command instruction information data and the self-state information data into the motion control strategy to obtain corresponding target joint angle data; Constructing a value evaluation strategy, inputting the command instruction information data and the self-state information data into the value evaluation strategy to obtain corresponding cumulative reward expected value data; Inputting the target joint angle data into the quadruped single-arm robot for motion control and calculating the instant reward value data; Adjust and optimize the motion control strategy and value evaluation strategy parameters based on the accumulated reward expectation value data and immediate reward value data; Repeat the above process until the quadruped single-arm robot reaches the ideal operating state in the simulation environment; A state assessment strategy is constructed to evaluate the linear velocity of the quadruped single-arm robot, and the state assessment strategy is trained using supervised learning in a simulation environment. The motion of the quadruped single-arm robot is controlled using optimized motion control strategy and state evaluation strategy.

2. A quadruped single-arm robot motion control method according to claim 1, characterized in that: The specific process of calculating the instant reward value data includes: Inputting the target joint angle data into the quadruped single-arm robot for motion control, and obtaining the current state data of the quadruped single-arm robot according to the interaction between the quadruped single-arm robot and the simulation environment; Construct a reward function, input the target joint angle data and the current state data of the quadruped single-arm robot into the reward function for calculation, and obtain the corresponding immediate action reward value.

3. A quadruped single-arm robot motion control method according to claim 2, characterized in that: The specific expression of the reward function is: r l ＝r vel -k c (r clear +r slip +r base +r vz +r joint +r smooth ) Where r l represents the reward function, r vel represents the speed following term, r clear represents the leg-lifting height term, r slip Represents the leg sliding term, r base represents the body orientation term, r vz represents the body velocity term, r joint represents the joint angle term, r smooth represents the motion smoothing term, k c Represents the course factor.

4. The motion control method of a quadruped single-arm robot according to claim 1, characterized in that: The training process of movement control strategy and value evaluation strategy includes: Using a proximal strategy optimization algorithm to train the motion control strategy and the value evaluation strategy; The trained motion control strategy is used to collect simulated motion data of the quadruped single-arm robot in a simulation environment, and the state evaluation strategy is trained using a supervised learning method.

5. The motion control method of a quadruped single-arm robot according to claim 1, characterized in that: The motion control strategy, the value evaluation strategy and the state assessment strategy are all neural networks.

6. A control system using the quadruped single-arm robot motion control method according to any one of claims 1 to 5, characterized in that: include: An acquisition module is used to obtain command information data and its own status information data during the movement of the quadruped single-arm robot; A first calculation module is used to construct a motion control strategy, input the command instruction information data and the self-state information data into the motion control strategy to obtain corresponding target joint angle data; Constructing a value evaluation strategy, inputting the command instruction information data and the self-state information data into the value evaluation strategy to obtain corresponding cumulative reward expected value data; A second calculation module is used to input the target joint angle data into the quadruped single-arm robot for motion control and calculate the immediate reward value data; An optimization module is used to adjust and optimize the motion control strategy and value evaluation strategy parameters based on the obtained cumulative reward expectation value data and instant reward value data; An adjustment module is used to repeat the above process until the quadruped single-arm robot reaches an ideal operating state in the simulation environment; The training module is used to build a state assessment strategy to evaluate the linear velocity of the quadruped single-arm robot and train the state assessment strategy using supervised learning in a simulation environment; The control module is used to control the motion of the quadruped single-arm robot using an optimized motion control strategy and a state evaluation strategy.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the quadruped single-arm robot motion control method according to any one of claims 1 to 5.

Images