Musculoskeletal robot control method and system based on deep reinforcement learning
By constructing a workspace and proximal policy optimization model for musculoskeletal robots based on deep reinforcement learning, the problem of high control difficulty of musculoskeletal robots is solved, and efficient motion control and accurate end effector position tracking are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-26
- Publication Date
- 2026-03-10
AI Technical Summary
Existing technologies struggle to effectively control musculoskeletal robots due to the high redundancy, strong coupling, and complex tendon distribution, coupled with a lack of accurate geometric and dynamic models, resulting in poor control performance and low training efficiency.
A deep reinforcement learning-based approach is used to construct the workspace of the musculoskeletal robot end effector, which is divided into multiple subspaces. A Top-k hybrid expert network is designed as the proximal policy optimization model, and a motion reward function is constructed. The proximal policy optimization model is updated through position error for control.
It effectively reduces the difficulty of strategy optimization, improves training efficiency and control accuracy, optimizes the position tracking accuracy and speed of the end effector of musculoskeletal robots, and overcomes the challenges of multi-muscle influence and tendon complexity.
Smart Images
Figure CN121625128A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of musculoskeletal robot control technology, and relates to a musculoskeletal robot control method and system based on deep reinforcement learning. Background Technology
[0002] Humanoid robots, representing the development of new productive forces and pioneering future industries, represent a new high ground in technological competition and a new engine for economic development. However, existing "joint-link" humanoid robots, attempting to simulate humans in terms of movement and behavior, suffer from poor compliance and other shortcomings that severely restrict natural, interactive, empathetic, and collaborative relationships between robots and humans. On the other hand, "muscle-skeletal" humanoid robots, based on human anatomy and neural mechanisms, can simulate the human musculoskeletal system and neural mechanisms from the inside out. This helps to explore the essential mechanisms by which humans achieve highly dexterous, compliant, and intelligent behavior, and possesses characteristics such as lightweight, compliance, and safety. They hold promise for achieving a more efficient and stable interactive system closer to humans, becoming an important development trend in humanoid robots.
[0003] However, the high redundancy, strong coupling, and strong nonlinearity of musculoskeletal robots pose significant challenges to control. Due to their high redundancy, musculoskeletal robot control requires solving for high-dimensional muscle control signals based on low-dimensional motion objectives. Therefore, the muscle control signals for a specific motion have infinite solutions, making rapid solution and optimization of muscle control signals difficult. Furthermore, musculoskeletal robots exhibit strong coupling; the movement of one joint is influenced by multiple muscles, and the output force of each muscle affects the movement of multiple joints. Therefore, it is impossible to decompose the motion control of the entire robot into individual control of each muscle, further increasing the difficulty of control. In addition, inspired by the arrangement and dynamics of human muscles, the tendon distribution and power transmission pathways of some musculoskeletal robots are complex, with significant friction between tendons and bones and other contact objects, and some muscle modules exhibiting strong nonlinearity. Therefore, it is difficult to establish accurate geometric and dynamic models for such musculoskeletal robots. On the other hand, reinforcement learning, as a model-free, data-driven policy optimization method, has received widespread attention in the field of intelligent control in recent years. However, existing research mostly focuses on the control of relatively simplified musculoskeletal robots, resulting in ineffective control under complex configurations. Therefore, there is an urgent need to propose a motion control method for musculoskeletal robots that does not require a precise dynamic model, in order to meet the motion control requirements of highly complex musculoskeletal robots. Summary of the Invention
[0004] The purpose of this invention is to solve the problems in the prior art where the movement of joints is affected by multiple muscles, making it impossible to decompose the motion control of the entire robot into the individual control of each muscle, which increases the difficulty of control. Furthermore, due to the complexity of tendons, it is difficult to establish accurate geometric and dynamic models, resulting in poor control performance and low training efficiency of existing models. The invention provides a musculoskeletal robot control method and system based on deep reinforcement learning.
[0005] To achieve the above objectives, the present invention employs the following technical solution: A musculoskeletal robot control method based on deep reinforcement learning includes the following steps: Construct the workspace of the musculoskeletal robot end effector, divide the workspace into multiple subspaces, and obtain the target points for musculoskeletal robot training based on the subspaces; Construct a proximal strategy optimization model, and obtain the execution actions of the musculoskeletal robot end effector based on the target point and the proximal strategy optimization model; A motion reward function is constructed, and the positional error between the executed action and the target point is calculated based on the motion reward function. The proximal policy optimization model is updated according to the positional error, and the musculoskeletal robot is subjected to motion control based on the updated proximal policy optimization model.
[0006] A further improvement of the present invention is that: The workspace for constructing the musculoskeletal robot end effector is divided into multiple subspaces, and target points for musculoskeletal robot training are obtained based on these subspaces, including: Let R be the radius of a sphere with the work center as the origin. The workspace of the musculoskeletal robot's end effector is expressed by the following formula:
[0007] The workspace is divided into 8 sections using the coordinate axis plane. spherical subspace: .
[0008] The geometric center of the subspace is selected as the target point for musculoskeletal robot training, and the coordinates of the target point are: .
[0009] The construction of the proximal strategy optimization model, based on the target point and the proximal strategy optimization model, obtains the execution actions of the musculoskeletal robot end effector, including: Construct a Top-k hybrid expert network to serve as the Actor and Critic in the proximal policy optimization model; Obtaining environmental observations A single-layer fully connected gate network is constructed based on a hybrid expert network to integrate environmental observations. As input to the fully connected gate network, obtain the expert output weights. ; Select the k highest weights from all weights, and activate the expert sequence number based on these k weights. ; Based on expert serial number Activate the expert network; Environmental observations As input to the activated expert network, the output signal of the expert network is obtained; Output signal and weights based on expert network Obtain the execution actions of the end effector of the musculoskeletal robot.
[0010] The acquisition of environmental observations ,include: Obtain the joint angles of a musculoskeletal robot Joint angular velocity End effector position End effector speed The difference between the end effector position and the target position As environmental observation quantity .
[0011] The construction of the motion reward function, and the calculation of the positional error between the executed action and the target point based on the motion reward function, include: Sub-reward items are constructed based on the end-effector position error between the musculoskeletal robot and the target point, the end-effector velocity, the joint angular velocity, and the control signals:
[0012] Construct the total reward function based on the sub-reward items:
[0013] Where the subscript t indicates time t, Indicates joint angular velocity; Indicates the speed of the end effector; This indicates the difference between the position of the end effector and the target position. The output signal represents the control strategy; Indicates the desired end-point control accuracy; - This represents the weighting coefficient for each sub-reward item.
[0014] The step of updating the near-end strategy optimization model based on position error includes:
[0015] in, For the neural network parameters that need to be optimized, and For the new strategy and the old strategy, For the dominant function, This is the clipping function.
[0016] A musculoskeletal robot based on deep reinforcement learning executes the control method described in this invention.
[0017] A musculoskeletal robot control system based on deep reinforcement learning includes: The environment construction module is used to construct the workspace of the musculoskeletal robot end effector, divide the workspace into multiple subspaces, and obtain the target points for musculoskeletal robot training based on the subspaces. The model building module is used to build a proximal policy optimization model, and obtain the execution actions of the musculoskeletal robot end effector based on the target point and the proximal policy optimization model. The reward function construction module is used to construct the motion reward function, calculate the position error between the executed action and the target point based on the motion reward function, update the proximal policy optimization model according to the position error, and perform motion control on the musculoskeletal robot based on the updated proximal policy optimization model.
[0018] A terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the methods described in this invention.
[0019] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in any one of the present invention.
[0020] Compared with the prior art, the present invention has the following beneficial effects: This invention discloses a musculoskeletal robot control method based on deep reinforcement learning. It constructs a workspace for the musculoskeletal robot's end effector, dividing the workspace into multiple subspaces. This effectively reduces the difficulty of policy optimization while maintaining good generalization ability and improving training efficiency. The proximal policy optimization model further enhances training efficiency while maintaining control accuracy. Simultaneously, this invention constructs a motion reward function and updates the proximal policy optimization model based on position error to reduce end effector position error. This overcomes the problems of poor control performance and low training efficiency in musculoskeletal robots caused by the influence of multiple muscles and tendon complexity.
[0021] Furthermore, in this invention, a Top-k hybrid expert network is constructed as the Actor and Critic in the proximal policy optimization model. Based on the hybrid expert model, the proximal policy optimization algorithm utilizes a sparse activation hybrid expert network structure, combined with the proximal policy optimization algorithm. Compared to networks with the same number of parameters, this reduces computational cost, improves training efficiency, and maintains control accuracy. Furthermore, in this invention, the constructed reward function optimizes the end effector position tracking accuracy, end effector velocity, joint angular velocity, and control signal strength in the motion control of the upper limb musculoskeletal robot, thereby improving control accuracy in the strategy for training the end effector position tracking of the musculoskeletal robot. Attached Figure Description
[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0023] Figure 1 This is a flowchart of a method according to an embodiment of the present invention.
[0024] Figure 2 This is a schematic diagram of the training process and workspace division of the present invention.
[0025] Figure 3 This is a reinforcement learning training reward graph of the present invention.
[0026] Figure 4 This is a schematic diagram illustrating the control effect of the present invention.
[0027] Figure 5 This is a diagram of the end position tracking error of the present invention (where a represents a schematic diagram of the difference between the end effector position and the target position; b represents a schematic diagram of the difference in end effector speed). Detailed Implementation
[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.
[0029] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.
[0030] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0031] In the description of the embodiments of the present invention, it should be noted that if terms such as "upper," "lower," "horizontal," or "inner" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of the invention is in use, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Furthermore, terms such as "first" and "second" are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0032] Furthermore, the use of the term "horizontal" does not imply that the component must be absolutely horizontal, but rather that it can be slightly tilted. For example, "horizontal" simply means that its direction is more horizontal than "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted.
[0033] In the description of the embodiments of the present invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention according to the specific circumstances.
[0034] The present invention will now be described in further detail with reference to the accompanying drawings: See Figure 1 This invention discloses a musculoskeletal robot control method based on deep reinforcement learning, aiming to improve the design and training efficiency of musculoskeletal robot control strategies, and includes the following three aspects: Step S1: Establish the workspace partitioning of the musculoskeletal robot end effector to reduce the optimization space of the reinforcement learning algorithm and reduce the difficulty of policy convergence; Step S2: Design a proximal policy optimization algorithm based on a hybrid expert model to accelerate policy convergence and alleviate gradient conflicts in multiple tasks; Step S3: Construct a reinforcement learning reward function suitable for motion control of musculoskeletal robots to reduce end effector position error.
[0035] Specifically, the following steps are included: Step 1: Construct the workspace of the musculoskeletal robot end effector, divide the workspace into multiple subspaces, and obtain the target points for musculoskeletal robot training based on the subspaces; For details, see Figure 2 The workspace of the end effector of the musculoskeletal robot can be constructed by considering it as a sphere with the work center as the origin and R as the radius of the sphere. The boundary of the workspace can be represented as:
[0036] Furthermore, the workspace of the musculoskeletal robot's end effector is partitioned, dividing the workspace into 8 sections using the coordinate plane. Given a spherical subspace, with its geometric center as the target point for reinforcement learning training, the coordinates of these 8 target points satisfy the following:
[0037] Therefore, the coordinates of the target point are:
[0038] This embodiment establishes a method for partitioning the workspace of a musculoskeletal robot end effector, dividing the entire workspace into eight subspaces, which effectively reduces the difficulty of strategy optimization, maintains good generalization ability, and improves training efficiency.
[0039] Step 2: Construct a proximal policy optimization model, and obtain the execution actions of the musculoskeletal robot end effector based on the target point and the proximal policy optimization model; In this embodiment, a near-end policy optimization algorithm based on a hybrid expert model is designed. A Top-k hybrid expert network is constructed as the Actor and Critic in the near-end policy optimization algorithm. The detailed network architecture design and training process are given below: Step 2.1: Select the joint angles of the musculoskeletal robot Joint angular velocity End effector position End effector speed The difference between the end effector position and the target position As an environmental observation measure:
[0040] Step 2.2: As Figure 2 As shown, a hybrid expert network is used as both an Actor and a Critic to construct a single-layer fully connected gate network for environmental observations. As input, expert output weights As output:
[0041] Step 2.3: Output weights from experts Select the k highest-ranking experts and obtain their corresponding network indices. :
[0042] Step 2.4: According to Activate the expert network and increase environmental observations As input to each expert network, experts are independent of each other:
[0043] Step 2.5: Control signals output by each expert network Weights output by the SUM network The final control signals input to the musculoskeletal robot system are obtained as follows:
[0044] This invention designs a proximal policy optimization algorithm based on a hybrid expert model. It utilizes a network structure of sparse activation hybrid experts and combines it with a proximal policy optimization algorithm. Compared with a network with the same number of parameters, it reduces the amount of computation, improves training efficiency, and maintains its control accuracy.
[0045] Step 3: Construct a motion reward function, calculate the position error between the executed action and the target point based on the motion reward function, update the proximal policy optimization model according to the position error, and perform motion control on the musculoskeletal robot based on the updated proximal policy optimization model.
[0046] In this embodiment, a reinforcement learning reward function suitable for the motion control of musculoskeletal robots is constructed as the reward function used in the simulation environment during the reinforcement learning process in step 2. The network loss is calculated through the reward, specifically including the following steps: Step 3.1: As Figure 2 As shown, in the simulation environment, the control effect of the musculoskeletal robot on the control strategy generates a reward value.
[0047] The total reward consists of 6 sub-reward items, from arrive These are the end-effector position error term, joint angular velocity term, end-effector velocity term, control signal strength term, end-effector proximity to target point reward term, and end-effector position error integral term. Then, based on the end-effector position error... Terminal velocity Joint angular velocity and control signals The constructed sub-reward items are as follows:
[0048] Step 3.2: Calculate the total reward value by weighted summation based on each sub-reward item:
[0049] in, arrive These are the weighting parameters for each reward item.
[0050] Furthermore, the Proximal Policy Optimization (PPO) algorithm is used as the reinforcement learning algorithm. The loss is calculated using the reward function obtained in the simulation environment, and then backpropagation is used to optimize and update the network weights.
[0051] This invention constructs a reinforcement learning reward function suitable for the motion control of musculoskeletal robots. In the motion control of upper limb musculoskeletal robots, it optimizes the end effector position tracking accuracy, end effector velocity, joint angular velocity, and control signal strength. In the strategy for training the end effector position tracking of musculoskeletal robots, it improves control accuracy, reduces convergence difficulty, and increases training efficiency.
[0052] This invention also discloses a model verification process, specifically including: During training, the update learning rate of the proximal policy optimization algorithm is set to... The data experience pool capacity is set to The data batch size used for each update is set to [value]. Data utilization batch is The maximum step size is set to Activate expert numbers Set as Training was conducted on a computer equipped with an Intel Core i9-13900KF CPU and an Nvidia RTX4090 GPU.
[0053] The reward function curves generated during reinforcement learning training are compared with those generated by training using the Proximal Policy Optimization (PPO) algorithm and the Soft Actor-Critic (SAC) algorithm without using a hybrid expert model. Figure 3As shown, the algorithm using a hybrid expert model converges faster, demonstrating the effectiveness of the hybrid expert model in accelerating training.
[0054] In a simulation environment, the trained strategy is used to control the movement of a musculoskeletal robot from a random initial position to an arbitrarily selected target position, as follows: Figure 4 As shown in the figure, the end effector position error and end speed change over time during this process are as follows: Figure 5 As shown, the end effector position error and end velocity of the musculoskeletal robot converge quickly to near zero, demonstrating the accuracy of the policy in controlling the musculoskeletal robot.
[0055] This invention also discloses a musculoskeletal robot control system based on deep reinforcement learning, comprising: The environment construction module is used to construct the workspace of the musculoskeletal robot end effector, divide the workspace into multiple subspaces, and obtain the target points for musculoskeletal robot training based on the subspaces. The model building module is used to build a proximal policy optimization model, and obtain the execution actions of the musculoskeletal robot end effector based on the target point and the proximal policy optimization model. The reward function construction module is used to construct the motion reward function, calculate the position error between the executed action and the target point based on the motion reward function, update the proximal policy optimization model according to the position error, and perform motion control on the musculoskeletal robot based on the updated proximal policy optimization model.
[0056] A schematic diagram of a terminal device according to an embodiment of the present invention. The terminal device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the various method embodiments described above. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in the various device embodiments described above.
[0057] The computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention.
[0058] The terminal device can be a desktop computer, laptop computer, cloud server, or other device with strong computing power. The terminal device may include, but is not limited to, a processor and memory.
[0059] The optimal choice for the processor is a multi-core high-speed central processing unit (CPU).
[0060] The memory can be used to store the computer program and / or module. The processor implements various functions of the terminal device by running or executing the computer program and / or module stored in the memory and calling the data stored in the memory.
[0061] If the modules / units integrated into the terminal device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.
[0062] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for controlling a musculoskeletal robot based on deep reinforcement learning, characterized by, The method comprises the following steps: constructing a workspace of a musculoskeletal robot end effector, dividing the workspace into multiple subspaces, and obtaining target points for training of the musculoskeletal robot based on the subspaces; constructing a proximal strategy optimization model, and obtaining an execution action of the musculoskeletal robot end effector based on the target points and the proximal strategy optimization model; constructing a motion reward function, calculating a position error between the execution action and the target points based on the motion reward function, updating the proximal strategy optimization model according to the position error, and performing motion control on the musculoskeletal robot based on the updated proximal strategy optimization model.
2. The method of claim 1, wherein, The method of constructing a workspace of a musculoskeletal robot end effector, dividing the workspace into multiple subspaces, and obtaining target points for training of the musculoskeletal robot based on the subspaces comprises: a sphere is made with the work center as the origin and R as the radius of the sphere, and the workspace of the musculoskeletal robot end effector is expressed by the following formula: The workspace is divided into 8 subspaces of spherical shape: The geometric center of the selected subspace is selected as the target point for the musculoskeletal robot training, and the target point coordinates are .
3. The method of claim 1, wherein, The method of constructing a proximal strategy optimization model and obtaining an execution action of the musculoskeletal robot end effector based on the target points and the proximal strategy optimization model comprises: a Top-k hybrid expert network is constructed as an Actor and a Critic in the proximal strategy optimization model; Obtaining an environmental observation , constructing a one-layer full-connection gate network based on the mixed expert network, taking the environmental observation as the input of the full-connection gate network, and obtaining expert output weights ; selecting the k weights with the highest values among all the weights, and activating the expert serial number based on the k weights ; Based on the expert number Activate the expert network; obtaining environmental observations obtaining the output signal of the expert network as an input to the activated expert network Output signals and weights based on expert network An execution action of a musculoskeletal robot end effector is acquired.
4. The method of claim 3, wherein the method is based on deep reinforcement learning. The acquisition environment observation quantity comprises: Obtaining joint angles of a musculoskeletal robot joint angular velocities end effector positions end effector velocities and differences between end effector positions and target positions as environmental observations .
5. The method of claim 1, wherein, The method of constructing a motion reward function and calculating a position error between the execution action and the target points based on the motion reward function comprises: a sub-reward item is constructed according to an end position error, an end velocity, an angular velocity of a joint, and a control signal between the musculoskeletal robot end effector and the target point: a total reward function is constructed according to the sub-reward item: where subscript t represents the t time, denotes joint angular velocity; denotes end effector velocity; denotes the difference between end effector position and target position; denotes the output signal of the control strategy; denotes the desired end control accuracy; - denotes the weighting coefficient of each sub-reward item.
6. The method of claim 5, wherein the method is based on deep reinforcement learning. The method of updating the proximal strategy optimization model according to the position error comprises: wherein, is a neural network parameter to be optimized, and is a new policy and an old policy, is a merit function, is a clipping function.
7. A musculoskeletal robot based on deep reinforcement learning, characterized by, performing the control method of claim 1.
8. A musculoskeletal robot control system based on deep reinforcement learning, characterized by, The method comprises: an environment construction module for constructing a workspace of a musculoskeletal robot end effector, dividing the workspace into multiple subspaces, and obtaining target points for training of the musculoskeletal robot based on the subspaces; a model construction module for constructing a proximal strategy optimization model and obtaining an execution action of the musculoskeletal robot end effector based on the target points and the proximal strategy optimization model; a reward function construction module for constructing a motion reward function, calculating a position error between the execution action and the target points based on the motion reward function, updating the proximal strategy optimization model according to the position error, and performing motion control on the musculoskeletal robot based on the updated proximal strategy optimization model.
9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1-7.
10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1-9. The computer program is executed by the processor to implement the steps of the method of any one of claims 1-7.