Robot motion control method and system based on multi-task reinforcement learning
By constructing a task similarity matrix and using clustering methods to group tasks, the problem of gradient interference in multi-task reinforcement learning is solved, thereby improving the stability and accuracy of robot control and making it suitable for complex multi-task scenarios.
Patent Information
- Application Number
- CN202511721774.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-21
- Publication Date
- 2026-01-23
AI Technical Summary
Existing multi-task reinforcement learning methods fail to explicitly model the relationships between tasks, leading to gradient interference and performance degradation, making it difficult to achieve stable training and efficient control in complex multi-task scenarios.
By constructing a control task similarity matrix and using clustering methods to group tasks, related tasks and conflicting tasks are identified, enabling knowledge sharing within groups and isolation between groups, thus optimizing the joint training of the policy network and the value network.
It improves the stability and overall performance of reinforcement learning models, enhances robot control accuracy and generalization ability, and reduces negative transfer problems caused by gradient conflicts.
Smart Images
Figure CN121374599A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of deep reinforcement learning technology, and in particular relates to a robot motion control method and system based on multi-task reinforcement learning. Background Technology
[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.
[0003] In the field of robot motion control, deep reinforcement learning (RL) has achieved significant results in single-task motion decision-making, such as the autonomous control of precise grasping by robotic arms and stable walking by quadruped robots. Nevertheless, most reinforcement learning methods are still limited to the learning paradigm of single motion tasks, making it difficult to generalize the motion skills learned in a certain task (such as joint coordination and dynamic balance strategies) to complex multi-task scenarios (such as robots needing to complete a composite task of "obstacle avoidance-grasping-placement" simultaneously). This limitation greatly hinders its practicality in training real-world robot systems (such as industrial collaborative robots and service robots).
[0004] To overcome this bottleneck, multi-task reinforcement learning has been proposed in the field of robot motion control. Its core is to improve the robot's overall performance in multiple motion tasks and enhance its adaptability to complex real-world scenarios through knowledge transfer between tasks. These methods can be broadly classified into four categories: Knowledge Extraction: Focusing on aggregating the specific knowledge of different motion tasks (such as assembly and sorting of robotic arms) into a unified control strategy; Structure sharing: By reusing modular motion control components (such as joint motion sub-networks and gait generation modules) to assemble strategies, the scalability in multi-task scenarios is improved; Representation learning: emphasizes learning shared motion state representations (such as robot pose and common features of joint dynamics) to facilitate cross-task policy learning and value estimation. Gradient operations: By adjusting the direction and scale of gradients, the stability of multi-task reinforcement learning in the optimization process can be enhanced (e.g., to alleviate the conflict between different motion tasks on joint torque constraints).
[0005] However, existing multi-task methods still have the following shortcomings in robot motion control: Failure to explicitly model inter-task relationships: Current methods generally assume that all tasks can share parameters and knowledge, ignoring the possible differences between tasks and failing to quantify the mutual influence between tasks.
[0006] Shared model structure leads to gradient interference: Different tasks will generate gradients with inconsistent directions during the optimization process, which will cause conflicts in model parameter updates, resulting in performance degradation and training instability.
[0007] The commonly used fully shared model architecture fails to adequately model the correlation and conflict between motion tasks (such as the fundamental difference in the requirements for robot joint stiffness and response speed between "high-speed running" and "fine operation"). This deficiency can lead to policy conflicts when significantly different motion tasks are jointly optimized, which in turn causes instability in the training process and ultimately results in the degradation of the robot's overall motion performance. Summary of the Invention
[0008] To address the technical problems mentioned above, this invention provides a robot motion control method and system based on multi-task reinforcement learning. By constructing a control task similarity matrix and using clustering methods to group tasks, it effectively identifies related and conflicting tasks, achieves knowledge sharing within groups and mutual isolation between groups, solves the negative interference problem of all task-shared models, thereby improving the stability and overall performance of the reinforcement learning model and enhancing the robot control accuracy.
[0009] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of this invention provides a robot motion control method based on multi-task reinforcement learning, comprising: Obtain robot control tasks and divide them into corresponding task groups. Each task group has an independent policy network and value network. The robot obtains its current state, receives control commands through the policy network of the corresponding task group, drives the robot to perform actions, receives environmental feedback, and the value network evaluates the value of the actions based on the environmental feedback and feeds it back to the policy network for policy optimization. The training of the policy network and the value network includes: sampling trajectories from multiple control tasks, calculating the pairwise similarity between control tasks, constructing a similarity matrix, and grouping the control tasks by clustering based on the similarity matrix to obtain several task groups. Within each task group, parameters are shared for joint training of the policy network and the value network.
[0010] Furthermore, the calculation steps of the similarity matrix include: Select the first control task and calculate the Q value of the first control task under the current parameters; Select the second control task, use the data samples from the second control task to update the parameters of the policy network and the value network, and then calculate the Q value again on the first control task. The relative change in the Q value of the first control task before and after the parameter update is calculated. The relative change magnitude of all control tasks is calculated to obtain a similarity matrix.
[0011] Furthermore, the relative change range is: ;in, It is the minimum value. The Q value of the first control task before parameter update. The Q value of the first control task after parameter update.
[0012] Furthermore, the grouping of control tasks includes: constructing positive and negative Laplacian matrices based on the similarity matrix to obtain a task spectrum representation, and using clustering on the task spectrum representation to group the control tasks.
[0013] A second aspect of the present invention provides a robot motion control system based on multi-task reinforcement learning, comprising: The task partitioning module is configured to: acquire robot control tasks, partition robot control tasks into corresponding task groups, and each task group has an independent policy network and value network. The motion control module is configured to: acquire the current robot state, obtain control commands through the policy network of the corresponding task group, drive the robot to perform actions, obtain environmental feedback, evaluate the value of the performed actions based on the environmental feedback, and feed it back to the policy network for policy optimization; The training of the policy network and the value network includes: sampling trajectories from multiple control tasks, calculating the pairwise similarity between control tasks, constructing a similarity matrix, and grouping the control tasks by clustering based on the similarity matrix to obtain several task groups. Within each task group, parameters are shared for joint training of the policy network and the value network.
[0014] Furthermore, the calculation steps of the similarity matrix include: Select the first control task and calculate the Q value of the first control task under the current parameters; Select the second control task, use the data samples from the second control task to update the parameters of the policy network and the value network, and then calculate the Q value again on the first control task. The relative change in the Q value of the first control task before and after the parameter update is calculated. The relative change magnitude of all control tasks is calculated to obtain a similarity matrix.
[0015] Furthermore, the relative change range is: ;in, It is the minimum value. The Q value of the first control task before parameter update. The Q value of the first control task after parameter update.
[0016] Furthermore, the grouping of control tasks includes: constructing positive and negative Laplacian matrices based on the similarity matrix to obtain a task spectrum representation, and using clustering on the task spectrum representation to group the control tasks.
[0017] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the robot motion control method based on multi-task reinforcement learning as described above.
[0018] A fourth aspect of the present invention provides a computer device including a computer-readable storage medium, a processor, and a computer program stored on the computer-readable storage medium and executable on the processor, wherein the processor executes the program to implement the steps of a robot motion control method based on multi-task reinforcement learning as described above.
[0019] Compared with the prior art, the beneficial effects of the present invention are: This invention constructs a control task similarity matrix and uses clustering methods to group tasks, effectively identifying related and conflicting tasks, achieving knowledge sharing within groups and mutual isolation between groups, solving the negative interference problem of all task-shared models, thereby improving the stability and overall performance of reinforcement learning models and enhancing robot control accuracy. Attached Figure Description
[0020] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0021] Figure 1 This is a flowchart of a robot motion control method based on multi-task reinforcement learning according to Embodiment 1 of the present invention; Figure 2 This is a flowchart illustrating the construction of the task affinity matrix in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the structure of a computer device according to Embodiment 4 of the present invention. Detailed Implementation
[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.
[0023] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0024] Example 1 This embodiment provides a robot motion control method based on multi-task reinforcement learning.
[0025] This embodiment provides a robot motion control method based on multi-task reinforcement learning, which aims to effectively quantify the relationships between tasks and reasonably group them according to the task relationships to achieve knowledge sharing within groups and mutual isolation between groups, thereby solving the negative interference problem of all task-shared models and improving the stability and overall performance of reinforcement learning models.
[0026] The robot motion control method based on multi-task reinforcement learning provided in this embodiment is not mutually exclusive with existing multi-task reinforcement learning methods, but rather has good complementarity and compatibility.
[0027] This embodiment provides a robot motion control method based on multi-task reinforcement learning, which can be directly embedded into the multi-task reinforcement learning framework based on Actor-Critic (AC).
[0028] This embodiment provides a robot motion control method based on multi-task reinforcement learning, which includes the following steps: Step 1, Task Grouping and Deployment: Obtain robot control tasks. Based on the clustering results during the training phase, divide the different control tasks of the robot (such as grasping, walking, and obstacle avoidance) into corresponding task groups. Each task group has an independent reinforcement learning model (including a policy network (Actor) and a value network (Critic)).
[0029] Step 2, State Awareness and Input: The robot obtains its current state (such as its own pose, environmental obstacles, and target position) through sensors (such as cameras, force sensors, and IMUs) and inputs it into the policy network of the corresponding task group.
[0030] Step 3, Action Generation and Execution: The policy network outputs control commands (such as joint angle sequences and motion speeds) to drive the robot to perform actions.
[0031] Step 4, Value Feedback and Iteration: The value network evaluates the Q value of an action based on environmental feedback (such as whether the task is completed or the energy consumption of the action), and uses the Q value as a value signal for policy updates, feeding it back to the policy network. The policy network continuously optimizes the policy by minimizing the entropy-regularized policy objective function (if online fine-tuning is required).
[0032] In this embodiment, as Figure 1 As shown, the training of a reinforcement learning model includes two main stages: the task grouping stage and the grouped multi-task reinforcement training stage.
[0033] In the task grouping phase, the similarity between tasks is first measured by evaluating the impact of one task's gradient update on the shared function on the Q-values of other tasks, thus constructing a task similarity matrix. This similarity matrix indicates the pairwise relationships between tasks. Based on this matrix, tasks are grouped using spectral clustering, which includes two key steps: constructing task spectral representations and applying k-means clustering to form the final groups. Specifically, eigenvalue decomposition is performed on the similarity matrix, and the resulting eigenvectors are used to obtain low-dimensional spectral embeddings for each task; then, the k-means algorithm is used to cluster these embeddings, allowing related tasks to be grouped together.
[0034] In the grouped multi-task reinforcement training phase, reinforcement learning models are trained independently within each task group based on the task grouping results. Each task group is assigned a dedicated SAC model to achieve centralized policy learning, promote knowledge sharing within the group, and reduce interference from unrelated tasks.
[0035] In the task grouping phase, trajectories are first sampled from multiple tasks, pairwise task similarity is calculated, and a similarity matrix (i.e., task affinity matrix) is constructed. Positive and negative Laplacian matrices are then constructed based on the similarity matrix to obtain the task spectral representation. Finally, k-means clustering is used to group the tasks based on the spectral representation. In the group training phase, independent policy and value networks are established for each task group. Tasks within the group share model parameters for joint training, thereby achieving intra-group knowledge transfer and inter-group isolation optimization.
[0036] like Figure 2 As shown, the construction of the task affinity matrix includes the following process: First, select the target task j, and calculate its Q value under the current model parameters. Then, select another task i and use the data samples from task i to update the parameters of the shared network; after the update, calculate the new Q value again on task j. By comparing the relative change W of Q-value before and after task j update. i,j The similarity matrix W is used to measure the degree of influence of task i on task j. If the update causes the Q value to increase, it indicates that the two tasks are highly similar; otherwise, a large change indicates that there are differences or conflicts between the two tasks. The above calculation is performed on all task pairs (i,j) in turn to obtain the task similarity matrix W.
[0037] The relative change range is as follows: ;in, It is a local minimum.
[0038] It should be noted that k-means clustering is a step in spectral clustering. Spectral clustering first constructs a spectral representation of the task, and then uses the obtained spectral representation as the coordinates of points in space to perform k-means clustering.
[0039] This embodiment provides a robot motion control method based on multi-task reinforcement learning. The key lies in measuring the task relationships and grouping tasks using spectral clustering, and optimizing the individual policy network of each group, thereby improving training stability and generalization performance.
[0040] This embodiment provides a robot motion control method based on multi-task reinforcement learning, which improves the overall training stability and generalization ability by quantitatively modeling the similarity between tasks and automating group training.
[0041] This embodiment provides a robot motion control method based on multi-task reinforcement learning, which can effectively model the relationship between tasks, significantly reduce gradient conflicts between tasks, improve training stability and overall performance, and achieve experimental results that are significantly better than those of existing technologies in multiple multi-task simulation environments.
[0042] This embodiment provides a robot motion control method based on multi-task reinforcement learning. By explicitly modeling task relationships, it reduces negative interference: by constructing a task similarity matrix and using spectral clustering to group tasks, it effectively identifies related and conflicting tasks, achieving intra-group sharing and inter-group isolation, thereby significantly mitigating the negative transfer problem caused by gradient conflicts between tasks.
[0043] This embodiment provides a robot motion control method based on multi-task reinforcement learning, which improves training stability and performance: by combining intra-group joint optimization and inter-group parameter isolation, this invention improves the convergence stability of the reinforcement learning model and significantly improves the overall performance in a multi-task environment.
[0044] This embodiment provides a robot motion control method based on multi-task reinforcement learning, which is versatile and embeddable: the task grouping strategy proposed in this invention can be flexibly embedded into existing multi-task reinforcement learning frameworks and used in conjunction with knowledge extraction, representation learning or gradient operation methods to further improve the system's performance and generalization ability.
[0045] Example 2 This embodiment provides a robot motion control system based on multi-task reinforcement learning, comprising: The task partitioning module is configured to: acquire robot control tasks, partition robot control tasks into corresponding task groups, and each task group has an independent policy network and value network. The motion control module is configured to: acquire the current robot state, obtain control commands through the policy network of the corresponding task group, drive the robot to perform actions, obtain environmental feedback, evaluate the value of the performed actions based on the environmental feedback, and feed it back to the policy network for policy optimization; The training of the policy network and the value network includes: sampling trajectories from multiple control tasks, calculating the pairwise similarity between control tasks, constructing a similarity matrix, and grouping the control tasks by clustering based on the similarity matrix to obtain several task groups. Within each task group, parameters are shared for joint training of the policy network and the value network.
[0046] Furthermore, the calculation steps of the similarity matrix include: Select the first control task and calculate the Q value of the first control task under the current parameters; Select the second control task, use the data samples from the second control task to update the parameters of the policy network and the value network, and then calculate the Q value again on the first control task. The relative change in the Q value of the first control task before and after the parameter update is calculated. The relative change magnitude of all control tasks is calculated to obtain a similarity matrix.
[0047] Furthermore, the relative change range is: ;in, It is the minimum value. The Q value of the first control task before parameter update. The Q value of the first control task after parameter update.
[0048] Furthermore, the grouping of control tasks includes: constructing positive and negative Laplacian matrices based on the similarity matrix to obtain a task spectrum representation, and using clustering on the control spectrum representation to group the control tasks.
[0049] It should be noted that each module in this embodiment corresponds one-to-one with each step in Embodiment 1, and their specific implementation processes are the same, so they will not be repeated here.
[0050] Example 3 This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a robot motion control method based on multi-task reinforcement learning as described in Embodiment 1 above.
[0051] Example 4 This embodiment provides a computer device, such as... Figure 3As shown, the system includes a computer-readable storage medium 1003, a processor 1001, a communication interface 1002, and a computer program stored on the computer-readable storage medium 1003 and executable on the processor 1001. The processor 1001, communication interface 1002, and computer-readable storage medium 1003 can be connected via a bus or other means. The communication interface 1002 is used to receive and send data. When the processor 1001 executes the program, it implements the steps of the robot motion control method based on multi-task reinforcement learning as described in Embodiment 1 above.
[0052] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A robot motion control method based on multi-task reinforcement learning, characterized in that, include: Obtain robot control tasks and divide them into corresponding task groups. Each task group has an independent policy network and value network. The robot obtains its current state, receives control commands through the policy network of the corresponding task group, drives the robot to perform actions, receives environmental feedback, and the value network evaluates the value of the actions based on the environmental feedback and feeds it back to the policy network for policy optimization. The training of the policy network and the value network includes: sampling trajectories from multiple control tasks, calculating the pairwise similarity between control tasks, constructing a similarity matrix, and grouping the control tasks by clustering based on the similarity matrix to obtain several task groups. Within each task group, parameters are shared for joint training of the policy network and the value network.
2. The robot motion control method based on multi-task reinforcement learning as described in claim 1, characterized in that, The steps for calculating the similarity matrix include: Select the first control task and calculate the Q value of the first control task under the current parameters; Select the second control task, use the data samples from the second control task to update the parameters of the policy network and the value network, and then calculate the Q value again on the first control task. The relative change in the Q value of the first control task before and after the parameter update is calculated. The relative change magnitude of all control tasks is calculated to obtain a similarity matrix.
3. The robot motion control method based on multi-task reinforcement learning as described in claim 2, characterized in that, The relative change range is: ;in, It is the minimum value. The Q value of the first control task before parameter update. The Q value of the first control task after parameter update.
4. The robot motion control method based on multi-task reinforcement learning as described in claim 1, characterized in that, The grouping of control tasks includes: constructing positive and negative Laplacian matrices based on the similarity matrix to obtain a task spectrum representation; using clustering on the control spectrum representation to group the control tasks.
5. A robot motion control system based on multi-task reinforcement learning, characterized in that, include: The task partitioning module is configured to: acquire robot control tasks, partition robot control tasks into corresponding task groups, and each task group has an independent policy network and value network. The motion control module is configured to: acquire the current robot state, obtain control commands through the policy network of the corresponding task group, drive the robot to perform actions, obtain environmental feedback, evaluate the value of the performed actions based on the environmental feedback, and feed it back to the policy network for policy optimization; The training of the policy network and the value network includes: sampling trajectories from multiple control tasks, calculating the pairwise similarity between control tasks, constructing a similarity matrix, and grouping the control tasks by clustering based on the similarity matrix to obtain several task groups. Within each task group, parameters are shared for joint training of the policy network and the value network.
6. A robot motion control system based on multi-task reinforcement learning as described in claim 5, characterized in that, The steps for calculating the similarity matrix include: Select the first control task and calculate the Q value of the first control task under the current parameters; Select the second control task, use the data samples from the second control task to update the parameters of the policy network and the value network, and then calculate the Q value again on the first control task. The relative change in the Q value of the first control task before and after the parameter update is calculated. The relative change magnitude of all control tasks is calculated to obtain a similarity matrix.
7. A robot motion control system based on multi-task reinforcement learning as described in claim 6, characterized in that, The relative change range is: ;in, It is the minimum value. The Q value of the first control task before parameter update. The Q value of the first control task after parameter update.
8. A robot motion control system based on multi-task reinforcement learning as described in claim 5, characterized in that, The grouping of control tasks includes: constructing positive and negative Laplacian matrices based on the similarity matrix to obtain a task spectrum representation; using clustering on the control spectrum representation to group the control tasks.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the robot motion control method based on multi-task reinforcement learning as described in any one of claims 1-4.
10. A computer device comprising a computer-readable storage medium, a processor, and a computer program stored on the computer-readable storage medium and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the robot motion control method based on multi-task reinforcement learning as described in any one of claims 1-4.