Efficient multi-dimensional mechanical arm operation priority weighting soft modularization method based on multi-task reinforcement learning

Abstract

In recent years, reinforcement learning has a significant effect in multiple fields, but a traditional method has limited adaptability in a dynamic environment or a multi-task situation. The invention provides a multi-task reinforcement learning method named as a priority weighted soft modularization (SM-PHT) method, which is used for enhancing the adaptability and the generalization ability of an intelligent agent, and relates to a multi-task reinforcement learning method named as SM-PHT (SM-PHT), which is used for enhancing the adaptability and the generalization ability of the intelligent agent. SM-PHT integrates three key mechanisms: priority weighted knowledge distillation, and student network robustness is improved through weighted fusion of multi-model knowledge; the hierarchical cache is used for respectively storing low-level experience and high-level parameter optimization efficiency; and task embedding: capturing environment characteristics to enrich task representation so as to promote knowledge transfer. In the Meta-World MT10 challenge, the success rate of SM-PHT is twice that of the existing optimal method, and the average reward is improved by 30%; in the MT50 challenge, the success rate and the average reward are both improved by about 10%. Results prove that the stability and generalization of the mechanical arm in complex tasks highlight the application potential in the field of actual multi-task mechanical arm operation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of reinforcement learning, multi-task decision making and robot control, and particularly relates to a priority-weighted soft modularization method based on multi-task reinforcement learning for efficiently solving robot operation problems in multi-task scenarios. BACKGROUND

[0002] In recent years, single-task reinforcement learning (RL) has made remarkable achievements in the fields of games, autonomous driving and industrial robot control as a cutting-edge technology. It allows agents to explore and optimize behaviors to maximize cumulative rewards through a trial-and-error mechanism. Unlike supervised or unsupervised learning, reinforcement learning does not rely on pre-labeled data sets, but adjusts model parameters based on environmental feedback. Combined with deep learning techniques, it can effectively handle high-dimensional state spaces and large-scale action spaces.

[0003] With the development of technology, the limitations of single-task reinforcement learning in adapting to changing environments have gradually emerged, and multi-task reinforcement learning (MTRL) has therefore received more attention. In the fields of robot operation, autonomous driving and game AI, multi-task reinforcement learning enables agents to adapt more quickly to new tasks and environmental changes, and simultaneously handle multiple tasks, improving generalization ability and meeting the diverse and dynamic needs of practical applications.

[0004] However, existing MTRL methods each have their limitations. Transfer-based methods, such as central policy and progressive neural networks, respectively face the risk of negative transfer and increased storage overhead as the network expands. Expert mixing algorithms, although capable of training independent expert models for different tasks and selecting appropriate expert modules through a gating mechanism, are relatively complex to manage and tune. Routing network algorithms select appropriate modules through a routing network based on input to avoid negative transfer and improve efficiency, but their implementation can be relatively complex. In contrast, soft modularization networks, due to their advantages in dynamically adjusting weight distribution and suppressing the growth of module numbers, show better performance and potential for efficiency improvement, and become the basis for further research and improvement. SUMMARY

[0005] To overcome the above limitations, the present application proposes a brand-new algorithm - a priority-weighted soft modularization method based on multi-task reinforcement learning (soft modularization with priority-weighted knowledge distillation, hierarchical buffer and task embedding, SM-PHT). The algorithm combines innovative priority-weighted knowledge distillation mechanisms, hierarchical buffer mechanisms, and task embedding mechanisms. This design not only improves the stability and generalization of the agent in complex tasks, but also demonstrates its potential in practical multi-task reinforcement learning applications.

[0006] The main contributions of the present application are as follows:

[0007] An innovative priority-weighted knowledge distillation mechanism is proposed, which integrates the knowledge of multiple high-performance models through a weighted distillation process to capture more extensive task features.

[0008] A hierarchical buffer mechanism containing "experience cache" and "parameter cache" is designed to help the agent extract key features from multiple strategies, promote knowledge transfer, and enhance model generalization ability.

[0009] A task embedding mechanism is introduced to capture detailed environmental features to enrich task representation, reveal the inherent relationship between tasks, and promote efficient knowledge transfer.

[0010] The purpose of the present application can be achieved by the following technical solutions:

[0011] Step S1: Determine the reinforcement learning problem to be solved and initialize the multi-task environment;

[0012] Step S2: Calculate the task embedding vector corresponding to each task by pre-training a large language model;

[0013] Step S3: Obtain experience data by interacting with the environment;

[0014] Step S4: Collect experience data and model parameters based on hierarchical buffer;

[0015] Step S5: Sample data from the experience cache, and perform reinforcement learning training and model optimization for the agent;

[0016] Step S6: If the fixed batch is reached, trigger priority-weighted knowledge distillation. Sample parameters from the parameter cache according to the priority, and perform supervised learning training and knowledge optimization for the agent through weighted distillation;

[0017] Step S7: Repeat steps S3 to S6 until the decision termination condition is met.

[0018] Specifically, step S1, environment initialization, includes:

[0019] Step S1.1: Determine the task space and total number of tasks in the environment, define the motion space, state space and reward function for each robotic arm task, and initialize the multi-task environment;

[0020] Step S1.2: Obtain the initial environment state s0 and task ID t0 after environment initialization;

[0021] Step S1.3: Randomly initialize the value network parameters ω0 and policy network parameters θ0 in the agent.

[0022] Specifically, step S2, calculating the task embedding vector, includes the following steps:

[0023] Step S2.1: For all tasks in the task space, n = 1, 2, ..., N;

[0024] Step S2.2: Obtain the predefined task ID, task name, task objective, and task description from step S1;

[0025] Step S2.3: Input the data from step S2.1 and extract the latent representation E for each task using the pre-trained large language model. n ←LLM(Task ID, Task Name, Task Description);

[0026] Step S2.4: Repeat steps S2.2 to S2.3 until all tasks can be represented using task embedding vectors.

[0027] Specifically, step S3, which involves interacting with the environment, can be divided into the following steps:

[0028] Step S3.1: Based on the current environmental state s t Use a policy network to predict the optimal action a t ;

[0029] Step S3.2: Interact with the environment using the predicted optimal action to obtain the next environment state s. t+1 and reward r;

[0030] Step S3.3: s t+1 Assigned to s t ;

[0031] Step S3.4: Repeat steps S3.1 to S3.3 until the maximum number of environmental interactions is reached.

[0032] Specifically, step S4, based on hierarchical caching, collects empirical data and model parameters as follows:

[0033] Step S4.1: Combine all tuples s generated in step S3. t ,a t ,r t and s t+1 Stored in the experience replay pool;

[0034] Step S4.2: Set the current value network parameters ω t and policy network parameters θ t Store them in the parameter replay pool and sort them according to their respective evaluation indicators γ;

[0035] Specifically, the process of reinforcement learning training and model optimization for the agent in step S5 can be divided into the following steps:

[0036] Step S5.1: Determine whether the amount of data in the experience replay pool has reached the minimum training requirement. If the minimum training requirement is met, continue; otherwise, go to step S3 to continue collecting experience data.

[0037] Step S5.2: Take a batch of training data from the experience replay pool;

[0038] Step S5.3: The agent performs self-evaluation and reinforcement learning to optimize the policy network and value network;

[0039] Step S5.4: Update the parameters of the policy network and the value network.

[0040] Specifically, the process of priority-weighted knowledge distillation performed by the agent in step S6 can be divided into the following steps:

[0041] Step S6.1: Take a batch of samples from the experience replay pool as the training data set x;

[0042] Step S6.2: Input the training data set x into the student model to obtain the prediction distribution S(x);

[0043] Step S6.3: via L hard (S(x),y)=-∑ i y i log(S(x)i) calculates the hard label loss;

[0044] Step S6.4: Input the training data set x into N teacher models to obtain the prediction distribution T i (x), i = 1, 2, ..., n, and their respective evaluation indicators γ i .

[0045] Step S6.5: via L soft (S(x),T(x))=-Σ i T(x) ilog(S(x)i) is used to calculate the soft label loss;

[0046] Step S6.6: Through Calculate the total loss function;

[0047] Step S6.7: Differentiate the total loss function and optimize the parameters of the student model through backpropagation and the optimizer.

[0048] The beneficial effects of the technical solution provided by the embodiments of the present invention are as follows: This invention overcomes the negative transfer risk and time or storage overhead issues faced by previous multi-task reinforcement learning methods when dealing with high-dimensional robotic arm manipulation problems, significantly improving the adaptability, stability, and efficiency of multi-task reinforcement learning systems in dynamic and complex environments. By introducing three innovative mechanisms—priority-weighted knowledge distillation (PWKD), hierarchical caching, and task embedding—SM-PHT can not only effectively extract and integrate knowledge from multiple high-performance models but also optimize data management and task representation during the learning process, thereby enhancing the agent's generalization ability when facing diverse tasks. Therefore, this invention provides a novel and effective solution to the adaptability and efficiency problems faced by traditional multi-task reinforcement learning algorithms and is expected to play an important role in the field of high-dimensional robotic arm manipulation. Attached Figure Description

[0049] Figure 1 This is a flowchart illustrating the priority-weighted soft modularization method based on multi-task reinforcement learning.

[0050] Figure 2 Architecture diagram of priority-weighted knowledge distillation mechanism.

[0051] Figure 3 This is a flowchart of a tiered caching mechanism.

[0052] Figure 4 A comparison diagram of the mathematical principles of task embedding vectors and one-hot vectors.

[0053] Figure 5 This is a schematic diagram of all tasks in the Meta-World experimental environment.

[0054] Figure 6 The training results are shown for a priority-weighted soft modularization method based on multi-task reinforcement learning.

[0055] Figure 7 This is a performance comparison chart of priority-weighted soft modularization methods based on multi-task reinforcement learning.

[0056] Figure 8 This is a performance distribution diagram of each task in the priority-weighted soft modularization method based on multi-task reinforcement learning. Detailed Implementation

[0057] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions in the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments.

[0058] Example 1:

[0059] This embodiment discloses a priority-weighted soft modular algorithm based on multi-task reinforcement learning for efficiently handling the challenges of robotic arm operations in multi-task scenarios. A detailed flowchart is shown below. Figure 1 As shown. Figure 1 It specifically demonstrates the entire process of intelligent agents interacting with their environment, self-reinforcing and learning, and knowledge distillation.

[0060] The priority-weighted soft modularization algorithm based on multi-task reinforcement learning disclosed in this embodiment for efficiently handling the difficult problems of robotic arm operation in multi-task scenarios includes the following steps:

[0061] Step S1: Determine the reinforcement learning problem to be solved and initialize the multi-task environment;

[0062] Step S2: Calculate the task embedding vectors corresponding to each task using a pre-trained large language model, such as... Figure 1 As shown in the top left corner;

[0063] Step S3: Obtain empirical data through interaction with the environment, such as Figure 1 As shown on the right;

[0064] Step S4: Collect empirical data and model parameters separately based on hierarchical caching, such as Figure 3 As shown;

[0065] Step S5: Sample data from the experience cache to train the agent using reinforcement learning and optimize the model;

[0066] Step S6: If a fixed batch is reached, trigger priority-weighted knowledge distillation. Sample parameters from the parameter cache according to priority, and perform supervised learning training and knowledge optimization on the agent through weighted distillation, such as... Figure 2 As shown;

[0067] Step S7: Repeat steps S3 to S6 until the decision termination condition is met.

[0068] Figure 4 It demonstrates the differences in principle and effect between task embedding vectors and one-hot vectors.

[0069] Figure 5 The Meta-World experimental environment used for virtual experiments in the current embodiment is shown, and the operation process and objectives of each task are explained in detail using diagrams.

[0070] Figure 6 The average total reward and average task success rate of the current embodiment configuration after 500 iterations in the Meta-World simulation environment are shown.

[0071] Figure 7 The current embodiment configuration and the comparison algorithm are shown to have significantly improved average total reward and average task success rate under various experimental settings.

[0072] Figure 8 The performance distribution of each task after the current implementation is completed is shown in the graph, and compared with the comparison algorithm, there is a significant improvement.

[0073] As can be seen from the above technical solutions, this invention has the following advantages: it significantly improves the adaptability, stability, and efficiency of multi-task reinforcement learning systems in dynamic and complex environments. By introducing three innovative mechanisms—priority-weighted knowledge distillation (PWKD), hierarchical caching, and task embedding—SM-PHT can not only effectively extract and integrate knowledge from multiple high-performance models, but also optimize data management and task representation during the learning process, thereby enhancing the agent's generalization ability when facing diverse tasks. Therefore, this invention provides a novel and effective solution to the adaptability and efficiency problems faced by traditional multi-task reinforcement learning algorithms, and is expected to play an important role in the field of high-dimensional robotic arm manipulation.

[0074] For those skilled in the art, the present invention is not limited to the exemplary details of the above embodiments. The scope of protection of the present invention should be defined by the scope of the appended claims and not by the description of the above embodiments. Any technical solutions based on the present invention and any equivalent modifications made thereto should fall within the scope of protection of the present invention.

Claims

1. A priority-weighted soft modularization method based on multi-task reinforcement learning is used to efficiently handle the challenges of robotic arm operation in multi-task scenarios, including the following steps: Step S1: Determine the reinforcement learning problem to be solved and initialize the multi-task environment; Step S2: Calculate the task embedding vectors corresponding to each task using a pre-trained large language model; Step S3: Obtain empirical data through interaction with the environment; Step S4: Collect empirical data and model parameters separately based on hierarchical caching; Step S5: Sample data from the experience cache to train the agent using reinforcement learning and optimize the model; Step S6: If a fixed batch is reached, trigger priority-weighted knowledge distillation. Sample parameters from the parameter cache according to priority, and perform supervised learning training and knowledge optimization on the agent through weighted distillation; Step S7: Repeat steps S3 to S6 until the decision termination condition is met.

2. The initialization phase of the priority-weighted soft modularization method based on multi-task reinforcement learning according to claim 1, characterized in that: Before implementing the SM-PHT algorithm, the specific reinforcement learning problem to be solved is first clarified as efficiently handling robotic arm operations in multi-task scenarios. These problems are formalized as a multi-task Markov Decision Process (MDP), which includes a set of tasks T and a corresponding MDP set M. Based on the selected task set and considering the correlation between tasks, the entire multi-task environment is initialized, setting the relevant parameters for the agent's interaction with the environment, such as the action space A, state space S, and reward function R.

3. The task embedding mechanism according to claim 1, characterized in that: Task embedding vectors for each task are calculated using a pre-trained large language model. To better capture semantic relationships and contextual information between tasks, the pre-trained large language model is used to extract latent representations for each task. First, multi-dimensional information such as task identifiers, names, and descriptions for all tasks is collected, and then input into the large language model to obtain the embedding vector for each task. These embedding vectors not only accurately reflect the unique attributes of each task but also reveal the inherent connections and common features between tasks, thereby enhancing knowledge transfer efficiency. This step is crucial for constructing effective task representations, which is helpful for subsequent strategy learning and optimization.

4. The environmental interaction process according to claim 1, characterized in that: An intelligent agent learns optimal policies by performing actions and receiving feedback (rewards or penalties) from the environment. During this process, the agent learns through multiple rounds, each round consisting of observing the current state, choosing an action, performing the action, and receiving new states and rewards. The datasets generated by these interactions form the basis of the agent's learning; they contain not only successful paths but also records failed experiences, helping the agent identify which actions yield the greatest cumulative reward.

5. The hierarchical caching mechanism according to claim 1, characterized in that: Hierarchical caching collects empirical data and model parameters separately. Specifically, low-level empirical samples are stored in an experience replay cache to ensure that the learning process can fully utilize rich historical interaction data, thereby improving the model's learning performance and generalization ability. On the other hand, high-level model parameters are stored in a parameter priority cache, sorted according to performance metrics (such as average reward or success probability). This allows for efficient management of important parameters and enhances the system's robustness and flexibility by dynamically adjusting their importance.

6. The priority-weighted knowledge distillation mechanism according to claim 1, characterized in that: Parameters are sampled from the parameter cache according to priority, and the agent is trained and optimized through weighted distillation. When a predetermined number of training epochs are reached, this mechanism is activated, selecting a certain number of network parameters from the parameter priority cache as teacher models. Then, weights are assigned to each teacher model based on their performance, and knowledge from high-performing models is integrated into the student model. This weighted distillation method not only accelerates the learning process of the student model but also significantly improves its final performance while mitigating overfitting.

Images