A multi-task reinforcement learning method based on parameter contribution degree and application thereof
By constructing a sparse parameter sharing structure and performing selective activation and gradient updates based on parameter contribution, the problem of negative transfer between tasks caused by indiscriminate updates of shared parameters in multi-task reinforcement learning is solved, thereby improving the success rate and stability of UAV path planning.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANXI UNIV
- Filing Date
- 2026-03-13
- Publication Date
- 2026-06-26
AI Technical Summary
In existing multi-task reinforcement learning methods, the indiscriminate updating of shared parameters leads to negative transfer between tasks, affecting the model's convergence speed and performance, and making it difficult to achieve fine-grained parameter-level control.
We employ a multi-task reinforcement learning method based on parameter contribution. By constructing a sparse parameter sharing structure, we selectively activate and update gradients according to the contribution of shared parameters in different tasks, dynamically adjust the participation relationship of shared policy parameters, and reduce interference between tasks.
It improves the success rate and stability of multi-task path planning, reduces parameter conflicts between path planning tasks, and enhances the model's learning efficiency and performance.
Smart Images
Figure CN122287759A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence and reinforcement learning technology, specifically relating to a multi-task reinforcement learning method based on parameter contribution and its application. Background Technology
[0002] With the rapid development of artificial intelligence technology, reinforcement learning has been widely applied in fields such as robot control, intelligent navigation, UAV path planning, and complex decision optimization. Compared with single-task learning, multi-task reinforcement learning can achieve knowledge transfer between different tasks by sharing network structures, improving sample utilization efficiency and reducing training costs, thus having significant value in practical engineering applications.
[0003] In existing multi-task reinforcement learning methods, a network structure combining a shared feature extraction layer and a task-specific output layer is typically used to achieve collaborative learning between different tasks by sharing some network parameters. However, when there are significant differences between different tasks, a unified parameter sharing mechanism may lead to mutual interference between tasks, resulting in negative transfer, which in turn affects the convergence speed and final performance of the model.
[0004] To mitigate task conflicts, existing techniques typically employ gradient constraints, regularization terms, or weighted strategies to regulate the multi-task learning process. These methods primarily coordinate at the overall gradient level, but they lack fine-grained differentiation of the importance of individual parameters within shared parameters across different tasks, making it difficult to achieve selective control at the parameter level.
[0005] Therefore, in the multi-task reinforcement learning framework, how to dynamically adjust the shared parameters according to the contribution of different tasks to the shared parameters, and realize a more refined parameter sharing mechanism, so as to reduce interference between tasks while ensuring the knowledge transfer effect, remains an urgent technical problem to be solved. Summary of the Invention
[0006] To address the problem of negative transfer between tasks caused by indiscriminate updates of shared parameters in existing multi-task reinforcement learning methods, this invention provides a multi-task reinforcement learning method based on parameter contribution and its application.
[0007] To achieve the above objectives, the present invention employs the following technical solutions:
[0008] A multi-task reinforcement learning method based on parameter contribution, the method comprising the following steps:
[0009] Step 1: Construct a multi-task reinforcement learning model. The model is used to generate action decisions through a policy function under environmental state input, and update the policy parameters based on the policy optimization objective function. The model includes policy parameters shared across tasks and task-specific parameters.
[0010] The specific operation of step 1 is as follows:
[0011] Assume there is a total One task, task Recorded as The multi-task reinforcement learning model includes a shared parameter layer and multiple task-specific parameter layers; let the parameters of the shared parameter layer be: This represents the set of policy parameters shared across tasks; task The specific parameters are ,Task The strategy parameters are expressed as follows: The shared parameter layer is used to extract common feature representations across tasks, while the task-specific parameter layer is used to generate the corresponding task's policy output or value evaluation results.
[0012] The multi-task reinforcement learning model generates action decisions based on the environmental state through a policy function. The policy function is expressed as:
[0013] in, Indicates task The corresponding action; Indicates task The corresponding strategy function; Representing state Action variables below; Indicates task The corresponding state;
[0014] The policy function is updated by minimizing the policy loss function. The policy optimization objective function is defined as follows:
[0015]
[0016] in, Represents the dominance function; Indicates task Next state – action pair Action value function; Indicates task Next state – action pair The degree of advantage relative to the baseline of state value; Indicates from the task Experience dataset Mid-sampled state – action pair The corresponding mathematical expectation.
[0017] Step 2: Calculate the parameter contribution of the shared strategy parameters in different tasks according to the objective function of the strategy optimization, and construct a task-related sparse parameter sharing structure based on the parameter contribution to determine the selective participation relationship of the shared strategy parameters in the strategy calculation of each task;
[0018] The specific operation of step 2 is as follows:
[0019] Step 2.1: The parameter contribution is calculated based on the squared gradient of the shared parameters with respect to the objective function of the task strategy optimization. The Middle The contribution of each shared parameter is expressed as:
[0020]
[0021] For the shared layer One shared parameter; This represents the statistical expectation based on the sampled data; the contribution of the shared parameters is used to measure the degree of influence of the shared parameters on the performance of the current task.
[0022] And form a contribution vector:
[0023]
[0024] Step 2.2: Generate a task-specific binary mask by sorting and filtering the parameter contribution evaluation values. Construct a sparse parameter sharing structure related to the task, as shown in the formula:
[0025]
[0026] in, Indicates the first Shared parameters in the task The contribution level below Sparsity;
[0027] And obtain the mask vector:
[0028] . .
[0029] Step 3: Based on the sparse parameter sharing structure, perform task-related gating control on the participation of shared policy parameters in the policy calculation and gradient update processes, and dynamically adjust the sparse parameter sharing structure according to the changes in parameter contribution during training to reduce inter-task interference in the multi-task policy learning process.
[0030] The specific operation of step 3 is as follows:
[0031] Step 3.1: During the forward propagation process, the shared parameters are selectively activated using a task-specific binary mask to obtain the task. The shared feature representation is calculated as follows:
[0032]
[0033] And construct a task based on the shared feature representation. The strategy function is generated and the corresponding action is produced:
[0034]
[0035] in, Indicates the state Extracted feature vectors, Indicates shared parameters The defined feature mapping function, This represents a task-specific policy mapping function. This indicates element-wise multiplication. Indicates task Next state The corresponding gated shared feature representation; This represents the Sigmoid activation function, used to compress the mapping results to a preset range to generate gated weights; This represents the bias parameters associated with the shared feature map.
[0036] Step 3.2: During backpropagation, the shared parameter gradient is selectively updated by performing an element-wise product operation with the task-specific binary mask, as shown below:
[0037]
[0038] in, Indicates task Gated gradients for shared parameters; Indicates task Next state Take action Action value function at time; Indicates task Next state The state value function.
[0039] Furthermore, gradients from multiple tasks are aggregated to ensure that inactive shared parameters do not participate in gradient updates under their respective tasks. The formula is as follows:
[0040] ,
[0041] in, This represents the aggregated update gradient with shared parameters; Indicates task The corresponding gradient aggregation weights, and the sum of the weights of each task is 1.
[0042] Step 3.3: Dynamically update the task-specific binary mask according to a preset training period. The update period is... One training step, and satisfying:
[0043]
[0044]
[0045] in, Indicates task In the The binary mask corresponding to each training iteration; Indicates task In the The binary mask corresponding to each training iteration.
[0046] The method of this invention can be applied to the training of UAV path planning models. In UAV path planning applications,
[0047] By using real-time environmental information acquired by the UAV as network input, and outputting flight control commands through a multi-task reinforcement learning network, autonomous path planning by the UAV is achieved under target arrival constraints, obstacle avoidance constraints, and flight boundary constraints. A selective sharing mechanism driven by parameter contribution can reduce parameter conflicts between different path planning tasks, thereby improving the success rate and stability of path planning.
[0048] The method of this invention can be deployed in an unmanned aerial vehicle (UAV) path planning system including a processor and memory. The memory stores a computer program for executing the method. When the processor runs the computer program, the aforementioned multi-task reinforcement learning method is executed.
[0049] Compared with the prior art, the present invention has the following advantages:
[0050] This invention's method can be applied to training UAV path planning models. In UAV path planning applications, real-time environmental state information acquired by the UAV is used as network input. A multi-task reinforcement learning network outputs flight control commands, enabling autonomous path planning by the UAV under target arrival constraints, obstacle avoidance constraints, and flight boundary constraints. Through a parameter contribution-driven selective sharing mechanism, parameter conflicts between different path planning tasks can be reduced, improving the success rate and stability of path planning.
[0051] The method of this invention can be deployed in an unmanned aerial vehicle (UAV) path planning system including a processor and memory. The memory stores a computer program for executing the method. When the processor runs the computer program, the aforementioned multi-task reinforcement learning method is executed. Attached Figure Description
[0052] Figure 1 This is an overall framework diagram of an embodiment of the present invention;
[0053] Figure 2 This is an environmental diagram of the UAV path planning according to an embodiment of the present invention;
[0054] Figure 3 This is a task design diagram of an embodiment of the present invention. Detailed Implementation
[0055] To gain a deeper understanding of this invention, we will provide a comprehensive and detailed description. However, this invention has various implementations and is not limited to the specific examples listed herein. These examples are presented to enhance a full understanding of the disclosure of this invention.
[0056] A multi-task reinforcement learning method based on parameter contribution, the method comprising the following steps: In this embodiment, considering the inclusion of A set of tasks for UAV path planning, tasks Recorded as Different missions may differ in terms of target location, obstacle distribution, and enemy drone layout, but the optimization objective remains the same: to enable the drone to reach the target area via the shortest possible path while satisfying boundary and safety constraints.
[0057] Step 1: Construct a multi-task reinforcement learning model. The model is used to generate action decisions through a policy function under environmental state input, and update the policy parameters based on the policy optimization objective function. The model includes policy parameters shared across tasks and task-specific parameters.
[0058] The specific operation of step 1 is as follows:
[0059] Assume there is a total One task, task Recorded as The multi-task reinforcement learning model includes a shared parameter layer and multiple task-specific parameter layers; let the parameters of the shared parameter layer be: This represents the set of policy parameters shared across tasks; task The specific parameters are ,Task The strategy parameters are expressed as follows: The shared parameter layer is used to extract common feature representations across tasks, while the task-specific parameter layer is used to generate the corresponding task's policy output or value evaluation results.
[0060] The multi-task reinforcement learning model generates action decisions based on the environmental state through a policy function. The policy function is expressed as:
[0061]
[0062] in, Indicates task The corresponding action; Indicates task The corresponding strategy function; Representing state Action variables below; Indicates task The corresponding state.
[0063] The policy function is updated by minimizing the policy loss function. The policy optimization objective function is defined as follows:
[0064]
[0065] in, Represents the dominance function; Indicates task Next state – action pair Action value function; Indicates task Next state – action pair The degree of advantage relative to the baseline of state value; Indicates from the task Experience dataset Mid-sampled state – action pair The corresponding mathematical expectation.
[0066] Step 2: Calculate the parameter contribution of the shared strategy parameters in different tasks according to the objective function of the strategy optimization, and construct a task-related sparse parameter sharing structure based on the parameter contribution to determine the selective participation relationship of the shared strategy parameters in the strategy calculation of each task;
[0067] The specific operation of step 2 is as follows:
[0068] Step 2.1: The parameter contribution is calculated based on the squared gradient of the shared parameters with respect to the objective function of the task strategy optimization. The Middle The contribution of each shared parameter is expressed as:
[0069]
[0070] For the shared layer One shared parameter; This represents the statistical expectation based on the sampled data; the contribution of the shared parameters is used to measure the degree of influence of the shared parameters on the performance of the current task.
[0071] And form a contribution vector:
[0072]
[0073] Step 2.2: Generate a task-specific binary mask by sorting and filtering the parameter contribution evaluation values. Construct a sparse parameter sharing structure related to the task, as shown in the formula:
[0074]
[0075] in, Indicates the first Shared parameters in the task The contribution level below Sparsity;
[0076] And obtain the mask vector:
[0077]
[0078] Step 3: Based on the sparse parameter sharing structure, perform task-related gating control on the participation of shared policy parameters in the policy calculation and gradient update processes, and dynamically adjust the sparse parameter sharing structure according to the changes in parameter contribution during training to reduce inter-task interference in the multi-task policy learning process.
[0079] The specific operation of step 3 is as follows:
[0080] Step 3.1: During the forward propagation process, the shared parameters are selectively activated using a task-specific binary mask to obtain the task. The shared feature representation is calculated as follows:
[0081]
[0082] And construct a task based on the shared feature representation. The strategy function is generated and the corresponding action is produced:
[0083]
[0084] in, Indicates the state Extracted feature vectors, Indicates shared parameters The defined feature mapping function, This represents a task-specific policy mapping function. This indicates element-wise multiplication. Indicates task Next state The corresponding gated shared feature representation; This represents the Sigmoid activation function, used to compress the mapping results to a preset range to generate gated weights; This represents the bias parameters associated with the shared feature map.
[0085] Step 3.2: During backpropagation, the shared parameter gradient is selectively updated by performing an element-wise product operation with the task-specific binary mask, as shown below:
[0086]
[0087] in, Indicates task Gated gradients for shared parameters; Indicates task Next state Take action Action value function at time; Indicates task Next state The state value function.
[0088] Furthermore, gradients from multiple tasks are aggregated to ensure that inactive shared parameters do not participate in gradient updates under their respective tasks. The formula is as follows:
[0089] ,
[0090] in, This represents the aggregated update gradient with shared parameters; Indicates task The corresponding gradient aggregation weights, and the sum of the weights of each task is 1.
[0091] Step 3.3: Dynamically update the task-specific binary mask according to a preset training period. The update period is... One training step, and satisfying:
[0092]
[0093]
[0094] in, Indicates task In the The binary mask corresponding to each training iteration; Indicates task In the The binary mask corresponding to each training iteration.
[0095] The method of this invention can be applied to the training of UAV path planning models. In UAV path planning applications,
[0096] By using real-time environmental information acquired by the UAV as network input, and outputting flight control commands through a multi-task reinforcement learning network, autonomous path planning by the UAV is achieved under target arrival constraints, obstacle avoidance constraints, and flight boundary constraints. A selective sharing mechanism driven by parameter contribution can reduce parameter conflicts between different path planning tasks, thereby improving the success rate and stability of path planning.
[0097] The method of this invention can be deployed in an unmanned aerial vehicle (UAV) path planning system including a processor and memory. The memory stores a computer program for executing the method. When the processor runs the computer program, the aforementioned multi-task reinforcement learning method is executed.
[0098] The hardware platform used in this invention is a computer equipped with an Intel(R) Core(TM) i9-14900KF processor and an NVIDIA GeForce RTX 4090 GPU. Our software platform is based on Python 3.8. Each experiment is run 10 times, and the average results are reported.
[0099] To verify the effectiveness of the sparse sharing multi-task reinforcement learning method based on parameter contribution proposed in this invention in UAV path planning tasks, a multi-task simulation experiment was conducted.
[0100] The experimental environment of this invention is a two-dimensional finite planar region, 600×700 in size. Several static obstacles and enemy drones are distributed within the region. The drones must reach the target area from the starting point while satisfying boundary and safety constraints. A total of 8 path planning tasks are set, each with different target locations, obstacle layouts, and environmental complexity. Each task has a maximum time step limit; if the target is not reached within the specified time, the task is considered a failure.
[0101] The main training parameters used are shown in Table 1.
[0102] Discount factor 0.99 Actor Learning Rate 1e-3 Critic learning rate 3e-3 Entropy coefficient and learning rate 3e-4 Batch size 128 Soft update coefficient 1e-2 Experience replay capacity 20000 Maximum number of training rounds 600 Maximum number of moves per round 1000 Number of test rounds 100
[0103] This invention was compared with the following two methods:
[0104] STL: Single-task approach, which trains an independent network for each UAV path planning task as a baseline.
[0105] MTSAC: Multi-task SAC hard sharing method, all parameters are shared between tasks.
[0106] This invention focuses on success rate Average shortest flight distance Average shortest flight time The algorithm performance is analyzed from three aspects.
[0107] Table 1 Comparison of success rates of different algorithms
[0108] Task STL MTSAC This invention task1 0.95 0.98 1 task2 0.87 0.98 1 task3 0.79 0.83 0.86 task4 0.53 0.78 0.9
[0109] As can be seen from Table 1, the present invention performs best on all tasks, with an average success rate of up to 90%, which indicates that the introduction of the parameter contribution sparse sharing mechanism improves the algorithm performance.
[0110] Table 2 Comparison of average shortest flight times for different algorithms
[0111] Task STL MTSAC This invention task1 221.9 202.8 198 task2 264.7 231.7 204 task3 290.2 220.8 203.5 task4 248.6 213.4 199.1
[0112] As can be seen from Table 2, the present invention achieves the shortest path on all tasks, and the introduction of the parameter contribution sparse sharing mechanism improves the knowledge negative transfer problem that exists in hard sharing.
[0113] Table 3 Comparison of average shortest flight distances for different algorithms
[0114] Task STL MTSAC FS-MTSAC task1 249.7 222.1 216.6 task2 286.3 252.7 219.6 task3 338.4 240.2 220.4 task4 306.2 232.9 213.8
[0115] As can be seen from Table 3, the present invention achieves the shortest flight time for all tasks.
[0116] Experimental results show that by introducing a sparse sharing mechanism based on parameter contribution, the gradient conflict problem in multi-task reinforcement learning can be effectively alleviated, the negative transfer effect between tasks can be reduced, and the performance of multi-task path planning can be improved.
[0117] Contents not described in detail in this specification are prior art known to those skilled in the art. Although illustrative specific embodiments of the invention have been described above to facilitate understanding by those skilled in the art, it should be understood that the invention is not limited to the scope of the specific embodiments. Various modifications are readily apparent to those skilled in the art as long as they fall within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of this invention are protected.
Claims
1. A multi-task reinforcement learning method based on parameter contribution, characterized in that, The method includes the following steps: Step 1: Construct a multi-task reinforcement learning model. The model is used to generate action decisions through a policy function under environmental state input, and update the policy parameters based on the policy optimization objective function. The model includes policy parameters shared across tasks and task-specific parameters. Step 2: Calculate the parameter contribution of the shared strategy parameters in different tasks according to the objective function of the strategy optimization, and construct a task-related sparse parameter sharing structure based on the parameter contribution to determine the selective participation relationship of the shared strategy parameters in the strategy calculation of each task; Step 3: Based on the sparse parameter sharing structure, perform task-related gating control on the participation of shared policy parameters in the policy calculation and gradient update processes, and dynamically adjust the sparse parameter sharing structure according to the changes in parameter contribution during training to reduce inter-task interference in the multi-task policy learning process.
2. The multi-task reinforcement learning method based on parameter contribution as described in claim 1, characterized in that, The specific operation of step 1 is as follows: Assume there is a total One task, task Recorded as The multi-task reinforcement learning model includes a shared parameter layer and multiple task-specific parameter layers; let the parameters of the shared parameter layer be: This represents the set of policy parameters shared across tasks; task The specific parameters are ,Task The strategy parameters are expressed as follows: The shared parameter layer is used to extract common feature representations across tasks, while the task-specific parameter layer is used to generate the corresponding task's policy output or value evaluation results. The multi-task reinforcement learning model generates action decisions based on the environmental state through a policy function. The policy function is expressed as: in, Indicates task The corresponding action; Indicates task The corresponding strategy function; Representing state Action variables below; Indicates task The corresponding state; The policy function is updated by minimizing the policy loss function. The policy optimization objective function is defined as follows: in, Represents the dominance function; Indicates task Next state – action pair Action value function; Indicates task Next state – action pair The degree of advantage relative to the baseline of state value; Indicates from the task Experience dataset Mid-sampled state – action pair The corresponding mathematical expectation.
3. The multi-task reinforcement learning method based on parameter contribution as described in claim 2, characterized in that, The specific operation of step 2 is as follows: Step 2.1: The parameter contribution is calculated based on the squared gradient of the shared parameters with respect to the objective function of the task strategy optimization. The Middle The contribution of each shared parameter is expressed as: For the shared layer One shared parameter; This represents the statistical expectation based on the sampled data; the contribution of the shared parameters is used to measure the degree of influence of the shared parameters on the performance of the current task. And form a contribution vector: Step 2.2: Generate a task-specific binary mask by sorting and filtering the parameter contribution evaluation values. Construct a sparse parameter sharing structure related to the task, as shown in the formula: in, Indicates the first Shared parameters in the task The contribution level below Sparsity; And obtain the mask vector: 。 4. The multi-task reinforcement learning method based on parameter contribution as described in claim 3, characterized in that, The specific operation of step 3 is as follows: Step 3.1: During the forward propagation process, the shared parameters are selectively activated using a task-specific binary mask to obtain the task. The shared feature representation is calculated as follows: And construct a task based on the shared feature representation. The strategy function is generated and the corresponding action is produced: in, Indicates the state Extracted feature vectors, Indicates shared parameters The defined feature mapping function, This represents a task-specific policy mapping function. This indicates element-wise multiplication. Indicates task Next state The corresponding gated shared feature representation; This represents the Sigmoid activation function, used to compress the mapping results to a preset range to generate gated weights; This represents the bias parameters associated with the shared feature map. Step 3.2: During backpropagation, the shared parameter gradient is selectively updated by performing an element-wise product operation with the task-specific binary mask, as shown below: in, Indicates task Gated gradients for shared parameters; Indicates task Next state Take action Action value function at time; Indicates task Next state The state value function. Furthermore, gradients from multiple tasks are aggregated to ensure that inactive shared parameters do not participate in gradient updates under their respective tasks. The formula is as follows: ， in, This represents the aggregated update gradient with shared parameters; Indicates task The corresponding gradient aggregation weights, and the sum of the weights of each task is 1; Step 3.3: Dynamically update the task-specific binary mask according to a preset training period. The update period is... One training step, and satisfying: in, Indicates task In the The binary mask corresponding to each training iteration; Indicates task In the The binary mask corresponding to each training iteration.
5. The application of a multi-task reinforcement learning method based on parameter contribution as described in any one of claims 1 to 4 in UAV path planning, characterized in that, The method is applied to the training of UAV path planning models. It uses the environmental state information acquired by the UAV in real time as input and outputs UAV flight control commands to realize autonomous path planning of UAV under target arrival constraints, obstacle avoidance constraints and flight boundary constraints.
6. A UAV path planning system, comprising a processor and a memory, characterized in that, The memory stores a computer program that, when executed on the processor, performs the method according to any one of claims 1 to 4.