Multitask self-learning strategy generation method based on reinforcement meta-learning

Abstract

The invention discloses a multi-task self-learning strategy generation method based on reinforcement meta-learning. The method comprises the following steps: S1, constructing a multi-task training environment and collecting an environment interaction track data set in the multi-task training environment; s2, generating a task embedding atlas; s3, constructing a multi-scale strategy generator according to the task embedding atlas; s4, generating a multi-scale learning schedule for each known task; s5, obtaining a meta-policy parameter prior set and a meta-level updating rule set; and S6, for a new target task, inputting an early environment interaction track of the target task into the task embedding vector model to obtain a target task embedding vector, jointly inputting the target task embedding vector and the task embedding atlas into the multi-scale strategy generator, and outputting a multi-scale learning schedule of the target task. According to the method, the migration initial performance of the model on the new task is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of multi-task self-learning technology, and in particular to a multi-task self-learning strategy generation method based on reinforcement meta-learning. Background Art

[0002] With the continuous development of multi-task learning and reinforcement learning technologies, how to efficiently generalize and transfer strategies across multiple tasks has become an important research direction. Traditional multi-task learning strategies mostly rely on statically designed training schedules and unified learning parameters, which makes it difficult to take into account the heterogeneity and dynamics between different tasks. In specific applications, such as autonomous driving, industrial control, and robotic operation scenarios, they often involve task sets with different structures, which places higher adaptability requirements on learning models.

[0003] Existing methods have attempted to introduce meta-learning mechanisms to improve the model's ability to quickly adapt to new tasks, but most existing technologies rely only on single-scale policy expression and simple task similarity calculations, lacking the ability to deeply model the structural relationships between tasks. In addition, existing methods mostly use manual settings or heuristics to handle task switching, resource allocation, and policy initialization, and are unable to dynamically perform optimal scheduling based on the task embedding structure. Especially when there are a large number of tasks and significant distribution differences, traditional methods are prone to problems such as decreased policy generalization performance, poor training stability, and resource waste, which seriously affect the scalability and actual deployment effect of the model.

[0004] Therefore, current technologies have the following defects in dealing with multi-task strategy generation: lack of dynamic modeling of task structural relationships, inability to achieve coordinated scheduling of multi-scale strategy parameters, and limited efficiency and generalization performance in adaptive learning of new tasks. A more efficient and structure-aware multi-task self-learning strategy generation method is urgently needed to overcome the above shortcomings. Summary of the Invention

[0005] One purpose of the present invention is to propose a multi-task self-learning strategy generation method based on reinforcement meta-learning, which significantly improves the initial performance of the model in migrating to new tasks.

[0006] A method for generating a multi-task self-learning strategy based on reinforcement meta-learning according to an embodiment of the present invention includes the following steps:

[0007] S1. Build a multi-task training environment and collect an environment interaction trajectory dataset in the multi-task training environment. The environment interaction trajectory dataset records the state, action, reward, and next state information of each known task in the multi-task training environment;

[0008] S2. Vectorize and encode the environment interaction trajectory dataset to obtain a set of task embedding vectors. Based on the set of task embedding vectors, a task embedding graph is generated. The task embedding graph is used to describe the similarity and structural relationship between known tasks.

[0009] S3. Construct a multi-scale strategy generator based on the task embedding graph. The multi-scale strategy generator includes a coarse-grained control module and a fine-grained parameter module. The coarse-grained control module outputs the task learning sequence, exploration intensity curve, and resource allocation plan. The fine-grained parameter module outputs the strategy initialization parameter set and the optimization hyperparameter set.

[0010] S4. Input the task embedding graph into the multi-scale strategy generator to generate a multi-scale learning schedule for each known task. The multi-scale learning schedule includes the task learning sequence, exploration intensity curve, and resource allocation plan generated by the coarse-grained control module, and the strategy initialization parameter set and optimization hyperparameter set generated by the fine-grained parameter module.

[0011] S5. Build a meta-policy network and use the task embedding graph and multi-scale learning schedule as input. Perform meta-training via reinforcement learning in a multi-task training environment to obtain a set of meta-policy parameter priors and a set of meta-level update rules.

[0012] S6. For a newly emerged target task, the early environment interaction trajectory of the target task is first input into the task embedding vector model to obtain the target task embedding vector. The target task embedding vector and the task embedding graph are then jointly input into the multi-scale strategy generator to output a multi-scale learning schedule for the target task.

[0013] Optionally, the S1 includes the following steps:

[0014] S11. Set a task set T, which contains multiple known tasks T i , each known task T i Both are defined as Markov decision processes, which are composed of the state space S i , action space A i , state transition probability distribution P i , immediate reward function R i and the discount factor γ i Composition, state space S i Represents the set of all possible states contained in the i-th known task, the action space A i Represents the set of all possible actions contained in the i-th known task, and the state transition probability distribution P i It is used to describe the probability relationship of transferring to the next state after the reinforcement learning agent is in a certain state and takes a certain action in the i-th known task, that is, taking action a in state s to transfer to the next state s′ The probability of immediate reward function R i Used to describe the immediate reward value obtained by taking action a in state s in the i-th known task, the discount factor γ i is the discount attenuation coefficient of the i-th known task, which is used to control the influence of long-term benefits in the optimization process;

[0015] S12. For each known task T i Deploy reinforcement learning agents in the environment to interact with the multi-task training environment, and each interaction process generates an environmental interaction trajectory. Environmental interaction trajectory is the jth interaction trajectory in the i-th known task. Environment interaction trajectory Contains a series of four-tuples of time sequence states, actions, rewards, and next states, specifically the state at time step t The action taken at time step t The immediate reward obtained at time step t and the next state to be transferred to after time step t Environment interaction trajectory dataset D traj is the set of all interaction trajectories for all known tasks, and the length of each interaction trajectory is T i,j , represents the number of time steps of the jth interaction trajectory in the i-th known task;

[0016] S13. Environment interaction trajectory dataset D traj All states, actions, and rewards in the task are normalized, and the global mean μ of all states in all known tasks is calculated. s and standard deviation σ s , each state Subtract the global mean μ of the state s and divided by the global standard deviation σ of the state s , get the normalized state; calculate the global mean μ of all actions in all known tasks respectively a and standard deviation σ a , every action Subtract the global mean μ of the action a and divided by the global standard deviation σ of the action a , get the normalized action; calculate the global mean μ of all rewards in all known tasks respectively r and standard deviation σ r , each instant reward Subtract the global mean of the reward μ r and divided by the global standard deviation of the reward σ r , get the normalized immediate reward;

[0017] S14. After normalization, all normalized states, normalized actions, and normalized rewards are combined with the next state in the original environment interaction trajectory to form the standardized environment interaction trajectory dataset D norm .

[0018] Optionally, S2 includes the following steps:

[0019] S21. Based on the standardized environment interaction trajectory dataset D norm Extract each known task T i The trajectory feature representation is represented by the task trajectory encoding network for each task T i A collection of task-level trajectories Encode and get task T i The corresponding task embedding vector e i , task embedding vector is a real-valued vector of dimension d, used to represent task T i Overall interactive behavior characteristics;

[0020] S22. All known tasks T i The task embedding vector e i Composed task embedding vector set E = e1, e2, ..., e N , where N is the number of known tasks in the task set T, and the task embedding vector set E is used as the basic input for modeling the structural relationship between tasks;

[0021] S23. Construct a task embedding graph G = (V, E), where the node set V of the graph is the embedding vector e of each task in the task set i , the edge set E of the graph is used to describe the similarity relationship between any two tasks. The weight of each edge represents the embedding distance or semantic similarity between tasks. Define task T i With Task T j The edge weight set w between i,j ;

[0022] S24. Normalized edge weight set w of task embedding graph i,j , generate the normalized similarity matrix

[0023] Optionally, S3 includes the following steps:

[0024] S31. Constructing a graph convolutional scheduling network G based on the task embedding graph G and the normalized similarity matrix W GCN , taking the task embedding vector set as the initial node feature, and using graph convolution operation to fuse adjacent task embedding information to generate context-aware task representation Context-aware task representation To express the task T i Dynamic distribution and impact relationship relative to other tasks in the task map;

[0025] S32. Representation based on context-aware tasks Construct task contrast loss module L contrast , each task T i As an anchor task, it has a high similarity task N pos (i) As a positive sample pair, low similarity task N neg (i) As negative sample pairs, calculate the structure preservation loss in the task representation space:

[0026]

[0027] Among them, sim(·,·) is the cosine similarity function, τ is the temperature coefficient, L contrast It is used to explicitly preserve the task distribution structure in the policy space and provide structural consistency guarantee for multi-task policy generation;

[0028] S33. Representing context-aware tasks Input multi-scale policy generator G multi The multi-scale strategy generator includes a coarse-grained control module G with information bottleneck regulation coarse-IB With the fine-grained parameter module G fine ; Among them, the coarse-grained control module G coarse-IB Task-based representation Compared with the task loss L contrast The gradient information of the dynamic compression representation dimension is introduced, the task information bottleneck mechanism is introduced, and the compressed representation z is calculated. i , so that the task learning sequence π order To meet the minimum representation redundancy and optimal task arrangement coordination, the optimization goal is:

[0029]

[0030] Among them, L explore is the exploration-exploitation adjustment loss, KL(·) is the Kullback-Leibler divergence, β is the bottleneck weight coefficient, is the compression encoder distribution, p(z) is the prior Gaussian distribution;

[0031] S34. The coarse-grained control module outputs the compressed task representation z i , and based on z i Generate task learning sequence π or der, exploration intensity curve η exp lore(t) and resource allocation vector ρa lloc, respectively represents the multi-task execution order, the exploration intensity change rule in the strategy training phase, and the training resource allocation weight of each task;

[0032] S35. Fine-grained parameter module G fine Based on the task compression representation z i And the coarse-grained results, output the corresponding strategy initialization parameter set Θ init and optimize the hyperparameter set Φ opt , where the optimized hyperparameter set Φ opt Including the learning rate α i , discount factor γ i and the target update coefficient λ i ;

[0033] S36. Set the task learning order π order , explore the intensity curve η explore (t), resource allocation vector ρ alloc , strategy initialization parameter set Θ init and optimize the hyperparameter set Φ opt Integrate into the final multi-scale learning schedule S multi .

[0034] Optionally, the S4 includes the following steps:

[0035] S41. Take the task embedding graph G and the normalized similarity matrix W as input and jointly represent the context-aware task Input to the multi-scale policy generator G multi , as each known task T i developing a multi-scale learning agenda;

[0036] S42. In the multi-scale strategy generator G multi In the coarse-grained control module G coarse-IB Using the compressed representation z of each task i , determine the task learning order π order , task learning order π order =[π1,π2,…,π N ] is the optimal execution order of each task in the task set T, π k Number the kth task to be executed;

[0037] S43. Coarse-grained control module G coarse-IB According to the compressed representation z i Further output exploration intensity curve η explore (t), exploration intensity curve η explore (t) is the exploration parameter change function of each task at different training stages, satisfying η explore(t)∈(0,1), and output the resource allocation vector ρ alloc =[ρ1,ρ2,…,ρ N ], resource allocation vector ρ alloc Medium ρ k For task T k The proportion of computing resources allocated during the training phase,

[0038]

[0039] S44. Fine-grained parameter module G fine Based on the compressed representation z of each task i And coarse-grained results, generate the strategy initialization parameters for each task Strategy initialization parameter set For task T i Initialization strategy parameters;

[0040] S45. Fine-grained parameter module G fine Synchronously output the optimized hyperparameter set Φ for each task opt =φ1,φ2,…,φ N , where φ i =(α i ,γ i ,λ i ), α i For task T i The learning rate, γ i For task T i The discount factor, λ i For task T i The target update coefficient of

[0041] S46. Summary of task learning order π order , explore the intensity curve η explore (t), resource allocation vector ρ alloc , strategy initialization parameter set Θ init and optimize the hyperparameter set Φ opt , generate a multi-scale learning schedule S for all known tasks in the task set T multi .

[0042] Optionally, the S5 includes the following steps:

[0043] S51. Constructing Meta-Strategy Network M meta , embed the task into the graph G, normalize the similarity matrix W, and represent the context-aware task and the multi-scale learning schedule S multiAs joint input, the task embedding graph G and the normalized similarity matrix W are used to express the structure and similarity relationship between tasks, and the context-aware task representation Used to reflect task T i Global and local features, multi-scale learning schedule S multi Used to schedule and initialize various parameters and execution order in the multi-task self-learning process;

[0044] S52. In a multi-task training environment, using the meta-policy network M meta Perform reinforcement learning meta-training on the task set T in parallel or batch. The meta-training goal is to maximize the expected cumulative reward of all tasks. The meta-gradient update method is used. The meta-policy network parameter is denoted as ψ. Each meta-training iteration updates the parameter according to the following formula:

[0045]

[0046] Among them, η meta is the meta-learning rate, is the meta-loss function, which takes multi-task cumulative reward, policy generalization performance and training stability as weighted objectives. All training samples and feedback are determined by the multi-scale learning schedule S multi Unified scheduling and sampling;

[0047] S53. After multiple rounds of meta-training, the meta-strategy network M meta Output meta-strategy parameter prior set Meta-policy parameter priors For task T i The initial policy parameters that can be transferred across tasks are kept consistent with task T i The initialization dimensions of the strategy parameters are consistent;

[0048] S54. Synchronously generate a set of meta-level update rules, which are used to describe the task T i The parameter update method used in the self-learning process, Including learning rate adjustment rules, meta-level replay weights and dynamic adaptation coefficients.

[0049] The beneficial effects of the present invention are:

[0050] (1) The present invention uses a task trajectory encoding network to perform a high-dimensional representation of the environmental interaction trajectory, and then constructs a task embedding graph and introduces a graph convolutional scheduling network. The similarity structure between tasks is encoded as a learnable adjacency weight, and the adjacent task information is integrated in the strategy generation process to generate context-aware task representations. The graph modeling and graph convolution mechanism can dynamically capture the distribution structure between tasks, effectively avoiding strategy redundancy and migration interference. The structure-preserving loss function further constrains the distribution stability of the task representation, thereby improving the distinguishability of the task representation and the structural consistency of the strategy generation.

[0051] (2) The present invention introduces a multi-scale modeling framework into the strategy generator, compresses the task representation and generates the task execution order, exploration intensity curve and resource allocation weight through the coarse-grained control module; at the same time, the fine-grained parameter module outputs the strategy initialization parameters and optimization hyperparameter set corresponding to each task. The coarse-grained module uses information bottleneck compression and KL divergence constraint to suppress representation redundancy and enhance the arrangement coordination between tasks, thereby obtaining a more reasonable task execution sequence. The fine-grained module combines task characteristics with coarse-grained results to output refined strategy configuration, which significantly improves the initial performance of the model in migrating to new tasks.

[0052] (3) The present invention proposes a meta-strategy network that takes task embedding graph, context-aware task representation and multi-scale learning schedule as joint input, and uses meta-gradient optimization to train to obtain a priori set of cross-task transferable policy parameters and a set of meta-level update rules. It not only realizes migration initialization in the policy parameter space, but also unifies the scheduling strategy for parameter updates, effectively improving the performance stability of the strategy in the early stages of unknown tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0054] Figure 1 This is a flowchart of a multi-task self-learning strategy generation method based on reinforcement meta-learning proposed in this invention. DETAILED DESCRIPTION

[0055] The present invention will now be described in further detail with reference to the accompanying drawings, which are simplified schematic diagrams that illustrate the basic structure of the present invention in a schematic manner.

[0056] refer to Figure 1 , a multi-task self-learning strategy generation method based on reinforcement meta-learning, including the following steps:

[0057] S1. Build a multi-task training environment and collect an environment interaction trajectory dataset in the multi-task training environment. The environment interaction trajectory dataset records the state, action, reward, and next state information of each known task in the multi-task training environment;

[0058] S2. Vectorize and encode the environment interaction trajectory dataset to obtain a set of task embedding vectors. Based on the set of task embedding vectors, a task embedding graph is generated. The task embedding graph is used to describe the similarity and structural relationship between known tasks.

[0059] S3. Construct a multi-scale strategy generator based on the task embedding graph. The multi-scale strategy generator includes a coarse-grained control module and a fine-grained parameter module. The coarse-grained control module outputs the task learning sequence, exploration intensity curve, and resource allocation plan. The fine-grained parameter module outputs the strategy initialization parameter set and the optimization hyperparameter set.

[0060] S4. Input the task embedding graph into the multi-scale strategy generator to generate a multi-scale learning schedule for each known task. The multi-scale learning schedule includes the task learning sequence, exploration intensity curve, and resource allocation plan generated by the coarse-grained control module, and the strategy initialization parameter set and optimization hyperparameter set generated by the fine-grained parameter module.

[0061] S5. Build a meta-policy network and use the task embedding graph and multi-scale learning schedule as input. Perform meta-training via reinforcement learning in a multi-task training environment to obtain a set of meta-policy parameter priors and a set of meta-level update rules.

[0062] S6. For a newly emerged target task, the early environment interaction trajectory of the target task is first input into the task embedding vector model to obtain the target task embedding vector. The target task embedding vector and the task embedding graph are then jointly input into the multi-scale strategy generator to output a multi-scale learning schedule for the target task.

[0063] In this embodiment, S1 includes the following steps:

[0064] S11. Set a task set T, which contains multiple known tasks T i , each known task T i Both are defined as Markov decision processes, which are composed of the state space S i , action space A i , state transition probability distribution P i , immediate reward function R i and the discount factor γ i Composition, state space S i Represents the set of all possible states contained in the i-th known task, the action space A iRepresents the set of all possible actions contained in the i-th known task, and the state transition probability distribution P i It is used to describe the probability relationship of transferring to the next state after the reinforcement learning agent is in a certain state and takes a certain action in the i-th known task, that is, taking action a in state s to transfer to the next state s ′ The probability of immediate reward function R i Used to describe the immediate reward value obtained by taking action a in state s in the i-th known task, the discount factor γ i is the discount attenuation coefficient of the i-th known task, which is used to control the influence of long-term benefits in the optimization process;

[0065] S12. For each known task T i Deploy reinforcement learning agents in the environment to interact with the multi-task training environment, and each interaction process generates an environmental interaction trajectory. Environmental interaction trajectory is the jth interaction trajectory in the i-th known task. Environment interaction trajectory Contains a series of four-tuples of time sequence states, actions, rewards, and next states, specifically the state at time step t The action taken at time step t The immediate reward obtained at time step t and the next state to be transferred to after time step t Environment interaction trajectory dataset D traj is the set of all interaction trajectories for all known tasks, and the length of each interaction trajectory is T i,j , represents the number of time steps of the jth interaction trajectory in the i-th known task;

[0066] S13. Environment interaction trajectory dataset D traj All states, actions, and rewards in the task are normalized, and the global mean μ of all states in all known tasks is calculated. s and standard deviation σ s , each state Subtract the global mean μ of the state s and divided by the global standard deviation σ of the state s , get the normalized state; calculate the global mean μ of all actions in all known tasks respectively a and standard deviation σ a , every action Subtract the global mean μ of the action a and divided by the global standard deviation σ of the action a , get the normalized action; calculate the global mean μ of all rewards in all known tasks respectively r and standard deviation σr , each instant reward Subtract the global mean of the reward μ r and divided by the global standard deviation of the reward σ r , get the normalized immediate reward;

[0067] S14. After normalization, all normalized states, normalized actions, and normalized rewards are combined with the next state in the original environment interaction trajectory to form the standardized environment interaction trajectory dataset D norm .

[0068] In this embodiment, S2 includes the following steps:

[0069] S21. Based on the standardized environment interaction trajectory dataset D norm Extract each known task T i The trajectory feature representation is represented by the task trajectory encoding network for each task T i A collection of task-level trajectories Encode and get task T i The corresponding task embedding vector e i , task embedding vector is a real-valued vector of dimension d, used to represent task T i Overall interactive behavior characteristics;

[0070] S22. All known tasks T i The task embedding vector e i Composed task embedding vector set E = e1, e2, ..., e N , where N is the number of known tasks in the task set T, and the task embedding vector set E is used as the basic input for modeling the structural relationship between tasks;

[0071] S23. Construct a task embedding graph G = (V, E), where the node set V of the graph is the embedding vector e of each task in the task set i , the edge set E of the graph is used to describe the similarity relationship between any two tasks. The weight of each edge represents the embedding distance or semantic similarity between tasks. Define task T i With Task T j The edge weight set w between i,j ;

[0072] S24. Normalized edge weight set w of task embedding graph i,j , generate the normalized similarity matrix

[0073] In this embodiment, S3 includes the following steps:

[0074] S31. Constructing a graph convolutional scheduling network G based on the task embedding graph G and the normalized similarity matrix W GCN , taking the task embedding vector set as the initial node feature, and using graph convolution operation to fuse adjacent task embedding information to generate context-aware task representation Context-aware task representation To express the task T i Dynamic distribution and impact relationship relative to other tasks in the task map;

[0075] S32. Representation based on context-aware tasks Construct task contrast loss module L contrast , each task T i As an anchor task, it has a high similarity task N pos (i) As a positive sample pair, low similarity task N neg (i) As negative sample pairs, calculate the structure preservation loss in the task representation space:

[0076]

[0077] Among them, sim(·,·) is the cosine similarity function, τ is the temperature coefficient, L contrast It is used to explicitly preserve the task distribution structure in the policy space and provide structural consistency guarantee for multi-task policy generation;

[0078] S33. Representing context-aware tasks Input multi-scale policy generator G multi The multi-scale strategy generator includes a coarse-grained control module G with information bottleneck regulation coarse-IB With the fine-grained parameter module G fine ; Among them, the coarse-grained control module G coarse-IB Task-based representation Compared with the task loss L contrast The gradient information of the dynamic compression representation dimension is introduced, the task information bottleneck mechanism is introduced, and the compressed representation z is calculated. i , so that the task learning sequence π order To meet the minimum representation redundancy and optimal task arrangement coordination, the optimization goal is:

[0079]

[0080] Among them, L explore is the exploration-exploitation adjustment loss, KL(·) is the Kullback-Leibler divergence, β is the bottleneck weight coefficient, is the compression encoder distribution, p(z) is the prior Gaussian distribution;

[0081] S34. The coarse-grained control module outputs the compressed task representation z i , and based on z i Generate task learning sequence π or der, exploration intensity curve η exp lore(t) and resource allocation vector ρ a lloc, respectively represents the multi-task execution order, the exploration intensity change rule in the strategy training phase, and the training resource allocation weight of each task;

[0082] S35. Fine-grained parameter module G fine Based on the task compression representation z i And the coarse-grained results, output the corresponding strategy initialization parameter set Θ init and optimize the hyperparameter set Φ opt , where the optimized hyperparameter set Φ opt Including the learning rate α i , discount factor γ i and the target update coefficient λ i ;

[0083] S36. Set the task learning order π order , explore the intensity curve η explore (t), resource allocation vector ρ alloc , strategy initialization parameter set Θ init and optimize the hyperparameter set Φ opt Integrate into the final multi-scale learning schedule S multi .

[0084] In this embodiment, S4 includes the following steps:

[0085] S41. Take the task embedding graph G and the normalized similarity matrix W as input and jointly represent the context-aware task Input to the multi-scale policy generator G multi , as each known task T i developing a multi-scale learning agenda;

[0086] S42. In the multi-scale strategy generator G multi In the coarse-grained control module G coarse-IB Using the compressed representation z of each task i , determine the task learning order π order , task learning order π order =[π1,π2,…,π N ] is the optimal execution order of each task in the task set T, π k Number the kth task to be executed;

[0087] S43. Coarse-grained control module G coarse-IBAccording to the compressed representation z i Further output exploration intensity curve η explore (t), exploration intensity curve η explore (t) is the exploration parameter change function of each task at different training stages, satisfying η explore (t)∈(0,1), and output the resource allocation vector ρ alloc =[ρ1,ρ2,…,ρ N ], resource allocation vector ρ alloc Medium ρ k For task T k The proportion of computing resources allocated during the training phase,

[0088]

[0089] S44. Fine-grained parameter module G fine Based on the compressed representation z of each task i And coarse-grained results, generate the strategy initialization parameters for each task Strategy initialization parameter set For task T i Initialization strategy parameters;

[0090] S45. Fine-grained parameter module G fine Synchronously output the optimized hyperparameter set Φ for each task opt =φ1,φ2,…,φ N , where φ i =(α i ,γ i ,λ i ), α i For task T i The learning rate, γ i For task T i The discount factor, λ i For task T i The target update coefficient of

[0091] S46. Summary of task learning order π order , explore the intensity curve η explore (t), resource allocation vector ρ alloc , strategy initialization parameter set Θ init and optimize the hyperparameter set Φ opt , generate a multi-scale learning schedule S for all known tasks in the task set T multi .

[0092] In this embodiment, S5 includes the following steps:

[0093] S51. Constructing Meta-Strategy Network M meta, embed the task into the graph G, normalize the similarity matrix W, and represent the context-aware task and the multi-scale learning schedule S multi As joint input, the task embedding graph G and the normalized similarity matrix W are used to express the structure and similarity relationship between tasks, and the context-aware task representation Used to reflect task T i Global and local features, multi-scale learning schedule S multi Used to schedule and initialize various parameters and execution order in the multi-task self-learning process;

[0094] S52. In a multi-task training environment, using the meta-policy network M meta Perform reinforcement learning meta-training on the task set T in parallel or batch. The meta-training goal is to maximize the expected cumulative reward of all tasks. The meta-gradient update method is used. The meta-policy network parameter is denoted as ψ. Each meta-training iteration updates the parameter according to the following formula:

[0095]

[0096] Among them, η meta is the meta-learning rate, is the meta-loss function, which takes multi-task cumulative reward, policy generalization performance and training stability as weighted objectives. All training samples and feedback are determined by the multi-scale learning schedule S multi Unified scheduling and sampling;

[0097] S53. After multiple rounds of meta-training, the meta-strategy network M meta Output meta-strategy parameter prior set Meta-policy parameter priors For task T i The initial policy parameters that can be transferred across tasks are kept consistent with task T i The initialization dimensions of the strategy parameters are consistent;

[0098] S54. Synchronously generate a set of meta-level update rules, which are used to describe the task T i The parameter update method used in the self-learning process, Including learning rate adjustment rules, meta-level replay weights and dynamic adaptation coefficients.

[0099] Example 1:

[0100] On September 12, 2024, at the A City Artificial Intelligence Joint Experimental Center, at an industrial project site called "Cross-Env RL", three engineers were responsible for debugging a cross-task learning control system. The goal was to achieve rapid deployment of adaptive strategies in a multi-scenario robot testing platform. The platform contains actual operating environments for nine different task types, covering typical task scenarios including precise assembly of industrial robotic arms, navigation and obstacle avoidance of service robots, load-bearing climbing of quadruped robots, and path planning of intelligent inspection vehicles.

[0101] The initial project approach used conventional policy migration methods for scheduling. Around 10:00 AM on September 13th, engineers attempted to migrate a policy trained on the "inspection vehicle path planning" task to the "robotic arm grasping" task. After the migration, the model exhibited abnormal behavior, with grasping trajectories shifting. This resulted in a grasping failure rate exceeding 60% during the task, and the arm failed to properly contact the target component seven out of ten consecutive trials. Zhou Yu noted in the experimental log: "The policy exhibited severe jitter during the cold start phase of the new scenario, indicating unstable target positioning, presumably due to a mismatch in initialization weights."

[0102] To solve the above problems, engineers decided to deploy the multi-task self-learning strategy generation system proposed in this invention for strategy scheduling. The system was named MetaSched-V2 and was deployed and launched at 13:45 on September 14, 2024. This system was first deployed on six known tasks, namely "obstacle avoidance for patrol vehicles", "assembly of robotic arms", "indoor navigation", "warehouse handling", "loaded climbing" and "quick retrieval". At the start of the system, the execution logs and trajectory data of each task in the recent period (August 28 to September 10, 2024) were imported, totaling about 28,000 valid records.

[0103] After deployment, the system automatically processes existing task data in the background. Starting at 2:10 PM on September 14, 2024, the system logs the data cleans and normalizes it, and creates a task similarity index. The annotation results show that the similarity between "inspection vehicle obstacle avoidance" and "indoor navigation" tasks is as high as 87%, while the similarity between "loaded climbing" and "robotic arm assembly" is only 12%. Based on these results, MetaSched-V2 automatically recommends a suggested execution order for each task and a priority strategy for task initialization.

[0104] At 9:35 AM on September 15, 2024, a new subtask of the "Warehouse Handling" task, "Grasping High-Rack Stacked Goods," was tested at the experimental site. This task had never appeared in the system's training task set before. The system recognized it as a new task and recorded the first interaction data at 9:36:02 AM (Device ID: MH-G3, State: Initial Position, Action: Right Arm Extended, Result: Grasping Failed). After collecting only eight initial interaction records, the system pushed a real-time notification: "New task embedding analysis completed. Beginning to generate a personalized policy learning path."

[0105] Starting at 9:40, the system called the two empirical models of "robotic arm assembly" and "warehouse handling" that were closest to the task structure among known tasks to initialize the strategy, and combined with the resource scheduling module to set the training resource ratio for the task to 12%. The system deployed the new strategy to the device MH-G3 for the first time at 9:45 and added a 3-second security verification countdown before execution.

[0106] After the new strategy was deployed, the engineering team conducted 30 rounds of experiments from 9:46 to 10:20. The results showed that the first-time grasp success rate jumped from 10% to 46.7%, and exceeded 70% after the 10th round. After the 18th round of system strategy fine-tuning, the target grasp success rate reached 83.3%. This is a significant improvement compared to the traditional transfer method (the third control group used a fixed strategy template, with a success rate of 37%). The experimental summary stated: "The strategy system demonstrated excellent rapid adaptability, quickly approaching the optimal action path while significantly reducing training time."

[0107] As the project progressed, the engineering team tried to let the system handle multiple task switching requests. From 13:10 to 16:00 on September 16, 2024, the platform conducted four rounds of task alternation training: switching from "warehouse handling" to "quick retrieval", then switching to "loaded climbing", and finally switching to "inspection vehicle obstacle avoidance". Traditional solutions generally have the problem of "re-cold start" in this type of cross-type task switching. On average, each round requires more than 30 minutes of retraining to achieve a performance of more than 60%. However, the strategy of the system of the present invention is initialized within 10 minutes after each round of switching. The average performance recovery speed of the switching strategy is 3.2 times that of the traditional solution. The time data in the platform log shows that the total switching time of the traditional method is 128 minutes, while the total time of MetaSched-V2 is only 38 minutes.

[0108] Furthermore, to ensure data security and system stability, the system retains complete version snapshot records at each strategy initialization and update node, generating real-time feedback during task execution. At 5:32 PM on September 17, 2024, during an unusual task execution, the MH-F2 device experienced an abnormal trajectory during a "Quick Pick" operation (an object slipped off the track). The system automatically rolled back to the strategy snapshot two cycles earlier and issued an alarm: "Strategy output deviates from the optimal path; angle deviation exceeded the limit detected," thus preventing a device collision.

[0109] By the end of the experiment on September 20, 2024, the system had completed the deployment of 12 types of task strategies, supported the initialization of 6 new tasks, and successfully generated 98 strategy scheduling logs. The overall comparison results showed that the method of the present invention shortened the time required for new task training by an average of 43%, improved the strategy cold start performance by an average of about 38%, and improved the task execution accuracy by more than 21%, which is far superior to the traditional reinforcement learning scheme and the meta-learning scheme for unstructured scheduling used in the experimental control group.

[0110] In summary, this embodiment takes a real industrial scenario as the background and completely reproduces the entire process of the present invention from task data collection, system embedding generation, strategy initialization, task scheduling to new task adaptation. The system shows significant efficiency, flexibility and stability in actual deployment, and effectively overcomes the problems of poor structural perception ability, high task switching cost and slow adaptation to new tasks in the existing technology in the multi-task reinforcement learning environment.

[0111] The present invention uses a task trajectory encoding network to perform high-dimensional representation of environmental interaction trajectories, then constructs a task embedding graph and introduces a graph convolutional scheduling network. The similarity structure between tasks is encoded as a learnable adjacency weight, and the adjacent task information is integrated in the strategy generation process to generate context-aware task representations. The graph modeling and graph convolution mechanism can dynamically capture the distribution structure between tasks, effectively avoiding strategy redundancy and migration interference. The structure-preserving loss function further constrains the distribution stability of the task representation, thereby improving the distinguishability of the task representation and the structural consistency of the strategy generation.

[0112] The present invention introduces a multi-scale modeling framework into the strategy generator, compresses task representations through a coarse-grained control module and generates task execution order, exploration intensity curve and resource allocation weight; at the same time, the fine-grained parameter module outputs the strategy initialization parameters and optimization hyperparameter set corresponding to each task. The coarse-grained module uses information bottleneck compression and KL divergence constraints to suppress representation redundancy and enhance the arrangement coordination between tasks, thereby obtaining a more reasonable task execution sequence. The fine-grained module combines task characteristics with coarse-grained results to output refined strategy configuration, which significantly improves the initial performance of the model in migrating to new tasks.

[0113] The present invention proposes a meta-policy network that takes task embedding graph, context-aware task representation and multi-scale learning schedule as joint input, and uses meta-gradient optimization to train to obtain a priori set of policy parameters and a set of meta-level update rules that are transferable across tasks. It not only realizes migration initialization in the policy parameter space, but also unifies the scheduling strategy for parameter updates, effectively improving the performance stability of the strategy in the early stages of unknown tasks.

[0114] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A multi-task self-learning strategy generation method based on reinforcement meta-learning, characterized by: The steps include: S1. Build a multi-task training environment and collect a dataset of environmental interaction trajectories in the multi-task training environment; S2. Vectorize and encode the environment interaction trajectory dataset to obtain a set of task embedding vectors, and generate a task embedding graph based on the set of task embedding vectors; S3. Construct a multi-scale strategy generator based on the task embedding graph; S4. Input the task embedding graph into the multi-scale strategy generator to generate a multi-scale learning schedule for each known task; S5. Build a meta-policy network and use the task embedding graph and multi-scale learning schedule as input. Perform meta-training via reinforcement learning in a multi-task training environment to obtain a set of meta-policy parameter priors and a set of meta-level update rules. S6. For newly emerged target tasks, the early environment interaction trajectory of the target task is input into the task embedding vector model to obtain the target task embedding vector. The target task embedding vector and the task embedding graph are then jointly input into the multi-scale strategy generator to output a multi-scale learning schedule for the target task.

2. A multi-task self-learning strategy generation method based on reinforcement meta-learning according to claim 1, characterized in that: Said S1 comprises the following steps: S11. Set a task set T, which contains multiple known tasks T i , each known task T i Both are defined as Markov decision processes, which are composed of the state space S i , action space A i , state transition probability distribution P i , immediate reward function R i and the discount factor γ i constitute; S12. For each known task T i Deploy reinforcement learning agents in the environment to interact with the multi-task training environment, and each interaction process generates an environmental interaction trajectory. Environmental interaction trajectory is the jth interaction trajectory in the i-th known task. Environment interaction trajectory Contains a series of four-tuples of time sequence states, actions, rewards, and next states, specifically the state at time step t The action taken at time step t The immediate reward obtained at time step t and the next state to be transferred to after time step t Environment interaction trajectory dataset D traj is the set of all interaction trajectories for all known tasks, and the length of each interaction trajectory is T i,j , represents the number of time steps of the jth interaction trajectory in the i-th known task; S13. Environment interaction trajectory dataset D traj All states, actions, and rewards in the task are normalized, and the global mean μ of all states in all known tasks is calculated. s and standard deviation σ s , each state Subtract the global mean μ of the state s and divided by the global standard deviation σ of the state s , get the normalized state; calculate the global mean μ of all actions in all known tasks respectively a and standard deviation σ a , every action Subtract the global mean μ of the action a and divided by the global standard deviation σ of the action a , get the normalized action; calculate the global mean μ of all rewards in all known tasks respectively r and standard deviation σ r , each instant reward Subtract the global mean of the reward μ r and divided by the global standard deviation of the reward σ r , get the normalized immediate reward; S14. After normalization, all normalized states, normalized actions, and normalized rewards are combined with the next state in the original environment interaction trajectory to form the standardized environment interaction trajectory dataset D norm .

3. The method for generating a multi-task self-learning strategy based on reinforcement meta-learning according to claim 2, characterized in that: The S2 comprises the following steps: S21. Based on the standardized environment interaction trajectory dataset D norm Extract each known task T i The trajectory feature representation is represented by the task trajectory encoding network for each task T i A collection of task-level trajectories Encode and get task T i The corresponding task embedding vector e i ; S22. All known tasks T i The task embedding vector e i The set of task embedding vectors E = {e1, e2, ..., e N }, where N is the number of known tasks in the task set T, and the task embedding vector set E is used as the basic input for modeling the structural relationship between tasks; S23. Construct a task embedding graph G = (V, E), where the node set V of the graph is the embedding vector e of each task in the task set i , the edge set E of the graph is used to describe the similarity relationship between any two tasks. The weight of each edge represents the embedding distance or semantic similarity between tasks. Define task T i With Task T j The edge weight set w between i,j ; S24. Normalized edge weight set w of task embedding graph i,j , generate the normalized similarity matrix 4. The method for generating a multi-task self-learning strategy based on reinforcement meta-learning according to claim 3, characterized in that: The S3 includes the following steps: S31. Constructing a graph convolutional scheduling network G based on the task embedding graph G and the normalized similarity matrix W GCN , taking the task embedding vector set as the initial node feature, and using graph convolution operation to fuse adjacent task embedding information to generate context-aware task representation Context-aware task representation To express the task T i Dynamic distribution and impact relationship relative to other tasks in the task map; S32. Representation based on context-aware tasks Construct task contrast loss module L contrast , each task T i As an anchor task, it has a high similarity task N pos (i) As a positive sample pair, low similarity task N neg (i) As negative sample pairs, calculate the structure preservation loss in the task representation space: Among them, sim(·,·) is the cosine similarity function, τ is the temperature coefficient, L contrast It is used to explicitly preserve the task distribution structure in the policy space and provide structural consistency guarantee for multi-task policy generation; S33. Representing context-aware tasks Input multi-scale policy generator G multi The multi-scale strategy generator includes a coarse-grained control module G with information bottleneck regulation coarse-IB With the fine-grained parameter module G fine ; Among them, the coarse-grained control module G coarse-IB Task-based representation Compared with the task loss L contrast The gradient information of the dynamic compression representation dimension is introduced, the task information bottleneck mechanism is introduced, and the compressed representation z is calculated. i , so that the task learning sequence π order To meet the minimum representation redundancy and optimal task arrangement coordination, the optimization goal is: Among them, L explore is the exploration-exploitation adjustment loss, KL(·) is the Kullback-Leibler divergence, β is the bottleneck weight coefficient, is the compression encoder distribution, p(z) is the prior Gaussian distribution; S34. The coarse-grained control module outputs the compressed task representation z i , and based on z i Generate task learning sequence π order , explore the intensity curve η explore (t) and resource allocation vector ρ alloc , respectively represent the multi-task execution order, the exploration intensity change law in the strategy training stage, and the training resource allocation weight of each task; S35. Fine-grained parameter module G fine Based on the task compression representation z i And the coarse-grained results, output the corresponding strategy initialization parameter set Θ init and optimize the hyperparameter set Φ opt , where the optimized hyperparameter set Φ opt Including the learning rate α i , discount factor γ i and the target update coefficient λ i ; S36. Set the task learning order π order , explore the intensity curve η explore (t), resource allocation vector ρ alloc , strategy initialization parameter set Θ init and optimize the hyperparameter set Φ opt Integrate into the final multi-scale learning schedule S multi .

5. The method for generating a multi-task self-learning strategy based on reinforcement meta-learning according to claim 1, characterized in that: The S4 comprises the following steps: S41. Take the task embedding graph G and the normalized similarity matrix W as input and jointly represent the context-aware task Input to the multi-scale policy generator G multi , as each known task T i developing a multi-scale learning agenda; S42. In the multi-scale strategy generator G multi In the coarse-grained control module G coarse-IB Using the compressed representation z of each task i , determine the task learning order π order , task learning order π order =[π1,π2,…,π N ] is the optimal execution order of each task in the task set T, π k Number the kth task to be executed; S43. Coarse-grained control module G coarse-IB According to the compressed representation z i Further output exploration intensity curve η explore (t), exploration intensity curve η explore (t) is the exploration parameter change function of each task at different training stages, satisfying η explore (t)∈(0,1), and output the resource allocation vector ρ alloc =[ρ1,ρ2,…,ρ N ], resource allocation vector ρ alloc Medium ρ k For task T k The proportion of computing resources allocated during the training phase, S44. Fine-grained parameter module G fine Based on the compressed representation z of each task i And coarse-grained results, generate the strategy initialization parameters for each task Strategy initialization parameter set For task T i Initialization strategy parameters; S45. Fine-grained parameter module G fine Synchronously output the optimized hyperparameter set Φ for each task opt =φ1,φ2,…,φ N , where φ i =(α i ,γ i ,λ i ), α i For task T i The learning rate, γ i For task T i The discount factor, λ i For task T i The target update coefficient of S46. Summary of task learning order π order , explore the intensity curve η explore (t), resource allocation vector ρ alloc , strategy initialization parameter set Θ init and optimize the hyperparameter set Φ opt , generate a multi-scale learning schedule S for all known tasks in the task set T multi .

6. The method for generating a multi-task self-learning strategy based on reinforcement meta-learning according to claim 5, characterized in that: The S5 comprises the following steps: S51. Constructing Meta-Strategy Network M meta , embed the task into the graph G, normalize the similarity matrix W, and represent the context-aware task and the multi-scale learning schedule S multi As joint input, the task embedding graph G and the normalized similarity matrix W are used to express the structure and similarity relationship between tasks, and the context-aware task representation Used to reflect task T i Global and local features, multi-scale learning schedule S multi Used to schedule and initialize various parameters and execution order in the multi-task self-learning process; S52. In a multi-task training environment, using the meta-policy network M meta Perform parallel or batch reinforcement learning meta-training on the task set T. The meta-training goal is to maximize the cumulative reward expectation of all tasks. The meta-gradient update method is used. The meta-policy network parameter is denoted as ψ. Each meta-training iteration updates the parameter according to the following formula: Among them, η meta is the meta-learning rate, is the meta-loss function, which takes multi-task cumulative reward, policy generalization performance and training stability as weighted objectives. All training samples and feedback are determined by the multi-scale learning schedule S multi Unified scheduling and sampling; S53. After multiple rounds of meta-training, the meta-strategy network M meta Output meta-strategy parameter prior set Meta-policy parameter priors For task T i The initial policy parameters that can be transferred across tasks are kept consistent with task T i The initialization dimensions of the strategy parameters are consistent; S54. Synchronously generate a set of meta-level update rules, which are used to describe the task T i The parameter update method used in the self-learning process, Including learning rate adjustment rules, meta-level replay weights and dynamic adaptation coefficients.

Images