Multi-agent meta-learning fast adaptation method

Abstract

A multi-agent meta-learning rapid adaptation method, the method comprises the following steps: constructing a multi-agent meta-learning rapid adaptation problem model in a dynamic communication environment; according to the multi-agent meta-learning rapid adaptation problem model in the dynamic communication environment, a learning framework adapted to the dynamic communication environment is designed, including a multi-task meta-learning module and a dynamic task rapid migration module; the learning framework adapted to the dynamic communication environment is trained for multi-task to obtain a meta model; new task training data is obtained, and the meta model is adapted and trained to obtain an adapted model; and the multi-agent is cooperatively controlled by using the adapted model. The above method can quickly activate the effective mode in the historical strategy when the communication condition mutates by establishing a multi-environment perception and experience reuse cooperation mechanism, significantly reduces the strategy adjustment cost, and provides a new technical path for the practicalization of the multi-agent system in the dynamic changing environment.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a rapid adaptation method for multi-agent learning. Background Technology

[0002] Multi-agent systems in real-world applications face core challenges arising from unknown dynamic changes, particularly policy mismatch caused by abrupt changes in the communication environment ("dynamic communication environment" specifically refers to the non-stationary adjustment of the communication rate, i.e., unpredictable changes in the number of objects an agent can communicate with). Traditional methods typically rely on the assumption of environmental stationarity, which conflicts with the complex conditions that may occur in real-world applications, such as communication interference and topology reconfiguration. When communication conditions change, the system needs to adapt quickly to the new environment. This places higher demands on the generalization and adaptive capabilities of algorithms.

[0003] Existing methods have limitations in handling such unknown dynamic challenges: First, when faced with untrained communication patterns, the lack of cross-environment feature extraction capabilities leads to policy failure; second, online learning-based retraining mechanisms consume a large number of interaction samples and computational resources, making it difficult to meet real-time response requirements. More importantly, dynamic communication environments require agents to maintain cooperative performance while achieving rapid transfer and adaptive reconstruction of cross-environment policy knowledge, which poses further challenges to the transfer efficiency and robustness of the algorithms. Summary of the Invention

[0004] Therefore, it is necessary to provide a rapid adaptation method for multi-agent learning to address the aforementioned technical problems.

[0005] A fast adaptation method for multi-agent learning, the method comprising: Constructing a multi-agent learning model for rapid adaptation in dynamic communication environments; Based on the multi-agent meta-learning fast adaptation problem model in the dynamic communication environment, a learning framework adapted to the dynamic communication environment is designed, including a multi-task meta-learning module and a dynamic task fast transfer module. The learning framework adapted to the dynamic communication environment is trained on multiple tasks to obtain a meta-model; Acquire training data for the new task and perform adaptive training on the meta-model to obtain the adapted model; The adapted model is used to perform collaborative control of the multiple agents.

[0006] In one embodiment, the construction of a multi-agent meta-learning fast adaptation problem model in a dynamic communication environment includes defining the composition of the multi-agents, discrete time steps, environmental information, and communication mode identifiers, and defining the observation space, communication messages, and policy functions of each agent.

[0007] In one embodiment, the multi-task training of the learning framework adapted to the dynamic communication environment to obtain the meta-model includes: Based on the communication mode identifier, a multi-task Markov decision model is constructed, with each task corresponding to a communication rate; all tasks share a unified policy network. Samples are collected sequentially from the experience pools of each task, and the shared parameters of the policy network are iteratively updated. Repeat the iteration until the shared parameters of the policy network no longer change, thus obtaining the meta-model.

[0008] In one embodiment, the step of acquiring new task training data and adapting the meta-model to obtain the adapted model includes: Meta-training sample data is obtained by randomly sampling from the experience pools of each task; Calculate the importance ratio between the new task training data and the meta-training sample data; Calculate the regularization coefficient; The meta-model strategy is optimized using the importance ratio and the regularization coefficient, and the meta-model parameters are updated.

[0009] In one embodiment, the step of sequentially collecting samples from each task experience pool and iteratively updating the shared parameters of the policy network includes: Step 501: Perform round-robin sampling on each task in sequence to obtain the sample set of each task; Step 502: Calculate the temporal difference loss for the current task based on the sample set for each task; Step 503: Perform gradient update and sharing on the policy network parameters; Step 504: Reset the task pointer and repeat steps 502 to 503 until all tasks have been traversed.

[0010] In one embodiment, calculating the importance ratio of the new task training data and the meta-training sample data includes: The new task training data samples and the meta-training sample data samples are respectively labeled; The labeled samples are trained using a logistic regression model to minimize the cross-entropy loss. The importance ratio is calculated using the method of minimizing cross-entropy loss.

[0011] In one embodiment, calculating the regularization coefficient includes: Calculate the standardized effective sample size based on the aforementioned meta-training sample data; The regularization coefficient is calculated based on the standardized sample size.

[0012] In one embodiment, optimizing the meta-model strategy and updating the meta-model parameters using the importance ratio and the regularization coefficient includes: Using the training data samples of the new task, specific feature learning is performed on the meta-model; Based on the importance ratio, the meta-model learned from the specific features is jointly optimized, and the parameters of the meta-model are updated.

[0013] The aforementioned multi-agent meta-learning fast adaptation method first abstracts the heterogeneous communication environment into a Markov decision task set, designs progressive multi-task training objectives, and enables the meta-model to extract common features of cross-task policies during alternating learning, possessing continuous generalization characteristics from weak to strong communication. Then, a logistic regression adaptation mechanism is introduced, which evaluates the similarity in distribution between the meta-training data and new task data, and utilizes an importance sampling mechanism to achieve efficient reuse of historical experience. With the support of a small number of new samples, cross-environment data association analysis and policy parameter orientation adjustment are completed, thereby improving adaptation efficiency and shortening the retraining cycle of the dynamic communication system. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating a fast adaptation method for multi-agent learning in one embodiment; Figure 2 System framework design for one embodiment; Figure 3 This is a multi-task training process in one embodiment; Figure 4 This is a fast migration framework in one embodiment. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0016] In one embodiment, a multi-agent learning fast adaptation method is described in the following flowchart: Figure 1 The method includes: Constructing a multi-agent learning model for rapid adaptation in dynamic communication environments; Based on the multi-agent learning fast adaptation problem model under the dynamic communication environment, a learning framework adapted to the dynamic communication environment (such as...) is designed. Figure 2 ), including a multi-task meta-learning module and a dynamic task fast migration module; Multi-task training is performed on the learning framework adapted to dynamic communication environments (e.g.) Figure 3 ), thus obtaining the meta-model; Acquire training data for the new task and perform adaptive training on the meta-model to obtain the adapted model (e.g., Figure 4 ); The adapted model is used to perform cooperative control of the multi-agent system. In one embodiment, the multi-agent meta-learning fast adaptation problem model in a dynamic constructivist communication environment includes defining the composition of the multi-agent system, discrete time steps, environmental information, and communication mode identifiers, and defining the observation space, communication messages, and policy functions for each agent.

[0017] Specifically, consider by Each of our intelligent agents and An adversarial system consisting of several adversary intelligent agents, whose interaction process occurs in discrete time steps. Expanding on the environmental conditions. Includes situational information and current communication mode identifier ,in For a predefined set of communication patterns, each pattern Corresponding to fixed communication rate The communication rate determines the upper limit of the number of connections an agent can establish, for example... At that time, each intelligent agent can interact with at most [number] other agents. Neighbor communication.

[0018] During system operation, the communication mode may abruptly change due to external interference. Define a communication mode switching event. After it is triggered, the current mode changes from Switch to Such mutations lead to a reconstruction of the communication adjacency matrix: for each pattern Corresponding adjacency matrix Row constraints are For example, when the communication rate switches from 80% to 30%, the adjacency matrix changes abruptly from a densely connected structure to a sparse hierarchical structure. This restructuring not only changes the number of connections but also alters the information propagation path and collaboration logic.

[0019] intelligent agent The observation space is expanded to ,in For local battlefield awareness information (such as relative positions of enemies and allies, resource status), For communication messages received based on the current adjacency matrix, This identifies the current communication rate. (Policy function) The goal is to maximize the global cumulative reward in the long run. .

[0020] In one embodiment, the multi-task training of the learning framework adapted to the dynamic communication environment to obtain the meta-model includes: Based on the communication mode identifier, a multi-task Markov decision model is constructed, with each task corresponding to a communication rate; all tasks share a unified policy network. Specifically, the core objective of multi-task meta-training is to jointly train multiple tasks with different communication rates, extract common policy features among the tasks, and accumulate diverse interaction trajectory data, thereby constructing a highly generalizable meta-model. This model can improve training efficiency and decision robustness in new environments by leveraging diverse patterns from historical experience when encountering unknown communication environments.

[0021] Tasks with different communication rates are uniformly modeled as a multi-task MDP framework with a shared policy space. A standard Markov decision process can be defined as a quintuple. ,in Let be the state transition probability. For the reward function, This is a discount factor. In multi-task extensions, each communication rate task... It inherits the basic MDP structure, but introduces differentiated communication constraints: (1) Among them, the adjacency matrix Dynamically constrain the communication links between intelligent agents and define the tasks. Each agent in the middle can have a maximum of 1,0 ... Communicating with each neighbor. For example, when As the value increases from 0.2 to 0.8, the system communication topology gradually changes from a star structure to a fully connected network.

[0022] All tasks share a unified policy network. Its parameters Policy updates are achieved through multi-task joint optimization. This design forces the policy network to extract common features (such as key node identification and information routing priority) from heterogeneous communication constraints, rather than memorizing the specificities of a single task. Specifically, policy updates must satisfy multi-task compatibility conditions: (2) in This is a communication feasibility indicator function. This is the temperature coefficient.

[0023] Samples are collected sequentially from the experience pools of each task, and the shared parameters of the policy network are iteratively updated. Repeat the iteration until the shared parameters of the policy network no longer change, thus obtaining the meta-model.

[0024] In one embodiment, the step of acquiring new task training data and adapting the meta-model to obtain the adapted model includes: Meta-training sample data is obtained by randomly sampling from the experience pools of each task; Calculate the importance ratio between the new task training data and the meta-training sample data; Specifically, in the adaptation process to new tasks, the meta-training strategy A small amount of new task data needs to be utilized. Rapid adjustments are possible. However, relying solely on new task data for policy optimization is extremely costly. Furthermore, limited sample size can negatively impact training accuracy and decision stability. To address these issues, a logistic regression model is introduced. By estimating the distributional similarity between new task data and the original training data, historical data can be efficiently reused, thereby improving the model's generalization ability and adaptation speed.

[0025] Importance ratio It is used to measure new task samples The estimation method for an index that compares to the similarity of historical data distributions is as follows. Assume the training distribution is... The test distribution is The importance ratio is Directly calculating the density ratio is somewhat difficult, so an approximate calculation is performed using a logistic regression model.

[0026] Calculate the regularization coefficient; The meta-model strategy is optimized using the importance ratio and the regularization coefficient, and the meta-model parameters are updated.

[0027] In one embodiment, calculating the importance ratio of the new task training data and the meta-training sample data includes: The new task training data samples and the meta-training sample data samples are respectively labeled; The labeled samples are trained using a logistic regression model to obtain a minimized cross-entropy loss; the importance ratio is then calculated using the minimized cross-entropy loss.

[0028] Specifically, from the meta-training experience pool Random sampling Trajectory Assigning labels Simultaneously from the new task buffer equal-sampling trajectory Assigning labels Next, a logistic regression model is used to train the labeled samples. The optimization objective of the logistic regression is to minimize the cross-entropy loss. (3) in, This is the weight vector of the logistic regression model. For feature mapping function, This is the regularization coefficient, used to prevent overfitting. The optimal solution to the above equation can be obtained by training a logistic regression model. This leads to the importance ratio: (4) This ratio quantifies the meta-training trajectory. Contribution to the update of the new task strategy. When the trajectory... When the feature distribution is highly similar to the new task, The weighting value tends towards 1, and vice versa. This data-driven weighting mechanism effectively mitigates the negative impact of distribution bias, allowing historical experience to be safely incorporated into the optimization of new strategies.

[0029] In one embodiment, the step of sequentially collecting samples from each task experience pool and iteratively updating the shared parameters of the policy network includes: Step 501: Perform round-robin sampling on each task in sequence to obtain the sample set of each task; Step 502: Calculate the temporal difference loss for the current task based on the sample set for each task; Step 503: Perform gradient update and sharing on the policy network parameters; Step 504: Reset the task pointer and repeat steps 502 to 503 until all tasks have been traversed.

[0030] Specifically, each iteration includes the following steps: (1) Task polling sampling: Create a circular task queue Tasks are selected sequentially in each round. From its experience pool Medium uniform sampling batch data ; (2) Loss calculation: Calculate the temporal difference loss of the current task. : (9) (3) Parameter gradient update: along the gradient direction Update shared parameters ; (4) Loop traversal mechanism: Reset the task pointer and repeat the above process until all tasks are traversed.

[0031] The objective function of this process is to minimize the average loss of all tasks: (10) The above process essentially forces the network parameters to be optimized through alternating optimization. The model converges to a solution space region that is friendly to all communication rate tasks. This alternating update strategy can effectively alleviate inter-task conflicts, allowing the model to retain task commonalities while avoiding overfitting to a single communication rate.

[0032] Experience pools for each task This task collects unique interaction trajectory data through its independent interaction process. Due to different communication rates This leads to differences in connection patterns between agents, resulting in significant diversity in trajectories across different experience pools: In sparse communication tasks (such as...) In the trajectory, the agent relies more on local information, and the choice of communication object is biased towards key nodes.

[0033] In intensive communication tasks (such as In this context, trajectories involve extensive global information exchange, and collaborative strategies place greater emphasis on group coordination.

[0034] Through multi-task joint training, the meta-model During the optimization process, the model simultaneously encounters these heterogeneous data, thereby learning policy primitives covering different communication patterns. For example, the model might learn a central node-based information relay policy from sparse tasks, while simultaneously learning distributed coordination rules from dense tasks. This diverse accumulation of experience provides a rich pool of policy candidates for rapid adaptation to subsequent new tasks.

[0035] In one embodiment, calculating the regularization coefficient includes: Calculate the standardized effective sample size based on the aforementioned meta-training sample data; The regularization coefficient is calculated based on the standardized sample size.

[0036] Specifically, to quantify the reuse effectiveness of historical experience, a standardized effective sample size (NESS) is introduced as a basis for adaptive adjustment: (7) NESS reflects the equivalent information content of the weighted meta-training data, and its value range is [0, 1]. When the distributions of new and old tasks are highly consistent, NESS approaches 1, indicating that historical experience can be fully trusted; when the distributions differ significantly, NESS approaches 0, suggesting that the weight of historical data needs to be reduced.

[0037] Based on the NESS index, the regularization coefficient Implement a dynamic adjustment strategy: (8) This mechanism forms a negative feedback adjustment loop: if the meta-training data is highly correlated with the new task (NESS→1), the regularization constraint is weakened, allowing the policy to absorb more historical experience; if the correlation is weak (NESS→0), the parameter conservatism is strengthened to avoid interference from outdated experience. This data-driven regularization strategy achieves the best trade-off between experience reuse and policy robustness.

[0038] In summary, the fast migration module combines logistic regression and importance ratio estimation to enable multi-agent systems to adapt to dynamic environments, aiming to improve policy migration efficiency, system stability, and robustness.

[0039] In one embodiment, optimizing the meta-model strategy and updating the meta-model parameters using the importance ratio and the regularization coefficient includes: Using the training data samples of the new task, specific feature learning is performed on the meta-model; Specifically, the focus is on learning task-specific features to ensure the strategy can adapt quickly with limited new data support. This process is achieved through the following formula: (5) Where regularization coefficient This stage controls the degree to which the model deviates from the initial meta-parameters. It's equivalent to performing a local search within the neighborhood of the prior distribution of the meta-model, avoiding policy oscillations caused by insufficient new data.

[0040] Based on the importance ratio, the meta-model learned from the specific features is jointly optimized, and the parameters of the meta-model are updated.

[0041] Specifically, the optimization objectives are as follows: (6) At this point, the optimization objective includes two key design elements: firstly, through... First, the meta-training trajectories are soft-filtered to retain experience highly relevant to the new task; second, the regularized anchor points are dynamically adjusted to reflect the results of the first stage. This forms a progressive optimization chain. This progressive structure not only ensures the stability of the initial adaptation, but also improves the generalization ability of the strategy by injecting historical data.

[0042] Specifically, the model training process is as follows: A two-stage progressive training architecture is adopted: the first stage builds the basic policy space through multi-task meta-training, and the second stage triggers a fast adaptation mechanism when changes in the communication environment are detected.

[0043] (1) Multi-task training phase 1. Task Initialization: Constructing the Communication Rate Spectrum Each corresponds to an independent experience pool. ; 2. Alternating Optimization: Traverse tasks using a round-robin mechanism, starting from each... Mid-sample batch data update shared parameters ; 3. Experience Accumulation: Keep the latest version of each task. A trajectory.

[0044] The training process is shown in the table below: (2) New task adaptation phase 1. Mutation detection: Monitoring communication rate When detected Time-triggered adaptation; 2. Data Collection: Collect interaction data for new tasks. ; 3. Two-stage optimization: Execute logistic regression-driven strategy fine-tuning.

[0045] The training process is shown in the table below.

[0046] The aforementioned multi-agent meta-learning fast adaptation method first abstracts the heterogeneous communication environment into a Markov decision task set, designs progressive multi-task training objectives, and enables the meta-model to extract common features of cross-task policies during alternating learning, possessing continuous generalization characteristics from weak to strong communication. Then, a logistic regression adaptation mechanism is introduced, which evaluates the similarity in distribution between the meta-training data and new task data, and utilizes an importance sampling mechanism to achieve efficient reuse of historical experience. With the support of a small number of new samples, cross-environment data association analysis and policy parameter orientation adjustment are completed, thereby improving adaptation efficiency and shortening the retraining cycle of the dynamic communication system.

[0047] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0048] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A fast adaptation method for multi-agent learning, characterized in that, The method includes: Constructing a multi-agent learning model for rapid adaptation in dynamic communication environments; Based on the multi-agent meta-learning fast adaptation problem model in the dynamic communication environment, a learning framework adapted to the dynamic communication environment is designed, including a multi-task meta-learning module and a dynamic task fast transfer module. The learning framework adapted to the dynamic communication environment is trained on multiple tasks to obtain a meta-model; Acquire training data for the new task and perform adaptive training on the meta-model to obtain the adapted model; The adapted model is used to perform collaborative control of the multiple agents.

2. The method according to claim 1, characterized in that, The construction of a multi-agent meta-learning fast adaptation problem model in a dynamic communication environment includes defining the composition of the multi-agents, discrete time steps, environmental information, and communication mode identifiers, and defining the observation space, communication messages, and policy functions of each agent.

3. The method according to claim 2, characterized in that, The multi-task training of the learning framework adapted to the dynamic communication environment to obtain the meta-model includes: Based on the communication mode identifier, a multi-task Markov decision model is constructed, with each task corresponding to a communication rate; all tasks share a unified policy network. Samples are collected sequentially from the experience pools of each task, and the shared parameters of the policy network are iteratively updated. Repeat the iteration until the shared parameters of the policy network no longer change, thus obtaining the meta-model.

4. The method according to claim 3, characterized in that, The process of acquiring new task training data and adapting the meta-model to obtain the adapted model includes: Meta-training sample data is obtained by randomly sampling from the experience pools of each task; Calculate the importance ratio between the new task training data and the meta-training sample data; Calculate the regularization coefficient; The meta-model strategy is optimized using the importance ratio and the regularization coefficient, and the meta-model parameters are updated.

5. The method according to claim 4, characterized in that, The step of sequentially collecting samples from each task experience pool and iteratively updating the shared parameters of the policy network includes: Step 501: Perform round-robin sampling on each task in sequence to obtain the sample set of each task; Step 502: Calculate the temporal difference loss for the current task based on the sample set for each task; Step 503: Perform gradient update and sharing on the policy network parameters; Step 504: Reset the task pointer and repeat steps 502 to 503 until all tasks have been traversed.

6. The method according to claim 5, characterized in that, The calculation of the importance ratio between the new task training data and the meta-training sample data includes: The new task training data samples and the meta-training sample data samples are respectively labeled; The labeled samples are trained using a logistic regression model to minimize the cross-entropy loss. The importance ratio is calculated using the method of minimizing cross-entropy loss.

7. The method according to claim 6, characterized in that, The calculation of the regularization coefficient includes: Calculate the standardized effective sample size based on the aforementioned meta-training sample data; The regularization coefficient is calculated based on the standardized sample size.

8. The method according to claim 7, characterized in that, The step of optimizing the meta-model strategy and updating the meta-model parameters using the importance ratio and the regularization coefficient includes: Using the training data samples of the new task, specific feature learning is performed on the meta-model; based on the importance ratio, the meta-model after specific feature learning is jointly optimized, and the parameters of the meta-model are updated.

Images