Multi-agent reinforcement learning generalization enhancement method and device, equipment and storage medium

Abstract

The invention provides a multi-agent reinforcement learning generalization enhancement method and device, equipment and a storage medium, and belongs to the technical field of agents, and the method comprises the steps: obtaining interaction data sets of a plurality of agents, and obtaining a plurality of interaction feature sets based on the interaction data sets of the plurality of agents; for each agent, performing complementary self-supervised learning based on the plurality of interactive feature groups and self feature data, and performing a self-supervised loss calculation operation to obtain a self-supervised learning loss corresponding to the agent; reinforcement learning is executed based on the multiple interaction feature groups, reinforcement learning loss calculation operation is carried out, and reinforcement learning loss corresponding to the agent is obtained; total loss is determined based on self-supervised learning loss and reinforcement learning loss; and training each agent based on the total loss until a preset stop condition is met, and obtaining a trained multi-agent reinforcement learning model. According to the invention, the generalization ability of multi-agent reinforcement learning can be improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent agents, and more particularly relates to a multi-agent reinforcement learning generalization enhancement method and device, equipment and a storage medium. BACKGROUND

[0002] Multi-agent reinforcement learning is an important machine learning technique, aiming to solve the problem of collaborative decision-making of multiple agents in a shared environment. This technique allows each agent to learn an optimal policy to maximize the common long-term reward, and has broad application prospects in complex scenarios such as autonomous driving, UAV swarm cooperative control, online multiplayer games, and smart grid scheduling.

[0003] However, the current multi-agent reinforcement learning technology faces a core technical bottleneck when moving from training environment to practical application: insufficient generalization ability. That is, a strategy model trained in a specific scenario will have a sharp decline in decision-making performance when deployed to an unknown new scenario where external conditions have changed, and even cause the system task to fail. Existing multi-agent reinforcement learning generalization techniques are difficult to effectively solve the problem of insufficient generalization ability of multi-agent reinforcement learning. SUMMARY

[0004] The purpose of the present application is to provide a multi-agent reinforcement learning generalization enhancement method, device, equipment and storage medium to improve the generalization ability of multi-agent reinforcement learning.

[0005] The first aspect of the embodiment of the present application provides a multi-agent reinforcement learning generalization enhancement method, comprising: obtaining an interaction data group of multiple agents, and obtaining multiple interaction feature groups based on the interaction data group of the multiple agents; for each agent, performing complementary self-supervised learning based on the multiple interaction feature groups and its own feature data, and performing self-supervised loss calculation operation to obtain the self-supervised learning loss corresponding to the agent; and performing reinforcement learning based on the multiple interaction feature groups, and performing reinforcement learning loss calculation operation to obtain the reinforcement learning loss corresponding to the agent; determining a total loss based on the self-supervised learning loss and the reinforcement learning loss; training each agent based on the total loss until a preset stopping condition is met, to obtain a trained multi-agent reinforcement learning model.

[0006] The second aspect of the embodiment of the present application provides a multi-agent reinforcement learning generalization enhancement device, comprising: a data acquisition module configured to obtain an interaction data group of multiple agents, and obtain multiple interaction feature groups based on the interaction data group of the multiple agents; The first loss determination module is configured to, for each agent, perform complementary self-supervised learning based on the multiple interaction feature groups and the self feature data, and perform a self-supervised loss calculation operation to obtain a self-supervised learning loss corresponding to the agent; and perform reinforcement learning based on the multiple interaction feature groups, and perform a reinforcement learning loss calculation operation to obtain a reinforcement learning loss corresponding to the agent. The second loss determination module is configured to determine a total loss based on the self-supervised learning loss and the reinforcement learning loss. The training module is configured to train each agent based on the total loss until a preset stop condition is met, to obtain a trained multi-agent reinforcement learning model.

[0007] In a third aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the steps of the multi-agent reinforcement learning generalization enhancement method described above are implemented.

[0008] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the multi-agent reinforcement learning generalization enhancement method described above are implemented.

[0009] The multi-agent reinforcement learning generalization enhancement method, device, equipment, and storage medium provided by the embodiments of the present application have the following advantages: The embodiments of the present application obtain the interaction data groups of multiple agents and obtain the interaction feature groups. For each agent, complementary self-supervised learning is performed based on the interaction feature groups and the self feature data, so that the model can learn more comprehensive system dynamic rules. Meanwhile, reinforcement learning is performed based on the interaction feature groups. The reinforcement learning can consider the influence of the joint behavior of the multiple agents on the reward, and utilize the joint information in the interaction feature groups. The self-supervised learning loss and the reinforcement learning loss are determined to obtain a total loss, and each agent is trained based on the total loss. As a result, the model can learn more rich and universal features and strategies in the training process, thereby effectively improving the generalization capability of the multi-agent reinforcement learning model in unknown new scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort based on these drawings.

[0011] Figure 1A flowchart illustrating a multi-agent reinforcement learning generalization enhancement method provided in an embodiment of this application; Figure 2 A flowchart illustrating another multi-agent reinforcement learning generalization enhancement method provided in an embodiment of this application; Figure 3 A schematic diagram of a complementary self-supervised learning process provided in an embodiment of this application; Figure 4 This is a structural block diagram of a multi-agent reinforcement learning generalization enhancement device provided in an embodiment of this application; Figure 5 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.

[0014] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a multi-agent reinforcement learning generalization enhancement method provided in an embodiment of this application. The method can be executed by an electronic device and may include: S101-S104.

[0015] S101: Obtain interaction data sets of multiple agents, and obtain multiple interaction feature sets based on the interaction data sets of multiple agents.

[0016] In this embodiment, multiple agents refer to independent decision-making units interacting in a shared environment, such as multiple vehicles in autonomous driving or multiple drones in a drone swarm. Each agent has its own observation, action, and decision-making objectives, but they need to collaborate to achieve common tasks, such as cooperative obstacle avoidance or joint exploration. An interaction data set refers to the raw data set generated by the interaction between agents and the environment, and between agents themselves. An interaction data set represents a complete state transition, which may include: the joint observation of all agents at time t, the joint action performed by all agents at time t, and the joint observation at time t+1 after the environment transition.

[0017] Joint observation at time t+1 after an environmental transition refers to the complete set of observation data acquired jointly by all agents after the environmental state is updated. Environmental transition refers to the state change of the environment caused by the actions of all agents at time t. For example, in autonomous driving, multiple vehicles simultaneously performing acceleration / steering actions at time t will cause changes in vehicle positions, spacing, and road relative relationships; this state change process is called environmental transition. Joint observation is not an independent observation by a single agent, but rather a summary of the observation results of all agents at time (t+1). Each agent can only see information from its own perspective, such as its own position and the surrounding local environment. Joint observation integrates the observation data of all agents to form complete data reflecting the entire multi-agent system and its environment.

[0018] In this embodiment, the interaction feature set refers to the abstract feature set extracted from the interaction data set that is more suitable for multi-agent model learning.

[0019] S102: For each agent, perform complementary self-supervised learning based on multiple interaction feature groups and its own feature data, and perform self-supervised loss calculation to obtain the self-supervised learning loss corresponding to the agent; and perform reinforcement learning based on multiple interaction feature groups, and perform reinforcement learning loss calculation to obtain the reinforcement learning loss corresponding to the agent.

[0020] In this embodiment, for each agent, distinct from the joint features of the interaction feature group, its own feature data is the agent's independent feature data. For example, it may include the agent's observation features (such as its own position and velocity) and action features (such as its own turning angle) at time t. Complementary self-supervised learning is a learning method that learns patterns through self-prediction of tasks without manual annotation. Its core is complementarity, focusing on both its own dynamics and the dynamics of other agents. The self-supervised learning loss in the complementary self-supervised learning process is used to measure the difference between the predicted and actual results.

[0021] In multi-agent systems, the decisions of a single agent are influenced not only by its own behavior but also by the behavior of other agents. This embodiment can learn more comprehensive system dynamics by simultaneously predicting the future states of itself and others, thereby improving its adaptability to unknown scenarios.

[0022] In this embodiment, reinforcement learning refers to the process by which multiple agents learn the optimal policy through trial and error. After an agent performs an action, the environment provides a reward, and the model optimizes the policy by maximizing the long-term cumulative reward. In this embodiment, reinforcement learning needs to consider the impact of the joint behavior of multiple agents on the reward. For example, the action of a single agent needs to cooperate with other agents to obtain a high reward; therefore, joint information from the interaction feature set is required. During the reinforcement learning process, the reinforcement learning loss can be calculated based on the difference between the expected cumulative reward and the actual reward. The reinforcement learning loss is used to measure the gap between the current policy and the optimal policy in reinforcement learning.

[0023] S103: Determine the total loss based on self-supervised learning loss and reinforcement learning loss.

[0024] In this embodiment, the total self-supervised learning loss can be obtained by summing the self-supervised learning losses of each agent and the total reinforcement learning loss can be obtained by summing the reinforcement learning losses of each agent. The total self-supervised learning loss and the total reinforcement learning loss are then weighted to determine the total loss. The weights for the weighting calculation can be set by the user.

[0025] S104: Train each agent based on the total loss until a preset stopping condition is met, and obtain the trained multi-agent reinforcement learning model.

[0026] In this embodiment, the gradient of the total loss with respect to all parameters of the model can be calculated based on the gradient descent algorithm, and the parameters of the agent can be updated. The preset stopping condition can be that the number of training rounds reaches a preset maximum value, or that the total loss value is less than a preset minimum value after N consecutive training rounds. When the preset stopping condition is met, the multi-agent reinforcement learning model after training is obtained.

[0027] See Figure 2 In one embodiment of this application, a batch of interactive data sets can be sampled from the experience replay buffer. Each interactive data set contains at least the observation data of each agent at time t. The shared feature extraction module can receive the observation data of each agent at time t, process it into a low-dimensional feature vector containing key information, and implement complementary self-supervised learning in the self-supervised learning module. Specifically, the self-supervised learning module uses the features output by the shared feature extraction module to perform a complementary future state prediction task and calculates the self-supervised learning loss. The reinforcement learning module is responsible for performing the standard multi-agent reinforcement learning task. It can learn the optimal decision strategy based on shared features and environmental rewards and calculate the reinforcement learning loss. The complementary multi-task balancing module receives the self-supervised learning loss calculated by the self-supervised learning module and the reinforcement learning loss calculated by the reinforcement learning module, and dynamically weights the two losses based on the uncertainty principle to generate a final total loss. The total loss is then backpropagated to collaboratively optimize the parameters of all modules.

[0028] This embodiment aims to address the practical deployment challenges of multi-agent systems in complex and dynamic environments. For example, in the field of autonomous driving, multiple intelligent vehicles need to make efficient and safe collaborative decisions on the road, such as maintaining safe platooning and cooperative lane changing in mixed traffic flow. The real problem is that road environments are constantly changing. The composition of the platoon may change at any time (new vehicles join or leave); the performance parameters of each vehicle are also not static, for example, differences in braking performance due to tire wear or load variations, and decreased perception accuracy due to sensor obstruction by dirt. Using the model trained in this embodiment, each vehicle in the platoon not only learns specific following or lane-changing strategies, but more importantly, it masters a general physical dynamic model of vehicle interactions through complementary self-supervised learning. When a vehicle's braking performance deteriorates, neighboring vehicles can predict changes in its behavior pattern by observing subtle dynamic changes and proactively adjust their own strategies (such as increasing the safe distance), thereby avoiding rear-end collisions. When new vehicles of different brands and with different driving habits join the convoy, the multi-agent reinforcement learning model, after inspection, can quickly adapt to the new members by leveraging its understanding of general dynamics, thus maintaining the stability and safety of the entire convoy.

[0029] As can be seen from the above, complementary self-supervised learning focuses on the dynamics of itself and other agents, enabling the model to learn more comprehensive system dynamic laws; reinforcement learning considers the impact of joint actions of multiple agents on rewards, utilizing joint information in interaction feature groups. This comprehensive learning approach allows the model to accumulate richer general knowledge during training, effectively improving the model's adaptability to unknown scenarios and significantly enhancing its generalization ability. In multi-agent systems, the decisions of a single agent are influenced by the combined actions of itself and other agents. The complementary self-supervised learning in this embodiment, by simultaneously predicting the future states of itself and others, allows the model to deeply understand the interaction laws between multiple agents. Taking autonomous driving as an example, when the braking performance of a vehicle deteriorates, nearby vehicles can observe its subtle dynamic changes, predict changes in its behavior patterns based on the general physical dynamic model learned by the model, and proactively adjust their own strategies, such as increasing the safe distance, thereby making more accurate decisions and effectively avoiding dangerous situations such as rear-end collisions.

[0030] In one embodiment of this application, in the complementary self-supervised learning task, besides predicting the next state characteristics of a teammate, the prediction target can also be other information that reflects the teammate's dynamics. For example, it can predict the specific action or action probability distribution that the teammate will perform in the next moment. It can predict relational information related to the teammate, such as the relative distance or relative speed in the next moment, or whether they are within communication range.

[0031] In addition to predicting the agent's complete next-moment observation, it can also predict the difference between observations, or only predict the most critical and frequently changing parts of the observation, such as its own health value and coordinates, in order to reduce the difficulty and computational cost of the prediction task.

[0032] In one embodiment of this application, the complementary multi-task balancing module can also be implemented in any of the following ways: Gradient-based balancing methods: In addition to uncertainty-based methods, gradient-based dynamic balancing strategies can also be employed. For example, the gradient norms of the self-supervised learning loss and reinforcement learning loss backpropagated to the shared feature extraction module can be monitored and normalized. By adjusting the loss weights, the gradient magnitudes of different tasks can be kept at a similar order of magnitude, ensuring a balanced learning rate.

[0033] Task performance-based balancing methods can dynamically adjust weights based on the performance improvement of each task. For example, a performance monitor can be set up to dynamically increase the weight of the reinforcement learning loss when the reward growth of the reinforcement learning task stagnates; conversely, when the prediction error of self-supervised learning decreases too quickly or too slowly, its weight can be adjusted accordingly, forming a feedback-based adaptive adjustment mechanism.

[0034] Course-based weight scheduling: A pre-defined schedule for weight changes can be implemented. For example, in the early stages of training, higher weights are assigned to self-supervised learning tasks to prioritize learning a robust state representation; as training progresses, the weights of self-supervised tasks are gradually reduced while the weights of reinforcement learning tasks are increased, smoothly transitioning the learning focus from representation learning to policy learning.

[0035] The aforementioned multi-agent reinforcement learning model employs a fully shared feature extraction module. In one embodiment of this application, it can also be a partially shared structure. For example, the parameters of the lower layers of the network are shared for extracting general basic features, while the higher layers of the network closer to the task module can have independent, task-specific feature processing layers for self-supervised learning and reinforcement learning tasks, respectively.

[0036] In the aforementioned multi-agent reinforcement learning model, the self-supervised learning module and the reinforcement learning module operate in parallel. In another embodiment, the prediction results or hidden states of the self-supervised learning module can be fed as additional input information to the reinforcement learning module. This allows the reinforcement learning module to make more forward-looking decisions by utilizing not only the current state but also predictions about the future.

[0037] In one embodiment of this application, for each agent, complementary self-supervised learning is performed based on multiple interaction feature groups and its own feature data, and a self-supervised loss calculation operation is performed to obtain the self-supervised learning loss corresponding to the agent, including: Based on its own feature data, the agent makes a prediction to determine its first predicted state features at time (t+1); the own feature data includes: the agent's observation features at time t and its action features at time t. Select any one of the multiple interaction feature groups as the target interaction feature group; the target interaction feature group contains at least the joint observation features of multiple agents at time t, the joint action features of multiple agents at time t, and the joint observation features of multiple agents at time (t+1). Based on the target interaction feature group, predict the second predicted state features of the remaining agents among multiple agents at time (t+1); The first predicted state feature and the second predicted state feature are concatenated to obtain the combined predicted feature vector; Under the premise of blocking gradient backpropagation, determine the actual state characteristics of the other agents at time (t+1); The actual state features of the agent itself at time (t+1) are concatenated with the actual state features of the other agents at time (t+1) to obtain a combined actual feature vector; The self-supervised learning loss of the agent is determined by combining the predicted feature vector and the actual feature vector.

[0038] In this embodiment, the self-feature data refers to the observation features and action features of a single agent at time t. The future state of the agent is determined by its current state and action. Therefore, by predicting its own state at time (t+1), the correlation between its own behavior and its own state changes can be learned. The first predicted state feature refers to the model's prediction result of its own state at time (t+1), such as the predicted position and velocity.

[0039] refer to Figure 3 In this embodiment, for each agent in the multi-agent group, intra-agent self-supervised learning and cross-agent self-supervised learning can be performed. Intra-agent self-supervised learning refers to the process of predicting the characteristics of the agent itself based on its own observation characteristics and additional information. Cross-agent self-supervised learning refers to the process of predicting the characteristics of other agents (i.e., the agent's teammates) based on its own observation characteristics and additional information. The additional information of the agent should at least contain the joint observation characteristics of multiple agents at time t, the joint action characteristics of multiple agents at time t, and the joint observation characteristics of multiple agents at time (t+1).

[0040] In a multi-agent system, the future state of a single agent is influenced not only by itself but also by the actions of other agents. Therefore, joint features are needed as a reference to predict the future states of other agents. In this embodiment, each agent can randomly select a target interaction feature group from multiple interaction feature groups as input for calculation. The target interaction feature group contains at least the joint observation features of multiple agents at time t, the joint action features of multiple agents at time t, and the joint observation features of multiple agents at time (t+1). Since the state of other agents at time (t+1) is determined by their own state and actions at time t, the joint observation features and joint action features of each agent in the target interaction feature group at time t can be used to predict the second predicted state features of the remaining agents at time (t+1).

[0041] In this embodiment, the first and second predicted state features are concatenated to obtain a complete vector, facilitating subsequent loss calculation. Correspondingly, the actual state features of the agent itself at time (t+1) and the actual state features of the other agents at time (t+1) should also be concatenated to obtain a combined actual feature vector. It should be noted that the actual state features of the other agents at time (t+1) are obtained under the premise of blocking gradient backpropagation. Blocking gradient backpropagation ensures that during subsequent gradient backpropagation, the gradient flows only to the parameters of the self-supervised learning module, and not to the parameters of the shared feature extraction module used to generate the true target. This provides a stable learning target for the prediction task. In other words, under the premise of blocking gradient backpropagation, the actual state data of the other agents at time (t+1) can be input into the shared feature extraction module to obtain the actual state features of the other agents at time (t+1). In this embodiment, the mean square error between the combined predicted feature vector and the combined target vector with gradient blocking applied can be calculated to obtain the self-supervised learning loss of the agent.

[0042] As can be seen from the above, this embodiment combines intra-agent self-supervised learning and cross-agent self-supervised learning, enabling each agent to predict its own first predicted state at time (t+1) based on its own feature data, learning the correlation between its own behavior and its own state changes. It can also randomly select target interaction feature groups from multiple interaction feature groups to predict the second predicted state features of other agents at time (t+1). This comprehensive prediction method allows agents to fully consider the impact of themselves and other agents on their future state, thereby learning more comprehensive and in-depth system dynamics in multi-agent systems, providing a richer knowledge base for subsequent decision-making. For example, in a drone swarm collaborative control scenario, each drone can not only understand the impact of its own flight attitude and actions on its state at the next moment, but also predict the flight state changes of other drones, thus better coordinating flight trajectories and avoiding collisions.

[0043] Multi-agent systems face various unknown new scenarios in practical applications, and the interactions between environmental conditions and agents are complex and ever-changing. In this embodiment, agents learn comprehensive system dynamics, enabling them to better understand and adapt to these changes. When deployed to a new scenario, even if environmental conditions or the composition of agents change, agents can still make reasonable predictions about their own and other agents' future states based on the learned patterns, thereby making decisions more adaptable to the new scenario and improving the model's generalization ability. Taking autonomous driving scenarios as an example, when vehicles of different models and performance join the fleet or the road environment changes, each intelligent vehicle can still make collaborative decisions based on its own dynamic patterns and those of other vehicles, ensuring driving safety and efficiency.

[0044] When calculating the self-supervised learning loss, this embodiment blocks gradient backpropagation. When acquiring the actual state features of the other agents at time (t+1), the gradient flows only to the parameters of the self-supervised learning module, and not to the parameters of the shared feature extraction module used to generate the true target. This avoids unreasonable updates to the shared feature extraction module by the gradient, providing a stable learning objective for the prediction task. A stable learning objective helps the model optimize parameters more efficiently and accurately, improving the effectiveness of self-supervised learning and thus enhancing the performance of the entire multi-agent reinforcement learning model.

[0045] To address the problem that existing technologies (such as traditional MARL algorithms and data augmentation methods) fail to learn the deep dynamic relationships within multi-agent systems, leading to over-coupling between the model and the training scenario and a sharp performance decline when faced with semantic changes in agent capabilities and team composition, this embodiment introduces cross-agent and intra-agent self-supervised learning tasks. This forces the model to shift from simply learning "how to do it" (policy) to simultaneously learning "why this happens" (system dynamics). By predicting its own and its teammates' future states, the model learns the potential, causal interaction patterns between agents. This profound understanding of the system's inherent dynamics allows policies to no longer rely on the surface features of the training scenario, but rather to make decisions at a more fundamental level of rules. Therefore, when the semantics of the test scenario change, the underlying dynamic rules still apply, enabling the model to adapt quickly, demonstrating strong generalization ability and decision robustness, and resulting in more stable and reliable system operation.

[0046] Addressing the shortcomings of inductive bias-based methods (such as equivariant graph neural networks), which are typically designed for specific types of generalization tasks, are highly hypothetical, have narrow applicability, and cannot cope with diverse semantic variations, this application does not rely on any prior assumptions about specific tasks or types of variations. The "system dynamics" it learns is a general concept applicable to various types of semantic variations, whether it's changes in agent capabilities, adjustments to team composition, or changes in environmental interaction rules. Furthermore, as a modular framework, this invention can be easily integrated into various mainstream MAR-L algorithms (including policy-based and value-based algorithms) as a plug-and-play generalization enhancement unit with broad applicability.

[0047] In one embodiment of this application, before acquiring the interaction data set of multiple smart agents, the method further includes: Initialize the experience replay buffer; the experience replay buffer is used to store the data sequence generated by the interaction between each agent and the environment. Acquire interaction data sets from multiple agents, including: Data is sampled from the initialized experience replay buffer to obtain interaction data sets from multiple agents.

[0048] In this embodiment, the experience replay buffer is essentially a data storage container used to store all data generated by the multi-agent interaction with the environment and other agents during the training process. The stored content can be time-series interaction data, such as agent observations, actions performed, environmental rewards, and the state at each time step. The data stored in the experience replay buffer is real data after extensive experimental testing. In this embodiment, a portion of the data can be randomly sampled from the experience replay buffer to obtain a group of interaction data from multiple agents.

[0049] In one embodiment of this application, determining the total loss based on self-supervised learning loss and reinforcement learning loss includes: The first loss is determined based on the self-supervised learning loss and the dynamically determined self-supervised learning weights; The second loss is determined based on the reinforcement learning loss and the dynamically determined reinforcement learning weights; The total loss is determined by the sum of the first loss and the second loss.

[0050] In this embodiment, the self-supervised learning weights and reinforcement learning weights can be preset based on experience or dynamically determined based on the training process. In one embodiment of this application, a method for dynamically determining the self-supervised learning weights and reinforcement learning weights is provided: The self-supervised learning weights are determined based on the uncertainty parameters of self-supervised learning; the uncertainty parameters of self-supervised learning are negatively correlated with the self-supervised learning weights. Reinforcement learning weights are determined based on the uncertainty parameters of reinforcement learning; the uncertainty parameters of reinforcement learning are negatively correlated with the reinforcement learning weights. Among them, the uncertainty parameters of self-supervised learning and reinforcement learning are dynamically adjusted during the training process of the multi-agent reinforcement learning model.

[0051] In this embodiment, the uncertainty parameter of self-supervised learning is denoted as... The uncertainty parameter of reinforcement learning is denoted as Uncertainty parameters of self-supervised learning Essentially, it's an indicator that measures the learning difficulty or unreliability of prediction results for a current self-supervised task. For example, in the early stages of training, the model cannot adapt to the dynamics of multi-agent systems, resulting in large prediction errors and unstable results. The error rate is relatively high; in the later stages of training, the model grasps the system's patterns, resulting in smaller prediction errors and more stable results. Smaller. Uncertainty parameters in reinforcement learning. Similarly, I will not elaborate further.

[0052] During the training and learning process of the model and It is not a fixed value; it updates with the model's learning state in each training round. Specifically, when configuring the experience replay buffer, the uncertainty parameters of the complementary multi-task balancing module can be initialized simultaneously to obtain the initialized parameters. and During model training, the model automatically adjusts its uncertainty parameter based on the learning difficulty of each task. When training a task (such as self-supervised learning) is difficult and the loss fluctuates significantly, its corresponding uncertainty increases. This will increase, causing its weight to decrease accordingly. This adaptive adjustment mechanism avoids any single task dominating the entire gradient update process, achieving harmonious coexistence and collaborative optimization among tasks. In subsequent calculations, the total loss is obtained, and a gradient descent algorithm is performed on this total loss. When the calculated gradient is backpropagated, not only are the parameters of all neural network modules updated, but the uncertainty parameters are also updated simultaneously. and Ultimately, the goal is to stably learn a policy model that can both accomplish the main task and deeply understand the system dynamics, while also possessing high generalization ability.

[0053] In this embodiment, reference Figure 3 ,in For the total loss of self-supervised learning, To enhance total learning loss, Indicates the self-supervised learning weights. This represents the reinforcement learning weights. To address potential conflicts between self-supervised learning and reinforcement learning tasks, this embodiment introduces a complementary multi-task balancing module for dynamic and adaptive loss weighting. The self-supervised learning weights and reinforcement learning weights can be determined based on the following method: , The above weight calculation formula ensures that the weights are inversely proportional to the uncertainty of the corresponding task; that is, the greater the uncertainty, the smaller the weight. This is because during the model's training and learning process... and These are not fixed values; they change as training progresses. Therefore, the weights for self-supervised learning and reinforcement learning will also change.

[0054] In this embodiment, the first loss can be expressed as The second loss can be expressed as In this embodiment, the total loss can be determined as the sum of the first loss and the second loss, or a regularization term can be added, for example: ,in, This is a regularization term used to prevent... and The weights are increased without limit during the optimization process, causing them to approach zero and halting the learning process, thus ensuring the stability and effectiveness of the entire dynamic equilibrium mechanism.

[0055] As can be seen from the above, this embodiment dynamically determines the self-supervised learning weights and reinforcement learning weights based on the uncertainty parameters of self-supervised learning and reinforcement learning, and the uncertainty parameters are negatively correlated with the corresponding weights. During model training, the uncertainty parameter of self-supervised learning measures the learning difficulty of the self-supervised task or the unreliability of the prediction results, and the same applies to the uncertainty parameter of reinforcement learning. As training progresses, when a task becomes difficult to train or experiences large loss fluctuations, its uncertainty increases, and its weight decreases accordingly. This adaptive adjustment mechanism avoids any single task dominating the entire gradient update process, achieving harmonious coexistence and collaborative optimization between self-supervised learning and reinforcement learning tasks. For example, in multi-agent policy learning, if the self-supervised learning task becomes difficult at a certain stage, its weight will automatically decrease, allowing the reinforcement learning task to play a greater role in gradient updates and ensuring the overall stability of model training. Since self-supervised learning and reinforcement learning tasks may have potential conflicts, this embodiment introduces a complementary multi-task balancing module for dynamic and adaptive loss weighting, solving this problem. This embodiment determines weights based on uncertainty parameters, enabling the model to automatically adjust weights according to the learning progress of different tasks. This ensures that both tasks receive reasonable learning and optimization during training, thereby improving the overall performance of the model. When determining the total loss, this embodiment incorporates a regularization term to prevent the self-supervised learning and reinforcement learning weights from increasing uncontrollably during optimization, which could lead to weights approaching zero and causing learning stagnation. The introduction of the regularization term ensures the stability and effectiveness of the entire dynamic equilibrium mechanism, guaranteeing that the model can learn continuously and stably during training, ultimately resulting in a high-performance policy model. For example, in training a multi-agent decision-making model in financial trading, the regularization term can prevent decision-making errors caused by extreme weight distribution, improving the model's stability and reliability.

[0056] In multi-task learning, there may be goal conflicts between different tasks (such as self-supervised learning and reinforcement learning in this application). Using simple weighted summation can easily lead to training instability, or even a seesaw effect—that is, performance improvement in one task leads to performance degradation in another. This invention introduces a "complementary multi-task balancing module," which utilizes the principle of uncertainty to achieve dynamic and adaptive weighting of the losses for different tasks. During training, the system can automatically assess the learning difficulty and uncertainty of each task and adjust its contribution to gradient updates accordingly, effectively mitigating conflicts between tasks. Figure 4 The experimental results show that this mechanism avoids the situation where the performance of the main task (reinforcement learning) is compromised due to over-optimization of the auxiliary task (self-supervised learning), ensuring that the two tasks can work together to achieve higher decision performance and a more stable training process.

[0057] Corresponding to the multi-agent reinforcement learning generalization enhancement method in the above embodiments, Figure 4 This is a structural block diagram of a multi-agent reinforcement learning generalization enhancement device provided in one embodiment of this application. For ease of explanation, only the parts relevant to the embodiment of this application are shown. References Figure 4 The multi-agent reinforcement learning generalization enhancement device 20 includes: a data acquisition module 21, a first loss determination module 22, a second loss determination module 23, and a training module 24.

[0058] Among them, the data acquisition module 21 is used to acquire the interaction data group of multiple intelligent agents, and obtain multiple interaction feature groups based on the interaction data group of multiple intelligent agents. The first loss determination module 22 is used to perform complementary self-supervised learning based on multiple interaction feature groups and its own feature data for each agent, and to perform self-supervised loss calculation to obtain the self-supervised learning loss corresponding to the agent; and to perform reinforcement learning based on multiple interaction feature groups, and to perform reinforcement learning loss calculation to obtain the reinforcement learning loss corresponding to the agent. The second loss determination module 23 is used to determine the total loss based on the self-supervised learning loss and the reinforcement learning loss; Training module 24 is used to train each agent based on the total loss until a preset stopping condition is met, thus obtaining the trained multi-agent reinforcement learning model.

[0059] In one embodiment of this application, the first loss determination module 22 is specifically used to make a prediction based on its own feature data to determine the first predicted state feature of the agent at time (t+1); the own feature data includes: the observation feature of the agent at time t and the action feature at time t; Select any one of the multiple interaction feature groups as the target interaction feature group; the target interaction feature group contains at least the joint observation features of multiple agents at time t, the joint action features of multiple agents at time t, and the joint observation features of multiple agents at time (t+1). Based on the target interaction feature group, predict the second predicted state features of the remaining agents among multiple agents at time (t+1); The first predicted state feature and the second predicted state feature are concatenated to obtain the combined predicted feature vector; Under the premise of blocking gradient backpropagation, determine the actual state characteristics of the other agents at time (t+1); The actual state features of the agent itself at time (t+1) are concatenated with the actual state features of the other agents at time (t+1) to obtain a combined actual feature vector; The self-supervised learning loss of the agent is determined by combining the predicted feature vector and the actual feature vector.

[0060] In one embodiment of this application, complementary self-supervised learning is implemented based on a self-supervised learning module; Blocking gradient backpropagation is used to restrict gradients to flow only to the self-supervised learning module during the gradient backpropagation process for training each agent.

[0061] In one embodiment of this application, the multi-agent reinforcement learning generalization enhancement device 20 further includes: an initialization module for initializing an experience replay buffer; the experience replay buffer is used to store data sequences generated by the interaction between each agent and the environment; Acquire interaction data sets from multiple agents, including: Data is sampled from the initialized experience replay buffer to obtain interaction data sets from multiple agents.

[0062] In one embodiment of this application, the second loss determination module 23 is specifically used to determine the first loss based on the self-supervised learning loss and the dynamically determined self-supervised learning weights; The second loss is determined based on the reinforcement learning loss and the dynamically determined reinforcement learning weights; The total loss is determined by the sum of the first loss and the second loss.

[0063] In one embodiment of this application, the multi-agent reinforcement learning generalization enhancement device 20 further includes: a weight determination module, used to determine the self-supervised learning weights based on the uncertainty parameters of self-supervised learning; the uncertainty parameters of self-supervised learning are negatively correlated with the self-supervised learning weights; Reinforcement learning weights are determined based on the uncertainty parameters of reinforcement learning; the uncertainty parameters of reinforcement learning are negatively correlated with the reinforcement learning weights. Among them, the uncertainty parameters of self-supervised learning and reinforcement learning are dynamically adjusted during the training process of the multi-agent reinforcement learning model.

[0064] In one embodiment of this application, each interaction feature group contains at least the joint observation features of multiple agents at time t; The joint observation features of multiple agents at time t are obtained by inputting the joint observation data of multiple agents at time t into the shared feature extraction module.

[0065] See Figure 5 , Figure 5 This is a schematic block diagram of an electronic device provided according to an embodiment of this application. Figure 5The electronic device 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to perform the functions of each module / unit in the above-described device embodiments, for example... Figure 4 The functions of the data acquisition module 21, the first loss determination module 22, the second loss determination module 23, and the training module 24 are shown.

[0066] It should be understood that, in the embodiments of this application, the processor 301 may be a central processing unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0067] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0068] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store device type information.

[0069] In specific implementations, the processor 301, input device 302, and output device 303 described in the embodiments of this application can execute the implementation methods described in the multi-agent reinforcement learning generalization enhancement method provided in the embodiments of this application, or they can execute the implementation methods of the electronic devices described in the embodiments of this application, which will not be elaborated here.

[0070] In another embodiment of this application, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. Alternatively, the computer program can instruct related hardware to complete the process. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0071] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., provided on the electronic device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the electronic device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0072] Those skilled in the art will recognize that the modules / units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0073] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0074] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules, units, or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces or modules / units, or it may be an electrical, mechanical, or other form of connection.

[0075] The modules / units described as separate components may or may not be physically separate. Similarly, the components shown as modules / units may or may not be physical modules / units; they may be located in one place or distributed across multiple network modules / units. Some or all of the modules / units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0076] Furthermore, the functional modules / units in the various embodiments of this application can be integrated into one processing module / unit, or each module / unit can exist physically separately, or two or more modules / units can be integrated into one module / unit. The integrated modules / units described above can be implemented in hardware or in the form of software functional modules / units.

[0077] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multi-agent reinforcement learning generalization enhancement method, characterized in that, The method comprises the following steps: obtaining a plurality of interaction data sets of a plurality of agents, and obtaining a plurality of interaction feature sets based on the plurality of interaction data sets of the plurality of agents; for each agent, performing complementary self-supervised learning based on the plurality of interaction feature sets and self feature data, and performing self-supervised loss calculation operation to obtain the self-supervised learning loss corresponding to the agent; and performing reinforcement learning based on the plurality of interaction feature sets, and performing reinforcement learning loss calculation operation to obtain the reinforcement learning loss corresponding to the agent; determining a total loss based on the self-supervised learning loss and the reinforcement learning loss; training each agent based on the total loss until a preset stopping condition is met, to obtain a trained multi-agent reinforcement learning model.

2. The multi-agent reinforcement learning generalization augmentation method of claim 1, wherein, The method further comprises the following steps: based on the self feature data, predicting to determine a first predicted state feature of the agent at time (t+1); the self feature data comprises an observation feature of the agent at time t and an action feature of the agent at time t; selecting any interaction feature set from the plurality of interaction feature sets as a target interaction feature set; the target interaction feature set at least contains joint observation features of the plurality of agents at time t, joint action features of the plurality of agents at time t and joint observation features of the plurality of agents at time (t+1); based on the target interaction feature set, predicting second predicted state features of the remaining agents in the plurality of agents at time (t+1); splicing the first predicted state feature and the second predicted state feature to obtain a combined predicted feature vector; determining actual state features of the remaining agents at time (t+1) under the premise of blocking gradient backpropagation; splicing the actual state feature of the agent itself at time (t+1) and the actual state features of the remaining agents at time (t+1) to obtain a combined actual feature vector; based on the combined predicted feature vector and the combined actual feature vector, determining the self-supervised learning loss corresponding to the agent.

3. The multi-agent reinforcement learning generalization augmentation method of claim 2, wherein, The complementary self-supervised learning is realized based on a self-supervised learning module. The blocking gradient backpropagation is used to limit the gradient to flow only to the self-supervised learning module in the gradient reverse propagation process of training each agent.

4. The multi-agent reinforcement learning generalization augmentation method of claim 1, wherein, Before the step of obtaining the plurality of interaction data sets of the plurality of agents, the method further comprises the following steps: initializing an experience replay buffer; the experience replay buffer is used to store data sequences generated by the interaction between each agent and the environment; the step of obtaining the plurality of interaction data sets of the plurality of agents comprises the following steps: sampling data from the experience replay buffer after initialization to obtain the plurality of interaction data sets of the plurality of agents.

5. The multi-agent reinforcement learning generalization augmentation method of any one of claims 1 to 4, wherein, The step of determining the total loss based on the self-supervised learning loss and the reinforcement learning loss comprises the following steps: determining a first loss based on the self-supervised learning loss and a dynamically determined self-supervised learning weight; determining a second loss based on the reinforcement learning loss and a dynamically determined reinforcement learning weight; determining the total loss based on the sum of the first loss and the second loss.

6. The multi-agent reinforcement learning generalization augmentation method of claim 5, wherein, Also comprising: determining the self-supervised learning weight based on the uncertainty parameter of the self-supervised learning; the uncertainty parameter of the self-supervised learning is negatively correlated with the self-supervised learning weight; determining the reinforcement learning weight based on the uncertainty parameter of the reinforcement learning; the uncertainty parameter of the reinforcement learning is negatively correlated with the reinforcement learning weight; wherein the uncertainty parameter of the self-supervised learning and the uncertainty parameter of the reinforcement learning are dynamically adjusted during the training process of the multi-agent reinforcement learning model.

7. The multi-agent reinforcement learning generalization augmentation method of any one of claims 1 to 4, wherein, Each of the interaction feature groups contains at least joint observation features of multiple agents at time t; The joint observation features of multiple agents at time t are obtained by inputting joint observation data of multiple agents at time t into a shared feature extraction module.

8. A multi-agent reinforcement learning generalization enhancement device, characterized by, Comprising: a data acquisition module configured to acquire interaction data groups of multiple agents, and obtain multiple interaction feature groups based on the interaction data groups of the multiple agents; a first loss determination module configured to, for each agent, perform complementary self-supervised learning based on the multiple interaction feature groups and its own feature data, and perform self-supervised loss calculation operation to obtain the self-supervised learning loss corresponding to the agent; and perform reinforcement learning based on the multiple interaction feature groups, and perform reinforcement learning loss calculation operation to obtain the reinforcement learning loss corresponding to the agent; a second loss determination module configured to determine a total loss based on the self-supervised learning loss and the reinforcement learning loss; a training module configured to train each agent based on the total loss until a preset stopping condition is met, and obtain a trained multi-agent reinforcement learning model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, The processor executes the computer program to realize the steps of the method of any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to realize the steps of the method of any one of claims 1 to 7.

Images