Multi-agent reinforcement learning method and device, electronic equipment, medium and product

Abstract

The application discloses a multi-agent reinforcement learning method and device, electronic equipment, medium and product, relates to the technical field of agents, and realizes function decoupling through a double critic architecture, provides two independent value evaluation channels for each agent, wherein a plastic critic outputs a short-term adaptive value judgment, a stability critic outputs a long-term stability value judgment, and the two value estimates are dynamically fused through a state-driven attention mechanism to guide policy generation, breaking the policy convergence feedback cycle under the homogeneous critic architecture and maintaining policy diversity of the multi-agent system.

Description

Technical Field

[0001] This application relates to the field of intelligent agent technology, and in particular to multi-agent reinforcement learning methods, devices, electronic devices, media, and products. Background Technology

[0002] In reinforcement learning, especially in multi-agent collaborative and adversarial scenarios, traditional methods employ a single or homogeneous critic architecture and use the same set of parameter update mechanisms, which cannot adapt to dynamic changes while preserving historical knowledge. This problem is more prominent in multi-agent environments due to increased non-stationarity, which restricts the robustness and strategy diversity of group decision-making. Summary of the Invention

[0003] The main objective of this application is to provide a multi-agent reinforcement learning method, apparatus, electronic device, medium, and product, which aims to solve the technical problem of insufficient policy diversity caused by the inherent conflict between stability and plasticity in multi-agent reinforcement learning.

[0004] To achieve the above objectives, this application proposes a multi-agent reinforcement learning method, which includes: For each agent in the multi-agent group, the action distribution of the agent is obtained based on the agent's current state and the policy decoding network. The action to be executed is sampled from the action distribution and sent to the environment. The environment reward and the next state are received. The transition sample containing the current state, the action to be executed, the environment reward and the next state are stored in the experience replay buffer. The current state includes the state information of the agent and the interaction information between the agent and the environment. In response to the satisfaction of parameter update conditions, a batch of transfer samples is sampled from the experience replay buffer, wherein each transfer sample includes a sample state, a sample action, a sample reward, and a sample next state; For each of the transition samples, the target value is determined based on the sample's next state, the target plasticity commentator, and the target stability commentator; Based on the transfer samples and the target value, the parameters of the plastic critic are optimized. After the parameter optimization of the plastic critic is completed, the weight parameters of the low-utility units are identified and reset based on the cumulative contribution utility of each hidden unit in the plastic critic, wherein the contribution utility of the low-utility unit is less than a preset threshold. Based on the transferred samples and the target value, the parameters of the stability commentator are optimized. When optimizing the parameters of the stability commentator, a penalty term associated with the historical importance of the parameters is constructed based on the elastic weight consolidation mechanism, and the update magnitude of the weight parameters is constrained based on the penalty term. The parameters of the policy decoding network are optimized based on the transferred samples; After updating the parameters of the plasticity critic and the stability critic, a soft update is performed on the target plasticity critic and the target stability critic.

[0005] Furthermore, to achieve the above objectives, this application also proposes a multi-agent reinforcement learning device, which includes: An interaction module is used to obtain the action distribution of each agent in a multi-agent system based on the agent's current state and the policy decoding network, sample the execution action from the action distribution, send the execution action to the environment, receive the environment reward and the next state, and store the transition sample containing the current state, the execution action, the environment reward and the next state in an experience replay buffer, wherein the current state includes the agent's state information and the interaction information between the agent and the environment; A parameter update module is used to sample a batch of transition samples from the experience replay buffer in response to meeting parameter update conditions. Each transition sample includes a sample state, a sample action, a sample reward, and a sample next state. For each transition sample, a target value is determined based on the sample next state, a target plasticity critic, and a target stability critic. Based on the transition samples and the target value, parameters of the plasticity critic are optimized. After parameter optimization of the plasticity critic, low-utility items are identified and reset based on the cumulative contribution utility of each hidden unit in the plasticity critic. The weight parameters of the unit are defined, wherein the contribution utility of the inefficient unit is less than a preset threshold; based on the transition samples and the target value, the parameters of the stability commentator are optimized, wherein, when optimizing the parameters of the stability commentator, a penalty term associated with the historical importance of the parameters is constructed based on the elastic weight consolidation mechanism, and the update magnitude of the weight parameters is constrained based on the penalty term; the parameters of the policy decoding network are optimized based on the transition samples; after completing the parameter updates of the plasticity commentator and the stability commentator, the target plasticity commentator and the target stability commentator are softly updated.

[0006] In addition, to achieve the above objectives, this application also proposes an electronic device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the multi-agent reinforcement learning method as described above.

[0007] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the multi-agent reinforcement learning method as described above.

[0008] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the multi-agent reinforcement learning method described above.

[0009] In this application, for each agent in a multi-agent system, based on the agent's current state and a policy decoding network, the agent's action distribution is obtained. An execution action is sampled from the action distribution, sent to the environment, and the environment reward and next state are received. A transition sample containing the current state, the execution action, the environment reward, and the next state is stored in an experience replay buffer, wherein the current state includes the agent's state information and the agent's interaction information with the environment. In response to satisfying parameter update conditions, a batch of transition samples is sampled from the experience replay buffer, wherein each transition sample includes a sample state, a sample action, a sample reward, and a sample next state. For each transition sample, based on the sample next state, the target plasticity commentator, and the target stability commentator... The following steps are taken: First, determine the target value. Second, optimize the parameters of the malleable commentator based on the transition samples and the target value. After optimizing the parameters of the malleable commentator, identify and reset the weight parameters of inefficient units based on the cumulative contribution utility of each hidden unit in the malleable commentator, where the contribution utility of the inefficient unit is less than a preset threshold. Third, optimize the parameters of the stability commentator based on the transition samples and the target value. When optimizing the parameters of the stability commentator, construct a penalty term related to the historical importance of the parameters based on an elastic weight consolidation mechanism, and constrain the update magnitude of the weight parameters based on the penalty term. Fourth, optimize the parameters of the policy decoding network based on the transition samples. Fifth, after updating the parameters of the malleable commentator and the stability commentator, perform a soft update on the target malleable commentator and the target stability commentator.

[0010] Compared to applying regularization or replay constraints only within a single commentator, this application achieves functional decoupling through a dual-commentator architecture. This dual-commentator architecture provides each agent with two independent value assessment channels: the plasticity commentator outputs short-term adaptive value judgments, and the stability commentator outputs long-term stability value judgments. These two value estimates are dynamically fused through a state-driven attention mechanism to guide policy generation. Because different agents operate in different local environments, their value assessments and policy outputs exhibit differentiated characteristics, breaking the policy convergence feedback loop under a homogeneous commentator architecture. This allows the multi-agent system to maintain policy diversity, improves group exploration efficiency, and enhances the robustness of collaborative decision-making. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating an embodiment of the multi-agent reinforcement learning method of this application. Figure 2 This is a flowchart illustrating Embodiment 2 of the multi-agent reinforcement learning method of this application; Figure 3 A simplified flowchart illustrating the multi-agent reinforcement learning method provided in this application embodiment; Figure 4 A comparison chart of test results for the multi-agent reinforcement learning method provided in the embodiments of this application; Figure 5 This is a comparison chart of ablation performance provided in the embodiments of this application, wherein, Figure 5 (a) in the figure is a comparison chart of the ablation experimental performance of the single commentator removal mechanism provided in the embodiments of this application. Figure 5 (b) in the figure is a comparison of the ablation experimental performance of the removal fusion module and the dual-commentator mechanism provided in the embodiments of this application; Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the multi-agent reinforcement learning method in the embodiments of this application.

[0014] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0015] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0016] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0017] In reinforcement learning, especially in complex decision-making scenarios involving multi-agent collaboration and adversarial competition, deep neural network models face an inherent conflict between stability and plasticity. Training a neural network involves searching for parameter configurations that minimize the loss function in a high-dimensional parameter space. When an agent learns continuously in a non-stationary environment, the data flow distribution drifts over time, and the gradient descent direction changes accordingly. If parameters are updated with a large step size along the current data gradient direction, the parameter vector will deviate from the local optimum region corresponding to the historical task, leading to a catastrophic loss of historical knowledge. Conversely, if stability is maintained through parameter regularization or constrained updates, the gradient update magnitude is limited, the parameter space exploration range is compressed, and the network's neural expressive power will gradually be lost due to plasticity degradation.

[0018] In a multi-agent environment, the aforementioned conflicts are further amplified. In a single-agent scenario, environmental dynamics arise solely from external factors. However, in a multi-agent system, each agent's policy update alters the state transition distribution and reward function faced by other agents. From each agent's perspective, the transition probability matrix continuously changes with the evolution of other agents' policies, and the convergence target of the value function itself becomes a moving target. When multiple agents employ a homogeneous critic architecture, their value assessments and policy updates form a positive feedback loop. Agent A's policy adjustment triggers environmental changes, and Agent B updates its policy to adapt to these changes, which in turn affects Agent A. In the absence of a differentiated value assessment mechanism, this feedback loop leads to convergent policies among the group, causing the multi-agent system to fall into a suboptimal equilibrium characterized by low diversity and low exploration efficiency.

[0019] Existing methods for mitigating the conflict between stability and plasticity mainly fall into three categories: regularization methods, replay methods, and parameter isolation methods. Regularization methods constrain updates by adding a parameter offset penalty term to the loss function, but this limits the exploration range of the parameter space. In multi-agent scenarios, the optimal policy itself migrates with the evolution of other agents, and overly strong parameter constraints will prevent the model from tracking the movement trajectory of the optimal policy point. Replay methods maintain memory by mixing historical and new samples, but the replay buffer capacity is limited, and historical samples come from outdated environmental dynamics; over-reliance on historical samples hinders adaptation to new policy patterns. Parameter isolation methods allocate independent parameter subspaces for different tasks, but in single-task continuous learning scenarios, task boundaries are blurred, making it difficult to effectively allocate parameter resources, and the increased number of parameters at the architectural level leads to excessively high deployment costs. A common limitation of these methods is that they all impose constraints within a single or homogeneous critic architecture, attempting to simultaneously satisfy the mutually exclusive functional requirements of rapid adaptation and knowledge preservation with a single value evaluation network.

[0020] This application structurally decouples stability and plasticity objectives at the network architecture level, constructing a heterogeneous dual-commentator system with complementary functions for each agent. The plasticity commentator is dedicated to capturing short-term signals in the dynamic changes of the environment and the evolution of the opponent's strategy, allowing its parameters to iterate rapidly to track the movement trajectory of the optimal point of the value function; the stability commentator is dedicated to maintaining stable decision-making patterns learned in historical interactions, preventing representational drift and policy degradation caused by the influx of new data.

[0021] To address the potential degradation of neural expressive capacity in plasticity commentators due to continuous and rapid updates, this application introduces a selective reset mechanism based on the cumulative contribution utility of hidden units. By calculating the weighted cumulative amount of the activation value and output connection weight magnitude of each hidden unit, the actual contribution of each unit to the network output is quantified, and units that have been in an inefficient state for a long time are dynamically identified, and their weight parameters are reset to their initial values, thereby continuously creating new plasticity space within the network.

[0022] For stability reviewers, this application employs a constrained optimization strategy based on elastic weight consolidation, suppressing drastic shifts in key parameters by constructing a penalty term that is aware of the historical importance of the parameters. The Fisher Information Matrix is ​​used to assign an importance score to each parameter in the stability reviewer, reflecting the parameter's sensitivity to the historical task loss function. When the network updates parameters on new samples, a penalty term proportional to the parameter's importance and its deviation from the historical optimum is added to the loss function, forcing the network to preferentially rely on low-importance parameters to adapt to new data, while high-importance parameters are strictly protected.

[0023] Compared to applying regularization or replay constraints only within a single commentator, this application achieves functional decoupling through a dual-commentator architecture. This dual-commentator architecture provides each agent with two independent value assessment channels: the plasticity commentator outputs short-term adaptive value judgments, and the stability commentator outputs long-term stability value judgments. These two value estimates are dynamically fused through a state-driven attention mechanism to guide policy generation. Because different agents operate in different local environments, their value assessments and policy outputs exhibit differentiated characteristics, breaking the policy convergence feedback loop under a homogeneous commentator architecture. This allows the multi-agent system to maintain policy diversity, improves group exploration efficiency, and enhances the robustness of collaborative decision-making.

[0024] For ease of understanding, the following explains some key terms in this embodiment: Multi-agent reinforcement learning is a machine learning paradigm in which multiple agents learn through trial and error in a shared environment to optimize their decision-making strategies and maximize long-term cumulative rewards. Each agent independently or collaboratively perceives the environment, performs actions, and receives feedback. An agent is an entity with the ability to perceive, make decisions, and act. Each agent selects actions based on its current state and learned policies.

[0025] The current state refers to the set of information perceived by an agent about its environment at a specific point in time. In multi-agent scenarios, the current state can include the agent's own local observation information, as well as global or local information generated by interactions with the environment or other agents.

[0026] The environment is the external system through which an agent interacts. The environment receives the agent's actions, updates its state according to its internal dynamics, and returns rewards and new states to the agent.

[0027] The policy decoding network receives the agent's state information as input and outputs a probability distribution of the agent's various possible actions in the current state. The agent selects an action based on this distribution. The action distribution, output by the policy decoding network, represents the probability of the agent taking each possible action in a given state. The executed action is the specific action sampled by the agent based on the action distribution; this action is then sent to the environment to influence its state.

[0028] Environmental rewards are feedback signals from the environment to the agent's actions. They are usually scalar values ​​used to measure the quality of the actions.

[0029] A transition sample is a data tuple that typically contains the agent's current state, the action performed, the environmental reward, and the next state. Transition samples are used to train reinforcement learning models. The next state is the new state the environment transitions to after the agent performs an action.

[0030] The experience replay buffer is a data structure that stores transfer samples. The agent stores the collected transfer samples in this buffer and randomly samples batches of samples from it during subsequent training.

[0031] Parameter update conditions refer to the conditions that trigger model parameter updates, such as reaching a preset number of training steps, collecting a sufficient number of transition samples, or meeting specific performance metrics. A batch is a set of transition samples sampled from the experience replay buffer for a single parameter update iteration.

[0032] The plasticity critic is a value assessment network designed to quickly adapt to environmental changes and new information. In this embodiment, its responsiveness to new data is maintained through specific parameter optimization and reset mechanisms. The parameters of the target plasticity critic are delayed or smoothed versions of the plasticity critic parameters, used to provide a relatively stable target value estimate to guide the learning of the plasticity critic.

[0033] The stability critic is a value assessment network designed to maintain the stability of learned knowledge and avoid catastrophic forgetting. In this embodiment, specific regularization or constraint mechanisms are used to limit the magnitude of parameter updates. The parameters of the target stability critic are delayed or smoothed versions of the stability critic parameters, used to provide another relatively stable estimate of the target value to guide the learning of the stability critic.

[0034] The target value is a reference value used to update the critic network (value function estimator), typically calculated based on the Bellman equation, and incorporates the estimated value of the environment reward and the next state.

[0035] Cumulative contribution utility is an indicator that measures the contribution of a hidden unit in a neural network to the network output or performance over a period of time, reflecting the importance of that unit. Low-utility units are hidden units whose cumulative contribution utility is below a preset threshold, indicating that their contribution to network performance is relatively small.

[0036] The elastic weight consolidation mechanism protects knowledge of old tasks by penalizing updates to important parameters by calculating the historical importance of the parameters.

[0037] Historical importance of parameters is an indicator that measures the contribution of each parameter in a neural network to past learning tasks.

[0038] Soft update is a method for updating the parameters of a target network. It involves slowly incorporating the parameters into the target network with a small learning rate, rather than directly copying them, which helps maintain the stability of the target network.

[0039] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions. The following description uses an electronic device as an example to illustrate this embodiment and the subsequent embodiments.

[0040] Based on this, embodiments of this application provide a multi-agent reinforcement learning method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the multi-agent reinforcement learning method of this application. In this embodiment, the multi-agent reinforcement learning method includes steps S10 to S70: Step S10: For each agent in the multi-agent group, based on the agent's current state and the policy decoding network, obtain the agent's action distribution, sample the action to be executed from the action distribution, send the action to the environment, and receive the environment reward and the next state. Store the transition sample containing the current state, the action, the environment reward and the next state in the experience replay buffer, wherein the current state includes the agent's state information and the interaction information between the agent and the environment; The current state refers to the information observed by the agent from the environment at the current moment, which includes two parts: first, the agent's own state information, such as physical quantities like the agent's position coordinates, movement speed, orientation angle, and joint angles; second, the interaction information between the agent and the environment, such as the relative distance to other agents, whether a collision has occurred, the location of objects in the environment, and the distribution of obstacles. These two parts of information together constitute the environmental perception input required for the agent's decision-making.

[0041] The current state is input into a pre-built policy decoding network. The policy decoding network is a neural network model that takes the current state as input and outputs an action distribution. The action distribution represents the probability that the agent will choose each candidate action in the current state. For a continuous action space, this distribution is typically Gaussian, and the policy decoding network outputs the mean and standard deviation parameters of this distribution.

[0042] A specific action is obtained by sampling from the action distribution according to probability. The purpose of sampling is to introduce a certain degree of randomness, allowing the agent to explore different action choices during training and avoid getting trapped in local optima too early.

[0043] The action is sent to the environment. Upon receiving the action, the environment updates its state according to its internal physical rules or state transition logic, and returns an immediate reward signal and the state information for the next moment. The immediate reward signal is an immediate evaluation of the agent's action choice; for example, completing the task earns a positive reward, while a collision earns a negative reward. The next state information is the new state the agent is in after the environment transitions.

[0044] The data generated from this interaction is organized into a transition sample, which contains four elements: the current state, the action performed, the environmental reward, and the next state information. This transition sample is then stored in the experience replay buffer. The experience replay buffer is a fixed-capacity data storage structure used to store historical interaction data for sampling during subsequent parameter update phases.

[0045] Step S20: In response to satisfying the parameter update condition, sample a batch of transfer samples from the experience replay buffer, wherein each transfer sample includes a sample state, a sample action, a sample reward, and a sample next state; During training, parameter updates are not performed immediately after each interaction. Instead, a parameter update process is triggered when the parameter update conditions are met. The parameter update conditions can be set according to actual needs. For example, a common practice is to set an update to be performed every fixed number of steps of interaction with the environment (e.g., every 50 steps). Another practice is to start the update when the number of samples stored in the experience replay buffer reaches the preset minimum batch size. This embodiment does not impose specific restrictions on this.

[0046] When the parameter update conditions are met, a batch of transition samples is randomly sampled from the experience replay buffer. The structure of each transition sample is consistent with the stored samples, including sample state, sample action, sample reward, and sample next state. The number of samples in a batch can be set according to computing resources and training stability, for example, 256 samples are taken, and there is no limit here.

[0047] Step S30: For each transfer sample, determine the target value based on the sample's next state, the target plasticity commentator, and the target stability commentator; For each transferred sample in the sampling batch, its corresponding target value needs to be calculated. The target value is a reference value used in reinforcement learning to calculate the commentator network loss function. Its role is to guide the commentator network's value estimation to be updated in a more accurate direction.

[0048] Specifically, the target value is calculated using the next state of the sample in the transition sample, combined with a pre-constructed target plasticity critic network and target stability critic network. The target plasticity critic network and the plasticity critics have the same network structure, but their parameter update methods differ; the plasticity critic network uses a soft update method to slowly follow parameter changes, providing a stable value estimation reference. The same relationship exists between the target stability critic network and the stability critics.

[0049] Step S40: Based on the transfer sample and the target value, optimize the parameters of the plasticity critic. After optimizing the parameters of the plasticity critic, identify and reset the weight parameters of the low-utility units based on the cumulative contribution utility of each hidden unit in the plasticity critic, wherein the contribution utility of the low-utility unit is less than a preset threshold. The parameters of the plasticity critic are optimized using the batch transfer samples obtained from sampling and the target value.

[0050] First, based on the sample state and sample action in the transition sample, a first value estimate is calculated by the plasticity critic. The first value estimate represents the assessment of the expected cumulative return of the combination of sample state and sample action under the current network parameters.

[0051] Next, the deviation between the first value estimate and the target value determined in step S30 is calculated, and the first basic loss component is determined based on the deviation. The first basic loss component can be calculated using common loss function forms such as mean square error, and there are no restrictions here.

[0052] Then, the backpropagation algorithm is executed based on the first basic loss component to calculate the gradient of the loss component with respect to the parameters of each layer of the plasticity critic, and the network parameters are updated using gradient descent optimizer, etc.

[0053] After completing the aforementioned routine parameter updates, a plasticity maintenance operation is further performed for the plasticity commentator. Specifically, the cumulative contribution utility of each hidden layer unit in the plasticity commentator is calculated. Cumulative contribution utility is a quantitative evaluation metric of the functional importance of a hidden unit over a training period. Hidden units with a cumulative contribution utility below a preset threshold are identified as inefficient units, meaning these units contribute little to the network's decision-making and may have entered a saturated or inert state. For identified inefficient units, their input and output connection weights are reset to preset initial values, such as small values ​​randomly sampled from a uniform or normal distribution. Through this reset operation, inefficient units gain a chance to relearn, and the network's neural expressive power is maintained.

[0054] Step S50: Based on the transferred samples and the target value, optimize the parameters of the stability commentator. When optimizing the parameters of the stability commentator, construct a penalty term related to the historical importance of the parameters based on the elastic weight consolidation mechanism, and constrain the update magnitude of the weight parameters based on the penalty term. The parameters of the stability critic are optimized using the batch transfer samples obtained from sampling and the target value.

[0055] First, based on the sample state and sample action in the transferred sample, a second value estimate is obtained by a stability commentator. The deviation between the second value estimate and the target value is calculated, and the second basic loss component is determined based on this deviation.

[0056] Unlike the plasticity commentator, the stability commentator incorporates an additional constraint term into its optimization objective. Based on the elastic weight consolidation mechanism, a penalty term is constructed for each parameter of the stability commentator, which is associated with the historical importance of the parameter. This penalty term applies a larger penalty value to parameters that were deemed important in historical tasks when their current value deviates significantly from their historical optimum, thereby limiting their update magnitude during backpropagation.

[0057] Specifically, the penalty components of each parameter are accumulated into the second basic loss component to obtain the total loss of the stability commentator. Backpropagation is then performed based on the total loss of the stability commentator to update the network parameters of the stability commentator. Through the elastic weight consolidation mechanism, the updates of historically important parameters are constrained, thereby protecting the learned historical decision knowledge from being overwritten by new data and suppressing catastrophic forgetting.

[0058] Step S60: Optimize the parameters of the policy decoding network based on the transferred samples; The parameters of the policy decoding network are optimized using sampled batch transfer samples. The optimization goal of the policy decoding network is to adjust its parameters so that, in the current state, the decisions guided by its output action distribution can obtain higher cumulative rewards. In specific implementation, based on the sample states in the transfer samples, the action distribution is regenerated through the policy decoding network and actions are sampled. Then, the updated plasticity critic network and stability critic network are used to evaluate the value of the actions, and the evaluation results are used as the basis for policy optimization.

[0059] The parameters of the policy decoding network can be updated using the gradient descent algorithm. Through multiple iterations, the policy decoding network gradually learns to choose a better action in a given state.

[0060] Step S70: After updating the parameters of the plasticity critic and the stability critic, perform a soft update on the target plasticity critic and the target stability critic.

[0061] After updating the parameters of the plasticity critic and the stability critic, a soft update is performed on the target plasticity critic network and the target stability critic network.

[0062] The specific method of soft update can be as follows: the parameters of the target network are slowly moved closer to the corresponding parameters according to a preset soft update coefficient. That is, the new parameters of the target network are equal to the soft update coefficient multiplied by the current parameters, plus one minus the soft update coefficient multiplied by the current parameters of the target network. The soft update coefficient is usually small, for example, 0.005.

[0063] Through soft updates, the parameters of the target network change very smoothly, avoiding abrupt changes in parameters, thus providing a stable reference benchmark for the calculation of target value and contributing to the convergence stability of the overall training process.

[0064] In this embodiment, each agent in the multi-agent system learns decision-making strategies that balance rapid adaptation and knowledge retention in dynamic, non-stationary environments by continuously interacting with the environment and updating network parameters. The plasticity critic maintains responsiveness to new changes by resetting inefficient units, while the stability critic protects historical decision knowledge through parameter importance constraints, thus mitigating the conflict between stability and plasticity in multi-agent reinforcement learning.

[0065] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 Step S30, the step of determining the target value based on the next state of the sample, the target plasticity critic, and the target stability critic, includes: Step S301: Input the next state of the sample into the first feature extraction layer, the second feature extraction layer and the third feature extraction layer respectively to obtain the first feature representation, the second feature representation and the third feature representation; The next state of the sample in the transition sample is taken as input and fed into three parallel feature extraction layers: the first feature extraction layer, the second feature extraction layer, and the third feature extraction layer. These three feature extraction layers can be fully connected neural network layers with independent parameters, and each performs feature mapping on the next state of the input sample.

[0066] After the mapping process of the first feature extraction layer, the first feature representation is output. The first feature representation is mainly used by the subsequent policy decoding network to generate the next action, and as a query vector in attention fusion.

[0067] After mapping processing by the second feature extraction layer, the second feature representation is output. This second feature representation is primarily used for value evaluation by the target plasticity critic network, and also serves as a key vector corresponding to the target plasticity critic in attention fusion.

[0068] After mapping processing by the third feature extraction layer, the third feature representation is output. This third feature representation is primarily used by the target stability critic network for value evaluation, and also serves as the key vector corresponding to the target stability critic in attention fusion.

[0069] Through three parallel feature extraction layers, the next state of the same sample is mapped to three different feature vectors, providing dedicated input representations for the policy network, the target plasticity critic network, and the target stability critic network, respectively.

[0070] Step S302: Input the first feature representation into the policy decoding network to obtain the next action distribution, sample the next action from the next action distribution, and calculate the log probability of the next action; The policy decoding network outputs a next action distribution based on the first feature representation of the input. This next action distribution represents the probability distribution of each candidate action predicted by the policy network to be selected in the next state of the sample. A specific next action is obtained by sampling from this next action distribution according to probability. The purpose of sampling is to simulate the action that the policy network would actually take in the next state of the sample, for subsequent value evaluation.

[0071] Simultaneously, the log probability of the next action obtained from this sampling under the current next action distribution is calculated. The log probability is the natural logarithm of the probability corresponding to the action in the action distribution, and is used to introduce an entropy regularization term in subsequent target value calculations to encourage the exploratory nature of the strategy.

[0072] Step S303: Input the second feature representation and the next action to the target plasticity critic to obtain the first value component, and input the third feature representation and the next action to the target stability critic to obtain the second value component; The second feature representation and the next action are input together into the target plasticity critic network. The target plasticity critic network is a target network copy of the main plasticity critic network, and its parameters are updated softly. The target plasticity critic network evaluates the value of the input sample's next state features and next action, and outputs a first value component. The first value component reflects the expected cumulative reward estimate of performing the next action in the sample's next state from the perspective of the target plasticity critic.

[0073] Simultaneously, the third feature representation and the next action are input together into the target stability critic network. The target stability critic network is a target network copy of the main stability critic network. The target stability critic network evaluates the value of the input and outputs a second value component. The second value component reflects the expected cumulative reward estimate for the same state-action pair from the perspective of the target stability critic.

[0074] Because the target plasticity critic network and the target stability critic network have different parameter update mechanisms and functional focuses, their value assessments of the same state-action pair may differ, providing an information basis for subsequent dynamic fusion.

[0075] Step S304: Using the first feature representation as the query and the second feature representation as the key, calculate the first attention weight; using the first feature representation as the query and the third feature representation as the key, calculate the second attention weight; and perform weighted fusion of the first value component and the second value component based on the first attention weight and the second attention weight to obtain the first fused value. Using the first feature representation as the query vector and the second feature representation as the key vector, the similarity between the two is calculated. Specifically, the dot product of the query vector and the first key vector is calculated, the dot product result is divided by a preset scaling factor (which is usually the square root of the feature vector dimension), and then the result is processed through a normalized exponential function to obtain the first attention weight.

[0076] Similarly, using the first feature representation as the query vector and the third feature representation as the key vector, the second attention weight is obtained using the same calculation method. It should be noted that the sum of the first and second attention weights is 1.

[0077] The first value component is weighted using a first attention weight, and the second value component is weighted using a second attention weight. The two weighted value components are then added together to obtain the first fused value. This fusion process can be expressed as: the first fused value equals the first attention weight multiplied by the first value component plus the second attention weight multiplied by the second value component.

[0078] Through the attention mechanism, the allocation of fusion weights is dynamically determined by the similarity between the first feature representation and the second feature representation (features specific to the plasticity critic), and the third feature representation (features specific to the stability critic). When the similarity between the first feature representation and the second feature representation is higher, the first attention weight increases, and the first fusion value focuses more on the evaluation of the target plasticity critic; conversely, it focuses more on the evaluation of the target stability critic.

[0079] In this embodiment, the formula for calculating attention weights can be:

[0080] Where z is the first feature representation, z i Let i be the representation of the i-th feature. Let z represent the attention weight corresponding to the i-th feature. i T d represents the transpose of the i-th feature representation. k This represents the dimension of the feature vector, which is the length of the first feature representation z.

[0081] The formula for weighted fusion of the first value component and the second value component based on the first attention weight and the second attention weight to obtain the first fused value can be:

[0082] in, The primary value lies in integration; , These are the first value component and the second value component, respectively. , These are the first attention weight and the second attention weight, respectively.

[0083] Step S305: Calculate the target value of the transferred sample based on the sample reward, the preset discount factor, the first fusion value, the preset temperature parameter, and the logarithmic probability of the next action.

[0084] Based on the sample reward in the transition sample, a preset discount factor, the first fusion value, a preset temperature parameter, and the log probability of the next action, the target value of the transition sample is calculated. The target value calculation follows the definition of target value in maximum entropy reinforcement learning. Specifically, the target value equals the sample reward plus the product of the discount factor and the following difference: the first fusion value minus the product of the temperature parameter and the log probability of the next action. The product of the temperature parameter and the log probability is the entropy regularization term, which encourages the policy to maintain a certain degree of randomness, promoting exploration.

[0085] The calculated target value will serve as the learning objective for optimizing both the plasticity critic parameters and the stability critic parameters. The target value comprehensively utilizes the differentiated evaluation capabilities of the two critics, the adaptive fusion capability of the attention mechanism, and the theoretical framework of maximum entropy reinforcement learning, providing an accurate and stable estimate of the target value.

[0086] In one feasible embodiment, step S40, the step of optimizing the parameters of the plasticity critic based on the transfer sample and the target value, includes: Step S401: Input the sample state into the first feature extraction layer, the second feature extraction layer and the third feature extraction layer respectively to obtain the fourth feature representation, the fifth feature representation and the sixth feature representation; The sample state in the transferred sample is used as input and fed into the first feature extraction layer, the second feature extraction layer and the third feature extraction layer respectively.

[0087] After the mapping process of the first feature extraction layer, the fourth feature representation is output, which is mainly used for parameter optimization of the subsequent policy decoding network. After the mapping process of the second feature extraction layer, the fifth feature representation is output, which is mainly used as the value evaluation input of the plasticity critic network. After the mapping process of the third feature extraction layer, the sixth feature representation is output, which is mainly used as the value evaluation input of the stability critic network. It should be noted that the first, second, third, fourth, fifth, and sixth features in this embodiment do not constitute a limitation on the order, but are only used as a distinction in name.

[0088] The sample state is transformed into three different feature vectors, providing input representations for the policy network, the plasticity critic network, and the stability critic network, respectively. The feature extraction process is structurally consistent with the feature extraction process for the next state of the sample, ensuring that the same feature extraction architecture is used in the training phase and the target value calculation phase. The specific process will not be elaborated here.

[0089] Step S402: Input the fifth feature representation and the sample action into the plasticity critic to obtain a first value estimate; and determine a first basic loss component based on the deviation between the first value estimate and the target value. The fifth feature representation and the sample actions in the transition sample are input together into the plasticity critic. The plasticity critic evaluates the value of the input state features and actions, and outputs a first value estimate, which represents the expected cumulative return of the combination of sample state and sample action under the current plasticity critic parameters.

[0090] The first value estimate is compared with the target value, and the deviation between the two is calculated. The method of calculating the deviation is not limited; for example, mean squared error can be used, which is the square of the difference between the first value estimate and the target value. This deviation serves as the first basic loss component, used to measure the gap between the plastic critic's current value estimate and the learning objective. The larger the first basic loss component, the further the plastic critic's value estimate deviates from the target value, and the greater the adjustment needed; conversely, a smaller component indicates a more accurate estimate.

[0091] Step S403: Perform backpropagation based on the first basic loss component to update the network parameters of the plasticity commentator.

[0092] Based on the first fundamental loss component, the backpropagation algorithm is executed. Starting from the output layer of the plasticity critic, the backpropagation algorithm calculates the gradient of the first fundamental loss component with respect to each parameter in the network, layer by layer. The gradient represents the rate of change of the loss component along each parameter direction, indicating how to adjust the parameters to reduce the loss component.

[0093] Using a gradient descent optimizer, such as the Adam optimizer, the weight and bias parameters of the plasticity critic are updated based on the calculated gradient. The updated parameters enable the plasticity critic to estimate the value of its output more closely to the target value when faced with the same input.

[0094] Through iterative execution of the above parameter updates, the plasticity critic gradually learns a more accurate value estimation ability. It should be noted that after the parameter updates are completed, an inefficient unit identification and reset operation will also be performed.

[0095] In one feasible embodiment, step S40, the step of identifying and resetting the weight parameters of low-utility units based on the cumulative contribution utility of each hidden unit in the plasticity critic, includes: Step S404: For each hidden unit of the plasticity critic, obtain the historical contribution utility of the hidden unit, and perform attenuation processing on the historical contribution utility using an exponential decay rate to obtain the attenuated historical utility contribution; wherein, the historical contribution utility is the cumulative contribution utility of the hidden unit in the historical iterations adjacent to the current iteration; For each hidden layer unit in the plasticity critic, first obtain the historical contribution utility of that hidden unit. Historical contribution utility refers to the cumulative contribution utility value calculated by that hidden unit at the end of the previous iteration preceding the current iteration. When this step is executed for the first time, the historical contribution utility of each hidden unit can be initialized to zero.

[0096] The acquired historical contribution utility is attenuated using a preset exponential decay rate. The specific operation of the attenuation process is as follows: the historical contribution utility is multiplied by the exponential decay rate to obtain the attenuated historical contribution utility. The exponential decay rate is between 0 and 1, for example, it can be set to 0.99. After attenuation, the value of the historical contribution utility is reduced proportionally. The further the iteration from the current time, the lower the residual weight of the contribution information in the current cumulative value. This embodiment allows the cumulative contribution utility to better reflect the functional performance of the hidden unit in recent iterations.

[0097] Step S405: Based on the activation value of the hidden unit and the weights of all output connections from the hidden unit to the next layer during the forward propagation process of the current iteration, calculate the contribution utility of the hidden unit in the current iteration. During the forward propagation of the current iteration, the activation values ​​of each hidden layer unit in the plasticity commentator, as well as the output connection weights from each hidden unit to all neurons in the next layer, are recorded.

[0098] For each hidden unit, its contribution utility in the current iteration is calculated. The contribution utility is calculated by multiplying the absolute value of the hidden unit's activation value in the current forward propagation by the sum of the absolute values ​​of all output connection weights from that hidden unit to the next layer. The absolute value of the activation value reflects the strength of the hidden unit's response to the current input sample. A larger activation value indicates a more active participation of the unit in feature encoding the current state. The sum of the absolute values ​​of the output connection weights reflects the influence of the hidden unit on neurons in the next layer. A larger sum of weights indicates a more significant impact of the unit's output on subsequent network computations. Multiplying both values ​​together provides a comprehensive measure of the hidden unit's contribution to the overall network output in the current iteration.

[0099] Step S406: The sum of the decayed historical utility contribution and the contribution utility of the current iteration round is determined as the cumulative contribution utility of the hidden unit in the current iteration round; The decayed historical utility contribution is added to the calculated contribution utility of the current iteration round, and the sum is determined as the cumulative contribution utility of the hidden unit after the end of the current iteration round. This cumulative contribution utility value serves as the input for the historical contribution utility of the next iteration round. Through this recursive summation method, the cumulative contribution utility achieves an exponential moving average of historical contribution information, preserving the smoothness of historical trends and enabling timely responses to changes in the unit's recent functional performance.

[0100] In this embodiment, the update formula for the cumulative contribution utility can be:

[0101] in, The cumulative contribution utility of the i-th hidden unit in layer l is used to quantify the historical contribution of this unit to the network output. η is the exponential decay rate, which ranges from 0 to 1 and is used to control the decay rate of historical contribution utility; 1-η represents the weighting coefficient of the contribution utility in the current iteration round, which is complementary to the exponential decay rate. is the activation value of the i-th hidden unit in the l-th layer during the forward propagation process at the current time t. It can be a single activation value or a cumulative activation value, and there is no restriction here. Let be the connection weight from the i-th hidden unit in layer l to the k-th unit in layer (l+1). t represents the total number of units in the (l+1)th layer; l represents the layer index number of the hidden layer; i represents the index number of the hidden unit in the current layer; t represents the current time or the current iteration round.

[0102] Step S407: The hidden units whose cumulative contribution utility is lower than the preset threshold are identified as low-utility units, and the weight parameters of the low-utility units are set to preset initial values.

[0103] After updating the cumulative contribution utility of all hidden units, the cumulative contribution utility value of each hidden unit is compared with a preset threshold. This preset threshold can be set according to the network size and training experience, and is not restricted here. For example, it can be set as a certain percentage of the average cumulative contribution utility of all hidden units.

[0104] Hidden units whose cumulative contribution utility is below a preset threshold are identified as low-utility units. Low-utility units refer to neurons whose actual contribution to the network's decision output remains consistently low over a long training period. These units are usually in a state where their activation values ​​approach zero or their output weights shrink, thus losing their ability to learn effectively.

[0105] For each identified inefficient unit, its input and output connection weights are reset to preset initial values. The initial values ​​can be set using a random initialization strategy, such as randomly sampling from a normal distribution with zero mean and small variance, or randomly selecting values ​​from a uniform distribution. After the weights are reset, the inefficient unit essentially gains a completely new learning starting point and can re-participate in gradient updates and feature learning in subsequent training iterations.

[0106] In this embodiment, by continuously clearing and reinitializing the functionally degenerated neurons in the plastic commentator, the overall neural expressive power of the network is maintained at a high level, thereby ensuring the plastic commentator's continuous responsiveness to new environmental changes.

[0107] In one feasible embodiment, step S50, the step of optimizing the parameters of the stability commentator based on the transferred samples and the target value, includes: Step S501: Input the sixth feature representation and the sample action into the stability critic to obtain the second value estimate; The sixth feature representation and the sample actions in the transition sample are input together into the stability commentator. The sixth feature representation is the output obtained by the third feature extraction layer after feature mapping of the sample state, and is used as the value evaluation input for the stability commentator.

[0108] The stability commentator evaluates the value of the sixth feature representation of the input and the sample action, and outputs a second value estimate, which represents the expected cumulative return of the combination of the sample state and sample action under the current stability commentator parameters.

[0109] Step S502: Determine the second basic loss component based on the deviation between the second value estimate and the target value; The second value estimate is compared with the target value, and the deviation between the two is calculated. The calculation method for the deviation is not detailed here; for example, mean squared error can be used, which is the square of the difference between the second value estimate and the target value. This deviation serves as the second basic loss component, measuring the gap between the stability commentator's current value estimate and the learning objective. The second basic loss component reflects the stability commentator's fitting requirements on the current batch of data.

[0110] Step S503: Determine the importance score of each parameter in the stability reviewer based on the Fisher information matrix, and for each parameter, construct the constraint penalty component of the parameter based on the deviation between the current value and the historical best value of the parameter, the importance score of the parameter, and the preset regularization strength coefficient. Based on the elastic weight consolidation mechanism, a constraint penalty component is constructed for each parameter of the stability commentator. Specifically, the process involves determining the importance score of each parameter in the stability commentator based on the Fisher information matrix. The importance score reflects the sensitivity of each parameter to the historical task loss function; that is, how much a small change in the parameter near its historical optimum will affect the loss function. A higher importance score indicates that the parameter is more critical to maintaining historical decision performance.

[0111] For each parameter, obtain its current value. The current value is the actual value of the parameter in the network before this update. Simultaneously, obtain the historical best value for that parameter. The historical best value is the parameter value saved at the end of the training phase of a previous task, representing the parameter configuration that performed optimally on that task. Calculate the deviation between the current value and the historical best value. The method of calculating the deviation is not limited; it can be calculated using the absolute value or the square of the difference between the two.

[0112] Multiplying the square of the deviation by the importance score, and then multiplying by the regularization strength coefficient, yields the constraint penalty component of this parameter. The regularization strength coefficient is a global hyperparameter used to control the overall weight of the penalty term in the total loss. The specific formula could be:

[0113] in, It is the total loss of elastic weight consolidation; is the second fundamental loss component, i.e., the Bellman error of the current batch of samples; i is the index number of the parameter; This is the regularization strength coefficient, used to control the weight of the EWC (Elastic Weight Consolidation) penalty term; This refers to the importance score of the i-th diagonal element of the Fisher information matrix, which is the i-th parameter. This is the current value of the i-th parameter; The historical optimal value of the i-th parameter is the parameter value saved at the end of the historical training phase. This is the constraint penalty component of the i-th parameter.

[0114] In this embodiment, each parameter is given a constraint penalty value that is proportional to its historical importance and current deviation. The more important the parameter is in the past, the stronger the penalty is when it deviates further from the historical optimal value.

[0115] Step S504: The constraint penalty components of all parameters of the stability critic are accumulated to the second basic loss component to obtain the total loss of the stability critic, and backpropagation is performed based on the total loss of the stability critic to update the network parameters of the stability critic.

[0116] The constraint penalty components of all parameters are summed, and the sum is added to the second basic loss component to obtain the total stability commentator loss. The formula for calculating the total loss can be expressed as: The backpropagation algorithm is executed based on this total loss. Backpropagation starts from the output layer of the stability commentator and calculates the gradient of the total loss with respect to each parameter in the network layer by layer. Since the total loss includes a constraint penalty component, a restorative force component pointing to the historical optimum is superimposed on the gradient of historically important parameters, thereby limiting the large shift of these parameters during the update process.

[0117] The gradient descent optimizer is used to update the weight and bias parameters of the stability reviewer based on the calculated gradient. After the update, the values ​​of historically important parameters remain near their historical optima, while the values ​​of less important parameters can be freely updated to adapt to new data.

[0118] In this embodiment, the total training loss can be:

[0119] in, .

[0120] in, Total training loss; The basic Bellman error loss includes the errors of fusion value and dual critic value; The elastic weighting consolidation penalty term only applies to stability commentators; The parameters for the critic network include parameters for malleable critics and parameters for stable critics; For the fusion module parameters, i.e., the trainable parameters of the cross-attention module; B represents a batch of transferred samples sampled from the experience replay buffer; E τ~B This represents the expected value of each transferred sample τ sampled from the batch transferred sample set B, i.e., the average value. For fusion value estimation; This represents the value estimate of the i-th commentator network for the current observed state o and action a. In this embodiment, when i is 1, it is the first value estimate of the plasticity commentator output, and when i is 2, it is the second value estimate of the stability commentator output. For target value; r t γ represents the instantaneous environmental reward signal obtained at the current time t, i.e., the sample reward in the transferred sample; γ represents the preset discount factor; V(o') represents the state value estimate for the next observed state o', calculated by the target network; β represents the preset equilibrium coefficient. Let represent the squared term of the timing difference error of the fusion module.

[0121] In this embodiment, the stability commentator effectively suppresses catastrophic forgetting while gradually learning the distribution of new data values, thus maintaining the stability of the model during long-term training.

[0122] In one feasible embodiment, step S503, the step of determining the importance scores of each parameter among the stability commentators based on the Fisher information matrix, includes: Step S5031: Calculate the gradient of the loss function of the stability commentator with respect to each parameter on each historical transfer sample; A batch of historical transition samples is sampled from the experience replay buffer. These historical transition samples are data generated by the agent in earlier interaction rounds, and their distribution represents the characteristics of the historical tasks that the model has experienced.

[0123] For each historical transfer sample, the sample is input into the stability commentator, and the loss function is calculated under the current parameters. The loss function can be the mean squared error, which is the squared deviation between the stability commentator's estimate of the sample's value and the corresponding target value.

[0124] Based on the loss function value, the gradient of the loss function with respect to each parameter in the stability review is calculated using the backpropagation algorithm. The gradient is a vector or matrix with the same shape as the parameter, where each element represents the instantaneous rate of change of the loss function along the direction of that parameter. A larger absolute value of the gradient indicates that a small change in that parameter on the current sample will cause a significant change in the loss function, meaning that the parameter is more sensitive to the value estimation of that sample.

[0125] By calculating gradients on multiple historical transition samples, the sensitivity distribution information of each parameter on different state-action pairs was obtained.

[0126] Step S5032: Squaring the gradient of each parameter and calculating the expected value on each historical transfer sample to obtain the Fisher information matrix; For each parameter, the gradient value of that parameter calculated in step S5031 on each historical transfer sample is squared. The purpose of squaring is to eliminate the positive or negative sign of the gradient, making the magnitude rather than the direction of the gradient the evaluation criterion. Since the gradients of the same parameter on different samples may have opposite directions (positive and negative values), directly averaging would cause the positive and negative values ​​to cancel each other out, failing to truly reflect the average sensitivity of the parameter. Through squaring, the influence of the gradient direction is removed, retaining only the magnitude information.

[0127] After performing gradient squaring operations on all historical transfer samples, the expected value of the squared values ​​obtained for each sample is calculated, i.e., the arithmetic mean of these squared values ​​is calculated. This expected value is used as an estimate of the element at the corresponding position in the Fisher information matrix.

[0128] Applying the above process to all parameters of the stability commentator, the complete Fisher information matrix is ​​constructed. The Fisher information matrix is ​​a square matrix whose dimension equals the total number of network parameters. In deep neural networks, the computation and storage overhead of the complete Fisher information matrix is ​​significant; therefore, only its diagonal portion is used in subsequent steps.

[0129] Step S5033: The diagonal element values ​​of the Fisher information matrix are used as the importance scores of each parameter.

[0130] Extract the diagonal elements from the Fisher information matrix. The diagonal elements of the Fisher information matrix correspond to the expected value of the squared gradient of each parameter, reflecting the average sensitivity of each parameter to the historical task loss function independently. The larger the value of the diagonal element, the more likely a small shift of the parameter near the historical optimum will lead to an increase in the historical task loss function. Therefore, the more critical the parameter is to maintaining historical decision performance.

[0131] The extracted diagonal element values ​​are directly used as the importance scores of the corresponding parameters. This importance score is a non-negative scalar value used as the weight coefficient of the constraint penalty component. The higher the importance score of a parameter, the greater the weight of its corresponding constraint penalty component, and the stronger the constraint force on regression to the historical optimum in subsequent parameter updates.

[0132] In this embodiment, importance weights are automatically assigned to each parameter based on data statistical characteristics, avoiding the subjectivity and inaccuracy of manually setting importance levels, and providing an objective quantitative basis for the implementation of the flexible weight consolidation mechanism.

[0133] For example, to help understand the implementation process of the multi-agent reinforcement learning method obtained by combining this embodiment with the above embodiment one, please refer to... Figure 3 , Figure 3 A simplified flowchart of a multi-agent reinforcement learning method is provided, specifically: Please refer to Figure 3 , Figure 3 A simplified flowchart of a multi-agent reinforcement learning method is provided, specifically: The state information is first input to the encoder, which performs feature extraction and dimensionality reduction on the raw state data to generate a low-dimensional feature representation z. This low-dimensional feature representation z is then split into three parallel processing branches: the first branch is directly used as input to the policy decoder to generate the action distribution; the second branch is processed by the first feature projection layer to generate the first feature branch z1 and input to the plasticity commentator; the third branch is processed by the second feature projection layer to generate the second feature branch z2 and input to the stability commentator.

[0134] After receiving the feature representation of the first branch, the policy decoder outputs the action distribution and samples it to determine the action to be executed. Once this action is applied to the environment, the environment returns an immediate reward signal. Simultaneously, the plasticity critic and the stability critic independently calculate the first value estimate Q1 and the second value estimate Q2 based on their respective feature branches z1 and z2 and the current action. The plasticity critic uses a continuous back-propagation (CBP) mechanism for parameter updates, calculating the contribution utility of each hidden unit and periodically resetting low-utility units to maintain the network's rapid adaptability to new environments. The stability critic employs a resilient weight consolidation mechanism, determining the importance score of each parameter based on the Fisher information matrix and imposing constraints on the updates of important parameters to prevent catastrophic forgetting of historical knowledge.

[0135] The fusion module uses the low-dimensional feature representation z of the first branch as the query q, the first feature branch z1 and the second feature branch z2 as keys k, and the first value estimate Q1 and the second value estimate Q2 as values ​​v. It performs attention calculation and weight allocation, outputting the first weight W1 and the second weight W2. The first value estimate Q1 is weighted based on the first weight W1, and the second value estimate Q2 is weighted based on the second weight W2. The two weighted value estimates are then added (⊕) to generate the fused value estimate Q. This fused value estimate Q is used to guide the policy gradient optimization of the policy decoder and, in conjunction with the reward signal, to calculate the temporal difference loss (TD loss), driving the parameter updates of the two commentators. The entire system continuously collects transfer samples through environmental interaction and stores them in the experience replay buffer. During the training phase, batch data is sampled for parameter optimization, forming a closed-loop iteration of environmental interaction and network updates.

[0136] In a specific implementation, the scheme of this embodiment is compared and tested in a standardized benchmark test environment for continuous control task reinforcement learning. The test is conducted on the CARLA vision reinforcement learning task in an autonomous driving scenario. The cumulative reward is used as the evaluation index, with the horizontal axis representing the number of training steps and the vertical axis representing the cumulative reward. The higher the value, the better the policy performance. Figure 4 This is a performance comparison graph between the multi-agent reinforcement learning method provided in this application embodiment and mainstream baseline algorithms. The horizontal axis represents the number of training steps, with a scale range of 0 to 10 and a unit of ×10. 4 The vertical axis represents the cumulative reward, with the scale from bottom to top being -30, 60, 150, 240, 330, and 420. Figure 4The graph contains seven solid line curves with lightly shaded confidence bands of the same color. From top to bottom in the legend, they are: dark red solid line corresponding to our scheme (Ours), cyan solid line corresponding to SAC (Soft Actor-Critic), purple solid line corresponding to DrQ (Data-regularized Q), light red solid line corresponding to DeepMDP (Deep Markov Decision Process), blue solid line corresponding to CURL (Contrastive Unsupervised Representations for Reinforcement Learning), orange solid line corresponding to DBC (Deep Bisimulation for Control), and green solid line corresponding to MLR (Meta-Learning for Reinforcement). Each solid line curve represents the average change trend of the cumulative reward of the corresponding algorithm during training, and the lightly shaded area of ​​the same color around it represents the training fluctuation range.

[0137] like Figure 5 As shown, Figure 5 This is a comparison chart of ablation test performance provided in the embodiments of this application, wherein, Figure 5 (a) in the figure is a comparison chart of the ablation experimental performance of the single commentator removal mechanism provided in the embodiments of this application. Figure 5 (b) in the figure is a comparison of the ablation experimental performance of the removal fusion module and the dual critic mechanism provided in the embodiments of this application. Figure 5 (a) and Figure 5 In (b) of the diagram, the horizontal axis represents the number of training steps, with a scale range of 0 to 10 and a unit of ×10. 4 The vertical axis represents the cumulative reward, with the scale from bottom to top being -30, 60, 150, 240, 330, and 420. Figure 5 Figure (a) contains four solid line curves with lightly shaded confidence bands of the same color. From top to bottom in the legend, they are: dark red solid line corresponding to our scheme (Ours), cyan solid line corresponding to the baseline method (Baseline), green solid line corresponding to the variant with elastic weight consolidation mechanism removed (w / o EWC), and orange solid line corresponding to the variant with continuous backpropagation mechanism removed (w / o CBP). Figure 5Figure (b) contains four solid line curves with lightly shaded confidence bands of the same color. From top to bottom in the legend, they are: dark red solid line corresponding to our approach (Ours), cyan solid line corresponding to the baseline method (Baseline), green solid line corresponding to the variant with the attention fusion module removed (w / o attention), and orange solid line corresponding to the variant with both critic mechanisms removed (w / o CBP & EWC). Each solid line curve represents the mean change trend of the cumulative reward of the corresponding model variant during training, and the lightly shaded area of ​​the same color around it represents the training fluctuation range.

[0138] Test results show that this solution (i.e.) Figure 4 and Figure 5 As shown in the figure, Ours significantly outperformed all the baselines throughout the training process. At the end of 100,000 frames of training, the episode return value of this scheme reached about 330, while the second-best SAC and DrQ only reached about 180, with a performance improvement of more than 80%. This advantage was particularly evident in the middle of training (40,000-60,000 frames), when the baseline methods generally entered a performance plateau, while this scheme continued to rise, demonstrating a stronger continuous learning ability.

[0139] like Figure 5 The ablation experiments shown further reveal the value of each component, such as Figure 5 As shown in (a), the variant with the elastic weight consolidation mechanism removed (w / o EWC, green curve) shows a rapid increase in performance in the early stages of training, but significant performance fluctuations occur in the later stages, verifying that while the continuous backpropagation mechanism can improve adaptability, the lack of stability constraints leads to policy degradation; the variant with the continuous backpropagation mechanism removed (w / o CBP, orange curve) converges smoothly but with a lower peak value, indicating that overemphasizing knowledge retention inhibits the speed of environment adaptation. Figure 5 As shown in (b), the variant that removes the attention fusion module (w / o attention, green curve) uses static weight averaging, and its final performance drops by about 30% compared to the complete scheme, demonstrating the necessity of dynamic adaptive fusion under high-dimensional non-stationary inputs; at the same time, the baseline method that removes the dual critic mechanism (w / o CBP&EWC, orange curve) has the lowest sample efficiency, highlighting the fundamental improvement of the heterogeneous critic architecture to the exploration-exploitation tradeoff.

[0140] In summary, by adopting a heterogeneous dual-commentator design at the architectural level, the goals of stability and plasticity are decoupled to independent network branches. Then, a state-driven attention mechanism is used to achieve dynamic value fusion, thereby enabling collaborative optimization of rapid adaptation and knowledge retention within a single model. This breaks through the inherent trade-off bottleneck of traditional homogeneous commentator architectures.

[0141] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the multi-agent reinforcement learning method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0142] This application also provides a multi-agent reinforcement learning device. The multi-agent reinforcement learning device provided by this application, employing the multi-agent reinforcement learning method described in the above embodiments, can solve the technical problem of insufficient policy diversity caused by the inherent conflict between stability and plasticity in multi-agent reinforcement learning. Compared with the prior art, the beneficial effects of the multi-agent reinforcement learning device provided by this application are the same as those of the multi-agent reinforcement learning method provided in the above embodiments, and other technical features in the multi-agent reinforcement learning device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0143] This application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the multi-agent reinforcement learning method in Embodiment 1 above.

[0144] The following is for reference. Figure 6 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of this application. The electronic devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0145] like Figure 6As shown, the electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication device 1009 allows the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although the diagrams show electronic devices with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented alternatively.

[0146] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0147] The electronic device provided in this application employs the multi-agent reinforcement learning method described in the above embodiments, which can solve the technical problem of insufficient policy diversity caused by the inherent conflict between stability and plasticity in multi-agent reinforcement learning. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the multi-agent reinforcement learning method provided in the above embodiments, and other technical features of the electronic device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0148] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0149] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0150] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the multi-agent reinforcement learning method described in the above embodiments.

[0151] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0152] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0153] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by an electronic device, enable the electronic device to implement the multi-agent reinforcement learning methods of the various embodiments described above.

[0154] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0155] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0156] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0157] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described multi-agent reinforcement learning method. This addresses the technical problem of insufficient policy diversity caused by the inherent conflict between stability and plasticity in multi-agent reinforcement learning. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the multi-agent reinforcement learning method provided in the above embodiments, and will not be elaborated upon here.

[0158] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the multi-agent reinforcement learning method described above.

[0159] The computer program product provided in this application can solve the technical problem of insufficient policy diversity caused by the inherent conflict between stability and plasticity in multi-agent reinforcement learning. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the multi-agent reinforcement learning method provided in the above embodiments, and will not be repeated here.

[0160] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A multi-agent reinforcement learning method, characterized in that, The multi-agent reinforcement learning method includes: For each agent in the multi-agent group, the action distribution of the agent is obtained based on the agent's current state and the policy decoding network. The action to be executed is sampled from the action distribution and sent to the environment. The environment reward and the next state are received. The transition sample containing the current state, the action to be executed, the environment reward and the next state are stored in the experience replay buffer. The current state includes the state information of the agent and the interaction information between the agent and the environment. In response to the satisfaction of parameter update conditions, a batch of transfer samples is sampled from the experience replay buffer, wherein each transfer sample includes a sample state, a sample action, a sample reward, and a sample next state; For each of the transition samples, the target value is determined based on the sample's next state, the target plasticity commentator, and the target stability commentator; Based on the transfer samples and the target value, the parameters of the plastic critic are optimized. After the parameter optimization of the plastic critic is completed, the weight parameters of the low-utility units are identified and reset based on the cumulative contribution utility of each hidden unit in the plastic critic, wherein the contribution utility of the low-utility unit is less than a preset threshold. Based on the transferred samples and the target value, the parameters of the stability commentator are optimized. When optimizing the parameters of the stability commentator, a penalty term associated with the historical importance of the parameters is constructed based on the elastic weight consolidation mechanism, and the update magnitude of the weight parameters is constrained based on the penalty term. The parameters of the policy decoding network are optimized based on the transferred samples; After updating the parameters of the plasticity critic and the stability critic, a soft update is performed on the target plasticity critic and the target stability critic.

2. The multi-agent reinforcement learning method as described in claim 1, characterized in that, The step of determining the target value based on the next state of the sample, the target plasticity critic, and the target stability critic includes: The next state of the sample is input into the first feature extraction layer, the second feature extraction layer and the third feature extraction layer respectively to obtain the first feature representation, the second feature representation and the third feature representation; The first feature representation is input into the policy decoding network to obtain the next action distribution. The next action is sampled from the next action distribution, and the log probability of the next action is calculated. The second feature representation and the next action are input to the target plasticity critic to obtain the first value component, and the third feature representation and the next action are input to the target stability critic to obtain the second value component; Using the first feature as the query and the second feature as the key, calculate the first attention weight; using the first feature as the query and the third feature as the key, calculate the second attention weight; and weight and fuse the first value component and the second value component based on the first attention weight and the second attention weight to obtain the first fused value. The target value of the transferred sample is calculated based on the sample reward, the preset discount factor, the first fusion value, the preset temperature parameter, and the logarithmic probability of the next action.

3. The multi-agent reinforcement learning method as described in claim 2, characterized in that, The step of optimizing the parameters of the plasticity critic based on the transferred samples and the target value includes: The sample states are respectively input into the first feature extraction layer, the second feature extraction layer and the third feature extraction layer to obtain the fourth feature representation, the fifth feature representation and the sixth feature representation; The fifth feature representation and the sample action are input into the plasticity critic to obtain a first value estimate. Based on the deviation between the first value estimate and the target value, a first basic loss component is determined. Backpropagation is performed based on the first basic loss component to update the network parameters of the plasticity commentator.

4. The multi-agent reinforcement learning method as described in claim 3, characterized in that, The step of identifying and resetting the weight parameters of low-utility units based on the cumulative contribution utility of each hidden unit in the plasticity critic includes: For each hidden unit of the plasticity commentator, the historical contribution utility of the hidden unit is obtained, and the historical contribution utility is decayed by an exponential decay rate to obtain the decayed historical utility contribution; wherein, the historical contribution utility is the cumulative contribution utility of the hidden unit in the historical iterations adjacent to the current iteration; Based on the activation value of the hidden unit and the weights of all output connections from the hidden unit to the next layer during the forward propagation process of the current iteration, the contribution utility of the hidden unit in the current iteration is calculated. The sum of the decayed historical utility contribution and the contribution utility of the current iteration round is determined as the cumulative contribution utility of the hidden unit in the current iteration round. Hidden units whose cumulative contribution utility is lower than the preset threshold are designated as low-utility units, and their weight parameters are set to preset initial values.

5. The multi-agent reinforcement learning method as described in claim 3, characterized in that, The step of optimizing the parameters of the stability commentator based on the transferred samples and the target value includes: The sixth feature representation and the sample action are input into the stability critic to obtain the second value estimate; Based on the deviation between the second value estimate and the target value, a second basic loss component is determined; The importance score of each parameter in the stability reviewer is determined based on the Fisher information matrix. For each parameter, a constraint penalty component is constructed based on the deviation between the current value and the historical best value of the parameter, the importance score of the parameter, and the preset regularization strength coefficient. The constraint penalty components of all parameters of the stability critic are summed to the second basic loss component to obtain the total loss of the stability critic. Backpropagation is then performed based on the total loss of the stability critic to update the network parameters of the stability critic.

6. The multi-agent reinforcement learning method as described in claim 5, characterized in that, The step of determining the importance scores of each parameter among the stability commentators based on the Fisher information matrix includes: Calculate the gradient of the loss function with respect to each parameter for each historical transition sample; The gradient of each parameter is squared, and the expected value is calculated on each historical transfer sample to obtain the Fisher information matrix. The diagonal elements of the Fisher information matrix are used as the importance scores for each parameter.

7. A multi-agent reinforcement learning device, characterized in that, The multi-agent reinforcement learning device includes: An interaction module is used to obtain the action distribution of each agent in a multi-agent system based on the agent's current state and the policy decoding network, sample the execution action from the action distribution, send the execution action to the environment, receive the environment reward and the next state, and store the transition sample containing the current state, the execution action, the environment reward and the next state in an experience replay buffer, wherein the current state includes the agent's state information and the interaction information between the agent and the environment; A parameter update module is used to sample a batch of transition samples from the experience replay buffer in response to meeting parameter update conditions. Each transition sample includes a sample state, a sample action, a sample reward, and a sample next state. For each transition sample, a target value is determined based on the sample next state, a target plasticity critic, and a target stability critic. Based on the transition samples and the target value, parameters of the plasticity critic are optimized. After parameter optimization of the plasticity critic, low-utility items are identified and reset based on the cumulative contribution utility of each hidden unit in the plasticity critic. The weight parameters of the unit are defined, wherein the contribution utility of the inefficient unit is less than a preset threshold; based on the transition samples and the target value, the parameters of the stability commentator are optimized, wherein, when optimizing the parameters of the stability commentator, a penalty term associated with the historical importance of the parameters is constructed based on the elastic weight consolidation mechanism, and the update magnitude of the weight parameters is constrained based on the penalty term; the parameters of the policy decoding network are optimized based on the transition samples; after completing the parameter updates of the plasticity commentator and the stability commentator, the target plasticity commentator and the target stability commentator are softly updated.

8. An electronic device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the multi-agent reinforcement learning method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the multi-agent reinforcement learning method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the multi-agent reinforcement learning method as described in any one of claims 1 to 6.

Images