A multi-agent complex system migration reinforcement learning method based on fusion differential suggestion

Abstract

The application discloses a kind of multi-agent complex system migration reinforcement learning method based on fusion difference proposal, belong to multi-agent complex system field.The application greatly reduces the time and space complexity of difference proposal method by introducing random network distillation model, simultaneously expands the difference proposal idea to the complex environment of high-dimensional state space, expands the application range of algorithm.On this basis, a kind of multi-agent proposal fusion method based on weight is designed, more fully utilizes the proposal from multiple agents obtained based on difference proposal model, so that agent can use the fusion difference proposal from multiple agents generated in multiple high-dimensional states slightly different from current state, improve the quality and efficiency of knowledge transfer, significantly improve the learning effect of multi-agent system early stage, promote agent to obtain faster convergence speed, better solution on task.

Description

Technical Field

[0001] This invention belongs to the field of multi-agent complex systems and relates to a transfer reinforcement learning method for multi-agent complex systems based on fusion differential proposals. Background Technology

[0002] With the advancement of science and technology, multi-agent complex systems have become a hot topic in the research of complex dynamic systems, attracting widespread attention from scientists and engineers in fields such as computer science, artificial intelligence, biology and ecology, and communication and control. They have also been widely applied in various industries, including unmanned communication systems, power grids, robotics, and game AI. With the increasing cost of labor and the rapid development of artificial intelligence algorithms, compared to traditional methods that rely on experts to design task strategies for multi-agent complex systems, strategies trained using reinforcement learning can learn how to complete tasks autonomously without expert guidance. This has led more and more researchers to favor reinforcement learning techniques for designing task strategies for multi-agent complex systems. The complex task environments and large number of agents in multi-agent complex systems place higher demands on the system's task-solving capabilities and learning efficiency. Therefore, designing efficient multi-agent reinforcement learning algorithms for specific multi-agent task scenarios is crucial.

[0003] Multi-agent reinforcement learning typically targets specific task scenarios, establishing corresponding policy learning networks for agents within the environment. Further design of reinforcement learning algorithms allows agents to adjust their network parameters based on reward feedback during interactions with the environment, thus better completing the pre-defined tasks of the multi-agent system. In real-world work scenarios, agents begin tasks without knowing environmental, task, or reward information, nor how to complete the task. Random exploration strategies are usually designed for agents to help them understand the environment and task. Therefore, agents learning from scratch require extensive training to adapt to the task environment and learn how to complete the task. For example, in a soccer game, the agents (players) do not know beforehand that scoring requires shooting the ball into the goal. Therefore, the agents move randomly on the field, learning to score through extensive training. Furthermore, in a multi-agent system, players also need to learn to cooperate to score. A good multi-agent policy typically requires tens of thousands of training rounds, consuming significant training time and computational resources. Therefore, designing a reinforcement learning algorithm to accelerate the learning efficiency of complex multi-agent systems is a worthwhile research problem.

[0004] Transfer learning is an effective means of improving the learning efficiency of a system by transferring knowledge acquired from experts or other processes to the current task. Traditional multi-agent reinforcement learning methods suffer from slow model training speed and low learning efficiency, largely due to the lack of effective communication and knowledge sharing among agents in the environment, where each agent learns independently. For example, in a soccer task, each agent learns how to score independently, which is equivalent to training multiple identical strategies simultaneously, resulting in redundant training and wasted resources. If one agent learns how to score and shares its knowledge with other agents, the learning efficiency of the entire team can be significantly accelerated. Therefore, multi-agent transfer reinforcement learning, which integrates transfer learning and reinforcement learning, is an effective means to accelerate the learning of complex multi-agent systems.

[0005] The core idea of ​​multi-agent transfer reinforcement learning is to help agents learn from knowledge on similar tasks when learning a target task, thereby improving their learning efficiency. Specifically, agents need to transfer knowledge with other agents in the environment, selecting appropriate agents to seek knowledge suggestions based on the specific task and the agent's environment, thus helping them learn better. In a multi-agent system, agents learn appropriate strategies to solve problems through interaction with the environment. However, differences in how different agents interact with the environment lead to differences in policy knowledge, and different agents also have varying degrees of familiarity with the same state space in the environment. More familiar agents possess knowledge that other agents have not yet learned. By transferring knowledge with agents who have acquired more knowledge in a particular state space, agents that perform poorly in that state space can learn, thereby accelerating the system's learning speed.

[0006] Existing multi-agent transfer reinforcement learning algorithms include: traditional methods relying on human experts; transfer methods relying on expert agents; knowledge transfer methods based on action suggestions, including Ad-Hoc, suggestion exchange, and bidirectional imitation; knowledge transfer methods based on value suggestions (PSAF); methods using neural networks to train suggestion policies, including LeCTR and HMAT; and the Confidence-Based Mitigation Policy (RCMP) method, which applies the ideas of multi-head networks to deep value networks (DQN). However, all of these methods share a common limitation: knowledge transfer can only occur when the agent providing knowledge (the teacher agent) experiences a state that is exactly the same as the state currently held by the agent seeking knowledge (the student agent). In complex environments with high-dimensional states, especially in the early stages of learning when knowledge transfer is most needed, it is not easy for two agents to experience the exact same state. This requirement greatly limits knowledge transfer between agents, thus affecting the learning performance of multi-agent systems. To address this limitation, some researchers have applied the concept of differential privacy to multi-agent transfer reinforcement learning, proposing the Differential Advice Method (DA-RL). This method allows student agents to utilize knowledge from other teacher agents generated on a similar state that differs slightly from the current query state (only one state dimension is different). This method increases the chances of knowledge transfer and significantly improves the learning efficiency of multi-agent systems. However, the difference proposal method still has the following drawbacks:

[0007] (1) This method requires recording every state experienced by each agent from the beginning of learning. When determining whether similar states have been experienced, it is necessary to compare the current query state of the student agent with all states experienced by the teacher agent in each state dimension. As the number of states experienced by the agent increases, the number of states to be stored and the time consumed by each comparison will also increase. At the same time, in order to determine the agent's familiarity with the current state and decide whether to seek and provide suggestions, this method needs to record the number of times each agent accesses each state. In summary, DA-RL is an algorithm with extremely high time and space complexity.

[0008] (2) The method's approach to judging similar states and familiarity with states makes it impossible to apply the idea of ​​difference proposal to complex environments with high-dimensional or continuous state spaces. The application scenarios of the method are greatly limited.

[0009] (3) Similar to this approach, existing suggestion methods either select only one agent for direct knowledge transfer or choose one suggestion from multiple agents when using suggestions. Neither method maximizes knowledge sharing among multiple agents and fails to fully leverage the advantages of multi-agent systems. Furthermore, since differential suggestion methods have yielded more suggestions than traditional suggestion methods, failing to fully utilize these suggestions is a waste of resources in multi-agent systems. Summary of the Invention

[0010] To address the issues of high time and space complexity of differential proposal algorithms, their inability to be applied to complex environments with high-dimensional or continuous state spaces, and their failure to maximize knowledge sharing among multiple agents, this invention proposes a multi-agent transfer reinforcement learning method based on fused differential proposals.

[0011] In this invention, unlike traditional differential proposal methods that record all experienced states and then compare them one by one according to dimension when judging similar states, this invention uses a stochastic network distillation model to calculate the similarity between states in order to reduce time and space complexity. The stochastic network distillation model consists of two neural networks with identical structures but different parameters. Using the stochastic network distillation model, only one forward propagation of the state input to these two neural networks is needed to determine whether the agent has experienced a similar state. This greatly reduces the space complexity of storing all experienced states and the time complexity of comparing them one by one according to dimension, extending the differential proposal method to complex environments with high-dimensional state spaces. In terms of proposal generation, teacher agents that have experienced similar states generate differential proposals, and then the output values ​​of the stochastic network distillation model are used to generate weights. The proposals from multiple teacher agents are then fused based on these weights as the final transferred knowledge. This method significantly improves the degree of effective knowledge transfer between agents, further enhancing the learning performance of multi-agent systems and accelerating the convergence process.

[0012] In this invention, the agent's task is to complete the task objectives set by the multi-agent system, and the solution is the policy network parameters for the agent to complete the task. Based on this, the method of this invention is roughly divided into three stages:

[0013] (1) Determining whether a knowledge transfer stage is required: In this invention, each agent is equipped with an independent random network distillation model. This model consists of two neural networks with identical structures: a prediction network and a target network. Before training begins, the two networks are randomly initialized with different weights, and then the parameters of the target network are fixed. The principle behind using the random network distillation model to determine whether an agent has experienced a state similar to or the same as its current state is that, during training, the state s is... tThe inputs are fed into two separate networks, resulting in two outputs. and f(s) t ), calculate their L1-Loss: As an error, a gradient descent algorithm is used to train the prediction network, making the prediction network predictive of state s. t The output of the algorithm approximates the target network. Through this training process, for an input state s... t If the random network distillation model has undergone training with exactly the same or similar states, then the output error between the prediction network and the target network will be smaller, while the current state s t The greater the difference between the predicted network and the training samples input to the random network distillation model, the greater the output error between the predicted network and the target network. The magnitude of the error between the two network outputs can be used to determine whether the agent has experienced a state s. t Similar or identical states. During the training of a multi-agent system, at a certain moment, agent i is in state s. t At this point, agent i will state s t If the error μ is small enough when input into the random network distillation model, it means that agent i has experienced states s. t If the states are exactly the same, agent i uses its own policy network to make decisions; if the error μ is relatively small, it means that agent i has experienced states s and s. t If the states are similar, agent i uses differential proposal to make self-propagation without knowledge transfer; if the error μ is large, it means that agent i has not experienced a state similar to s. t In similar situations, it is necessary to seek advice from others and transfer knowledge.

[0014] (2) Knowledge Transfer Stage: The knowledge of an agent is defined as the value function Q(s,a) of all actions in a given environmental state. Therefore, the content transferred is the Q value. When the student agent issues a knowledge transfer request, other agents will transfer the student agent's current state s. t Input the data into your own random network distillation model, and determine whether you have experienced state s based on the output error value. t The process involves considering whether the states are identical or similar. If the states are identical, knowledge transfer is performed directly, providing the knowledge Q(s,a) and error μ. If the states are similar, difference proposals are used to add Laplace noise to the knowledge Q(s,a) to ensure that the knowledge remains consistent across states s and s. t The availability of the student agent is assessed before knowledge transfer. If the student agent has not experienced the same or similar state, no knowledge transfer is performed. If there is more than one teacher agent capable of knowledge transfer, their respective errors μ are used as weights to fuse their knowledge as the final knowledge transfer to the student agent.

[0015] (3) Multi-agent reinforcement learning and stochastic network distillation model training stage: The learning process of the agent in the environment is defined as a Markov decision process, that is, the agent's learning policy is a mapping relationship from state to action to reward. During the learning process, the agent stores the state, action, reward, etc. obtained by interacting with the environment into the experience pool, and uses methods such as importance sampling to train the agent network to complete the policy update. The state of the trained agent network is used to train the agent's stochastic network distillation model. During the interaction between the agent and the environment, the agent first decides whether to perform knowledge transfer by judging whether to perform knowledge transfer. If knowledge transfer is performed, the agent selects an action based on the knowledge Q(s,a) obtained by knowledge transfer; if knowledge transfer is not performed, the agent selects an action by adopting its own policy.

[0016] This invention employs traditional deep reinforcement learning algorithms (e.g., DQN algorithm, DDQN algorithm, etc.) to solve the policy of a single agent in a multi-agent environment, and accelerates the task convergence speed through the transfer operation of agent policy knowledge.

[0017] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0018] A transfer reinforcement learning method for complex multi-agent systems based on fusion difference proposals is proposed, with the following specific steps:

[0019] Step 1: Initialize the multi-agent reinforcement learning model and the stochastic network distillation model

[0020] A multi-agent reinforcement learning task is represented by a tuple. <S,A i O i ,R i ,T,γ,n>. Where S represents the state space of the environment. A i O i and R i Let represent the action space, observation space, and reward function of agent i, respectively. T is the state transition function of the environment, γ is the reward discount factor, and n is the number of agents in the environment. All agents are represented by neural networks with identical structures, and each agent can obtain its own observation o. i ∈O i Due to the limitations of the task environment, if the state of the environment can be fully observed by every agent, it is a globally observable environment; otherwise, it is a partially observable environment. Partially observable environments increase the difficulty of learning for agents.

[0021] The actions of all agents constitute a joint action, denoted as . Each agent's policy π i It can be modeled as an agent in state s i Take action ai The probability, the joint policy of the agents is defined as π = (π / 2) 1 ,π 2 ,…,π n ),Right now

[0022] π=p(s,a)=p(a|s)=p((a 1 ,a 2 ,…,a n )|s) (1)

[0023] Each agent's reward function is influenced by the environmental state and the actions of other agents, i.e., r i =R i (s,a,s ′ The reward function is set so that only the agent itself can receive it. State transitions in the environment are determined by the previous state of the environment and the joint actions of the agents. For multi-agent systems, the goal of multi-agent reinforcement learning is to find a set of policy parameters that maximizes the expected cumulative reward.

[0024] At the same time, a random network distillation model is set for each agent. Each random network distillation model consists of a prediction network and a target network. The two have the same structure but different initialization parameters. Only the parameters of the prediction network are updated in each training session, while the parameters of the target network are not updated after initialization.

[0025] The specific steps are as follows:

[0026] 1.1) Randomly initialize the Q network parameters w of the agent. i Clear experience pool D.

[0027] 1.2) Randomly initialize the network parameters of the prediction network and the target network in the random network distillation model.

[0028] Step 2: Multi-agent interaction with the environment

[0029] 2.1) Observe the state s of agent i i .

[0030] 2.2) Obtain state s i Then, a random network distillation model is used to determine whether knowledge transfer should be performed. If knowledge transfer is not required, the state s is then... i Input the agent into the Q-network to obtain estimates of all actions, and use the ∈-greedy method to select action a to be executed. i If knowledge transfer is required, then use the transferred knowledge selection action a. i .

[0031] 2.3) Execute action a i Receive environmental rewards r i and new environmental statesi ′ , will (s i ,a i ,r i ,s i ′ Store it in the experience pool.

[0032] Step 3: Train the reinforcement learning model

[0033] 3.1) Collect m samples (s) from the experience pool D. j ,a j ,r j ,s j ′ ), j = 1,…,m. Calculate the target value y. j :

[0034]

[0035] Wherein, the target value y j Indicates that in state s j The real reward obtained at that time; a j ′ For state s j ′ The agent will take the action that maximizes the reward; the task ends when the agent reaches the termination state; γ is the decay factor; w i represents the network parameters; Q(·) represents the value function.

[0036] 3.2) The mean squared loss error function is used to update all parameters w of the agent's Q-network through backpropagation of the neural network's gradient. i :

[0037]

[0038] in, For state s j The value obtained by the time-policy network estimation.

[0039] Step 4: Train the random network distillation model

[0040] 4.1) Take the states s from the m samples in step 3.1) j Inputting the data into a random network distillation model yields the predicted network output value. and the target network output value f(s) j ).

[0041] 4.2) Calculate L1-Loss: As an error.

[0042] 4.3) Use the error μ to update all parameters of the prediction network through backpropagation of the gradient of the neural network.

[0043] 4.4) Stop iterative training when the preset number of training sessions or task completion indicators are reached.

[0044] Furthermore, in step 2, the specific process for determining whether knowledge transfer should be performed is as follows:

[0045] The agent uses a random network distillation model to determine if it has experienced a state that is the same as or similar to its current state. If it has, it chooses an action to interact with the environment based on its own policy or uses differential suggestions for self-suggestion; otherwise, it requests knowledge transfer from other agents. The specific steps are as follows:

[0046] (1) The state of agent i Input your own random network distillation model and calculate the error μ. i :

[0047]

[0048] (2) μ i With threshold ask self Comparison, threshold ask self The decision determines whether an agent requests suggestions from other agents to initiate knowledge transfer. This is done for ease of comparison with the threshold `ask`. self Compare the errors μ i Normalize:

[0049]

[0050] Where, μ max μ represents the maximum value during the entire convergence process of the random network distillation model. min This represents the minimum value during the entire convergence process of the random network distillation model.

[0051] (3) If μ i <ask self This indicates that the state currently accessed by the agent is not significantly different from states that have already been accessed, and can be considered similar or identical states. The agent can then make decisions independently.

[0052] The specific steps are as follows:

[0053] a. The agent generates its own Q-network. Value vector.

[0054] b. Put μ i With threshold ask none Compare, ask noneUsed to determine whether to perform differential self-suggestion, ask none <ask self If μ i <ask none Then it can be almost assumed that the agent has experienced a state similar to the current state. If the situation is exactly the same and the person is very experienced, then proceed directly according to... Use the ε-greedy strategy to select actions. If μ i >ask none This indicates that the agent has experienced a state similar to the current state, and then... Laplace noise is added to each state-action value of the vector as a difference proposal to ensure that the proposal generated in similar states is valid in the current state. It is still available online:

[0055]

[0056] Where ∈ represents the privacy budget, which determines the level of privacy protection; the smaller ∈ is, the greater the noise. ΔQ represents the suggestion sensitivity.

[0057]

[0058] Where s represents the current state and s′ represents a similar state. In fact, ΔQ cannot be calculated before the experiment because calculating ΔQ requires the final Q-value of the environment state. However, ΔQ is calculated based on the Q-value, which is calculated based on the reward; therefore, the reward can be used to estimate ΔQ:

[0059] ΔQ=(reward max -reward min )α (8)

[0060] Among them, reward max and reward min These represent the maximum and minimum rewards in the experiment, respectively, and α is the learning rate.

[0061] (4) If μ i >ask self This indicates that the difference between the state currently accessed by the agent and the states it has already accessed is too large, so the agent seeks advice from other agents. The state is then... The broadcast is sent to other intelligent agents, requesting knowledge transfer.

[0062] Furthermore, in step 2, after determining whether knowledge transfer should be performed, if a request is made to transfer knowledge to other intelligent agents, the specific process is as follows:

[0063] When other agents receive a request for knowledge transfer from agent i, they determine whether knowledge transfer can proceed based on the output error of their respective random network distillation models. If more than one agent is involved in knowledge transfer, their knowledge is merged to form the final knowledge transfer. The steps are as follows:

[0064] 1) Agent j will state Input your own random network distillation model and calculate the output error μ. j And use formula (5) to calculate the error μ j Normalization is performed.

[0065] 2) Put μ j With threshold ask self Comparison, threshold ask self The decision determines whether agent j will undergo knowledge transfer with agent i, who requested the advice.

[0066] 3) If μ j >ask self This indicates that the teacher's agent has experienced states similar to... The differences are too great, so knowledge transfer is not permitted.

[0067] 4) If μ j <ask self This indicates that the teacher agent has experienced the same or similar states, so knowledge transfer is possible, and further based on μ... j with ask none The relative sizes of the values ​​determine whether to perform direct knowledge transfer or transfer through difference suggestions. The specific steps are as follows:

[0068] a. If μ j <ask none This indicates that the teacher's intelligent agent has experienced states... In the same state, knowledge transfer can be carried out directly.

[0069] b. If μ j >ask none This indicates that the teacher's intelligent agent has experienced states... For similar states, use formula (6) to make differential suggestions to ensure that the suggestions generated in similar states are applicable to the current state. It is still usable.

[0070] 5) If there are more than one teacher agent capable of knowledge transfer, their knowledge is merged as the final knowledge transfer. The specific steps are as follows:

[0071] a. Providing suggested μ values ​​(μ1, μ2, ..., μ) for the agent. k Transform into 1-μ1, 1-μ2, ..., 1-μk The inputs are fed into the softmax module to generate weights ω1, ω2, ..., ω. k :

[0072]

[0073] b. Integrate knowledge:

[0074]

[0075] in, This represents the knowledge provided by each teacher agent.

[0076] 6) Student intelligent agent i uses knowledge Make a decision-making action a i This completes the knowledge transfer process.

[0077] The beneficial effects of this invention are as follows: This invention employs a novel knowledge transfer method in the training process of multi-agent reinforcement learning. First, by introducing a random network distillation model, the time and space complexity of the differential proposal method is greatly reduced, extending it to complex environments in high-dimensional state spaces. Furthermore, weight-based fusion proposals enhance the utilization of transferred knowledge by the agents, enabling student agents to use fused differential proposals generated by multiple agents on multiple high-dimensional states slightly different from the current state. This further improves the learning performance of the multi-agent system and accelerates the learning ability and speed of multi-agent reinforcement learning. Attached Figure Description

[0078] Figure 1 This is a flowchart illustrating the framework of the present invention.

[0079] Figure 2 This is the agent reinforcement learning model used in this invention.

[0080] Figure 3 This is the random network distillation model used in this invention.

[0081] Figure 4 This is a schematic diagram illustrating the determination of whether knowledge transfer should be performed and the knowledge transfer process designed for this invention. Specific implementation examples:

[0082] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0083] The method of this invention can be used to accelerate the training process of agents in a multi-agent environment. The method flow of this invention is as follows: Figure 1 As shown.

[0084] The DQN network used in this invention constructs the intelligent agent, and its reinforcement learning model is as follows: Figure 2 As shown.

[0085] The stochastic network distillation model used in this invention has the following structure: Figure 3 As shown.

[0086] The invention designs a method for determining whether knowledge transfer should be performed and the knowledge transfer process as follows: Figure 4 As shown.

[0087] The embodiments of the present invention will be described in detail below (e.g. Figure 1 As shown), the specific steps include:

[0088] 1) Initialization process: First, initialize the DQN network parameters of the agent and clear the experience pool.

[0089] 2) Main loop process:

[0090] 2.1) Observe the environmental state: Obtain the current environmental state through the agent's sensors.

[0091] 2.2) Calculate the output error: Input the observed state features into the agent random network distillation model and calculate the output error.

[0092] 2.3) Determine whether knowledge transfer should be performed based on the output error.

[0093] 2.4) Select and execute actions: Based on the knowledge transfer results, the agent can make its own decision and select actions or select actions based on the transferred knowledge. After executing the actions, the agent obtains environmental feedback rewards and the characteristics of the next state.

[0094] 2.5) Store experience: Store the agent's state, actions, rewards, and next state information into the experience pool.

[0095] 2.6) Network parameter update: Sample data is drawn from the experience pool, the loss is calculated, and the DQN network parameters are updated using gradient descent backpropagation.

[0096] 2.7) Random network distillation model training: Using the environmental state data sampled in the previous step, calculate the error, and use the gradient descent method to backpropagate and update the parameters of the prediction network in the random network distillation model.

[0097] When the loop condition is not met, that is, when the loop reaches the preset number of training times or the preset index, a good agent strategy scheme can be obtained in the multi-agent system.

[0098] This example demonstrates the determination of whether knowledge transfer should be performed and the knowledge transfer process as follows: Figure 4As shown in the figure, in a football game task with four attacking players, all players need to cooperate to complete the task of scoring a goal. In this method, the players, as individuals participating in the system training, can be abstracted as intelligent agents. The state of agent 1 (the player) is obtained through sensors, and the error value μ output by the random network distillation model is used. i Determine whether knowledge transfer is needed. If not, calculate the Q-value based on your own Q-network and determine whether differential suggestions are needed to obtain the Q-value; if so, input your own state to the other three agents. The other three agents obtain their respective error values ​​μ and knowledge Q-values ​​through a random network distillation model. Based on the weights, perform knowledge fusion to obtain the final Q-value as the transferred knowledge.

Claims

1. A transfer reinforcement learning method for multi-agent complex systems based on fusion difference proposals, characterized in that, The method includes the following steps: Step 1: Initialize the multi-agent reinforcement learning model and the stochastic network distillation model A multi-agent reinforcement learning task is represented by a tuple. ;in, Represents the state space of the environment. , and Representing intelligent agents respectively The action space, observation space, and reward function. Let the state transition function be the environment. As a reward discount factor, The number of agents in the environment; all agents are represented by neural networks with the same structure, and each agent can obtain its own observations. ; The actions of all agents constitute a joint action, denoted as . ; each agent's policy Modeling as an agent in state Take action below The probability, the joint policy of the agents is defined as ,Right now Each agent's reward function is influenced by the environmental state and the actions of other agents. The reward function is set so that only the agent itself can obtain it; the state transition of the environment is determined by the previous state of the environment and the joint action of the agent; for multi-agent systems, the goal of multi-agent reinforcement learning is to find a set of policy parameters that maximizes the expected cumulative reward. At the same time, a random network distillation model is set for each agent. Each random network distillation model consists of a prediction network and a target network. The two have the same structure but different initialization parameters. Only the parameters of the prediction network are updated in each training session, and the parameters of the target network are not updated after initialization. The intelligent agent is an athlete in a football game task; The specific steps are as follows: 1.1) Randomly initialize the intelligent agent Network parameters Clear the experience pool ; 1.2) Randomly initialize the network parameters of the prediction network and the target network in the random network distillation model; Step 2: Multi-agent interaction with the environment 2.1) Observing intelligent agents status ; 2.2) Obtain the state Then, a random network distillation model is used to determine whether knowledge transfer should be performed. If knowledge transfer is not required, the state is then... Input agent The network obtains estimates of all actions, and uses... Greedy algorithm for selecting actions to execute If knowledge transfer is required, then use the transferred knowledge selection action. ; 2.3) Execute the action Receive environmental awards and new environmental conditions ,Will Store in the experience pool; Step 3: Train the reinforcement learning model 3.1) From the experience pool Collection in China Sample ; Calculate the target value : Among them, the target value Indicates the state The real rewards obtained at that time; In the state The agent will take the action that yields the greatest reward when the task is estimated; the end of the task indicates that the agent has reached the termination state. It is the attenuation factor; For network parameters; Q (·) represents the value function; 3.2) The agent is updated using the mean squared loss error function through backpropagation of the gradient of the neural network. All parameters of the network : in, In the state The value obtained by the time-policy network estimation; Step 4: Train the random network distillation model 4.1) Take step 3.1) m The state in each sample Inputting the data into a random network distillation model yields the predicted network output value. and target network output value ; 4.2) Calculate L1-Loss: As for error; 4.3) Usage error All parameters of the prediction network are updated through backpropagation of the gradients of the neural network. 4.4) Stop iterative training when the preset number of training iterations or task completion indicators are reached; In step 2, the specific process for determining whether knowledge transfer should be performed is as follows: (1) The intelligent agent Current status Input the data into your own random network distillation model and calculate the error. : (2) With threshold Comparison, threshold The decision determines whether an agent requests advice from other agents to initiate knowledge transfer. (3) If This indicates that the state currently accessed by the agent is not significantly different from the states it has already accessed, and is therefore considered to be a similar or identical state; the agent makes its own decisions. (4) If This indicates that the difference between the state currently accessed by the agent and the states it has already accessed is too large, so the agent seeks advice from other agents; the state is then... Broadcast to other intelligent agents, requesting knowledge transfer; If more than one teacher agent is capable of knowledge transfer during the knowledge transfer process, their knowledge is merged as the final knowledge transfer. The specific steps are as follows: a. The agent that will give suggestions value Transform into The weights are generated by inputting them into the softmax module. : b. Integrate knowledge: in, This represents the knowledge provided by each teacher agent; In the process of determining whether to perform knowledge transfer, when hour, a. The agent generates its own Q-network. Value vector; b. With threshold Comparison, Used to determine whether to perform differential self-suggestion. ;like Then the agent has experienced a state different from the current state. If the situation is exactly the same and the person is very experienced, then proceed directly according to... use Strategy selection action; if This indicates that the agent has experienced a state similar to the current state, and then... Laplace noise is added to each state-action value of the vector as a difference proposal to ensure that the proposal generated in similar states is valid in the current state. It is still usable.

2. The transfer reinforcement learning method for multi-agent complex systems based on fusion difference proposals according to claim 1, characterized in that, Will With threshold The comparison process is as follows: in, The privacy budget determines the level of privacy protection. The smaller the value, the louder the noise. Indicates the sensitivity of the suggestion: in, This is the current state. They are in similar states; in fact, It is impossible to perform calculations before the experiment because calculations... The final environmental state is required. Value; but Based on Value calculation, The value is calculated based on the reward, therefore a reward estimate is used. : in, and These represent the maximum and minimum rewards in the experiment, respectively. This is the learning rate.

3. The transfer reinforcement learning method for multi-agent complex systems based on fusion difference proposals according to claim 1, characterized in that, In step 2, after determining whether knowledge transfer should be performed, if a request is made to transfer knowledge to other intelligent agents, the specific process is as follows: 1) Intelligent agent State Input your own random network distillation model and calculate the output error. ; 2) With threshold Comparison, threshold The decision is made by the intelligent agent. Is it consistent with the agent that requested the suggestion? Knowledge transfer occurs; 3) If This indicates that the teacher's agent has experienced states similar to... The differences are too great, so knowledge transfer is refused. 4) If This indicates that the teacher agent has experienced the same or similar states, and is therefore able to transfer knowledge. 5) Student Intelligent Agent Use of knowledge Making decision-making actions This completes the knowledge transfer process.

4. The transfer reinforcement learning method for multi-agent complex systems based on fusion difference proposals according to claim 3, characterized in that, In step 2, during the knowledge transfer process, when At that time, according to and The magnitude of the difference determines whether knowledge transfer should be performed directly or through difference suggestions: a. If This indicates that the teacher's intelligent agent has experienced states... In the same situation, knowledge transfer can be performed directly; b. If This indicates that the teacher's intelligent agent has experienced states... For similar states, use formula (5) to make differential suggestions to ensure that the suggestions generated in similar states are applicable to the current state. It is still usable.

5. The transfer reinforcement learning method for multi-agent complex systems based on fusion difference proposals according to claim 1, characterized in that, In step 2, the error is calculated. Then, for convenience with threshold Compare and assess the error Normalize: in, This represents the maximum value during the entire convergence process of the random network distillation model. This represents the minimum value during the entire convergence process of the random network distillation model.

6. The transfer reinforcement learning method for multi-agent complex systems based on fusion difference proposals according to claim 3, characterized in that, In step 2, the error is calculated. Then, formula (10) is used to measure the error. Normalize: in, This represents the maximum value during the entire convergence process of the random network distillation model. This represents the minimum value during the entire convergence process of the random network distillation model.

Images