Electric power communication system robust routing method based on multi-agent deep reinforcement learning

By adopting a robust routing method of multi-agent deep reinforcement learning in power communication systems, using Soft-Actor-Critic algorithm and multi-head attention neural network, the routing strategy in the existing technology cannot meet security, speed and The problem of robustness achieves fast, safe and robust routing decisions in power communication systems, significantly improving the system's anti-interference ability.

CN119071211AActive Publication Date: 2024-12-03SOUTH CHINA UNIV OF TECH

Patent Information

Application Number
CN202411041127.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-12-03
Estimated Expiration
2044-07-31

AI Technical Summary

Technical Problem

The routing policies of existing power communication systems cannot meet the requirements of security, speed and robustness at the same time, especially in the face of DDoS attacks or communication link disconnection, which cannot effectively maintain the stability of packet transmission.

Method used

Using a robust routing method based on multi-agent deep reinforcement learning, multi-head attention neural network is designed by establishing an independent routing policy network at each routing node, and using the Soft-Actor-Critic algorithm to alternately update the routing policy network and the critic network. and dual Q networks realize optimal routing decisions for each agent, reduce information exchange frequency and improve robustness.

Benefits of technology

Fast, secure and robust routing decisions in power communication systems are realized, and the ability to maintain packet transmission delay within the allowable range in the case of network attacks or failures, significantly improving the security and anti-interference ability of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071211A_ABST
    Figure CN119071211A_ABST
Patent Text Reader

Abstract

The invention discloses an electric power communication system robust routing method based on multi-agent deep reinforcement learning, and the method comprises the steps: building an information flow model of an electric power communication system, taking a routing node as an agent, designing an observation space, an action space and a reward function of the agent according to a routing optimization target, and carrying out the routing optimization. A routing optimization problem is converted into a multi-agent deep reinforcement learning problem, a routing strategy network and an improved commentator network are designed for each agent, parameters of the routing strategy network and the improved commentator network of each agent are alternately updated by adopting a Soft-Actor-Critic algorithm, an optimal routing strategy network of all agents is obtained, and the optimal routing strategy network of all agents is obtained. And the routing module is used for outputting an optimal routing decision, and when the power communication system is subjected to network attack or has a fault, different intelligent agents maintain the transmission delay of a normal measurement data packet within an allowable range through the cooperation of the optimal routing decision. The method can be applied to millisecond-level control electric power communication scenes, and the communication delay and the packet loss rate of an electric power communication system are minimized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of power communication systems and routing optimization, and in particular to a robust routing method for power communication systems based on multi-agent deep reinforcement learning. Background Art

[0002] Modern power systems present the "three highs" characteristics of high proportion of new energy access, high proportion of power electronic devices and high proportion of distributed energy, and the operation mode and dynamic mechanism have become more complex. The complexity of modern power systems has put forward higher requirements for information control. With the access of a high proportion of new energy, energy injection in the power system has become dispersed, and the volatility of generator power has increased. This means that during the operation of the power system, faster dispatching and control are needed to maintain the balance and stability of the system.

[0003] Fast and accurate information control of the power system requires a safe and reliable power communication system as support, and data packet routing is a key link to ensure communication reliability. The routing optimization problem refers to finding the optimal routing strategy for each routing node in the communication network to minimize the average transmission delay and packet loss rate of all data packets from the source node to the target node. The routing strategy of the power communication system needs to meet the requirements of security, speed and robustness. Security means minimizing the number of information exchanges between routing nodes and reducing the exposure to network attacks. Speed ​​means that the routing strategy must have the ability to adjust quickly. Robustness means that the routing strategy must have a certain anti-interference ability. When some information source nodes in the power communication system are attacked by DDoS or some communication links are disconnected, the transmission delay of normal data packets can be maintained within the allowable range and no packet loss occurs.

[0004] The commonly used routing strategies can be divided into fixed shortest path routing strategy and adaptive routing strategy. The fixed shortest path routing strategy uses a graph optimization algorithm to calculate the shortest path from each information source node to the target node, and establishes a fixed routing decision table for each routing node. However, this routing strategy cannot be dynamically adjusted according to the state of the communication system, and does not meet the robustness requirements of the smart grid. The link state routing strategy is a type of adaptive routing strategy widely used in computer communication networks. However, this routing strategy requires a large amount of forwarding state query information, which increases the exposure of network attacks. When the state of the communication system changes suddenly, it takes a long transition time to readjust to the optimal routing strategy, which cannot meet the safety and speed requirements of the power communication system. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a robust routing method for an electric power communication system based on multi-agent deep reinforcement learning. The method is based on the idea of ​​distributed decision-making and establishes a mutually independent routing strategy network for each routing node. Each routing node only needs local observation data to make routing decisions, avoiding the safety hazards caused by frequent information interactions between different nodes. Moreover, the state change of the electric power communication system is only reflected as a change in the node observation quantity. Each routing node does not need extra time to adjust the routing strategy, solving the problem that traditional routing methods cannot take into account safety, speed and robustness at the same time.

[0006] To achieve the above-mentioned purpose, the technical solution provided by the present invention is: a robust routing method for a power communication system based on multi-agent deep reinforcement learning, wherein the power communication system is composed of an information source node for collecting measurement data, a routing node for receiving and forwarding measurement data packets, and a communication link, and includes the following steps:

[0007] S1: Based on the measurement data collection mechanism of the information source node and the data packet transmission mechanism of the power communication system, an information flow model of the power communication system is established;

[0008] S2: Based on the information flow model, the routing nodes in the power communication system are regarded as intelligent agents, and the observation space, action space and reward function of the intelligent agents are designed according to the goal of routing optimization, so as to transform the routing optimization problem into a multi-agent deep reinforcement learning problem;

[0009] S3: For the multi-agent deep reinforcement learning problem in step S2, a routing strategy network and an improved critic network are designed for each agent; the routing strategy network is used to receive the observation vector obtained by the agent in the power communication system and output the corresponding routing decision vector, and the improved critic network is used to score the routing decision vector output by the routing strategy network; the routing strategy network is composed of three layers of fully connected neural networks and one layer of LSTM neural network connected in sequence; the improved critic network adds an attention mechanism on the basis of the traditional critic network, and is composed of a multi-head attention neural network and a Q-value network. The multi-head attention neural network takes the observation vectors and routing decision vectors of all agents in the power communication system as input, and uses the multi-head attention mechanism to calculate the attention weights between each pair of agents. The attention feature vector of each agent is obtained by weighted summation. The Q-value network adopts a dual Q network structure, and each Q network is composed of two layers of fully connected neural networks, which receive the attention feature vector and observation vector of the agent and output the score of the agent's routing decision;

[0010] S4: The Soft-Actor-Critic algorithm is used to alternately update the routing strategy network of each agent and improve the parameters of the critic network, and finally the optimal routing strategy network of all agents is obtained to output the optimal routing decision. When the power communication system is attacked by a network or fails, different agents coordinate the optimal routing decision to maintain the transmission delay of normal measurement data packets within the allowable range.

[0011] Further, in step S1, the power communication system uses an undirected graph G=<V,E> Represented by, where V represents the set of all routing nodes, and E represents the set of all communication links; each routing node is connected to a fixed number of information source nodes, receives the measurement data packets generated by the information source nodes and stores them in the buffer, each data packet has a routing node as the target node, and is removed from the system when the data packet reaches the corresponding target node; the communication link adopts a full-duplex communication mechanism, and the data packets at both ends of the communication link can be transmitted to the opposite side at the same time and share the transmission bandwidth; the state equation of the power communication system is:

[0012]

[0013] B ij (t) = B ij (t-1)-b ij (t)

[0014] Where N i Represents routing node v i The neighbor node set, B i (t) represents the routing node v at time t i The number of packets in the buffer, S ij (t), B ij (t) represents the communication link e at time t ij The number of packets being transmitted and the number of packets in the buffer; in the unit time from t-1 to t, u i (t) represents node v i The number of packets generated by the connected information source node, δ ji (t) represents the slave node v j The number of packets transmitted to node vi, ω lj (t),ω ki (t) represent the slave nodes v l 、v k Incoming communication link ij The number of packets, r ij (t) represents the communication link e ij Incoming node v i 、v j The total number of packets in the buffer, bij (t) represents the communication link e ij The number of packets transferred from the buffer to the transmission queue; when the number of packets in the communication link and routing node buffer exceeds the maximum capacity of the buffer, the overflow packets will be removed from the system according to the "first in, first out" principle.

[0015] Furthermore, in step S2, the routing optimization problem is transformed into a multi-agent reinforcement learning problem, where the power communication system is regarded as an interactive environment, the routing nodes are regarded as agents, and the observation space, action space and external rewards of the agents are defined as follows:

[0016] a. Observation space: Based on the security requirement, each agent can only obtain the status information of neighbor nodes and neighbor communication links. The neighbor nodes are agents directly connected to the agent through a communication link. The corresponding communication link is the neighbor communication link. Define the observation space O of agent i i = {U i ,L i ,H i}, where U i represents the number of data packets in the buffer of all neighbor nodes of agent i at the current moment, L i represents the remaining bandwidth of all communication links connected to agent i at the current moment, H i Represents the local state information of agent i, including the target nodes of the next n packets in its buffer and the number of packets forwarded to each neighboring node by the previous m routing decisions;

[0017] b. Action space: The action of agent i is the routing decision, and its action space A is defined i Number all neighbor nodes of agent i; action a for agent i i One-hot encoding is used, where the index with element value 1 corresponds to the next-hop neighbor node number. If agent i does not make a routing decision, then a i is an all-zero vector;

[0018] c. External rewards: Based on the goal of the routing optimization problem, the reward mechanism of agent i should minimize the total delay and packet loss rate of the forwarded data packet p to the target node of the data packet. The following are the external rewards r for the three possible situations in the routing process: i ext Modeling is performed as follows:

[0019] c1. Data packet p is transmitted to the target node within the maximum allowed delay: the reward r obtained by each agent i that forwards data packet p i ext =t i (p)-t d(p), where t i (p) is the time when agent i forwards packet p, t d (p) is the time when the data packet arrives at the destination node;

[0020] c2. Packet p is lost because the communication delay exceeds the maximum allowed value: the reward r obtained by each agent i that forwards packet p i ext =β p (t i (p)-t d (p)), where β p >1 is the penalty coefficient;

[0021] c3. Packet p is lost because the buffer is full: The reward obtained by agent i who forwarded packet p for the last time where t max (p) is the maximum allowed delay for data packet p;

[0022] In addition, the agent’s reward at the moment when no routing decision is taken is set to the average reward of the agent’s historical actions.

[0023] Further, in step S3, the multi-head attention neural network uses A ω It means that its input is the observations of all agents o = (o1, o2, ..., o i ,...,o N ) and routing decision a=(a1,a2,...,a i ,...,a N ), the output is the attention feature z of each agent i i , where o i is the observation of the ith agent, is the routing decision of the ith agent; the observation-action vector pair of each agent i is first passed through the embedding layer neural network f i (o i ,a i ) is transformed into dimension n h The encoding vector e i , the output of the k-th attention head of each agent i It can be expressed as the weighted sum of the value vectors of other agents, expressed as:

[0024]

[0025] In the formula, is the value vector of the kth attention head of the jth agent, e j is the encoding vector of the jth agent, is the value extraction matrix of the kth attention head, h is the nonlinear layer calculated element by element; the attention weight between any two agents i and j The dot product of the query vector of agent i and the key vector of agent k is obtained by softmax operation and is expressed as:

[0026]

[0027] In the formula, is the intermediate variable for calculating the attention weight, Represents query and key extraction matrices respectively, n q is the dimension of the agent query vector; after obtaining the output of each attention head k After that, we concatenate them in the feature dimension and transform them through projection to get the attention feature z of agent i. i ;

[0028] The Q value network is It uses a dual Q network structure to establish two review networks with the same structure but different initial parameters for each agent. It can avoid the overestimation problem in reinforcement learning. The input of the Q-value network is the observation value o of agent i. i , routing decision a i and the attention feature z i , output the future expected reward of agent i

[0029] The routing strategy network is used It is composed of three layers of fully connected networks with equal number of neurons and one layer of long short-term memory network, and its input is the observation value o of the intelligent agent. i , the output is the probability ρ of the agent taking various routing decisions i , for ρ i After Monte Carlo sampling and one-hot encoding, the agent’s routing decision vector a is obtained. i .

[0030] Further, in step S4, the Soft-Actor-Critic algorithm is used to alternately train and improve the parameters of the critic network and the routing strategy network. The parameters of the routing strategy network are θ = (θ1, θ2, ..., θ i ,...,θ N ) is to maximize the state value, where θ i Represents the parameters of the policy network of agent i, and the joint loss function L(θ) of all agents is expressed as follows:

[0031]

[0032] Where D is the buffer for storing all agent historical observation vectors and historical action vectors, o and a represent all agent joint observation and joint action samples extracted from the buffer, respectively. is the observation amount o of agent i i Routing Policy Network The action vector obtained by sampling, μ represents the policy entropy weight coefficient, and the parameters ω, The update goal is to minimize the temporal difference error, and the joint loss function of all agent-critic networks is It is expressed by the following formula:

[0033]

[0034] In the formula, q i represents the update target of the temporal difference, o' and r represent the joint observation samples and joint reward samples of all agents at the next moment extracted from the buffer, respectively, and z' i represents the attention feature of agent i at the next moment, r i is the reward obtained by agent i, γ is the discount coefficient of future rewards, a' i is the observation o' of agent i i Routing Policy Network The action vector obtained by sampling is used; the gradient descent method is used to alternately update the network parameters until convergence, and the optimal routing strategy network for all agents is obtained.

[0035] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0036] 1. The present invention adopts a centralized training and decentralized execution reinforcement learning framework to train the routing strategy network of each intelligent agent. After the training is completed, each routing node only needs local observation data to make routing decisions, which reduces the frequency of information exchange between different routing nodes and improves the data security of the power communication system.

[0037] 2. The present invention controls the time of routing decision to microseconds, and when the operating conditions of the communication system change, the adjustment of the routing strategy does not require additional time, thus meeting the requirements of the power communication system for routing speed.

[0038] 3. The routing strategy established by the present invention shows high robustness against network security incidents. When some information source nodes are attacked by DDoS and some communication links are disconnected, the transmission delay of other data packets can still be kept within the allowable range. Its robustness is significantly better than that of existing routing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1This is the topology diagram of the 3-area-42-node power communication system.

[0040] Figure 2 This is the Q value network structure diagram.

[0041] Figure 3 This is the network structure diagram of the routing strategy.

[0042] Figure 4 This is a diagram of the centralized training-distributed execution framework. DETAILED DESCRIPTION

[0043] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0044] This embodiment discloses a robust routing method for a power communication system based on multi-agent deep reinforcement learning, the details of which are as follows:

[0045] 1) Based on the measurement data collection mechanism of the information source node and the data packet transmission mechanism of the power communication system, an information flow model of the power communication system is established. The topology of the 3-region-42-node power communication system is as follows: Figure 1 As shown, nodes 1, 2, ..., 42 represent routing nodes, and C1, C2, and C3 represent the target nodes of the data packets generated by the information source nodes in each area. The data packet collection mechanism and the parameter settings of the power communication system are as follows:

[0046] a. Packet collection mechanism: All routing nodes will receive packets uploaded by information source nodes at a fixed frequency, assuming that all packets have the same size, and introduce the packet generation rate τ i Describes the number of data packets received by routing node i per unit time, τ i Obeys uniform distribution U(0.08,0.12).

[0047] b. Power communication system parameters: The time required for an information node to complete a routing decision is the simulation step length. The time required for a data packet to be transmitted from one end of the communication link to the other end is t ij It depends on the transmission distance and fiber parameters. In this paper, t ij ∈{3,4,5}, the unit is the simulation step. The maximum bandwidth of the communication link ranges from b ij ∈{6,7,8}, the buffer size range is c ij ∈{12,13,14}, the unit is the number of data packets. Since the information node is equipped with independent computing and storage devices, its buffer size can be regarded as infinite.

[0048] 2) Based on the information flow model, the routing nodes in the power communication system are regarded as intelligent agents, and the observation space, action space and reward function of the intelligent agents are designed according to the goal of routing optimization, and the routing optimization problem is transformed into a multi-agent deep reinforcement learning problem. The goal of the routing problem is to find a set of routing strategies that can achieve optimal information transmission performance under limited network capacity. The quality of communication service (QoS) is usually measured by four indicators: bandwidth, communication delay, jitter and packet loss. Bandwidth refers to the total amount of data packets transmitted from one end of the network to the other end per unit time, which mainly depends on the communication link parameters. Communication delay refers to the total delay for data packets to be transmitted from the source node to the target node, which can be expressed as follows:

[0049]

[0050] In the formula, Path(p) represents the set of all routing nodes and communication links in the transmission path of data p, T d (e ij ) represents the data packet p passing through the communication link e ij The transmission delay, T q (e ij ), T q (u i ) represent the data packet p in the communication link e ij and routing node v i Queuing delay in the buffer. Jitter represents the time difference between the maximum delay and the minimum delay of the same group of data packets. The larger the jitter, the more unstable the information transmission. Packet loss represents the number of data packets lost per unit time. Considering that the first three indicators are directly related to communication delay, the present invention selects the minimum average delay and packet loss rate of data packets per unit time as the optimization target of the routing problem, and establishes a routing strategy for each routing node so that the communication delay and packet loss rate of any data packet arriving at the node from the node to the target node are minimized.

[0051] Based on the extension of Markov decision process, multi-agent reinforcement learning problem can be expressed as Markov game process (MGP). Assume that the global state space of all agents is S, and the action space of all agents is A1, A2, ..., A N , there is a state transition function T:S×A1×A2×...×A N →P(S'), given the current global state and the joint actions of all agents, we get the probability distribution of the global state of all agents at the next moment. Each agent has a reward function R i :S×A1×A2×...×A N→R represents the external reward obtained from the environment given the global state and joint action of the agent. In most multi-agent reinforcement learning tasks, agents usually cannot obtain global state information. Let the local observation space of each agent i be O i , which contains part of the information in the global observation, namely O i ∈S. The goal of each agent i is to learn an optimal strategy π i * :O i →P(A i ), so that when making decisions according to the optimal strategy function, the expected cumulative reward is maximized Where γ is the conversion factor of the reward at the next moment relative to the reward at the current moment.

[0052] In the routing problem of the power communication system, the power communication system is regarded as an interactive environment, and the routing nodes are regarded as intelligent agents. Since the information source nodes do not have the routing decision function, they can be merged with the connected routing nodes. The observation space, action space and external reward of the intelligent agent are defined as follows.

[0053] a. Observation space: Based on security requirements, each agent can only obtain the status information of neighbor nodes and neighbor communication links. The neighbor nodes are agents directly connected to the agent through a communication link, and the corresponding communication link is the neighbor communication link. Define the observation space O of agent i i = {U i ,L i ,H i}, where U i represents the number of data packets in the buffer of all neighbor nodes of agent i at the current moment, L i represents the remaining bandwidth of all communication links connected to agent i at the current moment, H i Represents the local state information of agent i, including the target nodes of the next n data packets in its own buffer and the number of data packets forwarded to each neighboring node in the previous m routing decisions.

[0054] b. Action space: The action of agent i is the routing decision, and its action space A is defined i Number all neighbor nodes of agent i. Action a for agent i i One-hot encoding is used, where the index with element value 1 corresponds to the next-hop neighbor node number. If agent i does not make a routing decision, then a i is an all-zero vector.

[0055] c. External rewards: Based on the goal of the routing optimization problem, the reward mechanism of agent i should minimize the total delay and packet loss rate of the forwarded data packet p to the target node of the data packet. The following are the external rewards r for the three possible situations in the routing process: i ext Modeling.

[0056] c1. Data packet p is transmitted to the target node within the maximum allowed delay: the reward r obtained by each agent i that forwards data packet p i ext =t i (p)-t d (p), where t i (p) is the time when agent i forwards packet p, t d (p) is the time when the data packet arrives at the destination node.

[0057] c2. Packet p is lost because the communication delay exceeds the maximum allowed value: the reward r obtained by each agent i that forwards packet p i ext =β p (t i (p)-t d (p)), where β p >1 is the penalty coefficient.

[0058] c3. Packet p is lost because the buffer is full: The reward obtained by agent i who forwarded packet p for the last time where t max (p) is the maximum allowed delay for packet p.

[0059] In addition, the agent’s reward at the moment when no routing decision is taken is set to the average reward of the agent’s historical actions.

[0060] 3) For the multi-agent deep reinforcement learning problem in step 2), a routing strategy network and an improved critic network are designed for each agent; the routing strategy network is used to receive the observation vector obtained by the agent in the power communication system and output the corresponding routing decision vector, and the improved critic network is used to score the routing decision vector output by the routing strategy network; the routing strategy network is composed of three layers of fully connected neural networks and one layer of LSTM neural network connected in sequence; the improved critic network adds an attention mechanism on the basis of the traditional critic network, and is composed of a multi-head attention neural network and a Q-value network. The multi-head attention neural network takes the observation vectors and routing decision vectors of all agents in the power communication system as input, and uses the multi-head attention mechanism to calculate the attention weights between each pair of agents, and obtains the attention feature vector of each agent by weighted summation. The Q-value network adopts a dual Q network structure, each Q network is composed of two layers of fully connected neural networks, receives the attention feature vector and observation vector of the agent, and outputs the score of the agent's routing decision.

[0061] Multi-head attention neural network with A ω It means that its input is the observations of all agents o = (o1, o2, ..., o i ,...,o N ) and routing decision a=(a1,a2,...,a i ,...,a N ), the output is the attention feature z of each agent i i , where o i is the observation of the ith agent, a i is the routing decision for the i-th agent. Each observation-action vector pair of agent i first passes through the embedding layer neural network f i (o i ,a i ) is transformed into dimension n h The encoding vector e i , the output of the k-th attention head of each agent i It can be expressed as the weighted sum of the value vectors of other agents.

[0062]

[0063] In the formula, is the value vector of the kth attention head of the jth agent, e j is the encoding vector of the jth agent, is the value extraction matrix of the kth attention head, h is the nonlinear layer calculated element by element; the attention weight between any two agents i and j The dot product of the query vector of agent i and the key vector of agent k is obtained by softmax operation and is expressed as:

[0064]

[0065] In the formula, is the intermediate variable for calculating the attention weight, Represents query and key extraction matrices respectively, n q is the dimension of the agent query vector. After getting the output of each attention head k After that, we concatenate them in the feature dimension and transform them through projection to get the attention feature z of agent i. i :

[0066]

[0067] Where d is the number of attention heads, symbol Represents the concatenation of two vectors in their feature dimension W o is the projection matrix.

[0068] Q value network Indicates that Figure 2 As shown in Figure 1, it uses a dual Q network structure to establish two review networks with the same structure but different initial parameters for each agent. It can avoid the overestimation problem in reinforcement learning. The input of the Q-value network is the observation value o of agent i i , routing decision a i and the attention feature z i , output the future expected reward of agent i

[0069] Routing Policy Network It is represented by three layers of fully connected (FC) networks with equal number of neurons and one layer of long short-term memory (LSTM) network, such as Figure 3 As shown, its input is the observation o of the agent i , the output is the probability ρ of the agent taking various routing decisions i , for ρ i After Monte Carlo sampling and one-hot encoding, the agent’s routing decision vector a is obtained. i .

[0070] 4) The Soft-Actor-Critic algorithm is used to alternately update the parameters of the routing strategy network and the critic network of each agent, and finally the optimal routing strategy network of all agents is obtained to output the optimal routing decision. When the power communication system is attacked by a network or fails, different agents coordinate the optimal routing decision to maintain the transmission delay of normal measurement data packets within the allowable range.

[0071] Different from the traditional Actor-Critic algorithm which takes maximizing the expected cumulative reward as the update target, the SoftActor-Critic algorithm adds an entropy regularization term to the update target, which makes the agent's strategy more random while maximizing the expected cumulative reward. In order to reduce the interference of the non-static nature of the environment state transition in the multi-agent reinforcement learning problem on the training, the following is adopted: Figure 4 The centralized training-distributed execution framework shown in Figure 1 allows each agent to use global state information to train the routing strategy, but only uses local observation information to generate routing decisions. The Soft-Actor-Critic algorithm is used to alternately train and improve the parameters of the critic network and the routing strategy network. The parameters of the routing strategy network are θ = (θ1, θ2, ..., θ i ,...,θ N ) is to maximize the state value, θ i Representing the parameters of the policy network of agent i, the joint loss function L(θ) of all agents can be expressed as follows:

[0072]

[0073] Where D is the buffer for storing all agent historical observation vectors and historical action vectors, and o and a represent the joint observation and joint action samples of all agents extracted from the buffer, respectively. is the observation amount o of agent i i Routing Policy Network The action vector obtained by sampling, μ represents the policy entropy weight coefficient. Parameters of improved critic network The update goal is to minimize the temporal difference error. The joint loss function of all agent-critic networks can be expressed as follows:

[0074]

[0075] In the formula, q i represents the update target of the temporal difference, o' and r represent the joint observation samples and joint reward samples of all agents at the next moment extracted from the buffer, respectively, and z' i represents the attention feature of agent i at the next moment, r iis the reward obtained by agent i, γ is the discount coefficient of future rewards, a' i is the observation o' of agent i i Routing Policy Network The action vector obtained by sampling is used. The gradient descent method is used to alternately update the network parameters until convergence, and the optimal routing strategy network for all agents is obtained.

[0076] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A robust routing method for a power communication system based on multi-agent deep reinforcement learning, wherein the power communication system is composed of an information source node for collecting measurement data, a routing node for receiving and forwarding measurement data packets, and a communication link, and is characterized in that: The following steps are involved: S1: Based on the measurement data collection mechanism of the information source node and the data packet transmission mechanism of the power communication system, an information flow model of the power communication system is established; S2: Based on the information flow model, the routing nodes in the power communication system are regarded as intelligent agents, and the observation space, action space and reward function of the intelligent agents are designed according to the goal of routing optimization, so as to transform the routing optimization problem into a multi-agent deep reinforcement learning problem; S3: For the multi-agent deep reinforcement learning problem in step S2, a routing strategy network and an improved critic network are designed for each agent; the routing strategy network is used to receive the observation vector obtained by the agent in the power communication system and output the corresponding routing decision vector, and the improved critic network is used to score the routing decision vector output by the routing strategy network; the routing strategy network is composed of three layers of fully connected neural networks and one layer of LSTM neural network connected in sequence; the improved critic network adds an attention mechanism on the basis of the traditional critic network, and is composed of a multi-head attention neural network and a Q-value network. The multi-head attention neural network takes the observation vectors and routing decision vectors of all agents in the power communication system as input, and uses the multi-head attention mechanism to calculate the attention weights between each pair of agents. The attention feature vector of each agent is obtained by weighted summation. The Q-value network adopts a dual Q network structure, and each Q network is composed of two layers of fully connected neural networks, which receive the attention feature vector and observation vector of the agent and output the score of the agent's routing decision; S4: The Soft-Actor-Critic algorithm is used to alternately update the routing strategy network of each agent and improve the parameters of the critic network, and finally the optimal routing strategy network of all agents is obtained to output the optimal routing decision. When the power communication system is attacked by a network or fails, different agents coordinate the optimal routing decision to maintain the transmission delay of normal measurement data packets within the allowable range.

2. The robust routing method for power communication system based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: In step S1, the power communication system uses an undirected graph G =<V,E> Represented by, where V represents the set of all routing nodes, and E represents the set of all communication links; each routing node is connected to a fixed number of information source nodes, receives the measurement data packets generated by the information source nodes and stores them in the buffer, each data packet has a routing node as the target node, and is removed from the system when the data packet reaches the corresponding target node; the communication link adopts a full-duplex communication mechanism, and the data packets at both ends of the communication link can be transmitted to the opposite side at the same time and share the transmission bandwidth; the state equation of the power communication system is: B ij (t)=B ij (t-1)-b ij (t) Where N i Represents routing node v i The neighbor node set, B i (t) represents the routing node v at time t i The number of packets in the buffer, S ij (t), B ij (t) represents the communication link e at time t ij The number of packets being transmitted and the number of packets in the buffer; in the unit time from t-1 to t, u i (t) represents node v i The number of packets generated by the connected information source node, δ ji (t) represents the slave node v j Transfer to node v i The number of packets, ω lj (t),ω ki (t) represent the slave nodes v l 、v k Incoming communication link ij The number of packets, r ij (t) represents the communication link e ij Incoming node v i 、v j The total number of packets in the buffer, b ij (t) represents the communication link e ij The number of packets transferred from the buffer to the transmission queue; when the number of packets in the communication link and routing node buffer exceeds the maximum capacity of the buffer, the overflow packets will be removed from the system according to the "first in, first out" principle.

3. The robust routing method for power communication system based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: In step S2, the routing optimization problem is transformed into a multi-agent reinforcement learning problem, where the power communication system is regarded as an interactive environment, the routing nodes are regarded as agents, and the observation space, action space and external rewards of the agents are defined as follows: a. Observation space: Based on the security requirement, each agent can only obtain the status information of neighbor nodes and neighbor communication links. The neighbor nodes are agents directly connected to the agent through a communication link. The corresponding communication link is the neighbor communication link. Define the observation space O of agent i i = {U i ,L i ,H i }, where U i represents the number of data packets in the buffer of all neighbor nodes of agent i at the current moment, L i represents the remaining bandwidth of all communication links connected to agent i at the current moment, H i Represents the local state information of agent i, including the target nodes of the next n packets in its buffer and the number of packets forwarded to each neighboring node by the previous m routing decisions; b. Action space: The action of agent i is the routing decision, and its action space A is defined i Number all neighbor nodes of agent i; action a for agent i i One-hot encoding is used, where the index with element value 1 corresponds to the next-hop neighbor node number. If agent i does not make a routing decision, then a i is an all-zero vector; c. External rewards: Based on the goal of the routing optimization problem, the reward mechanism of agent i should minimize the total delay and packet loss rate of the forwarded data packet p to the target node of the data packet. The following are the external rewards r for the three possible situations in the routing process: i ext Modeling is performed as follows: c1. Data packet p is transmitted to the target node within the maximum allowed delay: the reward r obtained by each agent i that forwards data packet p i ext =t i (p)-t d (p), where t i (p) is the time when agent i forwards packet p, t d (p) is the time when the data packet arrives at the destination node; c2. Packet p is lost because the communication delay exceeds the maximum allowed value: the reward r obtained by each agent i that forwards packet p i ext =β p (t i (p)-t d (p)), where β p >1 is the penalty coefficient; c3. Packet p is lost because the buffer is full: The reward obtained by agent i who forwarded packet p for the last time where t max (p) is the maximum allowed delay for data packet p; In addition, the agent’s reward at the moment when no routing decision is taken is set to the average reward of the agent’s historical actions.

4. The robust routing method for power communication system based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: In step S3, the multi-head attention neural network uses A ω It means that its input is the observations of all agents o = (o1, o2, ..., o i ,...,o N ) and routing decision a=(a1,a2,...,a i ,...,a N ), the output is the attention feature z of each agent i i , where o i is the observation of the ith agent, is the routing decision of the ith agent; the observation-action vector pair of each agent i is first passed through the embedding layer neural network f i (o i ,a i ) is transformed into dimension n h The encoding vector e i , the output of the k-th attention head of each agent i It can be expressed as the weighted sum of the value vectors of other agents, expressed as: In the formula, is the value vector of the kth attention head of the jth agent, e j is the encoding vector of the jth agent, is the value extraction matrix of the kth attention head, h is the nonlinear layer calculated element by element; the attention weight between any two agents i and j The dot product of the query vector of agent i and the key vector of agent k is obtained by softmax operation and is expressed as: In the formula, is the intermediate variable for calculating the attention weight, W q (k) , Represents query and key extraction matrices respectively, n q is the dimension of the agent query vector; after obtaining the output of each attention head k After that, we concatenate them in the feature dimension and transform them through projection to get the attention feature z of agent i. i ; The Q value network is It uses a dual Q network structure to establish two review networks with the same structure but different initial parameters for each agent. It can avoid the overestimation problem in reinforcement learning. The input of the Q-value network is the observation value o of agent i. i , routing decision a i and the attention feature z i , output the future expected reward of agent i The routing strategy network is used It is composed of three layers of fully connected networks with equal number of neurons and one layer of long short-term memory network, and its input is the observation value o of the intelligent agent. i , the output is the probability ρ of the agent taking various routing decisions i , for ρ i After Monte Carlo sampling and one-hot encoding, the agent’s routing decision vector a is obtained. i .

5. The robust routing method for power communication system based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: In step S4, the Soft-Actor-Critic algorithm is used to alternately train and improve the parameters of the critic network and the routing strategy network. The parameters of the routing strategy network are θ = (θ1, θ2, ..., θ i ,...,θ N ) is to maximize the state value, where θ i Represents the parameters of the policy network of agent i, and the joint loss function L(θ) of all agents is expressed as follows: Where D is the buffer for storing all agent historical observation vectors and historical action vectors, o and a represent all agent joint observation and joint action samples extracted from the buffer, respectively. is the observation amount o of agent i i Routing Policy Network The action vector obtained by sampling, μ represents the policy entropy weight coefficient, and the parameters ω, The update goal is to minimize the temporal difference error, and the joint loss function of all agent-critic networks is It is expressed by the following formula: In the formula, q i represents the update target of the temporal difference, o' and r represent the joint observation samples and joint reward samples of all agents at the next moment extracted from the buffer, respectively, and z' i represents the attention feature of agent i at the next moment, r i is the reward obtained by agent i, γ is the discount coefficient of future rewards, a' i is the observation o' of agent i i Routing Policy Network The action vector obtained by sampling is used; the gradient descent method is used to alternately update the network parameters until convergence, and the optimal routing strategy network for all agents is obtained.

Citation Information

Patent Citations

  • Method for controlling routing actions based on multi-agent reinforcement learning routing strategy

    CN111416771A

  • Traffic light control method and system based on multi-agent reinforcement learning in control area

    CN115631638A

  • Multi-agent reinforcement learning algorithm based on multi-head attention mechanism communication

    CN116341611A

  • Method and apparatus for packet forwarding control with reinforcement learning

    WO2023272722A1

Cited By

  • Power agent cooperative acceleration method, device and equipment for model routing and medium

    CN120416061A