Robust routing method for power communication system based on multi-agent deep reinforcement learning

Abstract

The application discloses a kind of robust routing methods of power communication system based on multi-agent deep reinforcement learning, comprising: the information flow model of power communication system is established, routing node is regarded as agent, according to the target of routing optimization, the observation space, action space and reward function of agent are designed, the routing optimization problem is converted into multi-agent deep reinforcement learning problem, for each agent, routing strategy network and improved critic network are designed, the parameters of routing strategy network and improved critic network of each agent are alternately updated using Soft-Actor-Critic algorithm, the optimal routing strategy network of all agents is obtained, for outputting optimal routing decision, when power communication system is subjected to network attack or fault, the transmission delay of normal measurement data packet is maintained within the allowable range through the cooperation of optimal routing decision between different agents.The application can be applied to millisecond-level control power communication scene, so that the communication delay and packet loss rate of power communication system are minimized.

Description

Technical Field

[0001] The present invention relates to the technical field of power communication systems and routing optimization, and in particular to a robust routing method for power communication systems based on multi-agent deep reinforcement learning. Background Art

[0002] Modern power systems exhibit the "three highs" of high renewable energy access, high proportion of power electronic devices, and high proportion of distributed energy resources. Their operation and dynamic mechanisms have become increasingly complex. The complexity of modern power systems places higher demands on information control. With the high proportion of renewable energy access, energy injection into the power system becomes dispersed, and generator power fluctuations increase. This means that during power system operation, faster dispatch control is required to maintain system balance and stability.

[0003] Fast and accurate information control of the power system requires the support of a secure and reliable power communication system, and packet routing is a key link in ensuring communication reliability. The routing optimization problem refers to finding the optimal routing strategy for each routing node in the communication network, minimizing the average transmission delay and packet loss rate of all data packets transmitted from the source node to the destination node. The routing strategy of the power communication system must meet the requirements of security, speed, and robustness. Security refers to minimizing the number of information exchanges between routing nodes and reducing the exposure to network attacks. Speed ​​refers to the ability of the routing strategy to quickly adjust. Robustness refers to the ability of the routing strategy to resist interference to a certain extent. When some information source nodes in the power communication system are attacked by DDoS or some communication links are disconnected, the routing strategy can maintain the transmission delay of normal data packets within the allowable range and prevent packet loss.

[0004] Currently, commonly used routing strategies can be categorized as fixed shortest path routing and adaptive routing. The fixed shortest path routing strategy utilizes a graph optimization algorithm to calculate the shortest path from each information source node to the destination node and establishes a fixed routing decision table for each routing node. However, this routing strategy cannot dynamically adjust to the state of the communication system and does not meet the robustness requirements of smart grids. Link state routing is a widely used adaptive routing strategy in computer communication networks. However, this routing strategy requires a large amount of forwarding state query information, which increases exposure to network attacks. Furthermore, when the communication system state suddenly changes, readjusting to the optimal routing strategy requires a long transition time, failing to meet the security and speed requirements of power communication systems. Summary of the Invention

[0005] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a robust routing method for power communication systems based on multi-agent deep reinforcement learning. The method is based on the idea of ​​distributed decision-making and establishes a mutually independent routing strategy network for each routing node. Each routing node only needs local observation data to make routing decisions, avoiding the safety hazards caused by frequent information interaction between different nodes. In addition, the state change of the power communication system is only reflected as a change in the node observation quantity. Each routing node does not need extra time to adjust the routing strategy, which solves the problem that traditional routing methods cannot take into account safety, speed and robustness at the same time.

[0006] To achieve the above objectives, the present invention provides a technical solution: a robust routing method for a power communication system based on multi-agent deep reinforcement learning. The power communication system consists of an information source node for collecting measurement data, a routing node for receiving and forwarding measurement data packets, and a communication link, and includes the following steps:

[0007] S1: Based on the measurement data acquisition mechanism of the information source node and the data packet transmission mechanism of the power communication system, an information flow model of the power communication system is established;

[0008] S2: Based on the information flow model, the routing nodes in the power communication system are regarded as intelligent agents. The observation space, action space and reward function of the intelligent agents are designed according to the goal of routing optimization, and the routing optimization problem is transformed into a multi-agent deep reinforcement learning problem.

[0009] S3: For the multi-agent deep reinforcement learning problem in step S2, a routing strategy network and an improved critic network are designed for each agent; the routing strategy network is used to receive the observation vector obtained by the agent in the power communication system and output the corresponding routing decision vector, and the improved critic network is used to score the routing decision vector output by the routing strategy network; the routing strategy network is composed of three layers of fully connected neural networks and one layer of LSTM neural network connected in sequence; the improved critic network adds an attention mechanism on the basis of the traditional critic network, and is composed of a multi-head attention neural network and a Q-value network. The multi-head attention neural network takes the observation vectors and routing decision vectors of all agents in the power communication system as input, and uses the multi-head attention mechanism to calculate the attention weights between each pair of agents, and obtains the attention feature vector of each agent by weighted summation. The Q-value network adopts a dual-Q network structure, and each Q network is composed of two layers of fully connected neural networks, which receives the attention feature vector and observation vector of the agent and outputs the score of the agent's routing decision;

[0010] S4: The Soft-Actor-Critic algorithm is used to alternately update the routing strategy network of each agent and improve the parameters of the critic network. Ultimately, the optimal routing strategy network of all agents is obtained, which is used to output the optimal routing decision. When the power communication system suffers a network attack or a failure, the different agents coordinate the optimal routing decision to maintain the transmission delay of normal measurement data packets within the allowable range.

[0011] Furthermore, in step S1, the power communication system uses an undirected graph G=<V,E> Represented as follows, where V represents the set of all routing nodes and E represents the set of all communication links. Each routing node is connected to a fixed number of information source nodes, receives measurement data packets generated by the information source nodes and stores them in a buffer. Each data packet has a routing node as the target node. When the data packet reaches the corresponding target node, it is removed from the system. The communication link adopts a full-duplex communication mechanism. Data packets at both ends of the communication link can be transmitted to the opposite side at the same time and share the transmission bandwidth. The state equation of the power communication system is:

[0012]

[0013] B ij (t) = B ij (t-1)-b ij (t)

[0014] Where N i Represents routing node v i The neighbor node set, B i (t) represents the routing node v at time t i The number of packets in the buffer, S ij (t), B ij (t) represents the communication link e at time t ij The number of packets being transmitted and the number of packets in the buffer; in the unit time t-1→t, u i (t) represents node v i The number of data packets generated by the connected information source node, δ ji (t) represents the slave node v j The number of packets transmitted to node vi, ω lj (t),ω ki (t) represents the slave node v l 、v k Incoming communication link ij The number of packets, r ij (t) represents the communication link e ij Incoming node v i 、v j The total number of packets in the buffer, bij (t) represents the communication link e ij When the number of packets in the communication link and routing node buffer exceeds the maximum capacity of the buffer, the overflow packets will be removed from the system according to the "first in, first out" principle.

[0015] Furthermore, in step S2, the routing optimization problem is transformed into a multi-agent reinforcement learning problem, where the power communication system is regarded as an interactive environment, the routing nodes are regarded as agents, and the observation space, action space and external reward of the agents are defined as follows:

[0016] a. Observation space: Based on security requirements, each agent can only obtain the status information of neighbor nodes and neighbor communication links. The neighbor nodes are agents directly connected to the agent through a communication link. The corresponding communication link is the neighbor communication link. Define the observation space O of agent i i ={U i ,L i ,H i}, where U i represents the number of data packets in the buffer of all neighbor nodes of agent i at the current moment, L i represents the remaining bandwidth of all communication links connected to agent i at the current moment, H i Represents the local state information of agent i, including the destination nodes of the next n packets in its own buffer and the number of packets forwarded to each neighboring node in the previous m routing decisions;

[0017] b. Action space: The action of agent i is the routing decision, and its action space A is defined. i Number all neighbor nodes of agent i; action a for agent i i One-hot encoding is used, where the index with element value 1 corresponds to the next-hop neighbor node number. If agent i does not make a routing decision, then a i is an all-zero vector;

[0018] c. External rewards: Based on the goal of the routing optimization problem, the reward mechanism of agent i should minimize the total delay and packet loss rate of the forwarded data packet p to the target node of the data packet. The following are the external rewards r for three possible situations in the routing process: i ext Modeling is carried out as follows:

[0019] c1. Data packet p is transmitted to the target node within the maximum allowed delay: the reward r obtained by each agent i that forwards data packet p i ext =t i (p)-t d(p), where t i (p) is the time when agent i forwards packet p, t d (p) is the time when the data packet arrives at the destination node;

[0020] c2. Packet p is lost because the communication delay exceeds the maximum allowed value: the reward r obtained by each agent i that forwards packet p i ext =β p (t i (p)-t d (p)), where β p >1 is the penalty coefficient;

[0021] c3. Packet p is lost because the buffer is full: the reward obtained by agent i who forwarded packet p for the last time where t max (p) is the maximum allowed delay for data packet p;

[0022] In addition, the agent’s reward at the moment when no routing decision is taken is set to the average reward of the agent’s historical actions.

[0023] Further, in step S3, the multi-head attention neural network is used ω Indicates that its input is the observation of all agents o=(o1,o2,...,o i ,...,o N ) and routing decision a=(a1,a2,...,a i ,...,a N ), the output is the attention feature z of each agent i i , where o i is the observation of the ith agent, is the routing decision of the ith agent; the observation-action vector pair of each agent i is first passed through the embedding layer neural network f i (o i ,a i ) is transformed into dimension n h The encoding vector e i , the output of the k-th attention head of each agent i It can be expressed as the weighted sum of the value vectors of other agents, expressed as:

[0024]

[0025] Where, is the value vector of the kth attention head of the jth agent, e j is the encoding vector of the j-th agent, is the value extraction matrix of the kth attention head, h is the nonlinear layer calculated element by element; the attention weight between any two agents i and j The dot product of the query vector of agent i and the key vector of agent k is obtained through softmax operation and is expressed as:

[0026]

[0027] Where, is the intermediate variable for calculating the attention weight, Represents query and key extraction matrices respectively, n q is the dimension of the agent query vector; after obtaining the output of each attention head k After that, the attention feature z of agent i is obtained by concatenating them in the feature dimension and performing projection transformation. i ;

[0028] The Q value network is used It uses a dual Q network structure to establish two review networks with the same structure but different initial parameters for each agent. It can avoid the overestimation problem in reinforcement learning. The input of the Q value network is the observation value o of the agent i. i , routing decision a i and attention feature z i , output the future expected reward of agent i

[0029] The routing policy network is used It is composed of three layers of fully connected networks with equal number of neurons and one layer of long short-term memory network, and its input is the observation value o of the intelligent agent. i , the output is the probability ρ of the agent taking various routing decisions i , for ρ i After Monte Carlo sampling and one-hot encoding, the agent’s routing decision vector a is obtained. i .

[0030] Furthermore, in step S4, the parameters of the critic network and the routing strategy network are alternately trained using the Soft-Actor-Critic algorithm. The parameters of the routing strategy network are θ=(θ1,θ2,...,θ i ,...,θ N ) is to maximize the state value, where θ i Representing the parameters of the policy network of agent i, the joint loss function L(θ) of all agents is expressed as follows:

[0031]

[0032] Where D is the buffer that stores all agent historical observation vectors and historical action vectors, o and a represent all agent joint observation and joint action samples extracted from the buffer, respectively. is the observation quantity o of agent i i Routing Policy Network The action vector obtained by sampling, μ represents the policy entropy weight coefficient, and the parameters ω, The update goal is to minimize the temporal difference error, and the joint loss function of all agent-critic networks is It is expressed by the following formula:

[0033]

[0034] Where q i represents the update target of the temporal difference, o' and r represent the joint observation sample and joint reward sample of all agents at the next moment extracted from the buffer, respectively, z' i represents the attention feature of agent i at the next moment, r i is the reward obtained by agent i, γ is the discount coefficient of future rewards, a' i is the observation quantity o' of agent i i Routing Policy Network The action vectors obtained by sampling are used; the network parameters are updated alternately using the gradient descent method until convergence, and the optimal routing strategy network for all intelligent agents is obtained.

[0035] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0036] 1. The present invention adopts a reinforcement learning framework with centralized training and decentralized execution to train the routing strategy network of each intelligent agent. After training, each routing node only needs local observation data to make routing decisions, which reduces the frequency of information exchange between different routing nodes and improves the data security of the power communication system.

[0037] 2. The present invention controls the routing decision time to microseconds, and when the operating conditions of the communication system change, the adjustment of the routing strategy does not require additional time, meeting the requirements of the power communication system for routing speed.

[0038] 3. The routing strategy established by the present invention shows high robustness against network security incidents. When some information source nodes are attacked by DDoS and some communication links are disconnected, the transmission delay of other data packets can still be kept within the allowable range. Its robustness is significantly better than that of existing routing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1This is the topology diagram of the 3-area-42-node power communication system.

[0040] Figure 2 This is the Q-value network structure diagram.

[0041] Figure 3 This is the network structure diagram of the routing strategy.

[0042] Figure 4 This is a diagram of the centralized training-distributed execution framework. DETAILED DESCRIPTION

[0043] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0044] This embodiment discloses a robust routing method for a power communication system based on multi-agent deep reinforcement learning, the details of which are as follows:

[0045] 1) Based on the measurement data collection mechanism of the information source node and the data packet transmission mechanism of the power communication system, the information flow model of the power communication system is established. The topology of the 3-region-42-node power communication system is as follows Figure 1 As shown in the figure, nodes 1, 2, ..., 42 represent routing nodes, and C1, C2, and C3 represent the target nodes of the data packets generated by the information source nodes in each area. The data packet collection mechanism and the parameter settings of the power communication system are as follows:

[0046] a. Data packet collection mechanism: All routing nodes will receive data packets uploaded by the information source node at a fixed frequency, assuming that all data packets have the same size, and introduce the data packet generation rate τ i Describes the number of data packets received by routing node i per unit time, τ i Obeys uniform distribution U(0.08,0.12).

[0047] b. Power communication system parameters: The time required for an information node to complete a routing decision is the simulation step length. The time required for a data packet to be transmitted from one end of the communication link to the other end is t ij It depends on the transmission distance and fiber parameters. In this paper, t ij ∈{3,4,5}, the unit is the simulation step. The maximum bandwidth of the communication link ranges from b ij ∈{6,7,8}, the buffer size range is c ij ∈{12,13,14}, where the unit is the number of data packets. Since information nodes are equipped with independent computing and storage devices, their buffer sizes can be considered infinite.

[0048] 2) Based on the information flow model, the routing nodes in the power communication system are regarded as intelligent agents, and the observation space, action space and reward function of the intelligent agents are designed according to the goal of routing optimization, and the routing optimization problem is transformed into a multi-agent deep reinforcement learning problem. The goal of the routing problem is to find a set of routing strategies that can achieve optimal information transmission performance under limited network capacity. The quality of communication service (QoS) is usually measured by four indicators: bandwidth, communication delay, jitter and packet loss. Bandwidth refers to the total amount of data packets transmitted from one end of the network to the other end per unit time, which mainly depends on the communication link parameters. Communication delay refers to the total delay of the data packet from the source node to the destination node, which can be expressed as the following formula:

[0049]

[0050] Where Path(p) represents the set of all routing nodes and communication links in the transmission path of data p, T d (e ij ) represents the data packet p passing through the communication link e ij The transmission delay, T q (e ij ), T q (u i ) represent the data packet p in the communication link e ij and routing node v i Queuing delay in the buffer. Jitter represents the time difference between the maximum and minimum delays for the same set of data packets. Greater jitter indicates more unstable information transmission. Packet loss represents the number of data packets lost per unit time. Considering that the first three indicators are directly related to communication delay, the present invention selects the minimum average delay and packet loss rate of data packets per unit time as the optimization objectives of the routing problem. A routing strategy is established for each routing node to minimize the communication delay and packet loss rate of any data packet arriving at that node when it is transmitted from that node to the target node.

[0051] Based on the extension of Markov decision process, multi-agent reinforcement learning problem can be expressed as Markov game process (MGP). Assume that the global state space of all agents is S and the action space of all agents is A1, A2, ..., A N , there exists a state transition function T:S×A1×A2×...×A N →P(S'), given the current global state and the joint actions of all agents, we get the probability distribution of the global state of all agents at the next moment. Each agent has a reward function R i :S×A1×A2×...×A N→R represents the external reward obtained from the environment when the global state and joint action of the given agent are given. In most multi-agent reinforcement learning tasks, agents usually cannot obtain global state information. Let the local observation space of each agent i be O i , which contains part of the information in the global observation quantity, namely O i ∈S. The goal of each agent i is to learn an optimal strategy π i * :O i →P(A i ), so that when making decisions according to the optimal policy function, the expected cumulative reward is maximized Where γ is the conversion factor of the reward at the next moment relative to the reward at the current moment.

[0052] In the routing problem of power communication systems, the power communication system is considered an interactive environment, and routing nodes are considered intelligent agents. Since information source nodes do not have routing decision-making capabilities, they can be merged with connected routing nodes. The observation space, action space, and external reward of the agent are defined as follows.

[0053] a. Observation space: Based on security requirements, each agent can only obtain the status information of neighbor nodes and neighbor communication links. The neighbor nodes are agents directly connected to the agent through a communication link, and the corresponding communication link is the neighbor communication link. Define the observation space O of agent i i ={U i ,L i ,H i}, where U i represents the number of data packets in the buffer of all neighbor nodes of agent i at the current moment, L i represents the remaining bandwidth of all communication links connected to agent i at the current moment, H i Represents the local state information of agent i, including the target nodes of the next n data packets in its own buffer and the number of data packets forwarded to each neighboring node in the previous m routing decisions.

[0054] b. Action space: The action of agent i is the routing decision, and its action space A is defined. i Number all neighbor nodes of agent i. Action a for agent i i One-hot encoding is used, where the index with element value 1 corresponds to the next-hop neighbor node number. If agent i does not make a routing decision, then a i is an all-zero vector.

[0055] c. External rewards: Based on the goal of the routing optimization problem, the reward mechanism of agent i should minimize the total delay and packet loss rate of the forwarded data packet p to the target node of the data packet. The following are the external rewards r for three possible situations in the routing process: i ext Perform modeling.

[0056] c1. Data packet p is transmitted to the target node within the maximum allowed delay: the reward r obtained by each agent i that forwards data packet p i ext =t i (p)-t d (p), where t i (p) is the time when agent i forwards packet p, t d (p) is the time when the data packet arrives at the destination node.

[0057] c2. Packet p is lost because the communication delay exceeds the maximum allowed value: the reward r obtained by each agent i that forwards packet p i ext =β p (t i (p)-t d (p)), where β p >1 is the penalty coefficient.

[0058] c3. Packet p is lost because the buffer is full: the reward obtained by agent i who forwarded packet p for the last time where t max (p) is the maximum allowed delay of data packet p.

[0059] In addition, the agent’s reward at the moment when no routing decision is taken is set to the average reward of the agent’s historical actions.

[0060] 3) For the multi-agent deep reinforcement learning problem in step 2), a routing strategy network and an improved critic network are designed for each agent; the routing strategy network is used to receive the observation vector obtained by the agent in the power communication system and output the corresponding routing decision vector, and the improved critic network is used to score the routing decision vector output by the routing strategy network; the routing strategy network is composed of three layers of fully connected neural networks and one layer of LSTM neural network connected in sequence; the improved critic network adds an attention mechanism on the basis of the traditional critic network, and is composed of a multi-head attention neural network and a Q-value network. The multi-head attention neural network takes the observation vectors and routing decision vectors of all agents in the power communication system as input, and uses the multi-head attention mechanism to calculate the attention weight between each pair of agents, and obtains the attention feature vector of each agent by weighted summation. The Q-value network adopts a dual Q network structure, each Q network is composed of two layers of fully connected neural networks, receives the attention feature vector and observation vector of the agent, and outputs the score of the agent's routing decision.

[0061] Multi-head attention neural network with A ω Indicates that its input is the observation of all agents o=(o1,o2,...,o i ,...,o N ) and routing decision a=(a1,a2,...,a i ,...,a N ), the output is the attention feature z of each agent i i , where o i is the observation quantity of the i-th agent, a i The observation-action vector pair of each agent i is first passed through the embedding layer neural network f i (o i ,a i ) is transformed into dimension n h The encoding vector e i , the output of the k-th attention head of each agent i It can be expressed as the weighted sum of the value vectors of other agents,

[0062]

[0063] Where, is the value vector of the kth attention head of the jth agent, e j is the encoding vector of the j-th agent, is the value extraction matrix of the kth attention head, h is the nonlinear layer calculated element by element; the attention weight between any two agents i and j The dot product of the query vector of agent i and the key vector of agent k is obtained through softmax operation and is expressed as:

[0064]

[0065] Where, is the intermediate variable for calculating the attention weight, Represents query and key extraction matrices respectively, n q is the dimension of the agent query vector. After that, the attention feature z of agent i is obtained by concatenating them in the feature dimension and performing projection transformation. i :

[0066]

[0067] Where d is the number of attention heads, symbol Represents the concatenation of two vectors in their feature dimension W o is the projection matrix.

[0068] Q value network Indicates that Figure 2 As shown, it adopts a dual Q network structure and establishes two review networks with the same structure but different initial parameters for each agent. It can avoid the overestimation problem in reinforcement learning. The input of the Q value network is the observation value o of agent i i , routing decision a i and attention feature z i , output the future expected reward of agent i

[0069] Routing policy network It is represented by three layers of fully connected (FC) networks with equal number of neurons and one layer of long short-term memory (LSTM) network, as shown in Figure 3 As shown, its input is the observation amount o of the intelligent agent i , the output is the probability ρ of the agent taking various routing decisions i , for ρ i After Monte Carlo sampling and one-hot encoding, the agent’s routing decision vector a is obtained. i .

[0070] 4) The Soft-Actor-Critic algorithm is used to alternately update the parameters of each agent's routing strategy network and critic network, ultimately obtaining the optimal routing strategy network for all agents. This network is used to output the optimal routing decision. When the power communication system suffers a network attack or a failure, the different agents collaborate on the optimal routing decision to maintain the transmission delay of normal measurement data packets within an allowable range.

[0071] Different from the traditional Actor-Critic algorithm which takes maximizing the expected cumulative reward as the update target, the SoftActor-Critic algorithm adds an entropy regularization term to the update target, which makes the agent's strategy more random while maximizing the expected cumulative reward. In order to reduce the interference caused by the non-static nature of the environment state transition in the multi-agent reinforcement learning problem on the training, the following is adopted: Figure 4 The centralized training-distributed execution framework shown in the figure allows each agent to use global state information to train the routing strategy, but only uses local observation information to generate routing decisions. The Soft-Actor-Critic algorithm is used to alternately train and improve the parameters of the critic network and the routing strategy network. The parameters of the routing strategy network are θ = (θ1, θ2, ..., θ i ,...,θ N ) is to maximize the state value, θ i Representing the parameters of the policy network of agent i, the joint loss function L(θ) of all agents can be expressed as follows:

[0072]

[0073] Where D is the buffer that stores all agent historical observation vectors and historical action vectors, and o and a represent all agent joint observation and joint action samples extracted from the buffer, respectively. is the observation quantity o of agent i i Routing Policy Network The action vector obtained by sampling, μ represents the policy entropy weight coefficient. Parameters of improved critic network The update goal is to minimize the temporal difference error. The joint loss function of all agent-critic networks can be expressed as follows:

[0074]

[0075] Where q i represents the update target of the temporal difference, o' and r represent the joint observation sample and joint reward sample of all agents at the next moment extracted from the buffer, respectively, z' i represents the attention feature of agent i at the next moment, r iis the reward obtained by agent i, γ is the discount coefficient of future rewards, a' i is the observation quantity o' of agent i i Routing Policy Network The action vectors obtained by sampling are used. The network parameters are updated alternately using the gradient descent method until convergence, and the optimal routing strategy network for all agents is obtained.

[0076] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A robust routing method for a power communication system based on multi-agent deep reinforcement learning. The power communication system consists of information source nodes for collecting measurement data, routing nodes for receiving and forwarding measurement data packets, and communication links. The method is characterized by: The following steps are involved: S1: Based on the measurement data acquisition mechanism of the information source node and the data packet transmission mechanism of the power communication system, an information flow model of the power communication system is established; S2: Based on the information flow model, the routing nodes in the power communication system are regarded as intelligent agents. The observation space, action space and reward function of the intelligent agents are designed according to the goal of routing optimization, and the routing optimization problem is transformed into a multi-agent deep reinforcement learning problem. S3: For the multi-agent deep reinforcement learning problem in step S2, a routing strategy network and an improved critic network are designed for each agent; the routing strategy network is used to receive the observation vector obtained by the agent in the power communication system and output the corresponding routing decision vector, and the improved critic network is used to score the routing decision vector output by the routing strategy network; the routing strategy network is composed of three layers of fully connected neural networks and one layer of LSTM neural network connected in sequence; the improved critic network adds an attention mechanism on the basis of the traditional critic network, and is composed of a multi-head attention neural network and a Q-value network. The multi-head attention neural network takes the observation vectors and routing decision vectors of all agents in the power communication system as input, and uses the multi-head attention mechanism to calculate the attention weights between each pair of agents, and obtains the attention feature vector of each agent by weighted summation. The Q-value network adopts a dual-Q network structure, and each Q network is composed of two layers of fully connected neural networks, which receives the attention feature vector and observation vector of the agent and outputs the score of the agent's routing decision; S4: The Soft-Actor-Critic algorithm is used to alternately update the routing strategy network of each agent and improve the parameters of the critic network. Ultimately, the optimal routing strategy network of all agents is obtained, which is used to output the optimal routing decision. When the power communication system suffers a network attack or a failure, the different agents coordinate the optimal routing decision to maintain the transmission delay of normal measurement data packets within the allowable range.

2. The robust routing method for power communication system based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: In step S1, the power communication system uses an undirected graph G=<V,E> Represented as follows, where V represents the set of all routing nodes and E represents the set of all communication links. Each routing node is connected to a fixed number of information source nodes, receives measurement data packets generated by the information source nodes and stores them in a buffer. Each data packet has a routing node as the target node. When the data packet reaches the corresponding target node, it is removed from the system. The communication link adopts a full-duplex communication mechanism. Data packets at both ends of the communication link can be transmitted to the opposite side at the same time and share the transmission bandwidth. The state equation of the power communication system is: B ij (t)=B ij (t-1)-b ij (t) Where N i Represents routing node v i The neighbor node set, B i (t) represents the routing node v at time t i The number of packets in the buffer, S ij (t), B ij (t) represents the communication link e at time t ij The number of packets being transmitted and the number of packets in the buffer; in the unit time t-1→t, u i (t) represents node v i The number of data packets generated by the connected information source node, δ ji (t) represents the slave node v j Transmit to node v i The number of packets, ω lj (t),ω ki (t) represents the slave node v l 、v k Incoming communication link ij The number of packets, r ij (t) represents the communication link e ij Incoming node v i 、v j The total number of packets in the buffer, b ij (t) represents the communication link e ij The number of data packets transferred from the buffer to the transmission queue; when the number of data packets in the communication link and routing node buffer exceeds the maximum capacity of the buffer, the overflow data packets will be removed from the system according to the "first in first out" principle.

3. The robust routing method for power communication system based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: In step S2, the routing optimization problem is transformed into a multi-agent reinforcement learning problem, where the power communication system is regarded as an interactive environment, the routing nodes are regarded as agents, and the observation space, action space, and external rewards of the agents are defined as follows: a. Observation space: Based on security requirements, each agent can only obtain the status information of neighbor nodes and neighbor communication links. The neighbor nodes are agents directly connected to the agent through a communication link. The corresponding communication link is the neighbor communication link. Define the observation space O of agent i i ={U i ,L i ,H i }, where U i represents the number of data packets in the buffer of all neighbor nodes of agent i at the current moment, L i represents the remaining bandwidth of all communication links connected to agent i at the current moment, H i Represents the local state information of agent i, including the destination nodes of the next n packets in its own buffer and the number of packets forwarded to each neighboring node in the previous m routing decisions; b. Action space: The action of agent i is the routing decision, and its action space A is defined. i Number all neighbor nodes of agent i; make routing decision a for agent i i One-hot encoding is used, where the index with element value 1 corresponds to the next-hop neighbor node number. If agent i does not make a routing decision, then a i is an all-zero vector; c. External rewards: Based on the goal of the routing optimization problem, the reward mechanism of agent i should minimize the total delay and packet loss rate of the forwarded data packet p to the target node of the data packet. The following are the external rewards r for the three situations that occur in the routing process: i ext Modeling is carried out as follows: c1. Data packet p is transmitted to the target node within the maximum allowed delay: Each agent i that forwards data packet p receives an external reward r. i ext =t i (p)-t d (p), where t i (p) is the time when agent i forwards packet p, t d (p) is the time when the data packet arrives at the destination node; c2. Packet p is lost because the communication delay exceeds the maximum allowed value: Each agent i that forwards packet p receives an external reward r i ext =β p (t i (p)-t d (p)), where β p >1 is the penalty coefficient; c3. Packet p is lost because the buffer is full: the external reward r obtained by agent i who forwarded packet p for the last time i ext =-t max (p), where t max (p) is the maximum allowed delay for data packet p; The agent’s reward at the moment when no routing decision is taken is set to the average reward of the agent’s historical actions.

4. The robust routing method for power communication system based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: In step S3, the multi-head attention neural network is used ω Indicates that its input is the observation of all agents o=(o1,o2,...,o i ,...,o N ) and routing decision a=(a1,a2,...,a i ,...,a N ), the output is the attention feature z of each agent i i , where o i is the observation quantity of the i-th agent, a i The routing decision of the i-th agent is: the observation-action vector pair of each agent i is first passed through the embedding layer neural network f i (o i ,a i ) is transformed into dimension n h The encoding vector e i , the output of the k-th attention head of each agent i It can be expressed as the weighted sum of the value vectors of other agents, expressed as: Where, is the value vector of the kth attention head of the jth agent, e j is the encoding vector of the j-th agent, is the value extraction matrix of the kth attention head, h is the nonlinear layer calculated element by element; the attention weight between any two agents i and j The dot product of the query vector of agent i and the key vector of agent k is obtained through softmax operation and is expressed as: Where, is the intermediate variable for calculating the attention weight, Represents query and key extraction matrices respectively, n q is the dimension of the agent query vector; after obtaining the output of each attention head k After that, the attention feature z of agent i is obtained by concatenating them in the feature dimension and performing projection transformation. i ; The Q value network is used It uses a dual Q network structure to establish two review networks with the same structure but different initial parameters for each agent. It can avoid the overestimation problem in reinforcement learning. The input of the Q value network is the observation value o of the agent i. i , routing decision a i and attention feature z i , output the future expected reward of agent i The routing policy network is used It is composed of three layers of fully connected networks with equal number of neurons and one layer of long short-term memory network, and its input is the observation value o of the intelligent agent. i , the output is the probability ρ of the agent taking various routing decisions i , for ρ i After Monte Carlo sampling and one-hot encoding, the agent’s routing decision a is obtained. i .

5. The robust routing method for power communication system based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: In step S4, the Soft-Actor-Critic algorithm is used to alternately train and improve the parameters of the critic network and the routing strategy network. The parameters of the routing strategy network are θ = (θ1, θ2, ..., θ i ,...,θ N ) is to maximize the state value, where θ i Representing the parameters of the policy network of agent i, the joint loss function L(θ) of all agents is expressed as follows: Where a i is the routing decision of the i-th agent; z i is the attention feature of each agent i; D is the buffer for storing all agent historical observation vectors and historical action vectors, o and a represent all agent joint observation and joint action samples extracted from the buffer, respectively. is the observation quantity o of agent i i Routing Policy Network The action vector obtained by sampling, μ represents the policy entropy weight coefficient, and the parameters ω, The update goal is to minimize the temporal difference error, and the joint loss function of all agent-critic networks is It is expressed by the following formula: Where q i represents the update target of the temporal difference, o' and r represent the joint observation sample and joint reward sample of all agents at the next moment extracted from the buffer, respectively, z' i represents the attention feature of agent i at the next moment, r i is the reward obtained by agent i, γ is the discount coefficient of future rewards, a' i is the observation quantity o' of agent i i Routing Policy Network The action vectors obtained by sampling are used; the network parameters are updated alternately using the gradient descent method until convergence, and the optimal routing strategy network for all intelligent agents is obtained.

Images