Electric power communication network self-optimization-approaching control method and system based on graph reinforcement learning

Abstract

The invention relates to an electric power communication network self-optimization-approaching control method and system based on graph reinforcement learning, and the method comprises the steps: obtaining the topological information of an electric power communication network and the state information in a communication period as graph data, and extracting the edge features in the topological information and the node features in the state information respectively; obtaining branch feature vectors by splicing the edge features and the node features corresponding to each edge and the nodes at the two ends; the branch feature vectors are combined with topological information, deep features of nodes and connection characteristics of the nodes in the electric power communication network are mined, risk scores corresponding to the branch feature vectors are obtained, risk scores of the nodes and edges in the electric power communication network are further obtained, and the obtained risk scores serve as weights during DRL proxy reinforcement learning; and obtaining a combination of recommended nodes and edges through a strategy learned by the DRL agent, forming a recommendation result of a data stream transmission path, and transmitting state information of the electric power communication network by taking the recommendation result as the path.

Description

Technical Field

[0001] The present invention relates to the technical field of reinforcement learning, and in particular to a self-optimizing control method and system for a power communication network based on graph reinforcement learning. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Communication network self-optimization control refers to the dynamic optimization and configuration of network resources through adaptive mechanisms and intelligent algorithms within the network to maximize network performance. The power communication network needs to face complex scenarios such as large-scale distributed power generation, multi-energy complementarity, and power market transactions. Therefore, it has higher requirements in terms of uneven resource allocation, frequent network congestion, and rapid response and recovery capabilities in the face of sudden failures.

[0004] For the self-optimizing control process of power communication networks, some existing technologies focus on using advanced path planning algorithms to improve network efficiency and stability. However, most of these technologies focus on optimizing data transmission paths and fail to fully consider the risk factors in network operation. Risk factors in network operation include but are not limited to potential bottlenecks caused by uneven resource allocation, reduced service quality caused by network congestion, and vulnerability and recovery speed in the face of emergencies. This neglect of risk assessment limits the effectiveness of the control process and the overall resilience of the network. Summary of the invention

[0005] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a self-optimizing control method and system for power communication networks based on graph reinforcement learning. By integrating graph neural networks and algorithms, it aims to improve the intelligence level of network management. By embedding graph convolutional neural networks into a deep reinforcement learning framework, it monitors the network status in real time and makes intelligent decisions, enhances network resilience, and responds to sudden communication failures to provide the best routing alternatives.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A first aspect of the present invention provides a self-optimizing control method for a power communication network based on graph reinforcement learning, comprising the following steps:

[0008] The topology information of the power communication network and the state information during the communication are obtained as graph data, and the edge features in the topology information and the node features in the state information are extracted respectively. The branch feature vector is obtained by concatenating the edge features and node features corresponding to each edge and the two end nodes.

[0009] The branch feature vector is combined with the topological information to mine the deep-level features of each node and its connection characteristics in the power communication network, and the risk score corresponding to each branch feature vector is obtained. The risk score of each node and edge in the power communication network is further obtained, and the obtained risk score is used as the weight in the agent reinforcement learning.

[0010] Through the strategies learned by the agent, the recommended combination of nodes and edges is obtained to form the recommended result of the data stream transmission path, and the status information of the power communication network is transmitted along the recommended result.

[0011] Furthermore, the branch feature vector is passed through the deep learning network of the pre-weighted GCN to extract the grid information and risk quantification information and perform weighted analysis. The deep learning network of the pre-weighted GCN includes two layers of weighted GCN embedded in the evalnet and targetnet of the deep Q learning network, and these two layers of GCN are updated together with the deep Q learning network, and the evalnet outputs the action value a at time t t .

[0012] Further, agent reinforcement learning includes, the agent performs action a t , the environment state is determined by the current state s t , switch to the next state s t+1 , and feedback the current action reward value r t For the intelligent agent to learn.

[0013] Furthermore, the combination of nodes and edges forms the recommended result of the data flow transmission path, specifically: for each pair of source-destination nodes, its action set A sd Including k shortest paths, action space A a Designed for:

[0014]

[0015] Where: N s , N d They represent the source node and target node set in the network respectively, and k is the preset number of shortest paths.

[0016] Furthermore, the reward value r t , as shown below:

[0017] r t =ω1×BandwidthUtilization(t)-ω2×CongestionLevel(t)+ω3×PathSuccessRate(t)+ω4×RiskScore t (t)-ω5×HopCountFactor t ;

[0018] Where: BandwidthUtilization(t) is the degree of utilization of bandwidth resources in the network at time step t; CongestionLevel(t) is the degree of congestion in the network at time step t; PathSuccessRate(t) is the success rate of data transmission or routing request in the network at time step t; RiskScore is the risk assessment score, which is obtained by weighted summation of risk factors, and the formula is: RiskScore = ω q ×q l，t +ω p ×p u +ω a ×(1-a), where ω q ,ω p ,ω a is the weight of the corresponding risk indicator; HopScountFactor t Hop Count Factor is a decay function based on the hop count. The formula is HopCountFactor = ω h ×e -α·HopCount (ω h is the hop weight).

[0019] Furthermore, during agent reinforcement learning, the priority p is determined for each newly accumulated experience. i , calculate the probability P(i) of each experience being sampled according to the determined priority.

[0020] Furthermore, during agent reinforcement learning, by calculating a weight ω i Compensate for sampling deviation as shown in the following formula:

[0021]

[0022] Where N is the total number of experiences in the experience pool, β' is a hyperparameter used to control the scale of the weights, and as training progresses, β' gradually increases to 1 to gradually reduce the bias effect of the weights.

[0023] The second aspect of the present invention provides a self-optimizing control system for a power communication network based on graph reinforcement learning, including.

[0024] The feature extraction module is configured to: obtain the topology information of the power communication network and the state information during the communication as graph data, and respectively extract the edge features in the topology information and the node features in the state information;

[0025] The feature enhancement module is configured to: obtain a branch feature vector by concatenating edge features and node features corresponding to each edge and two end nodes;

[0026] The risk quantification module is configured as follows: the branch feature vector is combined with the topological information to mine the deep-level features of each node and its connection characteristics in the power communication network, and the risk score corresponding to each branch feature vector is obtained, and the risk score of each node and edge in the power communication network is further obtained;

[0027] The agent learning module is configured to: use the obtained risk score as the weight during the DRL agent reinforcement learning;

[0028] The path adjustment module is configured to: obtain the recommended combination of nodes and edges through the strategy learned by the agent, form a recommended result of the data flow transmission path, and use the recommended result as the path to transmit the status information of the power communication network.

[0029] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-mentioned graph reinforcement learning-based power communication network self-optimization control method.

[0030] The fourth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the above-mentioned graph reinforcement learning-based self-optimization control method for power communication networks are implemented.

[0031] Compared with the prior art, one or more of the above technical solutions have the following beneficial effects:

[0032] 1. The power communication network is abstracted into graph data, and based on the idea of ​​risk quantification and deep reinforcement learning, the problem of risk neglect in path planning and resource scheduling of the power communication network is solved. By deeply extracting and comprehensively analyzing the branch features containing nodes and edges, it not only focuses on the traffic scheduling between nodes, but also conducts deep feature mining on each node and its connection characteristics in the network, and guides the deep reinforcement learning algorithm to make intelligent decisions through quantitative risk assessment scores.

[0033] 2. While transmitting network architecture information, we make full use of the adaptability of reinforcement learning to quickly solve uncertain factors. In order to overcome the local optimal problem that may be caused by greedy strategies in traditional reinforcement learning, we introduce the Experience Replay Buffer. By replaying and evaluating the historical path selection within a period of time (that is, the time period of the u concise point), it helps to learn long-term optimization strategies and avoid the impact of short-term greedy behavior on the overall path selection optimization. The experience replay pool not only enhances the robustness of the learning process, but also improves the model's adaptability to complex network environments, ensuring that path selection meets both real-time needs and long-term interests. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0035] Figure 1 is an overall flow chart of a self-optimizing control method provided by one or more embodiments of the present invention;

[0036] Figure 2 It is a schematic diagram of a model of a self-optimizing control method for a power communication network based on graph deep reinforcement learning provided by one or more embodiments of the present invention;

[0037] Figure 3 is a schematic diagram of evaluation test results provided by one or more embodiments of the present invention;

[0038] Figure 4 is a schematic diagram of relative scores obtained with respect to theoretical fluids provided by one or more embodiments of the present invention;

[0039] Figure 5 is a graph of test results of link failure recovery capability provided by one or more embodiments of the present invention;

[0040] Figure 6 It is a structural diagram of a power communication network self-optimization system provided by one or more embodiments of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0042] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0043] Terminology explanation:

[0044] DQN, the full name of which is DeepQ-Network, is an algorithm that combines deep learning and reinforcement learning, and is mainly used to solve optimization problems in the decision-making process.

[0045] DRL agent, that is, the agent in Deep Reinforcement Learning (DRL), is an entity that performs actions and learns strategies in the environment.

[0046] Link betweenness: Link betweenness is a metric used to measure the importance of links in a network, and is often used in graph theory and network analysis. The higher the link betweenness, the more important the link is for the flow of information in the network. Specifically, link betweenness refers to the ratio of the number of paths that pass through the link to the total number of paths in the network. Link betweenness B(l)B(l) can be defined as: Where: st is the number of all shortest paths from node s to node t. st (l) is the number of paths that pass through link l in the shortest path from node s to node t.

[0047] Prioritized Experience Replay: Prioritized Experience Replay (PER) is an experience replay technique used in reinforcement learning. It improves learning efficiency by non-uniformly extracting experiences based on the importance of the experience. Compared with traditional uniform experience replay (UER), PER allows the algorithm to prioritize experiences for learning based on their importance, which can accelerate the learning process and improve the stability of learning. Priority is usually determined based on the absolute time difference error (TD Error) of the experience. The specific steps are as follows:

[0048] Calculation priority: Each newly accumulated experience determines its priority, and the priority is determined based on the absolute time difference error (TD Error) of the experience. TD Error reflects the difference between the Q value predicted by the algorithm and the actual observed reward. As shown in the following formula:

[0049] p t =|δ t ∣+∈;

[0050] Among them, δ t is the TD Error, ∈ is a small positive number that ensures that even experiences with zero TD error have some non-zero priority.

[0051] Calculate sampling probability: Calculate the probability of being extracted based on the priority of the experience. Experience with higher priority has a higher probability of being extracted.

[0052]

[0053] Among them, α is a hyperparameter that controls the degree of skewness of the sampling probability. When α = 0, PER degenerates into uniform experience replay (UER). As α increases, the PER mechanism will be more inclined to select those experiences with larger TD errors.

[0054] Calculate importance sampling weights: Due to the uneven sampling probability caused by the PER mechanism, a weight needs to be calculated to compensate for the sampling bias and ensure the unbiased nature of the learning objective.

[0055]

[0056] Where N is the total number of experiences in the experience pool and β is another hyperparameter that controls the scale of the weights. As training progresses, β is gradually increased to 1 to gradually reduce the bias effect of the weights.

[0057] The following embodiments propose a self-optimizing control method and system for power communication networks based on graph reinforcement learning, which aims to improve the intelligent level of network management by integrating graph neural networks and algorithms. Specifically, the model embeds graph convolutional neural networks into a deep reinforcement learning framework to monitor network status in real time and make intelligent decisions, enhance network resilience, and respond to sudden communication failures to provide the best routing alternatives.

[0058] Embodiment 1:

[0059] The self-optimizing control method of the power communication network based on graph reinforcement learning includes the following steps:

[0060] The topology information of the power communication network and the state information during the communication are obtained as graph data, and the edge features in the topology information and the node features in the state information are extracted respectively. The branch feature vector is obtained by concatenating the edge features and node features corresponding to each edge and the two end nodes.

[0061] The branch feature vector is combined with the topological information to mine the deep-level features of each node and its connection characteristics in the power communication network, and the risk score corresponding to each branch feature vector is obtained. The risk score of each node and edge in the power communication network is further obtained, and the obtained risk score is used as the weight of the DRL agent reinforcement learning.

[0062] Through the strategies learned by the DRL agent, the recommended combination of nodes and edges is obtained to form the recommended result of the data stream transmission path, and the status information of the power communication network is transmitted along the recommended result.

[0063] This embodiment collects real-time status information in the power communication network, including but not limited to link load, delay, packet loss rate, etc., and defines a state space based on this information;

[0064] Based on the current state of the network, define a series of possible actions, such as changing the transmission path of the data flow;

[0065] Set the reward function and use network performance indicators (such as throughput, latency, etc.) as feedback signals;

[0066] Using deep reinforcement learning algorithms, such as DQN or DDPG, to train an agent that can select the best action from the current state;

[0067] In this embodiment, an agent refers to an entity that can take actions to achieve a specific goal. Its characteristic is that it can learn through interaction with the environment, thereby optimizing its behavior to maximize the cumulative reward. It can be considered as an agent model that selects actions based on an action value function.

[0068] Through the strategies learned by the agent, the transmission path of the data flow is dynamically adjusted in the actual network environment to achieve the optimal network performance.

[0069] In this embodiment, the meanings of the parameters involved in the formula are shown in Table 1.

[0070] Table 1: Parameter meaning introduction

[0071]

[0072] The power communication network generally uses optical cables or optical fibers as transmission media, so this embodiment simply refers to the power communication network as an optical transmission network or optical network.

[0073] like Figure 1 As shown in the figure, for feature extraction in the power communication network, this example first abstracts the optical transmission network into graph data, which contains two parts: the network topology (edge ​​features) of the optical network and its optical cable feature data X (node ​​features). The abstracted data is input into the graph convolutional network for node feature extraction, and the node feature vector is output. Then, the feature vectors of the nodes at both ends of the branch and the branch features (edge ​​features) are concatenated and input into the branch feature extraction network composed of a fully connected layer to output the branch feature vector. Finally, the branch feature vector is input into the linear rectification function (ReLU) to obtain the risk analysis result, and then normalized as a quantitative risk indicator weight to output the risk analysis result of the branch. The feature types selected in this example are shown in Table 1.

[0074] In this embodiment, the extracted node features are combined with the branch features through feature concatenation to form a comprehensive feature vector. When considering the risk of passing through the branch, the impact of the node on the branch risk is also considered.

[0075] like Figure 2As shown in the figure, the extracted features are input into the downstream deep Q learning network. Two layers of weighted GCN are embedded in the eval net and target net of the network to form a "deep learning network with pre-weighted GCN" (i.e. RQGRL). The grid information and risk quantification information are further extracted and weighted analysis is performed. The two layers of GCN will be updated together with DQN, and finally the eval net will output the action value a at time t. t The agent performs action a t , the environment state is determined by the current state s t , switch to the next state s t+1 , and feedback the current action reward value r t For the intelligent agent to learn.

[0076] Let the model state at time step t be S t ,but:

[0077] S t = {G t , O l，t , L n，t , P p，t};

[0078] Where: At time step t, G t Indicates the connection status of the network topology; l，t is the occupied bandwidth ratio of link l at time step t; L n，t The data traffic processed by node n; P p，t is the current availability and bandwidth reservation of path p.

[0079] Intelligent agent refers to the hardware equipment and / or software program used to monitor, collect, analyze, calculate, send, receive and store power communication data in the power communication network.

[0080] For the state space definition of the DRL agent, this example is defined as: the state space includes the topology of the optical fiber communication network, link occupancy, node load and path status, and link betweenness, which represents the number of possible paths through the link. Specifically, the link betweenness is calculated as follows:

[0081] For each pair of nodes in the topology, k candidate paths are computed and a per-link counter is maintained that indicates the number of paths that go through the link. Thus, the betweenness on each link is the number of end-to-end paths that cross the link divided by the total number of paths.

[0082] Regarding the action space design of DRL, considering that in large-scale real networks, DRL agents often face high-dimensional action space problems, the number of candidate paths is limited here. By limiting the action set of each source-target node pair to k shortest paths, the originally possible huge action space is greatly reduced. Here, the action set of each source-target node pair is limited to k candidate paths.

[0083] In order to maintain a good trade-off between the flexibility of routing traffic and the cost of evaluating all possible actions, this embodiment selects the shortest path set in each pair of source-destination nodes (k=4).

[0084] In addition, this embodiment considers an additional link layer feature, namely the bandwidth allocated to the link after the routing operation is applied. This value corresponds to the bandwidth requirement of the current traffic request to be allocated. Similarly, connections not included in the path selected by the action will have this feature set to 0. For traffic requests with a limited number of discrete bandwidth requirements in the environment, a one-hot encoding of N elements is used to represent the allocated bandwidth, where N is the vector length.

[0085] For each pair of source-target nodes, its action set A sd It includes k shortest paths (these paths are selected as the best possible routing options based on the shortest hop count and risk weighted calculation)

[0086] In summary, the action space A in this example a Designed for:

[0087]

[0088] Where: N s , N d They represent the source node and target node set in the network respectively, and k is the preset number of shortest paths.

[0089] For the design of the reward function, the reward function of this example is designed as follows:

[0090] r t =ω1×BandwidthUtilization(t)-ω2×CongestionLevel(t)+ω3×PathSuccessRate(t)+ω4×RiskScore t (t)-ω5×HopCountFactor t ;

[0091] Where: BandwidthUtilization(t) is the degree of utilization of bandwidth resources in the network at time step t; CongestionLevel(t) is the degree of congestion in the network at time step t; PathSuccessRate(t) is the success rate of data transmission or routing request in the network at time step t; RiskScore is the risk assessment score, which is obtained by weighted summation of risk factors, and the formula is: RiskScore = ω q ×q l，t +ω p ×p u +ω a ×(1-a), where ω q ,ω p ,ω a HopScountFactor is the weight of the corresponding risk indicator. t Hop Count Factor is a decay function based on the hop count. The formula is HopCountFactor = ω h ×e -α·HopCount (ω h is the hop weight).

[0092] In order to optimize learning efficiency and stability, the Prioritized Experience Replay (PER) mechanism is introduced. The PER mechanism allows the algorithm to sample experiences non-uniformly according to the importance of the experience, thereby improving learning efficiency. When implementing the PER mechanism, it is first necessary to sample each newly accumulated experience (s t , a t , R t：t+n ,s t+n ) Determine its priority i , priority p i It is determined based on the empirical absolute time difference error (Tempora1Differ ence Error, TD), that is:

[0093] p i =|TD error |+ε;

[0094] Here ε is a small positive number that ensures that even experiences with zero TD error have a certain non-zero priority, thereby ensuring that all experiences have the opportunity to be replayed and learned. Once the priority of each experience is determined, the probability P(i) of each experience being sampled can be calculated based on these priorities. The sampling probability is allocated proportionally, that is:

[0095]

[0096] Here, α' is a hyperparameter that controls the degree of skewness of the sampling probability. When α' = 0, P ER degenerates into Uniform Experience Replay. As α' increases, the P ER mechanism will be more inclined to select experiences with larger TD errors.

[0097] Due to the uneven sampling probability caused by the PER mechanism, a weight ω needs to be calculated i To compensate for the sampling bias and ensure the unbiased nature of the learning objective. Importance sampling weight ω i Defined as:

[0098]

[0099] Where N is the total number of experiences in the experience pool, and β' is another hyperparameter that controls the scale of the weights. As training progresses, β' gradually increases to 1 to gradually reduce the bias effect of the weights. After each update of the network, the TD error of each experience needs to be recalculated and its priority in the experience pool is updated accordingly, which ensures that the algorithm always focuses on the most valuable experiences.

[0100] The effectiveness of this example is verified through experiments, and the experimental parameter settings adopted are shown in Table 2.

[0101] Table 2: Test parameter settings

[0102]

[0103]

[0104] To verify the effectiveness of this example, it is compared with the most advanced DRL-based solution. When the DRL agent correctly assigns a demand, it will immediately receive a reward, which is the bandwidth of the current communication demand if the demand is correctly assigned, otherwise it is zero. It is defined here that if the links on all paths selected by the DRL agent have sufficient available capacity to carry such a demand, then the demand is considered to be successfully assigned. On the contrary, if the demand is not successfully assigned, the attempt is considered to have failed and enters the next time step or starts a new simulation cycle. The experimental results are shown in Figure 3 and Figure 4 shown.

[0105] like Figure 3 As shown, a box plot can be observed containing the results of 1000 evaluation trials. The y-axis represents the agent score, which corresponds to the bandwidth allocated to the agent. Figure 4The cumulative distribution function (CDF) of the relative scores obtained with respect to the flow model is shown. A load-balanced routing strategy (LB) is used, which uniformly randomly selects a path among the shortest paths to distribute the traffic demand. At the same time, the DRL+GNN agent performance is compared with the theoretical fluid model (labeled as Theoretical Fluid). This model is a theoretical approach that considers that SDH demands can be split and proportional to the available capacity they have. The purpose of this path strategy is to avoid congestion on the links.

[0106] For example, paths with lower available capacity will carry a smaller proportion of traffic from new demands. It is worth noting that this model is not feasible because in real SDH scenarios, SDH demands are usually difficult to segment, which will affect network performance, especially under high load. In this experiment, it can also be seen that the agent proposed in this example is slightly better than other DRL agents because the former allocates 6.6% more bandwidth than the latter. This result shows that the agent proposed in this example has achieved significant progress in routing optimization performance, and its bandwidth allocation efficiency has been improved by 6.6% compared to the current state-of-the-art DRL agent. This means that when handling diverse traffic demands, the model proposed in this example can more effectively utilize network resources and ensure higher bandwidth utilization while maintaining network stability and efficiency.

[0107] Through experiments, it is verified whether the method proposed in this example can effectively avoid risk factors in the network, such as Figure 5 As shown in the figure, the selected risk influencing factor is F = {link congestion, link failure}. Link congestion and link failure can exist simultaneously or separately in the same scenario, depending on the operating state of the system and external disturbances. The indicator sets f1, f2, and f3 are all subsets of F. f1 means that the agent only considers the link congestion indicator, f2 means that the agent only considers the link failure indicator, and f3 = {f 31 , f 32}, indicating that the agent considers two indicators at the same time. 32 , f 32 Indicates the link congestion index and link failure index in case of f3.

[0108] This example performs the following operations on the power communication network topology:

[0109] (1) Gradually increase the traffic through these links over a specific time period until it reaches or exceeds 90% of the link capacity, simulating a highly congested state.

[0110] (2) Randomly select links in the network for fault simulation. The simulation process can be transient faults (links are temporarily interrupted and then restored) or permanent faults (links fail completely). Each fault affects one or more links.

[0111] Figure 5 The relationship between the average score (y-axis) and the number of link failures (x-axis) is shown. It can be seen that even in the extreme case of 10 concurrent link failures, the agent proposed in this example can maintain better performance than the theoretical baseline. This means that when facing network risks, the agent can not only effectively deal with link congestion, but also quickly identify and bypass link failures, thereby ensuring the quality and efficiency of network communications.

[0112] The above process is based on the idea of ​​risk quantification and deep reinforcement learning to achieve self-optimization control of power communication, and specifically solves the problem of risk neglect in path planning and resource scheduling of power communication networks. By deeply extracting and comprehensively analyzing the characteristics of the surrounding environment of the nodes, it not only focuses on the traffic scheduling between nodes, but also introduces risk assessment factors. The core is to use the GCN model to perform deep feature mining on each node and its connection characteristics in the network to obtain a more accurate state representation. On this basis, a multidimensional evaluation system including risk quantification is designed to guide the deep reinforcement learning algorithm to make intelligent decisions.

[0113] By using the post-graph neural network to embed the reinforcement learning framework, while using the fiber transmission grid structure information, the adaptability of reinforcement learning to uncertain factors is fully utilized to quickly solve. In order to overcome the local optimal problem that may be caused by greedy strategies in traditional reinforcement learning, the experience replay buffer is introduced. By replaying and evaluating the historical path selection within a period of time (that is, the time period of the u concise point), this mechanism helps to learn long-term optimization strategies and avoids the impact of short-term greedy behavior on the overall path selection optimization. The experience replay pool not only enhances the robustness of the learning process, but also improves the model's adaptability to complex network environments, ensuring that the path selection meets both real-time needs and long-term interests.

[0114] Embodiment 2:

[0115] The self-optimizing control system of power communication network based on graph reinforcement learning includes:

[0116] The feature extraction module is configured to: obtain the topology information of the power communication network and the state information during the communication as graph data, and respectively extract the edge features in the topology information and the node features in the state information;

[0117] The feature enhancement module is configured to: obtain a branch feature vector by concatenating edge features and node features corresponding to each edge and two end nodes;

[0118] The risk quantification module is configured as follows: the branch feature vector is combined with the topological information to mine the deep-level features of each node and its connection characteristics in the power communication network, and the risk score corresponding to each branch feature vector is obtained, and the risk score of each node and edge in the power communication network is further obtained;

[0119] The agent learning module is configured to: use the obtained risk score as the weight during the DRL agent reinforcement learning;

[0120] The path adjustment module is configured to: obtain the recommended combination of nodes and edges through the strategy learned by the DRL agent, form a recommended result of the data flow transmission path, and use the recommended result as the path to transmit the status information of the power communication network.

[0121] Embodiment three:

[0122] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps in the method for self-optimization control of a power communication network based on graph reinforcement learning as described in the above-mentioned embodiment 2 are implemented.

[0123] Embodiment 4:

[0124] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the method for self-optimization control of a power communication network based on graph reinforcement learning as described in the second embodiment above are implemented.

[0125] The steps involved in the above embodiments 2 to 4 correspond to those in embodiment 1. For the specific implementation, please refer to the relevant description of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0126] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A self-optimizing control method for power communication networks based on graph reinforcement learning, characterized in that: The following steps are involved: The topology information of the power communication network and the state information during the communication are obtained as graph data, and the edge features in the topology information and the node features in the state information are extracted respectively. The branch feature vector is obtained by concatenating the edge features and node features corresponding to each edge and the two end nodes. The branch feature vector is combined with the topological information to mine the deep-level features of each node and its connection characteristics in the power communication network, and the risk score corresponding to each branch feature vector is obtained. The risk score of each node and edge in the power communication network is further obtained, and the obtained risk score is used as the weight in the agent reinforcement learning. Through the strategies learned by the agent, the recommended combination of nodes and edges is obtained to form the recommended result of the data stream transmission path, and the status information of the power communication network is transmitted along the recommended result.

2. The method for self-optimizing control of a power communication network based on graph reinforcement learning according to claim 1, characterized in that: The branch feature vector passes through the deep learning network of the pre-weighted GCN to extract the grid information and risk quantification information and perform weighted analysis. The deep learning network of the pre-weighted GCN includes two layers of weighted GCN embedded in the eval net and target net of the deep Q learning network, and these two layers of GCN are updated together with the deep Q learning network. The eval net outputs the action value a at time t t .

3. The self-optimizing control method for electric power communication network based on graph reinforcement learning according to claim 1, characterized in that: Agent reinforcement learning, including, the agent performs actions a t , the environment state is determined by the current state s t , switch to the next state s t+1 , and feedback the current action reward value r t For the intelligent agent to learn.

4. The method for self-optimizing control of a power communication network based on graph reinforcement learning according to claim 1, characterized in that: The combination of nodes and edges forms the recommended result of the data flow transmission path, specifically: for each pair of source-destination nodes, its action set A sd Including k shortest paths, action space A a Designed for: Where: N s ,N d They represent the source node and target node set in the network respectively, and k is the preset number of shortest paths.

5. The self-optimizing control method for electric power communication network based on graph reinforcement learning according to claim 1, characterized in that: Reward value r t , as shown below: r t =ω1×BandwidthUtilization(t)-ω2×CongestionLevel(t)+ω3×PathSuccessRate(t)+ω4×RiskScore t (t)-ω5×HopCountFactor t ; Where: BandwidthUtilization(t) is the degree to which the bandwidth resources of the network are utilized at time step t; CongestionLevel(t) is the degree of congestion of the network at time step t; PathSuccessRate(t) is the success rate of data transmission or routing request in the network at time step t; RiskScore is the risk assessment score, which is obtained by weighted summation of risk factors. The formula is: RiskScore = ω q ×q l,t +ω p ×p u +ω a ×(1-a), where ω q ,ω p ,ω a is the weight of the corresponding risk indicator; HopScountFactor t is the hop factor, which is a decay function based on the hop count.

6. The method for self-optimizing control of a power communication network based on graph reinforcement learning according to claim 1, characterized in that: During agent reinforcement learning, a priority p is determined for each newly accumulated experience i , calculate the probability P(i) of each experience being sampled according to the determined priority.

7. The self-optimizing control method for electric power communication network based on graph reinforcement learning according to claim 1, characterized in that: During agent reinforcement learning, by calculating the weight ω i Compensate for sampling deviation as shown in the following formula: Where N is the total number of experiences in the experience pool, β' is a hyperparameter used to control the scale of the weights, and as training progresses, β' gradually increases to 1 to gradually reduce the bias effect of the weights.

8. A self-optimizing control system for electric power communication networks based on graph reinforcement learning, characterized in that: include: The feature extraction module is configured to: obtain the topology information of the power communication network and the state information during the communication as graph data, and respectively extract the edge features in the topology information and the node features in the state information; The feature enhancement module is configured to: obtain a branch feature vector by concatenating edge features and node features corresponding to each edge and two end nodes; The risk quantification module is configured as follows: the branch feature vector is combined with the topological information to mine the deep-level features of each node and its connection characteristics in the power communication network, and the risk score corresponding to each branch feature vector is obtained, and the risk score of each node and edge in the power communication network is further obtained; The agent learning module is configured to: use the obtained risk score as a weight for agent reinforcement learning; The path adjustment module is configured to: obtain the recommended combination of nodes and edges through the strategy learned by the agent, form a recommended result of the data flow transmission path, and use the recommended result as the path to transmit the status information of the power communication network.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, which, when executed by a processor, implements the steps in the self-optimizing control method for a power communication network based on graph reinforcement learning as described in any one of claims 1 to 7.

10. A computer device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the self-optimizing control method for a power communication network based on graph reinforcement learning as described in any one of claims 1 to 7 are implemented.

Images