Mutual supply and mutual aid hybrid control method based on honeycomb transverse topological architecture

Abstract

The invention provides a mutual supply and mutual aid hybrid control method based on a honeycomb-shaped transverse topological framework, and the method comprises the following steps: carrying out the abstract representation of the transverse topological framework of a honeycomb-shaped power distribution network through a graph theory, and enabling each vertex to represent a main transformer substation of each regional grid in the honeycomb-shaped power distribution network, the regional grids are connected through a flexible interconnection device; constructing an effective node set through a screening method based on honeycomb-shaped effective ring topology; and representing the hybrid discrete-continuous action space model through a conditional variation auto-encoder and an embedded table, and embedding an approximate tidal current diagram neural network into a reward function of a parameterized action Markov decision process to realize optimal cooperative mutual supply and mutual aid of the honeycomb transverse topology. According to the method, the overall load balancing rate of the honeycomb-shaped power distribution network can be greatly improved on the basis of meeting the power flow constraint, the safe and optimized operation of the power grid is ensured, and the application of the deep reinforcement learning method in the power grid mixing problem is expanded.

Description

Technical Field

[0001] The present invention relates to the field of power system operation and control, and particularly to a mutual supply and mutual assistance hybrid control method based on a honeycomb-shaped horizontal topology architecture. Background Art

[0002] Due to its relatively simple power supply structure, rigid load characteristics, and mostly single-way power receiving operation characteristics, the traditional extensive distribution network often adopts a "top-down" vertical management and planning mode. With the large-scale grid connection of renewable resources, the resource density on each side of the source, grid, load, and storage has increased significantly, which easily leads to an increase in the uncertainty of power flow. The power users' requirements for power supply reliability are also constantly increasing, and the construction of the distribution network has gradually entered the stage of refined development. Under this background, due to the relatively rough granularity of the traditional vertical management mode, it is easy to cause a decline in resource interactivity and a reduction in the operation effect of the distribution network, and may even cause safety problems such as power fluctuations and voltage over-limit, making it difficult to achieve refined management of the distribution network. Therefore, on the basis of retaining the vertical management mode of the distribution network, adding a horizontal management mode is an inevitable trend in the construction and development of the distribution network.

[0003] In this mode, the power supply area of the distribution network is divided into several non-overlapping unit power supply grids of the same voltage level, and different power supply grids can achieve coordinated mutual supply and mutual assistance regulation through formulated strategies, so as to achieve full-range and refined horizontal management of the distribution network. The emergence of the new horizontal management mode can effectively enhance power supply reliability, improve dispatching flexibility, and enhance the lean management level. However, most of the existing research is based on the vertical management concept, ignoring the problems that may occur in the distribution network during the horizontal control process, and there is little research on the horizontal coordinated mutual supply and mutual assistance strategy of the distribution network. Therefore, for the research on the horizontal coordinated mutual supply and mutual assistance regulation strategy, the research ideas under the vertical architecture cannot be directly migrated.

[0004] At the same time, due to the extremely uneven spatio-temporal distribution of distributed renewable energy, it is easy to cause uneven load rates of regional power supply grids, which will not only affect the utilization rate of grid substations, but may even lead to problems such as uneconomical operation of lightly loaded transformers and unsafe operation of heavily loaded transformers. If the grid load rate is not regulated in time, it may further cause serious impacts such as large-scale power outages in the regional power grid, thus making it fall into a fault state and bringing huge safety hazards to the distribution system. Therefore, the load balancing problem between multi-region grids in the horizontal control mode is the content that needs to be focused on.

[0005] Different from the petal-shaped distribution network and the diamond-shaped distribution network, the honeycomb-shaped distribution network does not require traditional switching operations, but uses the hexagonal characteristics of the topology to achieve power supply guarantee of the power system with high economic benefits. In view of this, the present invention proposes a mutual supply and mutual assistance hybrid control method based on a honeycomb-shaped horizontal topology architecture. Summary of the Invention

[0006] In response to the increasingly refined development requirements of modern distribution networks, the present invention proposes a mutual supply and mutual assistance hybrid control method based on a honeycomb-shaped horizontal topological structure. Based on the flexible interconnection devices at the distribution layer, the highly interconnected honeycomb-shaped distribution network structure can balance reliability and stability while taking into account excellent scalability and flexibility, meeting the increasingly refined development requirements of modern distribution networks, achieving load balancing in the honeycomb-shaped distribution network, and solving the problems raised in the above background technology. The technical solutions provided by the present invention are as follows: A mutual supply and mutual assistance hybrid control method based on a honeycomb-shaped horizontal topological structure, comprising the following steps: Step 1, abstractly represent the horizontal topological structure of the honeycomb-shaped distribution network through graph theory, where each vertex represents the main substation of each regional grid in the honeycomb-shaped distribution network, and the regional grids are connected through flexible interconnection devices, represented as , where V is the set of all nodes in the honeycomb-shaped distribution network, E is the set of connection lines between nodes, A is the adjacency matrix representing the connection relationship of nodes; Step 2, construct an effective node set through a screening method for honeycomb-shaped effective circular topologies based on complex network theory and entropy-TOPSIS; Step 3, characterize the hybrid discrete-continuous action space model through a conditional variational autoencoder and an embedding table, and embed an approximate power flow graph neural network into the reward function of the parameterized action Markov decision process to achieve optimal collaborative mutual supply and mutual assistance of the honeycomb-shaped horizontal topology.

[0007] Preferably, step 2 specifically includes the following steps: Step 2.1, select the following indicators to construct a screening system, including the degree centrality index DC , the closeness centrality index CC , the betweenness centrality index LBC and the electrical betweenness centrality index EBC ; Step 2.2, the screening process of the honeycomb-shaped effective topology based on entropy-TOPSIS, the specific steps are as follows: Step 2.2.1, construct a multi-attribute decision matrix: ; Step 2.2.2, standardize and normalize X : to obtain the matrix ; Step 2.2.3, calculate thej Information entropy of each index: Step 2.2.4: Calculate the difference coefficient of each index , and calculate the index weights ; Step 2.2.5: Construct a weighted normalized decision matrix ; Step 2.2.6: Calculate the positive ideal solution and the negative ideal solution: Step 2.2.7: Calculate the distance between the index of each node and the optimal solution and the worst solution: Step 2.2.8: Calculate the relative closeness of each node: Rank the importance according to the relative closeness, and select the top 70% of the nodes to construct an effective node set.

[0008] Preferably, in Step 3, the hybrid discrete - continuous action space model is characterized by a conditional variational auto - encoder and an embedding table, specifically: The hybrid discrete - continuous action space model includes two processes: encoding and decoding. Before encoding, an learnable embedding table is used to construct G the latent space of discrete actions, where each row of the embedding table represents a discrete action corresponding to a one - dimensional continuous vector, and the total dimension is d 1 ; The conditional variational auto - encoder is used for the encoding operation. Taking the state and the embedding table as conditions, the continuous action parameters corresponding to the discrete action are mapped to the latent variable to construct a latent representation space. The encoding process is expressed as: The decoding process is expressed as: where, ξ are the embedding table parameters, is the encoding process, Φ are the encoding parameters, is the decoding process, ψ are the decoding parameters, is the predicted continuous action parameter, and respectively represent the transformation network and the reconstruction fully-connected layer.

[0009] Preferably, the embedding table and the conditional variational autoencoder network and their parameters are trained by minimizing the loss function: where the first term on the right is the L2 reconstruction loss, and the second term is the KL divergence between the variational posterior of the latent representation variable and the Gaussian prior, and the two together constitute the loss function of the conditional variational autoencoder; represents the mathematical expectation; D is the replay buffer, represents the continuous action parameter from the replay buffer.

[0010] Preferably, the hybrid discrete-continuous action space model further uses a sub-network after the decoder of the conditional variational autoencoder to predict the state residual. For any sample its state residual and its predicted state residual are defined as: The total training loss function of the final hybrid action representation model is: where, ϑ is the weight corresponding to the regularization term.

[0011] Preferably, the approximate power flow graph neural network consists of a masked encoder and an approximate power flow convolutional layer. The binary masked encoder characterizes the known features and the unknown features, sets the known features to 0, and sets the unknown features to 1. The mapping operation of the masked encoder is:

[0012]

[0013] where and are weight matrices, and are biases, and are trained iteratively; represents the activation function; The input node feature matrix X is combined with the mask encoder M to obtain the encoded graph feature matrix of the k-th layer: ; The approximate flow convolution layer has K layers, and iteratively processes the effective graph features to predict the node feature matrix after the final action X K . In the message passing stage, the iterative update process of each node is described by the following formula: where t is the number of iterations in the message passing stage; is the neighbor node of node i ; is the node feature of the hidden layer, is the feature of the connection line between node i and node j, and is updated through the transfer function and the node update function ; is the node feature after each iteration of message passing update; is the feature vector after message passing; is the final node feature update function, is the sum of all ; The feature vector is fed into the multi-layer perceptron to obtain the graph feature matrix after message passing , which is combined with the encoded graph feature matrix and sent to the topological adaptive graph convolution operator layer for processing to obtain a new encoded graph feature : where the matrix S k is the normalized adjacency matrix; is the trainable weight matrix; the hyperparameter D is the convolution layer of adjacent nodes considering the distance from the target node D ;

[0014] Preferably, in step 3, the Twin Delayed Deep Deterministic Policy Gradient algorithm (TD3) is further used to train the reinforcement learning agent. TD3 adopts a total of six network structures, where the Actor network and the Target-Actor network are used to train the policy network, and the Critic1 network, the Crtic2 network, the Target-Critic1 network, and the Target-Crtic2 network are used to estimate the action value function Q of the current state and action. Neural networks are used to approximate the six networks to obtain the corresponding parameters as , , , , and .

[0015] Preferably, the training process of TD3 is as follows: During each policy iteration, the honeycomb distribution network generates different states , which are input into the Actor network to obtain actions . Gaussian noise is added to the actions to obtain ; then, is input into the environment to obtain the next state and the reward ; the latent policy π ϕ output by the latent action vector e and z x are used to decode the latent variables using a conditional variational autoencoder to obtain the corresponding mixed actions g and x g , and these data are stored in the replay buffer in the form of , Done is the final state of the environment; The Critic1 network and the Crtic2 network take the latent action as the input to approximate the mixed discrete-continuous action value function ; the corresponding Target-Critic1 network and the Target-Crtic2 network are used to estimate the target values y 1 and y 2 ; during the training process, is used, and the following formula is used to calculate the target value: Strategy Indicates the Target-Critic network for estimating the target value; selects the minimum value between the two estimated values as the target value y t : Regularizes the target value by introducing the method of target policy smoothing: where r t is the reward function, γ is the discount factor, with a value range from 0 to 1, ϵ is the clipped noise, and c is the boundary of the Gaussian function ; The training objectives of the Critic1 network and the Crtic2 network are to regress to the learned target value y t , obtain the loss functions of the two networks, and use the gradient descent method to train the two neural networks: where represents the mathematical expectation, is the learning rate of the network parameters; is the exponential smoothing parameter for updating the learning rate of the target network parameters ; The ultimate goal of the Actor network is to find the action that maximizes the Q value, approximate the policy function using a parameterized neural network, and use the gradient descent method to train the Actor network. The Target Actor network is also updated using the exponential smoothing parameter. The detailed formulas are as follows: where M is the total number of training iterations; λ ϕ is the learning rate of the Actor network; η is the exponential smoothing parameter for updating the learning rate of the target Actor network and the target soft Actor network parameters ; Before training the Actor network, multiple rounds of TD3 Critic network training are required.

[0016] Compared with the prior art, the beneficial effects achieved by the present invention are: By introducing the mixed discrete-continuous action representation model, the reward volatility is reduced, and the final converged reward value is higher than the algorithm before the introduction of the model, which shows the effectiveness of the proposed mixed discrete-continuous action representation model in solving the mixed discrete-continuous action problem.

[0017] Based on the flexible interconnected equipment foundation of the distribution layer, the highly interconnected honeycomb distribution network structure can ensure reliability and stability while taking into account excellent scalability and flexibility. Unlike the petal-shaped distribution network and the diamond-shaped distribution network, the honeycomb distribution network does not require traditional switching operations, but instead uses the hexagonal characteristics of the topology to achieve power supply security for the power system with high economic benefits. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 is a flow chart of the method provided by the present invention; Figure 2 It is a framework diagram of the method provided by the present invention; Figure 3 It is a structural diagram of the hybrid discrete-continuous action representation model provided by the present invention; Figure 4 It is a structural diagram of the approximate power flow diagram neural network provided by the present invention; Figure 5 This is a performance comparison chart of the HDCAR-TD3 algorithm provided by the present invention and other mainstream deep reinforcement learning algorithms; Figure 6 This is a performance comparison result diagram of the HDCAR-TD3 algorithm provided by the present invention and other mainstream deep reinforcement learning algorithms for mixed discrete-continuous action spaces. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] To make the above objects, features, and effects of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] Embodiment 1: A mutual supply and mutual assistance hybrid control method based on a honeycomb-shaped lateral topological structure, as Figure 1 shown, includes: Step 1, abstractly represent the lateral topological structure of the honeycomb-shaped distribution network through graph theory, where each vertex represents the main substation of each regional grid in the honeycomb-shaped distribution network, and the regional grids are connected through flexible interconnection devices; Step 2, construct an effective node set through a screening method for honeycomb-shaped effective circular topologies based on complex network theory and entropy-TOPSIS; Step 3, model the hybrid discrete-continuous action space through a conditional variational autoencoder and an embedding table, and embed an approximate power flow graph neural network into the reward function of the parameterized action Markov decision process to achieve optimal cooperative mutual supply and mutual assistance of the honeycomb-shaped lateral topology.

[0022] The specific framework diagram is as Figure 2 shown. Based on graph theory, the honeycomb-shaped distribution network is represented by , the node set V represents the main substation of each regional grid in the honeycomb-shaped distribution network, the edge set E represents the branch with a flexible interconnection device, and the feature matrix F reflects the load rate feature of each node i in .

[0023] The screening method for honeycomb-shaped effective circular topologies based on complex network theory and entropy-TOPSIS in Step 2 specifically includes the following steps: Step 2.1, in order to eliminate invalid honeycomb topological structures, using the idea of multi-attribute decision-making, in terms of index selection, consider constructing a multi-index screening system by combining the topological structure and electrical characteristics of the honeycomb-shaped distribution network, including degree centrality index, closeness centrality index, betweenness centrality index, and electrical betweenness centrality index.

[0024] Degree centrality index: The larger the degree of a node, the more critical the position of the node in the network topological structure. Its calculation formula is: (1) where is the degree of node i , is the total number of edges connecting node i to other nodes j in the network, that is , where when there is an edge between node i and node j , otherwise ; N is the number of nodes in the network topology structure.

[0025] Closeness centrality index: is the number of nodes in the network N and the average shortest electrical distance L The reciprocal of the product, which reflects the degree of the node being in the center of the topological structure, and its calculation formula is: (2) Among them, the shortest electrical distance is the line with the minimum sum of impedance values between two nodes, L is the average shortest electrical distance of the network, is the node i and the node j 's shortest distance.

[0026] Betweenness centrality index: The number of times the shortest electrical distance between two non - adjacent nodes in the power grid after normalization passes through a certain node. The more times a certain node is passed through during the power network transmission, the more important the position of this point in the network. Its calculation formula is: (3) Among them, represents the number of times the shortest electrical distance between two non - adjacent nodes passes through the node i .

[0027] Electrical betweenness centrality index: In addition to considering the importance of the honeycomb - shaped network topology and associated lines, the particularity of the engines and loads inside the nodes also needs to be considered. Its calculation formula is: (4) Among them, and respectively represent the weight coefficients of the generator and load inside the node i ; n is the total number of nodes inside the node i ; and respectively represent the numbers of the generator and load inside the node i ; is the active power of the k - th generator in node i, is the active power of the k - th load in node i; is the betweenness centrality of node i; is the weight coefficient of the node transmission load rate, and its calculation formula is: (5) Among them, is the load rate of the node i , and respectively represent the upper and lower limits of the load rate of node i.

[0028] Step 2.2, the honeycomb effective topology screening process based on entropy value - TOPSIS, the specific steps are as follows: Step 2.2.1, construct a multi - attribute decision matrix: ; Step 2.2.2, according to formulas (6) and (7), perform standardization and normalization on X : (6) (7) to obtain the matrix ; Step 2.2.3, calculate the information entropy of the j th index: (8) Step 2.2.4, calculate the information redundancy value of each index, that is, the difference coefficient , and calculate the index weight ; Step 2.2.5, construct a weighted normalized decision matrix ; Step 2.2.6, calculate the positive ideal solution and the negative ideal solution: (9) (10) Step 2.2.7, calculate the distance between the index of each node and the optimal solution and the worst solution: (11) Step 2.2.8, calculate the relative closeness of each node: (12) Rank the importance according to the relative closeness, and select the top 70% of the nodes to construct an effective node set.

[0029] Step 3 specifically includes the best honeycomb network reconstruction method based on the hybrid discrete - continuous action representation model, the optimal collaborative mutual supply and mutual assistance strategy based on the approximate power flow graph neural network, and the optimal hybrid control strategy based on the HDCAR - TD3 algorithm, which are detailed below.

[0030] The best honeycomb network reconstruction method based on the hybrid discrete - continuous action representation model: Since the transition between states in the honeycomb network topology reconstruction depends entirely on the control decision and the current state, this process can be described as a Markov decision process. The basic Markov decision process can be described by a 5-tuple: (13) where, S is the state space, is the state of the honeycomb distribution network at time t and mainly includes topology structure information and node feature information.

[0031] A is the action space. The reinforcement learning agent executes action according to . Assuming a learning random policy , which is a state-based action probability distribution, the state value function and action value function under π can be described as follows: ; .

[0032] P is the state transition probability matrix. The agent observes the state t of the current environment at each moment , and simulates the state of the next moment's environment after being affected by this decision action to take an action .

[0033] R is the reward function, which is the goal that the reinforcement learning agent wants to achieve. In the present invention, the learning goal of the reinforcement learning agent is to achieve the best load balance of the honeycomb distribution network after the action.

[0034] γ is the discount factor, and its value range is from 0 to 1. The higher the γ value, the more the Markov decision process focuses on the long-term cumulative reward, and vice versa, it emphasizes more on the reward of the current behavior.

[0035] Since the reconstructed topology is represented by discrete actions and the load rate supply is represented by continuous actions, and there is a certain strong dependence between them, the action space of the Markov decision process is expanded. The expanded Markov decision process is called a parameterized action Markov decision process.

[0036] In the parameterized action Markov decision process, it is assumed that there is a finite set of discrete actions G , and each discrete action corresponds to a set of continuous actions respectively, such as Figure 3As shown, the Hybrid Discrete-Continuous Action Representation (HDCAR) model mainly consists of two processes: encoding and decoding. Before encoding, a latent space for discrete actions is constructed through a learnable embedding table Build G latent spaces for discrete actions, d 1 is the total dimension of the corresponding continuous vector, where each row of the embedding table represents G the one-dimensional continuous vector corresponding to the discrete action; then, an encoding operation is performed on the processed hybrid action latent space. Here, a conditional variational autoencoder is used, with the state and the embedding table as conditions, to map the continuous action parameters corresponding to the discrete action into the latent variable (14) The decoding process is expressed as: (15) (16) where, ξ are the embedding table parameters, is the encoding process, Φ are the encoding parameters, is the decoding process, ψ are the decoding parameters, is the predicted continuous action parameter, and represent the transformation network and the reconstruction fully connected layer, respectively.

[0037] The embedding table, the conditional variational autoencoder network, and their parameters are trained by minimizing the loss function: (17) (18) where the first term on the right side of Equation (17) is the L2 reconstruction loss, and the second term is the KL divergence between the variational posterior and the Gaussian prior of the latent representation variable ; the two together constitute the loss function of the conditional variational autoencoder; represents the mathematical expectation; D is the replay buffer, and samples are placed in the replay buffer in the form of , represents the continuous action parameter from the replay buffer.

[0038] The hybrid discrete-continuous action representation model uses a subnetwork after the decoder of the conditional variational autoencoder to predict the state residual. For any sample , its state residual and its predicted state residual can be defined as: (19) (20) Further use the following squared L2-norm prediction loss as the regularization term: (21) Combining Equation (18) and Equation (21), the total training loss function of the final hybrid action representation model is: (22) where ϑ is the weight corresponding to the regularization term.

[0039] Optimal collaborative mutual supply and mutual assistance strategy based on approximate power flow graph neural network: As mentioned above, the learning objective of the agent is to achieve the best load balancing rate after taking actions. At each moment, the immediate reward provided by the environment to the agent is: (23) where is the actual load rate of each regional grid substation in the honeycomb-shaped distribution network; is the average load rate of the honeycomb-shaped distribution network.

[0040] The topological information of the honeycomb-shaped distribution network can be represented by an undirected connected graph where V is the set of all nodes in the honeycomb-shaped distribution network, E is the set of connection lines between nodes. The adjacency matrix A represents the connection relationship between nodes. Since the adjacency matrix can reflect the physical topological information of the honeycomb-shaped distribution network, it is part of the input of the approximate power flow graph neural network (APFC-GNN).

[0041] In addition, the node feature matrix X and the edge feature matrix X E are another part of the input, where the node feature matrix contains the voltage magnitude feature, voltage magnitude feature, active power feature, reactive power feature, and load rate feature of each node, and the edge feature matrix contains the resistance and reactance features of each power line. The specific APFC-GNN structure is as shown in Figure 4 and consists of a masked encoder and approximate power flow convolutional layers: Mask Encoder: In traditional power flow calculations, the known and unknown features of different types of nodes are different. APFC-GNN needs to predict the values of unknown features while fixing the known features. Therefore, a binary mask encoder is introduced to represent the known and unknown features, enabling APFC-GNN to identify the feature values to be predicted. In the present invention, the known features are set to 0 and the unknown features are set to 1. The encoding mask layer is composed of two fully connected layer binary masks, and its mapping operation is: (24) (25) where and are weight matrices, and are biases, and they can all be trained through iteration; represents the activation function.

[0042] Combining the input node feature matrix X with the mask encoder M can obtain the encoded graph feature matrix of the k-th layer: (26) Approximate Power Flow Convolutional Layer: This layer consists of K interconnected approximate power flow convolutional layers. Its function is to iteratively process the effective graph features obtained through the mask encoder M in sequence, thereby predicting the node feature matrix X K after the final action. The output node feature matrix is similar to the input node feature matrix and contains the voltage amplitude feature, power feature, and load rate feature of each node.

[0043] Each approximate power flow convolutional layer utilizes the characteristic of message passing in the graph neural network that it can aggregate and update graph feature information by using the features of neighboring nodes and the graph structure. Each approximate power flow convolutional layer utilizes the characteristics of the message passing network in the graph neural network to aggregate and update the graph feature information by using the features of adjacent nodes and the graph structure. In the message passing stage, the iterative update process of each node can be described by the following formula: (27) (28) (29) where, t is the number of iterations in the message passing stage; is the neighbor node of node i ; is the node feature of the hidden layer, is the feature of the connection line between node i and node j, which can be updated through the transfer function and the node update function for update; is the node feature after message passing update in each iteration; is the feature vector after message passing, is the final node feature update function, is all sum.

[0044] These vectors are fed into a multi-layer perceptron (MLP) to obtain the messages transmitted to the nodes, where the ReLU activation function is used.

[0045] (30) where and are weight matrices, and are biases.

[0046] The graph feature matrix after message passing is combined with the encoded graph feature matrix and then sent to the topological adaptive graph convolution operator layer for processing to obtain a new encoded graph feature ; (31) where the matrix S k is the normalized adjacency matrix; is the trainable weight matrix; the hyperparameter D represents the convolutional layer considering adjacent nodes at a distance of D from the target node. D The larger the value of

[0047] To better evaluate the difference between the predicted unknown feature value of the model and the actual power flow attribute feature corresponding to the agent's action, the error of the unknown feature is used as the evaluation index.

[0048] (32) To verify the feasibility of the proposed APFC-GNN approximate power flow, a transformation topology is selected in this embodiment for example analysis. During the training process of the model, the AdamW optimizer is used, with a learning rate of 0.001 and a dropout rate of 0.2. In addition, five third-order approximate power flow convolutional layers are used, and the hidden layer dimension is 512. According to the data in Table 1, it can be clearly seen that the proposed APFC-GNN method has similar performance and higher accuracy compared with the traditional power flow calculation method. At the same time, the execution times of the two methods are 0.7073 s and 1.4946 s respectively, which indicates that the proposed approximate power flow graph neural network method is reduced by about half compared with the traditional method, meeting the real-time requirements of the hybrid control strategy.

[0049] Table 1 Training results of the approximate power flow graph neural network

[0050] Optimal hybrid control strategy based on the HDCAR-TD3 algorithm: Considering the complexity of the honeycomb-shaped distribution network, the twin-delayed deep deterministic policy gradient algorithm (TD3) is used to train the reinforcement learning agent. As Figure 2 shown in the lower left part, TD3 adopts a total of six network structures, where the Actor network and the Target-Actor network are used to train the policy network, while the Critic1 network, the Crtic2 network, the Target-Critic1 network, and the Target-Crtic2 network are used to estimate the action value function of the current state and action, that is, the Q value. The above six networks are approximated using neural networks, and their corresponding parameters are 、 、 、 、 and . The TD3 training process considering the HDCAR model is as follows: Training sample generation: During each policy iteration, the honeycomb-shaped distribution network generates different states , which are input into the Actor network to obtain actions . To ensure a certain degree of exploration, Gaussian noise is added to the actions to obtain ; then, is input into the environment to obtain the next state and the reward . The data generated during the interaction is called experience and stored in the replay buffer DIn it; Done is the final state of the environment, which has only two values: True and False. When the experience data generated by the interaction between the agent and the environment is a trajectory, Done = True means that the number of rounds is equal to the set number of rounds of the trajectory, that is, this interaction process ends, and it is necessary to return to the initial state. The agent reselects actions and collects new experiences; Done = False means that this piece of experience is in the trajectory process and the interaction process has not ended yet. Due to the HDCAR model, the potential policy is obtained in this process π ϕ The output potential action vector e and z x , use equations (15) and (16) to decode the latent variables to obtain the corresponding mixed action g and x g . Therefore, the form of the experience stored in the replay buffer is .

[0051] Training of the Critic network: The Critic network aims to approximate the action value function, also known as the Q function. Therefore, we use ω a parameterized neural network to approximate the mixed discrete-continuous action value function. Using only one Q network for training easily leads to an overestimation bias of the Q value. Therefore, the target value network and the clipped double Q-learning algorithm (CDQL) are used to solve this problem. In short, the double Critic network and take the potential action as the input to approximate the mixed discrete-continuous action value function . The corresponding Target-Critic network is and , which is used to estimate the target value y 1 and y 2 . During the training process, use , and the following formula is used to calculate the target value of the double Critic network: (33) (34) Due to the slow change of the Actor-critic strategy, the current network and the target network are too similar to perform independent estimation and there is little improvement. To separate action selection and Q-value update, thereby reducing the possibility of overestimation, it is therefore alternately used for action selection and value update. Specifically, when selecting an action from Q1, Q2 is used to update the Q-value, and similarly, when selecting an action from Q2, Q1 is used to update the Q-value. The policy represents the Target-Critic network used to estimate the target value.

[0052] To avoid introducing additional overestimation in the target update, the minimum value between the two estimated values is selected as the target update of CDQL y t : (35) By introducing the method of target policy smoothing to regularize the target value, it can be expressed as follows: (36) where, ϵ represents the clipped noise. Adding noise at the target value is equivalent to adding a regularization term in the Critic. c is the boundary of the Gaussian function of.

[0053] For the two Critic networks, their training objectives are both to regress to this learned target value y t , so the loss functions of the two networks can be obtained, and the two neural networks are trained using the gradient descent method: (37) (38) (39) where, is the learning rate of the network parameters; is the exponential smoothing parameter used to update the learning rate of the target network parameters .

[0054] Training of the Actor network: As mentioned above, the offline-trained Actor network needs to provide deterministic actions according to the environmental conditions. Since there are two Q networks in TD3 and the ultimate goal of the Actor network is to find the action that maximizes the Q-value, the Actor can freely choose one of the Q networks to maximize. Generally, the Actor makes Maximize the output. Therefore, a parametric neural network is used to approximate the policy function, and the Actor network is trained using gradient descent. The Target Actor network is also updated using an exponential smoothing parameter. The detailed formulas are as follows: (40) (41) (42) (43) Among them, M is the total number of training iterations; λ ϕ is the learning rate of the Actor network; η is the exponential smoothing parameter, which is the learning rate for updating the target Actor network and the target soft Actor network parameters .

[0055] The Actor network continuously corrects the policy according to the evaluation results of the Critic network. If there is a large error in the trained Critic network, the learning process of the Actor network will also be affected by the previous error. Therefore, a delayed policy update method is adopted in the TD3 algorithm. Specifically, before training the Actor network, multiple rounds of training of the Critic network of TD3 are required.

[0056] In this embodiment, the construction of the typical honeycomb-shaped distribution network and the collection of data samples in the upper left first part are completed through Pandapower and Simbetch, and the writing and testing of the proposed policy method are completed through Python. To verify the superiority of the proposed hybrid discrete-continuous action representation model, the proposed algorithm is compared with mainstream deep reinforcement learning algorithms as baselines, including algorithms based on DDPG and algorithms based on TD3. For fair comparison, all baseline algorithms have the same network structure as the proposed HDCAR-TD3 algorithm. For each baseline, the policy interacts with the environment 500,000 times, and 5 experiments are conducted simultaneously and the average results are shown. To test the optimization performance of the proposed hybrid discrete-continuous action representation model, the present invention introduces it into the DDPG algorithm, which is called HDCAR-DDPG. Figure 2 shows the training curves of the agent under different algorithms, where the solid line represents the average cumulative reward every 500 time steps, and the light shaded area represents the standard deviation of five trials. Figure 5

[0057] Figure 5 ​​As shown, the proposed HDCAR-TD3 algorithm exhibits a smoother training process with the least fluctuations, and the reward value gradually converges to around -1100, while the performance of other baselines deteriorates to varying degrees. During the training process, TD3 has a faster learning speed than HDCAR-TD3, but the reward value drops from approximately 250,000 times to around -1400. The learning speeds of DDPG and HDCAR-DDPG are slower than those of TD3 and HDCAR-TD3, and at the same time, they exhibit obvious reward fluctuations, making it difficult for the agent to converge to the optimal stable value. The reason is that both the TD3 and DDPG algorithms are deep reinforcement learning algorithms only applicable to solving problems in continuous action spaces, so their learning performance for the optimal policy in a hybrid discrete-continuous action space with strong dependencies is poor. However, after introducing the proposed hybrid discrete-continuous action representation model, both HDCAR-DDPG and HDCAR-TD3 show a certain degree of performance improvement, with reduced volatility, and the final convergent reward value is higher than that of the algorithms before the model was introduced, indicating the effectiveness of the proposed hybrid discrete-continuous action representation (HDCAR) model in solving hybrid discrete-continuous action problems.

[0058] Furthermore, the proposed HDCAR-TD3 algorithm is compared with existing mainstream deep reinforcement learning baselines for hybrid discrete-continuous action spaces (such as PDQN-TD3, HHQN-TD3, and HPPO) to evaluate its effectiveness in terms of training performance. As Figure 6 shown, the hybrid control strategy trained by HDCAR-TD3 is superior to the strategies trained using the other three mainstream deep reinforcement learning baselines for hybrid discrete-continuous action spaces, and the proposed method has the highest reward value and the lowest fluctuation value. The learning speed of HHQN-TD3 is faster than that of HDCAR-TD3, but the reward value drops from approximately 200,000 times. Except that PDQN-TD3 failed to train successfully, the other algorithms converged, but HPPO and HHQN-TD3 converged to a lower value of -1500, which means that they were unable to learn the optimal policy in the hybrid discrete-continuous action space with strong correlations, demonstrating the poor generality of these two algorithms. The above comparison results strongly prove the effectiveness and reliability of the method proposed in the present invention in a hybrid discrete-continuous action space with strong correlations.

[0059] In addition, the present invention compares the performance of the proposed HDCAR-TD3 algorithm with mainstream deep reinforcement learning algorithms (DRL) and advanced intelligent algorithms (ADIA), such as genetic algorithms (GA) and differential evolution algorithms (DE), through two specific cases. The present invention considers two cases of load imbalance, including the following situations.

[0060] Case 1: The load rate of the honeycomb-shaped distribution network grid S1 is lower than the normal range [60%, 75%]. Case 2: The load rate of the honeycomb-shaped distribution network grid S1 is higher than the normal range [60%, 75%].

[0061] Table 2 Optimal strategy solutions under different algorithms

[0062] Table 2 shows the optimal strategy solution results of various algorithms in two cases. The 0 and 1 in the optimal network reconstruction correspond to Figure 2 the on / off states of the SOP in the first part. The load rate supply quantity is the load rate supply quantity between the regional grids of the honeycomb-shaped distribution network under different discrete actions, where the positive and negative signs represent the supply / receive load rates respectively. As shown in bold in Table 2, the action strategy adopted by HDCAR-TD3 has the best improvement effect on the load imbalance degree of the honeycomb-shaped distribution network in two different cases. Advanced intelligent algorithms and other deep reinforcement learning baselines for the hybrid discrete-continuous action space can alleviate the load imbalance of the honeycomb-shaped distribution network to a certain extent quickly, but they do not adopt the optimal strategy, and their calculation time is too long to meet the real-time requirements of load balancing and regulation services. The above conclusions prove that the proposed HDCAR-TD3 algorithm has strong effectiveness and superiority in the hybrid control problem of the cellular horizontal architecture with a strongly correlated hybrid discrete-continuous action space.

[0063] Example 2: The computer-readable storage medium of this embodiment stores a computer program, and when the program is executed by a processor, it implements the steps in a mutual supply and mutual assistance hybrid control method based on the honeycomb-shaped horizontal topology architecture in Example 1.

[0064] The computer-readable storage medium of this embodiment can be the internal storage unit of the terminal, such as the hard disk or memory of the terminal; the computer-readable storage medium of this embodiment can also be the external storage device of the terminal, such as the plug-in hard disk, smart memory card, secure digital card, flash card, etc. equipped on the terminal; further, the computer-readable storage medium can also include both the internal storage unit and the external storage device of the terminal.

[0065] The computer-readable storage medium of this embodiment is used to store the computer program and other programs and data required by the terminal, and the computer-readable storage medium can also be used to temporarily store the data that has been output or will be output.

[0066] Embodiment 3: The computer device in this embodiment includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in a mutual supply and mutual assistance hybrid control method based on a honeycomb-shaped horizontal topology architecture in Embodiment 1.

[0067] In this embodiment, the processor may be a central processing unit, or may also be other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.; the memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A part of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0068] Those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disc, etc., and includes several instructions to enable a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0069] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, for those skilled in the art, they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A mutual supply and mutual assistance hybrid control method based on a honeycomb-shaped horizontal topology architecture, characterized in that, It includes the following steps: Step 1: Abstractly represent the horizontal topological structure of the honeycomb-shaped distribution network through graph theory. Each vertex represents the main substation of each regional grid in the honeycomb-shaped distribution network, and the regional grids are connected through flexible interconnection devices, expressed as , where V is the set of all nodes in the honeycomb-shaped distribution network, E is the set of connecting lines between nodes, A is the adjacency matrix representing the connection relationship of nodes; Step 2: Construct an effective node set through a screening method for a honeycomb-shaped effective loop topology based on complex network theory and entropy-TOPSIS; Step 3: Characterize the hybrid discrete-continuous action space model through a conditional variational autoencoder and an embedding table, and embed an approximate power flow graph neural network into the reward function of a parameterized action Markov decision process to achieve optimal collaborative mutual supply and mutual assistance of the honeycomb-shaped horizontal topology.

2. The mutual supply and mutual assistance hybrid control method based on a honeycomb-shaped horizontal topology architecture according to claim 1, characterized in that, Step 2 specifically includes the following steps: Step 2.1, select the following indicators to construct a screening system, including the degree centrality indicator DC , the closeness centrality indicator CC , the betweenness centrality indicator LBC and the electrical betweenness centrality indicator EBC ; Step 2.2: The screening process of the honeycomb-shaped effective topology based on entropy-TOPSIS, and the specific steps are as follows: Step 2.2.1, construct a multi-attribute decision matrix: ; Step 2.2.2, for X perform standardization and normalization: ； ； Obtain a matrix ; Step 2.2.3, calculate the information entropy of the j th index: ； Step 2.2.4, calculate the difference coefficient of each indicator , and calculate the indicator weight ; Step 2.2.5, construct the weighted normalized decision matrix ; Step 2.2.6: Calculate the positive ideal solution and the negative ideal solution; ； ； Step 2.2.7: Calculate the distances between the indicators of each node and the optimal solution and the worst solution; ； Step 2.2.8: Calculate the relative closeness of each node; ； Rank the importance according to the relative closeness, and select the top 70% of the nodes to construct an effective node set.

3. The mutual supply and mutual assistance hybrid control method based on a honeycomb-shaped horizontal topology architecture according to claim 1, characterized in that, In Step 3, the hybrid discrete-continuous action space model is characterized through a conditional variational autoencoder and an embedding table, specifically as follows: The hybrid discrete - continuous action space model consists of two processes: encoding and decoding. Before encoding, a latent space for discrete actions is constructed through a learnable embedding table Build G a latent space for discrete actions, where each row of the embedding table corresponds to a one - dimensional continuous vector, and the total dimension is d 1 ; Perform an encoding operation using a conditional variational autoencoder, with the state and the embedding table as conditions, map the continuous action parameters corresponding to the discrete action to the latent variables to construct a latent representation space. The encoding process is expressed as: ； The decoding process is expressed as: ； ； Among them, ξ is the embedding table parameter, is the encoding process, Φ is the encoding parameter, is the decoding process, ψ is the decoding parameter, is the predicted continuous action parameter, and respectively represent the conversion network and the reconstruction fully connected layer.

4. The mutual supply and mutual assistance hybrid control method based on a honeycomb-shaped horizontal topology architecture according to claim 3, characterized in that, The embedding table and the conditional variational autoencoder network and their parameters are trained through a minimized loss function: ； ； Among them, the first term on the right is the L2 reconstruction loss, and the second term is the KL divergence between the variational posterior of the latent representation variable and the Gaussian prior. The two together constitute the loss function of the conditional variational autoencoder; denotes the mathematical expectation; D is the replay buffer, representing the continuous action parameters from the replay buffer.

5. A mutual supply and mutual assistance hybrid control method based on a honeycomb-shaped horizontal topology architecture according to claim 4, characterized in that, The hybrid discrete - continuous action space model further uses a sub - network after the decoder of the conditional variational auto - encoder to predict the state residual, for any sample , its state residual and its predicted state residual are defined as: ； ； Use the squared L2 norm prediction loss as a regularization term: ； The total training loss function of the final hybrid action representation model is: ； Among them, ϑ is the weight corresponding to the regularization term.

6. A mutual supply and mutual assistance hybrid control method based on a honeycomb-shaped horizontal topology architecture according to claim 4, characterized in that, The approximate power flow graph neural network consists of a masked encoder and an approximate power flow convolutional layer. The binary masked encoder characterizes the known features and the unknown features, sets the known features to 0, and sets the unknown features to 1. The mapping operation of the masked encoder is: ； ； Among them and are weight matrices, and are biases, and are trained through iteration; represents an activation function; The input node feature matrix X is combined with the mask encoder M to obtain the encoded graph feature matrix of the k-th layer: ； The approximate current convolution layer has K layers, which iteratively process the effective graph features in sequence to predict the node feature matrix after the final action X K , and in the message passing stage, the iterative update process of each node is described by the following formula: ； ； ； Among them, t is the number of iterations in the message passing phase; is the i neighbor node of node is the node feature of the hidden layer, is the feature of the connection line between node i and node j, and is updated through the transfer function and the node update function ; is the node feature after the message passing update in each iteration; is the feature vector after the message passing; is the final node feature update function, is all sum; Feature vector is fed into a multi-layer perceptron to obtain a graph feature matrix after message passing , combined with the encoded graph feature matrix , and sent to the topological adaptive graph convolution operator layer for processing to obtain a new encoded graph feature : ； Among them, the matrix S k is the normalized adjacency matrix; is the trainable weight matrix; the hyperparameter D takes into account the convolutional layer of adjacent nodes at a distance D from the target node.

7. A mutual supply and mutual assistance hybrid control method based on a honeycomb-shaped horizontal topology architecture according to claim 3, characterized in that, Step 3 further uses the Twin Delayed Deep Deterministic Policy Gradient algorithm TD3 to train the reinforcement learning agent. TD3 adopts a total of six network structures. Among them, the Actor network and the Target-Actor network are used to train the policy network, and the Critic1 network, the Crtic2 network, the Target-Critic1 network, and the Target-Crtic2 network are used to estimate the action value function Q of the current state and action. Neural networks are used to approximate the six networks to obtain the corresponding parameters as , , , , and .

8. A mutual supply and mutual assistance hybrid control method based on a honeycomb-shaped horizontal topology architecture according to claim 7, characterized in that, The training process of TD3 is as follows: During each policy iteration, the honeycomb distribution network generates different states , which are input into the Actor network to obtain actions . Gaussian noise is added to the actions to obtain ; then is input into the environment to obtain the next state and the reward ; the latent policy π ϕ is obtained through the hybrid discrete - continuous action space model e output latent action vectors z x , and the latent variables are decoded using a conditional variational auto - encoder to obtain the corresponding hybrid actions g and x g . These data are stored in the replay buffer in the form of , Done is the final state of the environment; Critic1 network and Crtic2 network take the potential action as the input to approximate the mixed discrete - continuous action value function ; the corresponding Target - Critic1 network and Target - Crtic2 network are used to estimate the target value y 1 and y 2 ; during the training process, use , and adopt the following formula to calculate the target value: ； ； Strategy Indicates the Target-Critic network for estimating the target value; selects the minimum value between two estimated values as the target value y t : ； Regularize the target value by introducing a method of target policy smoothing: ； Among them, r t is the reward function, γ is the discount factor, with a value range of 0 to 1, ϵ is the clipped noise, where c is the boundary of the Gaussian function ; The training objectives of the Critic1 network and the Critic2 network are to regress to the learning target values y t , obtain the loss functions of the two networks, and use the gradient descent method to train the two neural networks: ； ； ； Among them, represents the mathematical expectation, is the learning rate of the network parameters; is the exponential smoothing parameter for updating the learning rate of the target network parameters ; The ultimate goal of the Actor network is to find the action that maximizes the Q value, approximate the policy function using a parameterized neural network, and train the Actor network using the gradient descent method. The Target Actor network is also updated using an exponential smoothing parameter. The detailed formulas are as follows: ； ； ； ； Among them, M is the total number of training iterations; λ ϕ is the learning rate of the Actor network; η is the exponential smoothing parameter for updating the learning rate of the target Actor network and the target soft Actor network parameters ; Before training the Actor network, multiple rounds of training of the Critic network of TD3 are required.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in a mutual supply and mutual assistance hybrid control method based on a honeycomb-shaped horizontal topology architecture as described in any one of claims 1-8.

10. A computer device, comprising a processor, a memory, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in a mutual supply and mutual assistance hybrid control method based on a honeycomb-shaped horizontal topology architecture as described in any one of claims 1-8.

Images