Optimization control method for ubiquitous heterogeneous network topology of ultra-high voltage converter station panoramic monitoring

Abstract

The application discloses an optimization control method for a ubiquitous heterogeneous network topology of panoramic monitoring of an extra-high voltage converter station, belongs to the technical field of panoramic monitoring of the extra-high voltage converter station, and solves the problem of congestion of panoramic monitoring data transmission caused by poor real-time performance and low reliability of the communication network reconstruction of the extra-high voltage converter station when a network fault occurs; a topology control algorithm based on deep reinforcement learning sequentially constructs a topology structure of the heterogeneous network; a framework combining deep reinforcement learning and Monte Carlo tree search is adopted to sequentially construct the network according to pre-defined topology rules; a deep convolutional neural network is trained to predict transmission flow of a partially established topology and guide the Monte Carlo tree to expand search in a more promising area in a search space; and the search result of the Monte Carlo tree strengthens the learning of the deep convolutional neural network so as to obtain more accurate prediction in the next iteration.

Description

Technical Field

[0001] This invention belongs to the field of panoramic monitoring technology for ultra-high voltage converter stations, and relates to a ubiquitous heterogeneous network topology optimization control method for panoramic monitoring of ultra-high voltage converter stations. Background Technology

[0002] Ultra-high voltage direct current (UHVDC) transmission features large transmission capacity, high voltage, and can be used for asynchronous power system interconnection, offering greater economic, technical, and environmental advantages compared to traditional high-voltage AC transmission systems. Due to the diverse types of power equipment and complex information structures within converter stations, the safe operation of the DC transmission system is closely related to the equipment status. Therefore, to ensure the safe and stable operation of UHVDC converter stations, comprehensive monitoring of a large number of power devices is necessary. The status signal parameters applicable to the core links of UHVDC protection that need to be monitored by the protection devices of UHV converter stations are as follows: A. Monitoring the status of the outlet pressure plate; B. Temperature measurement of the terminal blocks inside the cabinet; C. Monitoring the front panel of the secondary equipment inside the cabinet; D. Operating temperature of the secondary equipment inside the cabinet; E. Operating voltage of the secondary equipment inside the cabinet; F. Fiber optic light intensity monitoring; G. Cable insulation detection; H. Outlet circuit detection; I. Auxiliary contact position; J. Cable status detection; K. Detection of environmental parameters, such as temperature and humidity; L. Corrosion status of the terminals. However, different power equipment uses different data transmission methods, requiring multiple networks to coordinate transmission, resulting in a heterogeneous network for the converter station. Because the concurrent processing of various network communications and service data in the panoramic monitoring of UHVDC converter stations is diverse, an unreasonable heterogeneous network topology can lead to dynamic imbalances in data flow access, resulting in poor overall data transmission performance. In severe cases, this can cause data congestion and affect network reliability. Therefore, it is necessary to optimize the heterogeneous network topology to improve its structure and transmission performance.

[0003] In recent years, the topology optimization problem of heterogeneous networks has received widespread attention. The 2015 publication "Logical Topology Design of Multi-Interface Multi-Channel Wireless Mesh Networks" (Bao Xuecai et al., Small and Microcomputer Systems) proposed a logical topology design method under topology reliability constraints. This method uses topology reliability and network path hop count as constraints, with maximum capacity and minimum interference as optimization objectives. It integrates shortest path and minimum spanning tree algorithms into the disjoint path calculation process to obtain an optimized logical topology, but this only improves network robustness. The 2019 publication "Adaptive Protection and Self-Healing Control Method for Distribution Networks Based on Dynamic Topology Analysis" (Zhang Anlong, Power System Protection and Control) proposed a new adaptive distributed topology control algorithm. By adjusting the transmission capacity of nodes under different states, it ensures network connectivity during node failures. Regarding data transmission, tree topology can transmit collected data better than other network topologies and has strong anti-interference capabilities. The 2019 publication, "Improving the Capacity of a Mesh LoRa Network by Spreading-Factor-Based Network Clustering" (Zhu G, Liao CH, Sakdejayont T, et al. IEEE Access), proposes a tree-based algorithm with a set of heuristic rules for constructing tree topologies in multi-hop wireless networks. The advantage of tree topologies lies in their efficient data transmission and aggregation through non-leaf nodes. Throughput in heterogeneous networks is a primary criterion for evaluating the merits of existing network models. Currently, many network topologies have been constructed for good data transmission, such as cluster-based and tree-based topologies. The performance of these topologies in heterogeneous networks demonstrates that topology quality significantly impacts data transmission. Therefore, network topology construction should be based on specific network requirements, accommodating various network types and transmission performance as much as possible.

[0004] The aforementioned network topology optimization methods consider the quality and security of data transmission, but the optimization process is time-consuming when the topology changes, making it difficult to meet the performance requirements of power industry data transmission networks. Finding the optimal reliability topology in the heterogeneous network of an UHVDC transmission system is essentially a combinatorial problem. The 2018 publication "Research on Minimum Spanning Tree Algorithm for Ventilation Networks Based on Weight Matrix" (Tu Peng et al., Journal of Railway Science and Engineering) proposes a minimum spanning tree topology optimization method. This method uses heuristic rules to reduce the number of candidate searches, thus obtaining a suboptimal solution to some extent. However, it still falls short of meeting the real-time and reliability requirements for rapid reconfiguration of the communication network when a UHVDC converter station experiences a network failure. Furthermore, because the search space for all possible topology configurations is extremely large, the complexity of achieving the optimal network configuration through exhaustive search increases exponentially. Summary of the Invention

[0005] The purpose of this invention is to design a ubiquitous heterogeneous network topology optimization control method for panoramic monitoring of UHV converter stations, so as to solve the problem of poor real-time performance and low reliability of panoramic monitoring data transmission caused by the reconstruction of communication networks when UHV converter stations experience network failures.

[0006] The present invention solves the above-mentioned technical problems through the following technical solutions:

[0007] A ubiquitous heterogeneous network topology optimization control method for panoramic monitoring of ultra-high voltage converter stations includes the following steps:

[0008] S1. Model the heterogeneous network of the UHV converter station as a tree structure, wherein the tree structure has a master station v0 and N-1 data transmission nodes {v1, v2, ..., v...} N-1 Each data transmission node has a unique path to the master station v0;

[0009] S2. Using the main station v0 as the root node of the tree structure, and recursively searching the Monte Carlo tree for each state with the root node as the initial state to obtain the training dataset.

[0010] S3. Input the training dataset obtained from the search into the deep convolutional neural network for training to obtain the value function and policy function, which are used to guide the Monte Carlo tree to recursively search for states with expected rewards and in turn update the training dataset collected by the deep convolutional neural network.

[0011] S4. After training is complete, starting from the initial state s0 = 0, select a sequentially from the strategies predicted by the deep convolutional neural network. t ~π(s) t The action at point ) and the update of state s t+1 =T(s) t at This process continues until a complete tree is reached, thus obtaining the heterogeneous network topology.

[0012] The method of this invention is based on a deep reinforcement learning-based topology control algorithm to sequentially construct the topology of heterogeneous networks. DRL-TC employs a framework combining deep reinforcement learning and Monte Carlo tree search, constructing the network sequentially according to predefined topology rules. A deep convolutional neural network is trained to predict the transmission traffic of a partially constructed topology and guides MCTS to search more promising regions in the search space. The search results of MCTS enhance the learning of the deep convolutional neural network, enabling more accurate predictions in the next iteration. This invention can determine the near-optimal topology of heterogeneous networks from a reliability perspective without relying on domain-specific knowledge beyond the topology rules, adapting to dynamic environmental changes and reconfiguring the network accordingly.

[0013] Furthermore, the method for modeling the heterogeneous network of the UHV converter station as a tree structure in step S1 is as follows: In each round of data acquisition, node v i Will The bit data is forwarded to its parent node, where i∈{1,2,...,N-1}; It is made by v i Self-generated data, data set It comes from v i The child nodes, the a() function is an aggregation function; a transport model is adopted. Transmission traffic, in the transmission model, the transmission traffic of nodes related to the topology consists of two parts: data processing and transmission time. and They are at node v i The time consumption for processing each bit of data and the time consumption for transmitting each bit of data are specified. The time consumption for transmitting each bit of data depends on the distance to the parent node, and its calculation formula is as follows: in, It is node v i Its parent node v j The Euclidean distance between them, ρ is the power amplification constant in the link budget that takes into account the effects of shadow fading.

[0014] Furthermore, the Monte Carlo tree recursive search method described in step S2 is as follows: each node in the Monte Carlo tree represents a 5-tuple of data (s, a, M(s, a), π(s), Q). π (s, a)); At each search step t < N, select the action that maximizes the confidence upper bound. When the search reaches the terminal state t = N, obtain the reward and propagate back along the search path to the root state of all visited states and the actions taken, Q on the path. πThe value is updated accordingly by the average value on the node; where s is the state of the heterogeneous network; a is the action in that state; M(s, a) is the total number of visits to (s, a) on the search tree; π(s) is the prior probability of the effective action predicted by the deep convolutional neural network; Q π (s, a) is the state-action value, representing the expected reward for starting from state s and taking action a.

[0015] Furthermore, the calculation formula for the action of maximizing the confidence upper limit is as follows: in, is the access count of state s, without considering actions, and c is a hyperparameter that controls the search level.

[0016] Furthermore, the deep convolutional neural network described in step S3 includes a deep Vgg16 module, a fully connected layer with softmax for the policy, and a fully connected layer with ReLU activation for the value function; the deep Vgg16 module consists of two convolutional layers with 64 convolutional filters, two convolutional layers with 128 convolutional filters, three convolutional layers with 256 convolutional filters, and six convolutional layers with 512 convolutional filters, each convolutional filter having a 3×3 kernel and a max pooling layer.

[0017] Furthermore, the value function described in step S3 satisfies the Bellman equation, indicating that the value of the current state is the reward of that state plus the expected reward of the next state. The formula for the value function is:

[0018] Furthermore, the strategy function formula mentioned in step S3 is as follows:

[0019] The advantages of this invention are:

[0020] The method of this invention is based on a deep reinforcement learning-based topology control algorithm to sequentially construct the topology of heterogeneous networks. DRL-TC employs a framework combining deep reinforcement learning and Monte Carlo tree search, constructing the network sequentially according to predefined topology rules. A deep convolutional neural network is trained to predict the transmission traffic of a partially constructed topology and guides MCTS to search more promising regions in the search space. The search results of MCTS enhance the learning of the deep convolutional neural network, enabling more accurate predictions in the next iteration. This invention can determine the near-optimal topology of heterogeneous networks from a reliability perspective without relying on domain-specific knowledge beyond the topology rules, adapting to dynamic environmental changes and reconfiguring the network accordingly. Attached Figure Description

[0021] Figure 1 This is a heterogeneous network model according to an embodiment of the present invention;

[0022] Figure 2 This is a heterogeneous network tree structure according to an embodiment of the present invention;

[0023] Figure 3 This is a description of two steps in a finite-time Markov decision process according to an embodiment of the present invention;

[0024] Figure 4 This describes the Monte Carlo tree search process according to an embodiment of the present invention;

[0025] Figure 5 This is the structure of a deep convolutional neural network according to an embodiment of the present invention;

[0026] Figure 6 This describes the convergence and performance of the DRL-TC algorithm proposed in this embodiment of the invention.

[0027] Figure 7 This is an evolution of the training process in the embodiments of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments:

[0030] Example 1

[0031] 1. Establish a heterogeneous network model for the UHV converter station.

[0032] To ensure the stable operation of ultra-high voltage converter stations, comprehensive monitoring of numerous devices within the station is necessary. However, different devices use different networks for data transmission, resulting in a heterogeneous network. Figure 1 As shown. To address the dynamic imbalance in data flow access caused by unreasonable heterogeneous network topology connections, it is necessary to optimize the heterogeneous network topology to meet network communication performance requirements.

[0033] Table 1. Symbols used in the network model

[0034]

[0035] This embodiment takes a ±1100 kV converter station as an example, targeting... Figure 1 The heterogeneous network model shown depicts the heterogeneous network as a tree structure, such as... Figure 2 As shown, this structure has one master station v0 and N-1 data transmission nodes {v1, v2, ..., v...} N-1}, where each node has a unique path to the main station v0. V = {v0, v1, ..., v} N-1} is the set of all vertices, and E is the set of directed edges.

[0036] In each round of data collection, node v i Need to The bit data is forwarded to its parent node. Calculated using formula (1):

[0037]

[0038] in, It is made by v i Self-generated data, data set It comes from v i The child nodes, the a() function is an aggregate function, i∈{1,2,...,N-1}.

[0039] This embodiment uses the transmission model shown in formula (2) to transmit traffic, where the topology-related node transmission traffic mainly consists of two parts: data processing (including data reception) and transmission time. The model is shown in formula (2):

[0040]

[0041] in, and They are at node v i The transmission time per bit for data processing and transmission is calculated. The transmission time per bit depends on the distance to the parent node and is further modeled as shown in Equation (3):

[0042]

[0043] in, It is node v i Its parent node (or main site) v j The Euclidean distance between them, ρ is the power amplification constant in the link budget that takes into account the effects of shadow fading.

[0044] To apply reinforcement learning in this embodiment, an effective tree structure is first constructed with the master station v0 as the root node. In each step, a node that is not yet connected is selected and connected to a node in the tree or the master station, until all nodes are connected. Figure 3 As shown, this process can be described by a finite-domain Markov decision process (MDP) with fully observable 4-tuples {S, A, T, R}. At each step t∈[0, N], the state s of the system is... t ∈S is the current adjacency matrix of the network. In a t The action at point A is to choose which node to connect to in the tree, or equivalently... Where node v i To connect to node v in the tree j (or the main station). Then, the system evolves to the next state s. t+1 In this case, there is a deterministic transition matrix T(s, a). Upon reaching the terminal state s... N Before all nodes are connected to the tree, the reward at step t is uncertain. Then the lifetime of the heterogeneous network is determined. Return as a reward for each action along the state trajectory.

[0045] The energy efficiency topology optimization framework proposed in this invention follows the following general settings:

[0046] (1) Node v i generated The data size is a random number extracted from a specific distribution in the DRL-TC algorithm;

[0047] (2) The aggregation function a() can be any deterministic function. In this invention, the summation method is used.

[0048] (3) The designed topology control algorithm should be applicable to other network objectives, such as minimizing the overall network time consumption or maximizing the network throughput.

[0049] Will be able to node v i The total traffic transmitted at that location is expressed as Since it is assumed that the main site v0 has no restrictions, therefore This invention defines the lifetime of a heterogeneous network as the minimum transmission traffic of all nodes based on the total number of transmission rounds. This maximization of the lifetime of a heterogeneous network can be expressed as:

[0050]

[0051]

[0052]

[0053]

[0054] Where δ(S) is the set of edges. If v iIt is v j A subset of v, then v i =1, otherwise 0. Constraint (4b) ensures all nodes are connected, and constraint (4c) ensures each node can only transmit to one parent node at a time. To approximate the complexity of the problem, if the topology is considered as an undirected spanning tree, then according to Cayley's formula, the number of all possible spanning trees in the network is N. N-2 While heuristic rules can reduce the number of search candidates, enumerating all possible solutions remains infeasible for reasonable N values. This invention proposes a real-time DRL-TC algorithm that focuses on more promising regions in the search space with limited computational resources and approaches the optimal solution with increased computational power.

[0055] 2. Topology optimization algorithm based on deep reinforcement learning

[0056] 2.1 Reinforcement Learning

[0057] Reinforcement learning teaches the agent to take actions in a dynamic environment to maximize reward signals. In step t, the agent performs actions in the environment and receives an immediate reward r. t Receive observations of the environmental state. The action to be taken is determined by a strategy. The strategy can be dependent on s t It can be a set of deterministic actions, or it can be a random strategy using a set of action probabilities.

[0058] Reward r t The agent is told how much the current environmental state necessitates the objective; this is given by a reward function, which may depend on s. t a t and s t+1 When the goal is achieved, it produces a high value; otherwise, it produces a low value. A series of states and actions is called trajectory motion τ, and the sum of discounts on all reward values ​​collected along a trajectory is called the reward, as shown in formula (5):

[0059]

[0060] Here, γ is a discount factor that reduces the value of future rewards. When γ < 1, the reward obtained now is more valuable than the reward obtained later. The return can be a finite-level return collected over the maximum number of time steps, and γ = 1 can be used if needed. Alternatively, the reward can be an unlimited, infinite-level return, in which case γ < 1 is required.

[0061] The value function satisfies the Bellman equation, which states that the value of the current state is the reward of that state plus the expected reward of the next state. The value function and policy function are shown in equations (6) and (7):

[0062]

[0063]

[0064] The main problem in reinforcement learning is finding a strategy that maximizes this expected reward, and its algorithms typically use approximation functions.

[0065] 2.2 Monte Carlo Tree Search

[0066] This embodiment uses the aforementioned deep convolutional neural network to approximate the policy function and value function. The deep convolutional neural network requires a training dataset of states, policies, and values ​​to fit the deep convolutional neural network as a function approximator. One approach is to enumerate and collect all states and their values ​​as the training dataset. However, this approach becomes infeasible when the state space is large, as it overfits the deep convolutional neural network. Instead of using heuristics to reduce the number of search candidates, this embodiment uses MCTS to efficiently collect the training dataset in more promising regions of the search space. Each node on the search tree represents a 5-tuple of data (s, a, M(s, a), π(s), Q). π Let M(s, a) be the state of the heterogeneous network, a be the action in that state, M(s, a) be the total number of visits to (s, a) on the search tree, π(s) be the prior probability of the effective action predicted by the deep convolutional neural network, and Q be the probability of the action. π (s, a) is the state-action value, defined as the expected reward of starting from state s and taking action a, calculated using formula (6). At each search step t < N, the action that maximizes the upper confidence limit (UCB) is selected, as shown in formula (8):

[0067]

[0068] in, Let be the visit count of state s, disregarding actions, and c be a hyperparameter controlling the search level. Intuitively, this selection policy initially favors actions with higher prior probabilities π, but asymptotically favors actions with higher state-action values ​​Q. π The action. When the search reaches the terminating state (i.e., t=N), a reward is obtained and the search propagates back along the search path to the root state of all visited states and the actions taken, Q on the path. π The values ​​are updated accordingly by the average value on the nodes. The details of MCTS are described in Algorithm 1, as shown in Table 2.

[0069] Table 2 shows the MCTS subroutine of the proposed DRL-TC algorithm.

[0070]

[0071]

[0072] Each search begins in a specific state and recursively searches for the next state until a new leaf state or terminal state is reached. By performing multiple MCTSs in each state, the posterior visit counts M(s) are collected as part of the training dataset used to update the deep convolutional neural network in the next iteration.

[0073] 2.3 Deep Convolutional Neural Networks

[0074] A stochastic policy π(s) defines the distribution of effective actions in a state, under which the system moves from state s. t Until terminal state s N Generate state and action trajectory h(s) t )={s t a t , ..., s N-1 a N-1 s N Value function V π (s) is defined as the expected reward for all possible trajectories starting from state s, and is calculated by formula (7).

[0075] This embodiment uses a deep convolutional neural network to approximate the policy function and value function f. Θ (s) (parameterized by Θ) approximates the optimal value function V * (s)=max π V π (s) and the optimal strategy π * (s). For example... Figure 5 As shown, the input to the deep convolutional neural network is the training dataset {(s, π(s), V}. π To maintain the feasibility of training multi-layer neural networks while significantly improving the representational power of deep convolutional neural networks, this embodiment employs a deep Vgg16 module. This module consists of two convolutional layers with 64 convolutional filters, two convolutional layers with 128 convolutional filters, three convolutional layers with 256 convolutional filters, and six convolutional layers with 512 convolutional filters. Each convolutional filter has a 3×3 kernel, followed by a max-pooling layer. Then, the deep convolutional neural network is divided into two branches of convolutional layers, followed by fully connected layers with softmax and ReLU activations for policy and value functions, respectively. The deep convolutional neural network (π(s), V π (s))=f Θ (s) The policy and value of each predicted state contain prior information that guides MCTS to search for states with high rewards and in turn collects the training dataset for deep convolutional neural networks.

[0076] Once the deep convolutional neural network (π(s)) is trained, V π (s))=f Θ (s) In order to obtain the tree-like topology of the heterogeneous network, this embodiment starts from the root state s0 = 0, and then sequentially selects a from the strategies predicted by the deep convolutional neural network. t ~π(s) t The action at point ) and the update of state s t+1 =T(s) t a t This process continues until a complete tree is reached. This embodiment notes that this topology construction is a stochastic process, and once the deep convolutional neural network has been trained for a sufficient number of iterations, it will converge to a solution.

[0077] 2.4 Self-configured DRL-TC algorithm

[0078] The self-configuration and self-optimization characteristics are known as SON (Self-Organizing Network), which better adapts to the flattening and flexibility of network structures, and therefore have attracted widespread attention. In short, the DRL-TC proposed in this embodiment alternates between training a deep convolutional neural network (DCNN) and MCTS (Multi-Channel Search Theory), where the DCNN provides a prior policy to guide the MCTS, and then the MCTS returns posterior visit counts and state values ​​to update the DCNN. In this way, with limited computational resources, the proposed DRL-TC algorithm will focus more on promising search regions and converge to a solution with higher rewards.

[0079] The proposed DRL-TC algorithm can also adapt to dynamic changes in the environment. For example, when nodes are suddenly added or removed, topology rules may make some actions effective or ineffective. In a new run of MCTS, the policy π returned by the deep convolutional neural network for this state will be renormalized for all effective actions. Therefore, the new prior policy π(s) reflects the changes in the network but is still related to historical data. MCTS will collect a new training dataset and use it to update the deep convolutional neural network. Assuming that network changes are slower than training time (depending on available computational resources), the proposed DRL-TC algorithm can track the dynamic changes in the network and reconfigure the network topology accordingly. Algorithm 2 describes the complete algorithm of the proposed DRL-TC, as shown in Table 3.

[0080] Table 3 proposes the DRL-TC algorithm.

[0081]

[0082] The complete algorithm of DRL-TC:

[0083] Step 1: Model the heterogeneous network as a tree structure, which has a master station v0 and N-1 data transmission nodes {v1, v2, ..., v...}. N-1 Each node has a unique path to the main station v0;

[0084] Step 2: Initialize the dataset Define N i N represents the number of iterations. e N represents the number of episodes. m Indicates the number of tree searches;

[0085] Step 3: Initialize i = 1, i ∈ [1, N] i ];

[0086] Step 4, let s i，0 =0 indicates that the i-th iteration starts from the root state;

[0087] Step 4.1: Initialize e = 1, e ∈ [1, N] e ], let the access count

[0088] Step 4.2: Initialize m = 1, m ∈ [1, N] m ];

[0089] Step 4.3: Check the current system state s i,t Perform the MCTS recursive search described in step 5 to obtain the training dataset, t∈[0,N];

[0090] Step 4.3: Check the current system state s i,t Perform the MCTS recursive search described in step 5 to obtain the training dataset, t∈[0,N];

[0091] Step 5: If t = N, it means s i,t If the terminal state is reached, proceed to step 5.1; if M(s) i,t If ) = 0, it means that the state has not been visited, proceed to step 5.2; otherwise, proceed to step 5.3;

[0092] Step 5.1: Obtain reward r and proceed to step 5.6;

[0093] Step 5.2: Use a deep convolutional neural network to obtain the predicted policy and value for each state, and obtain the state s. i,t For all valid actions, renormalize π(s) for all valid actions;

[0094] Step 5.2.1, let Represents the state s in the i-th iteration i,t Visited once;

[0095] Step 5.2.2: Calculate the value function using the following formula, and return to step 4.3 to continue execution;

[0096]

[0097] Wherein, the value function V π (s) represents the state s i,t The expected reward for all possible initial trajectories;

[0098] Step 5.3: Initialize the confidence upper limit

[0099] Step 5.3.1: Calculate the upper confidence limit for all valid actions obtained in Step 5.2 using the following formula;

[0100]

[0101] Among them, U(s) i,t ,a) represents s i,t The upper confidence limit of action a in a state, Q π (s i,t a) is the state-action value, c is the hyperparameter of the control level, and M(s) i,t a) is the choice s i,t The number of times action 'a' is performed in a given state;

[0102] Step 5.4: Select action a from step 5.3.1 that maximizes the upper confidence limit. t ←argmax a U(s i,t ,a), and in the next state s i,t+1 ←Τ(s i,t a) Perform the recursive search described in step 5, where T is the state transition matrix;

[0103] Step 5.5: Update the tree state as follows;

[0104]

[0105] M(s i,t ,a)←M(s i,t ,a)+1

[0106] M(s i,t )←M(s i,t )+1

[0107] Step 5.6: Let m = m + 1. If m = N m +1, continue to step 6; otherwise, return to step 4.3;

[0108] Step 6: Normalize the access count obtained from the recursive search in Step 5, and then normalize the resulting M(s). i,t ) and state s i,t Sum function V π (s) are recorded in dataset E;

[0109] Step 7, if s i,t For the terminal state, obtain the reward r, and update the value function of all states in e using r; otherwise, select the action with the most visits in that state, and transform to the next state s using the state transition matrix T. i,t+1 ;

[0110] Step 8: Let e ​​= e + 1. If e = N e +1, proceed to step 9; otherwise, return to step 4.1 and continue execution.

[0111] Step 9: Shuffle the data E and input it into the deep convolutional neural network described in step 5.2 for training, where the batch size is B and the learning rate is α, to obtain the training result policy function and value function;

[0112] Step 10: Starting from the root state s0 = 0, sequentially select a from the prediction strategies of the deep convolutional neural network trained in Step 9. t ~π(s) t The action at point ) and the update of state s t+1 =T(s) t ,a t This process continues until a complete tree is reached, resulting in the topology of the i-th iteration, where t∈[0,N];

[0113] Step 11: Let i = i + 1. If i = N i +1, iteration ends; otherwise, return to step 4 to continue iteration.

[0114] 3. Simulation Results and Analysis

[0115] 3.1 Simulation Settings

[0116] To evaluate the performance of the DRL-TC algorithm, simulation tests were conducted on a heterogeneous network of a ±1100 kV converter station. This heterogeneous network consists of a master node and 12 nodes distributed within a circular area with a radius of 1000 m, uniformly generating 500 to 1000 bits of sensing data in each transmission round. This embodiment assumes that all nodes have sufficient time to transmit data in each round. The data transmission throughput of each unit of all nodes is set to... The power amplification constant is set to ρ = 1.

[0117] In each iteration of the algorithm, from each state N m=Collect N in MCTS with 100 searches e =10 training samples. Batch size B=16, learning rate α=10 -6 This embodiment uses the ADAM optimizer to train a deep convolutional neural network. After each training iteration, 100 network topologies are constructed using the deep convolutional neural network, and the results are averaged to evaluate the performance of the algorithm.

[0118] 3.2 Results Analysis

[0119] First, this embodiment demonstrates the convergence and performance of the proposed DRL-TC algorithm. Figure 6 The solid lines in the table represent the network latency of the deep convolutional neural network after each training iteration, with the algorithm converging after approximately 50 iterations. Table 4 compares the performance of the proposed DRL-TC algorithm with three heuristics: star topology (all nodes are connected to the master station), random topology (each node randomly selects a node to connect to), and minimum spanning tree (MST) topology, where MST is weighted by the Euclidean distance between nodes. The star topology exhibits the longest network latency due to higher transmission traffic at edge nodes far from the master station. The random topology shows a shorter average network latency, but the differences are significant. The MST topology further reduces network latency by shortening the overall transmission distance. The proposed DRL-TC algorithm in this embodiment significantly outperforms these heuristics, and the algorithm has a short convergence time.

[0120] Table 4 Performance Comparison of DRL-TC Algorithm and Three Heuristic Methods

[0121]

[0122] Figure 7 The proposed DRL-TC demonstrates its ability to adapt to sudden changes in heterogeneous networks, showing the average network latency after each training iteration. Figure 7 Points A to D on the curve show the 100 topological superpositions given by the DRL-TC algorithm after the 1st, 62nd, 63rd, and 100th iterations. Point A indicates that in the first iteration, DRL-TC randomly explores the search space. Because the deep convolutional neural network has no prior information about the state values, the network latency is relatively high. Point B indicates that the network gradually converges after multiple iterations. Point C indicates that it can quickly adapt when the heterogeneous network structure changes. Point D indicates that the algorithm converges to the optimal solution after 100 iterations.

[0123] This invention proposes a novel and unified heterogeneous network topology optimization algorithm based on deep reinforcement learning. The proposed DRL-TC algorithm can adapt to environmental changes and exhibits significantly better data transmission performance than other heuristic algorithms, demonstrating excellent adaptability to network topology changes and enhancing network reliability. The DRL-MCTS framework has great potential in heterogeneous networks, enabling online training without intervening in network services. Furthermore, with the continuous improvement of computing power, this invention anticipates that in the 5G era, DRL-MCTS will see other promising topology control applications in self-organizing and fully automated IoT networks.

[0124] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A ubiquitous heterogeneous network topology optimization control method for panoramic monitoring of ultra-high voltage converter stations, characterized in that, Includes the following steps: S1. Model the heterogeneous network of the UHV converter station as a tree structure, wherein the tree structure has a master station v0 and N-1 data transmission nodes {v1, v2, ..., v...} N-1 Each data transmission node has a unique path to the master station v0; S2. Using the main station v0 as the root node of the tree structure, and recursively searching the Monte Carlo tree for each state with the root node as the initial state to obtain the training dataset. S3. Input the training dataset obtained from the search into the deep convolutional neural network for training to obtain the value function and policy function, which are used to guide the Monte Carlo tree to recursively search for states with expected rewards and in turn update the training dataset collected by the deep convolutional neural network. S4. After training is complete, starting from the initial state s0 = 0, select a sequentially from the strategies predicted by the deep convolutional neural network. t ~π(s) t The action at point ) and the update of state s t+1 =T(s) t ,a t This process continues until a complete tree is reached, thus obtaining the heterogeneous network topology.

2. The ubiquitous heterogeneous network topology optimization control method for panoramic monitoring of UHV converter stations according to claim 1, characterized in that, The method for modeling the heterogeneous network of the UHV converter station as a tree structure in step S1 is as follows: In each round of data acquisition, node v i Will Bit data is forwarded to its parent node, where i∈{1,2,...,N-1}; It is made by v i Self-generated data, data set It comes from v i The child nodes, the a() function is an aggregation function; a transport model is adopted. Transmission traffic, in the transmission model, the transmission traffic of nodes related to the topology consists of two parts: data processing and transmission time. and They are at node v i The time consumption for processing each bit of data and the time consumption for transmitting each bit of data are specified. The time consumption for transmitting each bit of data depends on the distance to the parent node, and its calculation formula is as follows: in, It is node v i Its parent node v j The Euclidean distance between them, ρ is the power amplification constant in the link budget that takes into account the effects of shadow fading.

3. The ubiquitous heterogeneous network topology optimization control method for panoramic monitoring of UHV converter stations according to claim 1, characterized in that, The method of Monte Carlo tree recursive search described in step S2 is as follows: Each node on the Monte Carlo tree represents 5-tuple data (s, a, M(s, a), π(s), Q π (s, a)); At each search step t < N, select the action that maximizes the upper confidence bound. When the search reaches the termination state t = N, obtain the reward and propagate it back along the search path to the root state of all visited states and the actions taken. The Q π value is correspondingly updated by the average value on the node; where s is the state of the heterogeneous network; a is the action in this state; M(s, a) is the total number of visits to (s, a) on the search tree; π(s) is the prior probability of the effective action predicted by the deep convolutional neural network; Q π (s, a) is the state-action value, representing the expected reward starting from state s and taking action a.

4. The ubiquitous heterogeneous network topology optimization control method for panoramic monitoring of UHV converter stations according to claim 3, characterized in that, The formula for calculating the action of maximizing the confidence upper limit is as follows: in, is the access count of state s, without considering actions, and c is a hyperparameter that controls the search level.

5. The ubiquitous heterogeneous network topology optimization control method for panoramic monitoring of UHV converter stations according to claim 1, characterized in that, The deep convolutional neural network described in step S3 includes a deep Vgg16 module, a fully connected layer with softmax for the policy, and a fully connected layer with ReLU activation for the value function; the deep Vgg16 module consists of two convolutional layers with 64 convolutional filters, two convolutional layers with 128 convolutional filters, three convolutional layers with 256 convolutional filters, and six convolutional layers with 512 convolutional filters, each convolutional filter having a 3×3 kernel and a max pooling layer.

6. The ubiquitous heterogeneous network topology optimization control method for panoramic monitoring of UHV converter stations according to claim 1, characterized in that, The value function described in step S3 satisfies the Bellman equation, meaning that the value of the current state is the reward of that state plus the expected reward of the next state. The formula for the value function is:

7. The ubiquitous heterogeneous network topology optimization control method for panoramic monitoring of UHV converter stations according to claim 1, characterized in that, The policy function formula mentioned in step S3 is:

Images