Distributed test network topology optimization method based on deep reinforcement learning

Abstract

The invention provides a distributed test network topology optimization method based on deep reinforcement learning. The method comprises the following steps: constructing a graph data structure; performing initialization processing; configuring reward calculation selection parameters and initialization action and state selection parameters; a first algorithm and a second algorithm are sequentially carried out, the accuracy rate is output, if the accuracy rate is larger than a preset accuracy rate value, the reward calculation selection parameter is represented as the first algorithm, otherwise, the reward calculation selection parameter is represented as the second algorithm, and the process is repeated. According to the method, deep reinforcement learning is used for replacing manual work, and deployment scheme optimization is carried out on a node network of a large-scale test. According to the method, labor can be effectively saved, the method can adapt to field conditions to quickly perform recalculation, and condition changes of a small number of nodes in large-scale calculation can be optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed node networks, and in particular to a distributed test network topology optimization method based on deep reinforcement learning. Background Art

[0002] In a distributed node network, the network topology affects important performance indicators, including link utilization, throughput, and latency. In large-scale distributed simulation experiments across regions, the network transmission conditions between different distributed nodes vary greatly. In order to improve the performance of the entire test system, a more complete test node deployment plan needs to be given. In practice, manual optimization methods are usually used, which cannot guarantee the optimal solution. This application proposes a method for solving the problem of cross-domain joint test node deployment using a deep reinforcement learning (DRL) algorithm, and uses an A2C algorithm to find the optimal solution for the node deployment plan.

[0003] In industrial simulation technology, using a single device for simulation tests can no longer meet the needs. At present, simulation of a certain system has become an important research direction in computer simulation, and is applied to fields such as electricity and aerospace. In order to conduct complete and sufficient simulation tests on a certain system, a distributed simulation system is usually used. The simulation platform is built through a distributed system, and the devices to be tested are connected to achieve comprehensive and realistic simulation tests on a system composed of multiple targets. In distributed simulation platform technology, due to the different communication forms of various devices involved in the test, there is heterogeneity. Experimenters usually use nodes as transfer stations to re-encode the information sent by different types of devices and forward it to the devices to be received. In actual applications, the devices participating in the test often come from different regions, and the actual distance is large. There is a phenomenon of cross-regional testing, which makes the transmission delay between nodes large. The node deployment plan in the distributed network will directly affect the performance of the entire cross-domain joint test system. In order to minimize the transmission delay and reduce the load pressure of a single node, it is necessary to optimize the network topology according to the data transmission requirements and adopt a reasonable node deployment plan.

[0004] In current technical applications, the node deployment selection of distributed test networks is manually completed by test personnel with the help of prior knowledge. For small-scale tests in a single region, the characteristics of low latency and low concurrency make the impact of the test network environment on the results negligible.

[0005] As the scale of the test increases and the number of devices involved in the test increases, the number of nodes required for the test and the data interaction relationship between the nodes will expand rapidly, which makes the distributed network highly complex. This leads to two problems. First, since the test facilities are located in different regions, there will be inevitable network delays between links. If the accumulated delays are too high, it will affect the correctness of the test results. Second, as the number of nodes expands, a single node is prone to high data concurrency, and the data traffic is too large to exceed the forwarding capacity, resulting in data loss or even node downtime. Therefore, in large-scale distributed simulation experiments, the network structure needs to be optimized.

[0006] In small-scale experiments, manually configuring nodes can still meet the needs, but in large-scale cross-domain simulation experiments, it is very difficult to optimize the complex and large-scale distributed network topology. Summary of the invention

[0007] The technical problem to be solved by the present invention is how to accurately realize the automatic configuration of nodes to meet the needs of large-scale cross-domain simulation experiments; in view of this, the present invention provides a distributed test network topology optimization method based on deep reinforcement learning.

[0008] The technical solution adopted by the present invention is a distributed test network topology optimization method based on deep reinforcement learning, comprising: Step S1, constructing a graph data structure, including a set of nodes in a distributed network and a set of communication relationships between nodes; Step S2, initializing a graph neural network and a buffer, wherein the graph neural network is used for quantitative evaluation; Step S3, initializing constants; Step S4, configuring reward calculation selection parameters and initializing action and state selection parameters, wherein the value of the reward calculation selection parameter represents the acquisition source of the parameter, and the value of the action and state selection parameter represents whether the current state is a complete space or a compressed space; Step S5, executing a preset number of steps of the first algorithm according to the reward calculation selection parameter and the action and state selection parameter to determine the policy gradient and adjust the policy function parameter, and if the reward calculation selection parameter indicates that the parameter source is the verifier of the first algorithm, recording the corresponding data into the buffer; Step S6, executing the operation of the second algorithm, inputting the data and threshold of the buffer into a preset graphical neural network, and outputting an updated graphical neural network; Step S7, using the configured policy function to generate an action sequence and a related network structure, executing Verifier to check the validity of the current graphical neural network, calculating the corresponding target value, and adding the target value to the buffer; Step S8, execute the operation of the second algorithm, input the trained network of the second algorithm and the generated target value data, and output the accuracy. If it is greater than the preset accuracy value, the reward calculation selection parameter is characterized as the first algorithm, otherwise it is the second algorithm, and repeat steps S5 to S8.

[0009] In one embodiment, in step S3, the constants include: a preset number of steps, a number of epochs, and a threshold.

[0010] In one implementation, the first algorithm is A2C, and the second algorithm is GNN.

[0011] In one embodiment, the preset accuracy value is specifically configured to be at least 90%.

[0012] In one embodiment, in the network of the second algorithm trained by the input, the training process includes: randomizing the buffer, collecting data pairs and representing them as icon data; Adding a label to the chart data, wherein the label is used to indicate the current network status; Using the optimizer to update the input GNN layer parameters for each pair of data in the graph data; Repeat the above steps for a total of the stated number of epochs.

[0013] Another aspect of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the distributed experimental network topology optimization method based on deep reinforcement learning as described in any one of the above items.

[0014] Another aspect of the present invention further provides a computer storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the distributed test network topology optimization method based on deep reinforcement learning as described in any one of the above items are implemented.

[0015] Compared with the prior art, the present invention has at least the following advantages: The embodiment of the present invention uses deep reinforcement learning to replace manual work to optimize the deployment plan of the node network of large-scale experiments. This method can effectively save labor, and can adapt to the on-site situation to quickly recalculate, and can optimize the situation changes of a few nodes in large-scale calculations. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of a process flow of a distributed test network topology optimization method based on deep reinforcement learning according to an embodiment of the present invention; Figure 2 A logic schematic diagram for implementing a distributed test network topology optimization method based on deep reinforcement learning according to an embodiment of the present invention; Figure 3 The figure is a schematic diagram of the composition of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose, the present invention is described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0018] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms (such as those defined in commonly used dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined in this article.

[0019] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0020] An embodiment of the present invention provides a distributed test network topology optimization method based on deep reinforcement learning, comprising: Step S1, constructing a graph data structure, including a set of nodes in a distributed network and a set of communication relationships between nodes; Step S2, initializing a graph neural network and a buffer, wherein the graph neural network is used for quantitative evaluation; Step S3, initializing constants; Step S4, configuring reward calculation selection parameters and initializing action and state selection parameters, wherein the value of the reward calculation selection parameter represents the acquisition source of the parameter, and the value of the action and state selection parameter represents whether the current state is a complete space or a compressed space; Step S5, executing a preset number of steps of the first algorithm according to the reward calculation selection parameter and the action and state selection parameter to determine the policy gradient and adjust the policy function parameter, and if the reward calculation selection parameter indicates that the parameter source is the verifier of the first algorithm, recording the corresponding data into the buffer; Step S6, executing the operation of the second algorithm, inputting the data and threshold of the buffer into a preset graphical neural network, and outputting an updated graphical neural network; Step S7, using the configured policy function to generate an action sequence and a related network structure, executing Verifier to check the validity of the current graphical neural network, calculating the corresponding target value, and adding the target value to the buffer; Step S8, execute the operation of the second algorithm, input the trained network of the second algorithm and the generated target value data, and output the accuracy. If it is greater than the preset accuracy value, the reward calculation selection parameter is characterized as the first algorithm, otherwise it is the second algorithm, and repeat steps S5 to S8.

[0021] In step S3, the constants include: a preset number of steps, a number of epochs, and a threshold value.

[0022] In this embodiment, the first algorithm is A2C, and the second algorithm is GNN.

[0023] In this embodiment, the preset accuracy value is specifically configured to be at least 90%.

[0024] In this embodiment, the second algorithm that has been trained is input into the network, and the training process includes: randomizing the buffer, collecting data pairs and representing them as icon data; Adding a label to the chart data, wherein the label is used to indicate the current network status; Using the optimizer to update the input GNN layer parameters for each pair of data in the graph data; Repeat the above steps for a total of the stated number of epochs.

[0025] refer to Figure 1 as well as Figure 2 , the method provided by this implementation will be further described below.

[0026] This application proposes a solution that uses deep reinforcement learning (DRL) to optimize the deployment scheme of distributed simulation test nodes. Among the many DRL algorithms, we chose the A2C algorithm to solve this type of problem for the following reasons. First, the A2C algorithm has the characteristic of low variance. In the problem of network structure optimization, the optimization difficulty of different structures varies greatly. Compared with other DRL algorithms, the training process of A2C is more stable and more adaptable to the larger variance between samples. Second, the A2C algorithm is an online learning algorithm that can update the strategy at each time step without waiting for the end of the entire round, so it has better real-time performance. In distributed simulation experiments, algorithms with better real-time performance are conducive to quickly solving various emergencies. Third, the A2C algorithm has fewer hyperparameters and is easier to adjust, which is convenient for test personnel to quickly adjust on site according to actual conditions. A2C (Advantage Actor-Critic) is an algorithm for deep reinforcement learning.

[0027] It combines the ideas of Actor-Critic method and Advantage learning. In A2C, there are two main components: actor and critic. The actor is responsible for selecting actions, while the critic evaluates the quality of these actions.

[0028] The A2C algorithm uses a neural network to approximate the actor and critic, and achieves learning by optimizing these two networks. The advantage of the A2C algorithm is that it can be parallelized because the actor and critic can be updated independently during training. This makes A2C more computationally efficient because it can collect data and make updates in multiple environments in parallel. In addition, A2C also utilizes the concept of Advantage learning. By calculating the advantage of each action, the value of the action can be more accurately evaluated, thereby improving the efficiency of learning. In general, the A2C algorithm is widely used in deep reinforcement learning because it combines the advantages of the Actor-Critic method and has good performance in efficiency and performance.

[0029] The node deployment problem in cross-region distributed simulation experiments is essentially a graph search problem. The network connections between nodes can be summarized into an undirected fully connected graph, and determining the node deployment plan is to select nodes and edges in the graph to form a subgraph.

[0030] In order to solve such problems, we first need to abstract the network topology and represent the network topology as a graph G = (V, E), where the node set V = {1, 2, ..., N} and the edge set E = {(m, n), m, n V}. The adjacent matrix x of the graph G is defined as a connectivity variable, that is, if i and j are connected, then (x)ij = 1, otherwise (x)ij = 0.

[0031] The indicators for measuring network topology performance are as follows: Efficiency(x)=αPressure(x)+βCost(x,x_0)+Delay(x) (1) Distance(e)≤ (2) Load(e)≤ (3) M(x)=True (4) In (1), x represents the optimized network structure, Efficiency(x) represents the efficiency index of the distributed network, Pressure(x) represents the maximum data transmission volume of a single link under the topology structure, and is used to describe the pressure of high-concurrency nodes. x0 represents the network structure before optimization, Cost(x,x_0) is the number of nodes that need to be moved when the graph is transformed from the initial topology x0 to x, which is the cost of the optimization process, Delay(x) refers to the average delay between nodes in the network, and α and β are the weights of the two when evaluating the network operation efficiency.

[0032] The constraints that the network structure needs to meet are shown in formulas (2), (3), and (4). In constraint (2), Distance(e) represents the distance between the two end nodes of a given edge e in the network topology. D represents the maximum acceptable physical distance between two nodes under the condition that physical delay does not interfere with the test accuracy. For each link (i; j), the two end nodes vi and vj must be within the range of the connection distance D. In constraint (3), Load(e) represents the load of link e between nodes. is the maximum allowable load. In actual experiments, due to the high concurrency of experimental data, it is necessary to limit the link load between two nodes to prevent data congestion. Condition (4) is the abstract feasibility requirement of the topology, which is determined according to the specific requirements of the experiment, such as the number of data forwarding restrictions, etc. The algorithm flow used in this embodiment is as follows Figure 1 As shown in the figure, the algorithm consists of three main parts: representation layer, DRL agent and verifier. In the figure, s represents the state, a represents the action, and r represents the reward function. T(s,a) represents the transition function. \pi is the target value calculated by the verifier. The A2C algorithm uses the representation layer to compress the state and action into s0 and a0, and uses the GNN classifier ~ f to learn the true target function f.

[0033] A specific embodiment process is as follows: A graph data structure is constructed to represent the overall structure of a distributed network, including the node set V of the distributed network and the communication relationship set E between the nodes, which are represented and stored using points and edges in the data structure respectively.

[0034] Select appropriate parameters to initialize the graph neural network, use the policy function parameter theta to build an evaluation function for quantitative evaluation of the optimization scheme, input the current state of the network and the operation being performed, output the quantitative evaluation value of the operation, and initialize the buffer R.

[0035] Initialization constants, such as the number of time steps T, the number of epochs N, and the threshold Q.

[0036] Set the reward calculation selection parameter C_r=1. When C_r = 1, it means that the reward parameter c is obtained from the Verifier in the A2V algorithm. When C_r = 2, it means that it is obtained from GNN.

[0037] Initialize the action and state selection C_as. When C_as = 1, it indicates the full space. When C_as = 2, it indicates the compressed space.

[0038] Repeat the following steps: According to the parameters C_r and C_as, execute the A2C algorithm T steps, calculate the policy gradient and adjust the policy function parameter theta. If C_r = 1, the data {a^t, s^t, r^t}^T _{t=1} is recorded to the buffer R.

[0039] Execute GNN training operation. Input the data in buffer R and threshold Q into the existing graph neural network f(t), and then output the new graph neural network f(t+1).

[0040] Use the policy function to generate action sequences {a_1, a_2, … a_n} and related network structures {s_1, s_2, … s_n}. Then execute Verifier to check the validity of the graph, calculate the corresponding target value {r_1, r_2, … r_n}$, and add the data to the buffer R.

[0041] Execute the GNN training operation. Input the trained GNN neural network and the data generated in the previous step. Then output the accuracy. If it is greater than 90%, C_r = 1, otherwise C_r = 2.

[0042] Among them, the GNN training process in 6.2 and 6.4 is as follows: 1. Randomize the buffer R and collect data pairs n, expressed as D = {(s_i, r_i)}^n _{i=1} 2. Add labels to the chart data, using 1 and 0 to indicate good and bad network status.

[0043] 3. Use the optimizer to update the input GNN layer parameters for each pair of data (s_i, r_i) in 4. Repeat the above steps N times Compared with the prior art, this embodiment has at least the following effects: The present invention proposes a method for local network optimization based on deep reinforcement learning to avoid errors in the results of large-scale distributed simulation experiments due to poor network performance.

[0044] This paper proposes a solution that uses deep reinforcement learning to replace manual work to optimize the deployment plan of node networks in large-scale experiments. This method can effectively save labor, adapt to the on-site situation and quickly recalculate, and can optimize the changes in the situation of a few nodes in large-scale calculations.

[0045] A second embodiment of the present invention is an electronic device, such as Figure 3 As shown, it can be understood as a physical device, including a processor and a memory storing instructions executable by the processor. When the instructions are executed by the processor, the following operations are performed: Step S1, constructing a graph data structure, including a set of nodes in a distributed network and a set of communication relationships between nodes; Step S2, initializing a graph neural network and a buffer, wherein the graph neural network is used for quantitative evaluation; Step S3, initializing constants; Step S4, configuring reward calculation selection parameters and initializing action and state selection parameters, wherein the value of the reward calculation selection parameter represents the acquisition source of the parameter, and the value of the action and state selection parameter represents whether the current state is a complete space or a compressed space; Step S5, executing a preset number of steps of the first algorithm according to the reward calculation selection parameter and the action and state selection parameter to determine the policy gradient and adjust the policy function parameter, and if the reward calculation selection parameter indicates that the parameter source is the verifier of the first algorithm, recording the corresponding data into the buffer; Step S6, executing the operation of the second algorithm, inputting the data and threshold of the buffer into a preset graphical neural network, and outputting an updated graphical neural network; Step S7, using the configured policy function to generate an action sequence and a related network structure, executing Verifier to check the validity of the current graphical neural network, calculating the corresponding target value, and adding the target value to the buffer; Step S8, execute the operation of the second algorithm, input the trained network of the second algorithm and the generated target value data, and output the accuracy. If it is greater than the preset accuracy value, the reward calculation selection parameter is characterized as the first algorithm, otherwise it is the second algorithm, and repeat steps S5 to S8.

[0046] The third embodiment of the present invention, the process of the distributed test network topology optimization method based on deep reinforcement learning in this embodiment is the same as that of the first and second embodiments, the difference is that in engineering implementation, this embodiment can be implemented by means of software plus the necessary general hardware platform, of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the method of the present invention can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), including a number of instructions for a device to execute the method described in the embodiment of the present invention.

[0047] Through the description of the specific implementation methods, a deeper and more specific understanding of the technical means and effects adopted by the present invention to achieve the predetermined purpose should be obtained. However, the accompanying drawings are only for reference and illustration purposes and are not intended to limit the present invention.

Claims

1. A distributed test network topology optimization method based on deep reinforcement learning, characterized in that: include: Step S1, constructing a graph data structure, including a set of nodes in a distributed network and a set of communication relationships between nodes; Step S2, initializing a graph neural network and a buffer, wherein the graph neural network is used for quantitative evaluation; Step S3, initializing constants; Step S4, configuring reward calculation selection parameters and initializing action and state selection parameters, wherein the value of the reward calculation selection parameter represents the acquisition source of the parameter, and the value of the action and state selection parameter represents whether the current state is a complete space or a compressed space; Step S5, executing a preset number of steps of the first algorithm according to the reward calculation selection parameter and the action and state selection parameter to determine the policy gradient and adjust the policy function parameter, and if the reward calculation selection parameter indicates that the parameter source is the verifier of the first algorithm, recording the corresponding data into the buffer; Step S6, executing the operation of the second algorithm, inputting the data and threshold of the buffer into a preset graphical neural network, and outputting an updated graphical neural network; Step S7, using the configured policy function to generate an action sequence and a related network structure, executing Verifier to check the validity of the current graphical neural network, calculating the corresponding target value, and adding the target value to the buffer; Step S8, execute the operation of the second algorithm, input the trained network of the second algorithm and the generated target value data, and output the accuracy. If it is greater than the preset accuracy value, the reward calculation selection parameter is characterized as the first algorithm, otherwise it is the second algorithm; repeat steps S5 to S8.

2. The distributed test network topology optimization method based on deep reinforcement learning according to claim 1 is characterized in that: In step S3, the constants include: a preset number of steps, a number of epochs, and a threshold value.

3. The distributed test network topology optimization method based on deep reinforcement learning according to claim 2 is characterized in that: The first algorithm is A2C, and the second algorithm is GNN.

4. The distributed test network topology optimization method based on deep reinforcement learning according to claim 1 is characterized in that: The preset accuracy value is specifically configured to be at least 90%.

5. The distributed test network topology optimization method based on deep reinforcement learning according to claim 3 is characterized in that: The training process of the inputted network of the second algorithm after training comprises: randomizing the buffer, collecting data pairs and representing them as icon data; Adding a label to the chart data, wherein the label is used to indicate the current network status; Using the optimizer to update the input GNN layer parameters for each pair of data in the graph data; Repeat the above steps for a total of the stated number of epochs.

6. An electronic device, characterized in that: The electronic device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the distributed test network topology optimization method based on deep reinforcement learning as described in any one of claims 1 to 5 are implemented.

7. A computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the distributed test network topology optimization method based on deep reinforcement learning as described in any one of claims 1 to 5.

Images