A network loop-free full coverage path set selection method based on deep reinforcement learning
Through the Actor-Critic algorithm and mask mechanism of deep reinforcement learning, a ring-free and low-redundant network full coverage path set is generated, which solves the problems of poor adaptability and high redundancy in the existing technology, and improves network measurement efficiency and deployment effect.
Patent Information
- Application Number
- CN202310576565.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-22
- Publication Date
- 2025-08-26
- Estimated Expiration
- 2043-05-22
AI Technical Summary
The existing network full coverage path set selection method has poor adaptability, high path set redundancy and contains loops, resulting in high deployment difficulty and network performance impact.
Using a deep reinforcement learning method, the actor network and the critical network are trained through the Actor-Critic algorithm, combined with the mask mechanism, a ring-free and low redundancy network full coverage path set is generated, and the path selection is optimized using the Markov decision-making process.
The generated path set reduces deployment difficulty and traffic load, improves network measurement efficiency, and reduces redundancy and loop count.
Smart Images
Figure CN116599886B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of network measurement technology, and specifically relates to a network loop-free full coverage path set selection method based on deep reinforcement learning. Background Art
[0002] With the rapid expansion of computer networks, fully measuring every path in the network is inefficient, or even impossible. To improve efficiency, it is essential to select the most valuable and representative paths within the network. The goal of path selection is to find a path set that contains as few paths as possible and covers the entire network. By selecting a reasonable path set to cover the entire network, we can not only obtain the real-time traffic status of individual devices (routers and switches) or device chains, but also gain a global view of the network to make optimal control decisions. Therefore, selecting a network-wide path set is a fundamental problem in network measurement and control. Current methods for selecting network-wide path sets have the following shortcomings: ① The start and end nodes of the paths in the path set are internal nodes in the network. This makes these methods ineffective in some situations where the network cannot be operated internally, resulting in poor adaptability. ② The paths in the path set contain loops, which is not conducive to practical deployment. ③ The selected path set has high redundancy, meaning that a link is included in multiple paths. This leads to high measurement traffic and affects network performance. Summary of the Invention
[0003] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a network loop-free full coverage path set selection method based on deep reinforcement learning.
[0004] To achieve the above object, the present invention is implemented through the following technical solutions:
[0005] A method for selecting a network loop-free full coverage path set based on deep reinforcement learning, comprising the following steps:
[0006] Step 1: Establish an optimization objective based on the size and redundancy of the path set. Then, establish constraints based on edge node characteristics, full coverage characteristics, and loop-free characteristics to obtain a network full coverage path set selection model.
[0007] Step 2: Convert the network full coverage path set selection model into a Markov decision process, determine the state space and action space representation, and design the reward signal function based on the redundancy of the path set;
[0008] Step 3: Use the Actor-Critic algorithm to train an actor network and a critic network. The actor network selects the next link based on the currently generated path and uses a masking mechanism to ensure that the generated path is acyclic. The critic network learns a value function based on the data collected by the actor network's interaction with the environment, thereby helping the actor network to update its policy.
[0009] Step 4: After training is completed, the agent gradually generates paths through the actor network. When the entire network is covered, a set of fully covered paths is obtained.
[0010] The optimization goal described in step 1 is to minimize the sum of the size of the path set P and the redundancy of the network G with respect to the path set P. The corresponding optimization goal expression is:
[0011]
[0012] Where P represents the set of generated paths; α and β are balance factors; represents the redundancy of network G with respect to path set P, The expression is as follows:
[0013]
[0014] Among them, V is the node set of network G, Link l uv Regarding the redundancy of the path set P, The expression is as follows:
[0015]
[0016] in, If there is a link between node u and node v, and path p i Contains link l uv ,but otherwise
[0017] The edge node characteristics described in step 1 represent the path p in the path set P i The starting and ending nodes must be edge nodes of the network. The constraint expressions corresponding to the edge node characteristics are as follows:
[0018]
[0019] in, If the node v(v∈V) is on the path p i On, then otherwise h vIf node v is an edge node, then h v =1, otherwise h v =0;d v represents the degree of node v;
[0020] The full coverage property described in step 1 means that the links in the path set P must cover the link set L of the entire network G. G , the constraint expression corresponding to the full coverage feature is as follows:
[0021]
[0022] in, If there is a link between node u and node v, and path p i Contains link l uv ,but otherwise
[0023] The acyclic property described in step 1 represents the path p in the path set P. i It must be a simple path and cannot contain loops. The constraint expression corresponding to the acyclic property is as follows:
[0024]
[0025]
[0026] in, If the node v(v∈V) is on the path p i On, then otherwise If there is a link between node u and node v, and path p i Contains link l uv ,but otherwise
[0027] The Markov decision process described in step 2 can be expressed as MDP=<S,A,T,R,γ> , where S is the agent's state space; A is the agent's action space; T is the state transition probability of the agent from one state to the next; R is the reward the agent receives from executing the action selected from the current action; γ is the discount factor, and its value range is [0,1]. A γ close to 1 focuses more on long-term cumulative rewards, while a γ close to 0 focuses more on short-term rewards.
[0028] The state space expression described in step 2 is as follows:
[0029]
[0030] in, Indicates the last link of the path currently being built. Indicates the size of the path set currently being built, Indicates the number of links currently visited. Indicates the path currently being built;
[0031] The expression of the action space described in step 2 is as follows:
[0032] A={a1,a2,a3,...,a N}
[0033] Where N is the number of links in the network, action a i Indicates that the i-th link is selected based on the path currently being built and added to the current path to form a new path;
[0034] The reward signal function described in step 2 is expressed as follows:
[0035]
[0036] Among them, M represents the path set currently being constructed, represents the set of edge nodes of network G, Represents the set of links in the network G that are included in the path set M, L G represents the link set of network G;
[0037] The goal of the Actor-Critic algorithm described in step 3 is to find a path generation strategy that minimizes the size of the path set and the redundancy of the path set for the network. The network structure of the Actor-Critic algorithm consists of two neural networks: the actor network and the critic network. The actor network and the critic network consist of three layers of fully connected networks, and the activation functions of the first and second layers are relu functions. The actor network learns a path selection strategy π(a|s; θ) (θ is a parameter) based on the input state s. This strategy gives the probability distribution p of the next selected link. θ (s t ), the expression is as follows:
[0038]
[0039] in, is the probability distribution p θ (s t ) represents the probability of selecting the i-th link based on the current path generation scheme.
[0040] The mask mechanism described in step 3 refers to the probability distribution p of the next link output by the actor network being selected θ (s t ) is masked to ensure that the new path formed by the next link selected and the current constructed path does not contain a loop, thereby obtaining the probability distribution of all valid links Finally renormalize Its expression is as follows:
[0041]
[0042]
[0043] in, is the probability distribution The i-th component of E va Represents the set of links that are connected to the currently constructed path and do not form a loop; is the probability distribution p θ (s t ) represents the probability of selecting the i-th link based on the current path generation scheme.
[0044] Data Γ collected by the interaction between the actor network and the environment described in step 3 t It can be expressed as:
[0045] Γ t =(s t ,a t ,r t ,s t+1 )
[0046] Among them, s t is the state of the agent at time t, a t is the agent in state s t Action taken, r t is the agent in state s t Execute action a t The reward obtained after t+1 is the agent in state s t Execute action a t The next state reached.
[0047] The Actor-Critic algorithm training process described in step 3 is as follows:
[0048] Step 1: Initialize the parameters θ of the actor network, the parameters φ of the critic network, and the number of epochs of algorithm training.
[0049] Step 2: During each epoch of training, the next link to be selected is sampled from the probability distribution of the action returned by the current actor network. The actions calculated by the actor network are iteratively executed until a path set selection solution that meets the constraints is successfully generated.
[0050] Step 3: Collect N path sets and select solution D k , D k The expression is as follows:
[0051] D k ={τ i}
[0052] Among them, τ i Represents the i-th path set selection plan.
[0053] Step 4: Calculate the reward after taking the action The expression is as follows:
[0054]
[0055] Among them, R(s t′ , a t′ , s t′+1 ) is the agent in state s t′ Take action a t′ Rewards received afterward.
[0056] Step 5: Select a solution based on the generated path set and calculate the advantage estimate of the path set generation process using the following formula:
[0057]
[0058] Among them, r t and v t are the reward obtained in step t and the output of the critic network, v t+1 is the output of the critic network at step t+1, γ is the discount factor, and λ is a smoothing parameter to reduce variance;
[0059] Step 6: Estimate based on strengths Computing the policy gradient of the actor network The formula is as follows:
[0060]
[0061] Step 7: Use gradient descent to calculate the policy update, which is as follows:
[0062]
[0063] where αk is the learning rate;
[0064] Step 8: Calculate the critic network parameters φ by minimizing the mean square error k+1 , the formula is as follows:
[0065]
[0066] Where V φ (s t ) is the critic network input state s t The output after.
[0067] Compared with the prior art, the present invention has the following beneficial effects:
[0068] The present invention provides a network loop-free full-coverage path set selection method based on deep reinforcement learning. In view of the shortcomings of the current network full-coverage path set selection method, such as poor adaptability, difficult deployment, high path set redundancy, and generated paths containing loops, an optimization target is established according to the size and redundancy of the path set, and a corresponding network full-coverage path set selection model is established. The network full-coverage path set selection model is then converted into a Markov decision process. Then, an Actor-Critic algorithm is used to train a neural network model, and the loop-free nature of the generated paths is guaranteed through a mask mechanism, thereby generating a loop-free network full-coverage path set solution. The path set has low redundancy. The method of the present invention can not only reduce the deployment difficulty and deployment overhead, but also reduce the traffic load introduced by measurement, thereby improving the efficiency of network measurement. BRIEF DESCRIPTION OF THE DRAWINGS
[0069] Figure 1 The network structure model diagram for the full coverage path set selection process;
[0070] Figure 2 This is the Airtel network topology diagram;
[0071] Figure 3 A redundancy comparison diagram of the full coverage path set generated by the algorithm of the present invention and the random algorithm;
[0072] Figure 4 A comparison chart of the sizes of the full coverage path sets generated by the algorithm of the present invention and the random algorithm;
[0073] Figure 5 A comparison chart of the number of concentrated rings in the full coverage path generated by the algorithm of the present invention and the heuristic algorithm based on DFS; DETAILED DESCRIPTION
[0074] To facilitate understanding of the present invention, the present invention will be described more fully below. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.
[0075] like Figure 1 、 Figure 2 As shown, the network loop-free full coverage path set selection method based on deep reinforcement learning in this embodiment includes the following steps:
[0076] Step 1: Establish an optimization objective based on the size and redundancy of the path set. Then, establish constraints based on edge node characteristics, full coverage characteristics, and loop-free characteristics to obtain a network full coverage path set selection model.
[0077] Step 2: Convert the network full coverage path set selection model into a Markov decision process, determine the state space and action space representation, and design the reward signal function based on the redundancy of the path set;
[0078] Step 3: Use the Actor-Critic algorithm to train an actor network and a critic network. The actor network selects the next link based on the currently generated path and uses a masking mechanism to ensure that the generated path is acyclic. The critic network learns a value function based on the data collected by the actor network's interaction with the environment, thereby helping the actor network to update its policy.
[0079] Step 4: After training is completed, the agent gradually generates paths through the actor network. When the entire network is covered, a set of fully covered paths is obtained.
[0080] The optimization goal described in step 1 is to minimize the sum of the size of the path set P and the redundancy of the network G with respect to the path set P. The corresponding optimization goal expression is:
[0081]
[0082] Where P represents the set of generated paths; α and β are balance factors; represents the redundancy of network G with respect to path set P, The expression is as follows:
[0083]
[0084] Among them, V is the node set of network G, Link l uv Regarding the redundancy of the path set P, The expression is as follows:
[0085]
[0086] in, If there is a link between node u and node v, and path p i Contains link l uv ,but otherwise
[0087] The edge node characteristics described in step 1 represent the path p in the path set P i The starting and ending nodes must be edge nodes of the network. The constraint expressions corresponding to the edge node characteristics are as follows:
[0088]
[0089] in, If the node v(v∈V) is on the path p i On, then otherwise h v If node v is an edge node, then h v =1, otherwise h v =0;d v represents the degree of node v;
[0090] The full coverage property described in step 1 means that the links in the path set P must cover the link set L of the entire network G. G , the constraint expression corresponding to the full coverage feature is as follows:
[0091]
[0092] in, If there is a link between node u and node v, and path p i Contains link l uv ,but otherwise
[0093] The acyclic property described in step 1 represents the path p in the path set P. i It must be a simple path and cannot contain loops. The constraint expression corresponding to the acyclic property is as follows:
[0094]
[0095]
[0096] in, If the node v(v∈V) is on the path p i On, then otherwise If there is a link between node u and node v, and path p i Contains link l uv ,but otherwise
[0097] The Markov decision process described in step 2 can be expressed as MDP=<S,A,T,R,γ> , where S is the agent's state space; A is the agent's action space; T is the state transition probability of the agent from one state to the next; R is the reward the agent receives from executing the action selected from the current action; γ is the discount factor, and its value range is [0, 1]. A γ close to 1 focuses more on long-term cumulative rewards, while a γ close to 0 focuses more on short-term rewards.
[0098] The state space expression described in step 2 is as follows:
[0099]
[0100] in, Indicates the last link of the path currently being built. Indicates the size of the path set currently being built, Indicates the number of links currently visited. Indicates the path currently being built.
[0101] Furthermore, the initial state s0 of the agent is expressed as follows:
[0102]
[0103]
[0104]
[0105]
[0106]
[0107] Where EL represents the set of edge links in the network G; random(EL) means randomly selecting a link from the EL set;
[0108] The expression of the action space described in step 2 is as follows:
[0109] A={a1,a2,a3,…,a N}
[0110] Where N is the number of links in the network, action a iIndicates that the i-th link is selected based on the path currently being built and added to the current path to form a new path.
[0111] The reward signal function described in step 2 is expressed as follows:
[0112]
[0113] Among them, M represents the path set currently being constructed, represents the set of edge nodes of network G, Represents the set of links in the network G that are included in the path set M, L G represents the link set of network G;
[0114] The goal of the Actor-Critic algorithm described in step 3 is to find a path generation strategy that minimizes the size of the path set and the redundancy of the path set for the network. The network structure of the Actor-Critic algorithm consists of two neural networks: the actor network and the critic network. The actor network and the critic network consist of three layers of fully connected networks, and the activation functions of the first and second layers are relu functions. The actor network learns a path selection strategy π(a|s; θ) (θ is a parameter) based on the input state s. This strategy gives the probability distribution p of the next selected link. θ (s t ), the expression is as follows:
[0115]
[0116] in, is the probability distribution p θ (s t ) represents the probability of selecting the i-th link based on the current path generation scheme.
[0117] The mask mechanism described in step 3 refers to the probability distribution p of the next link output by the actor network being selected θ (s t ) is masked to ensure that the new path formed by the next link selected and the current constructed path does not contain a loop, thereby obtaining the probability distribution of all valid links Finally renormalize Its expression is as follows:
[0118]
[0119]
[0120] in, is the probability distribution The i-th component of E va Represents the set of links that are connected to the currently constructed path and do not form a loop; is the probability distribution p θ (s t ) represents the probability of selecting the i-th link based on the current path generation scheme.
[0121] Furthermore, the normalization Refers to the probability distribution after mask processing Perform softmax operation, the expression is as follows:
[0122]
[0123] Data Γ collected by the interaction between the actor network and the environment described in step 3 t It can be expressed as:
[0124] Γ t =(s t , a t , r t , s t+1 )
[0125] Among them, s t is the state of the agent at time t, a t is the agent in state s t Action taken, r t is the agent in state s t Execute action a t The reward obtained after t+1 is the agent in state s t Execute action a t The next state reached.
[0126] The Actor-Critic algorithm training process described in step 3 is as follows:
[0127] Step 1: Initialize the parameters θ of the actor network, the parameters φ of the critic network, and the number of epochs of algorithm training.
[0128] Step 2: During each epoch of training, the next link to be selected is sampled from the probability distribution of the action returned by the current actor network. The actions calculated by the actor network are iteratively executed until a path set selection solution that meets the constraints is successfully generated.
[0129] Step 3: Collect N path sets and select solution D k , D k The expression is as follows:
[0130] Dk ={τ i}
[0131] Among them, τ i Represents the i-th path set selection plan.
[0132] Step 4: Calculate the reward after taking the action The expression is as follows:
[0133]
[0134] Among them, R(s t′ , a t′ , s t′+1 ) is the agent in state s t′ Take action a t′ Rewards received afterward.
[0135] Step 5: Select a solution based on the generated path set and calculate the advantage estimate of the path set generation process using the following formula:
[0136]
[0137] Among them, r t and v t are the reward obtained in step t and the output of the critic network, v t+1 is the output of the critic network at step t+1, γ is the discount factor, and λ is a smoothing parameter to reduce variance;
[0138] Step 6: Estimate based on strengths Computing the policy gradient of the actor network The formula is as follows:
[0139]
[0140] Step 7: Use gradient descent to calculate the policy update, which is as follows:
[0141]
[0142] where α k is the learning rate;
[0143] Step 8: Calculate the critic network parameters φ by minimizing the mean square error k+1 , the formula is as follows:
[0144]
[0145] Where V φ (s t ) is the critic network input state st The output after.
[0146] In the above technical solution, the balance factor α=1, β=1, the discount factor γ=0.98, and the algorithm training epoch number is initialized to 2048.
[0147] The network full coverage path set selection result is as follows Figure 3 、 Figure 4 and Figure 5 As can be seen from the figure, the size and redundancy of the full coverage path set generated by the algorithm of the present invention are much lower than those of the full coverage path set generated by the random algorithm. In addition, the full coverage path set generated by the algorithm of the present invention does not contain rings, while the number of rings in the full coverage path set generated by the DFS-based heuristic algorithm is higher.
[0148] The technical features of the above embodiments can be combined in any manner. To simplify the description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction between the combinations of these technical features, they should be considered to be within the scope of this specification.
[0149] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.
Claims
1. A method for selecting a network loop-free full coverage path set based on deep reinforcement learning, characterized in that: include: Step 1: Establish an optimization objective based on the size and redundancy of the path set. Then, establish constraints based on edge node characteristics, full coverage characteristics, and loop-free characteristics to obtain a network full coverage path set selection model. Step 2: Convert the network full coverage path set selection model into a Markov decision process, determine the state space and action space representation, and design the reward signal function based on the redundancy of the path set; Step 3: Use the Actor-Critic algorithm to train an actor network and a critic network. The actor network selects the next link based on the currently generated path and uses a masking mechanism to ensure that the generated path is acyclic. The critic network learns a value function based on the data collected by the actor network's interaction with the environment, thereby helping the actor network to update its policy. Step 4: After training is completed, the agent gradually generates paths through the actor network. When the entire network is covered, a set of fully covered paths is obtained. The optimization goal in step 1 is to minimize the sum of the size of the path set P and the redundancy of the network G with respect to the path set P. The corresponding optimization goal expression is: Where P represents the set of generated paths; α and β are balance factors; represents the redundancy of network G with respect to path set P, The expression is as follows: Among them, V is the node set of network G, Link l uv Regarding the redundancy of the path set P, The expression is as follows: in, If there is a link between node u and node v, and path p i Contains link l uv ,but otherwise The reward signal function described in step 2 is expressed as follows: Among them, M represents the path set currently being constructed, represents the set of edge nodes of network G, Represents the set of links in the network G that are included in the path set M, L G represents the link set of network G; The mask mechanism described in step 3 refers to the probability distribution p of the next link output by the actor network being selected θ (s t ) is masked to ensure that the new path formed by the next link selected and the current constructed path does not contain a loop, thereby obtaining the probability distribution of all valid links Finally renormalize Its expression is as follows: in, is the probability distribution The i-th component of E va Represents the set of links that are connected to the currently constructed path and do not form a loop; is the probability distribution p θ (s t ) represents the probability of selecting the i-th link based on the current path generation scheme.
2. The method for selecting a network loop-free full coverage path set based on deep reinforcement learning according to claim 1, characterized in that: The edge node characteristics described in step 1 represent the path p in the path set P i The starting and ending nodes must be edge nodes of the network. The constraint expressions corresponding to the edge node characteristics are as follows: in, If node v is on path p i On, then otherwise h v If node v is an edge node, then h v =1, otherwise h v =0;d v Represents the degree of node v.
3. The method for selecting a network loop-free full coverage path set based on deep reinforcement learning according to claim 1, characterized in that: The full coverage property described in step 1 means that the links in the path set P must cover the link set L of the entire network G. G , the constraint expression corresponding to the full coverage feature is as follows: in, If there is a link between node u and node v, and path p i Contains link l uv ,but otherwise 4. The method for selecting a network loop-free full coverage path set based on deep reinforcement learning according to claim 1, characterized in that: The acyclic property described in step 1 represents the path p in the path set P. i It must be a simple path and cannot contain loops. The constraint expression corresponding to the acyclic property is as follows: in, If node v is on path p i On, then otherwise If there is a link between node u and node v, and path p i Contains link l uv ,but otherwise 5. The method for selecting a network loop-free full coverage path set based on deep reinforcement learning according to claim 1, characterized in that: The state space expression described in step 2 is as follows: in, Indicates the last link of the path currently being built. Indicates the size of the path set currently being built, Indicates the number of links currently visited. Indicates the path currently being built.
6. The method for selecting a network loop-free full coverage path set based on deep reinforcement learning according to claim 1, characterized in that: The expression of the action space described in step 2 is as follows: <h2 style=";text-align:left;direction:ltr">A = {a1,a2,a3,...,a<h2 style=";text-align:left;direction:ltr"> N <h2 style=";text-align:left;direction:ltr">} Where N is the number of links in the network, action a i Indicates that the i-th link is selected based on the path currently being built and added to the current path to form a new path.