Strategy routing optimization method based on reinforcement learning

Through the strategy routing optimization method based on reinforcement learning, the problem of searching for QoS guaranteed paths in large-scale communication networks is solved, and fast and convenient multi-attribute path search is achieved, which improves network performance and resource utilization.

CN119030913APending Publication Date: 2024-11-26NANJING UNIV OF AERONAUTICS & ASTRONAUTICS
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411080232.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-07
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In large-scale communication networks, searching out two or more network paths with QoS guarantees for setting policy routing is a complete problem of NP. It is difficult for the prior art to effectively meet the service quality needs of network applications and utilize network idle resources.

Method used

Adopting a policy routing optimization method based on reinforcement learning, by defining a multi-attribute optimization path search algorithm, using the reinforcement learning engine in the knowledge plane to train a policy routing set, and setting a policy router for the data plane in the control plane to realize end-to-end QoS ensure.

Benefits of technology

It realizes the rapid and convenient search of a multi-attribute path set that meets QoS needs in large and complex networks, improves network transmission performance and resource utilization, and has the advantages of rapid convergence and high performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119030913A_ABST
    Figure CN119030913A_ABST
Patent Text Reader

Abstract

Policy routing (PBR) is more flexible than IP routing, QoS of network services can be better met, idle network resources can be better utilized, but it is difficult to determine to search two or more network paths with QoS guarantee for setting the policy routing in a large-scale communication network. The method for supporting the multi-attribute optimized path search based on reinforcement learning solves the problem of searching a plurality of path sets meeting the QoS requirement based on reinforcement learning, and has the advantages of being convenient to apply, good in performance, rapid in convergence and the like.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of network communication, and in particular is a method for searching a multi-attribute optimized network path set with QoS guarantee in a complex communication network by using reinforcement learning. Background Art

[0002] The IP routing algorithm can determine the "shortest and lowest cost" path, but the path selected by the algorithm cannot meet the needs of some network applications in many cases. For example, the quality of service (QoS) of some network applications cannot be met, and there are a lot of idle resources in the network that cannot be used. Policy-based routing (PBR) is more flexible than IP routing. It allows routing decisions or packet forwarding to be made according to the definition and matching of policy conditions (such as source address, protocol type, service type, etc.), and then sends packets to a specific path or destination, which can better meet the QoS of network services and utilize network idle resources. To set up a PBR path, two or more multi-attribute optimized routing sets must be searched in the network. Since each network link has performance indicators such as bandwidth, delay, packet loss rate, and security, searching for an end-to-end network path with QoS guarantee is an NP-complete problem. Reinforcement learning is a goal-oriented machine learning that maximizes rewards through the interaction between the environment and the agent. The agent has the ability to self-learn and can adapt to network scenarios of different complexities. According to the real-time network environment and performance indicators, it can adaptively adjust the strategy to select the appropriate transmission node and optimize the network transmission performance.

[0003] The present invention proposes a strategy routing optimization method based on reinforcement learning, which can search for a set of multi-attribute paths with QoS guarantee in a large complex network.

[0004] The significance and importance of the present invention lies in that, in order to solve the problem of searching two or more network paths with QoS guarantee for setting policy routing in large-scale communication networks, the present invention proposes a policy routing optimization method based on reinforcement learning, which has the advantages of convenient application, good performance, and fast convergence. Summary of the invention

[0005] [Purpose of the invention]

[0006] The present invention proposes a policy routing optimization method based on reinforcement learning, which searches for two or more network paths with QoS guarantee for setting policy routing in a large-scale communication network.

[0007] [Technical solution]

[0008] The technical solution of the present invention is:

[0009] 1. A system that supports searching for multi-attribute optimization paths based on reinforcement learning is proposed, which is characterized by:

[0010] A. Figure 1 To support the system architecture for searching multi-attribute optimized paths, the system architecture can be divided into three layers: data plane, control plane and knowledge plane, among which: first, the data plane transmits packets based on policy routing, and the system transmits the initial network status information big data to the reinforcement learning (RL) engine of the knowledge plane; second, the RL engine in the knowledge plane is trained with information such as network status to obtain an optimized policy routing set, and transmits this information to the controller of the control plane; third, the controller sets the policy routing (PBR) router of the data plane according to the optimized policy routing information.

[0011] B. The controller in the control plane includes the logic of controlling the router and sets up the PBR router so that the network can support two or more PBR-based paths to ensure the user's network application needs; the knowledge plane adopts an algorithm based on reinforcement learning to search for optimized paths. By collecting network information and application requirements, it learns and trains a set of paths suitable for policy routing with QoS guarantees.

[0012] 2. According to technical solution 1, a strategy routing optimization method OPR-RL based on reinforcement learning is proposed, which is characterized by

[0013] A. Definition of P g = {B, D, L, S, H} represents the bandwidth set, delay set, packet loss rate set, security set and node set of all links in the network G respectively. Definition Represent link bandwidth, delay, packet loss rate, security and path node set respectively. In order to ensure that the path solved by the algorithm has QoS requirement guarantee, the QoS requirement guarantee is defined as Condition = {b min ,d max , l max ,s min ,h max}, respectively represent the minimum bandwidth requirement, maximum delay limit, maximum packet loss rate limit, minimum security limit and maximum node hop limit. The link attributes of the searched path must satisfy formula (1):

[0014]

[0015] B. When the agent searches for a path, the set of optional nodes V and link parameters P observed when it is at the current node can be considered as its state S; the set of next-hop routes inferred by the agent based on the current state S can be considered as the action space A; when the agent moves, it is required to maximize future rewards, and its state transition probability from one node to an adjacent node is stable and only depends on the current position and has nothing to do with the previous position, searching for a multi-attribute optimization path with QoS guarantee k The process is a Markov decision process (MDP). For different application requirements, the algorithm needs to consider the impact of path attributes separately, and define θ, β, y, δ, ε, a total of 5 parameters, representing P g = the importance ratio of {B, D, L, S, H} in path selection, and the sum of these five parameters is 1, as shown in formula (2):

[0016] Coefficient={θ, β, γ, δ, ε|θ+β+γ+δ+ε=1} (2)

[0017] The maximum and minimum normalization method of formula (3) is used for processing, where x = {b kj , d kj , l kj ,s kj}, X = {B, D, L, S}:

[0018] M x =(x-min{X}) / (max{X}-min{X}) (3)

[0019] In the multi-attribute optimization path search process, the reward function determines the reward that the agent can obtain when selecting a node, and indirectly affects the agent's next choice of node. The reward function is expressed by formula (4), and the reward value it returns represents P g The overall benefit of the search path. The larger the value, the more the link meets the path requirements of the current application.

[0020] Reward=αM b +βM d +γM l +δM s (4)

[0021] C. Optimized Policy-based Routing based on Reinforcement Learning (OPR-RL), first initialize the Q-Table based on the state space S and action space A obtained from the knowledge plane.

[0022] Initialize the current state to the source node s = v s , use the action selection strategy to select the next hop node. If the next hop node exists, use formula (5) to update the Q-Table and continue to iterate until the destination node, where R is the reward value of the reward function formula (4) and γ is the discount factor. If the next hop node does not exist or passes through a repeated node, R is a large negative reward to avoid the node being selected again. If the next hop node is the destination node v d , storing the ordered nodes that the path passes through to the pathList collection and execute the next step. After that, the iterative difference is calculated. If the difference reaches the convergence threshold, the iteration is exited, otherwise, the next iteration is continued.

[0023] Q(S t , A t )←Q(S t , A t )+α[R+y*MaxQ(S t+1 , A t+1 )-Q(S t , A t )] (5)

[0024] In addition, during the iteration process, we dynamically adjust the selection probability e and learning efficiency α of the action strategy based on the number of paths searched, as shown in Equation (6). Where ∑{path} represents the number of paths currently searched, rate represents the change rate, step represents the number of steps the current agent moves, and e initial and α initial Represents the initialization value of probability and efficiency. This dynamic strategy indicates that as the number of searched paths and the number of steps the agent moves increases, the e value gradually increases and the α value gradually decreases. In the early iteration of the algorithm, there is a greater possibility to randomly explore links, which helps to find the destination node faster; in the later iteration of the algorithm, it will rely more on the knowledge that has been learned, reducing random and meaningless searches.

[0025]

[0026] After the learning engine determines a set of two or more end-to-end paths, it first uses formula (1) to determine the QoS requirements of the paths. If the requirements are not met, the path is directly discarded. If the requirements are met, the comprehensive performance function shown in formula (7) is used to calculate the performance of all paths:

[0027] Fitness(path k )=θN b +βN d +γN l +δNS +εN h (7)

[0028] Among them, in order to make Fitness(path k ) values ​​are meaningful, and the parameters with different dimensions need to be normalized, as shown in formula (8). After the paths are comprehensively sorted, several better paths ranked in the top are returned. If all paths do not meet the defined QoS requirements, several paths closest to constraint (1) and the attribute information of the several paths are returned to the controller for backup selection.

[0029]

[0030] D. The pseudo code description of the OPR-RL algorithm is as follows:

[0031]

[0032]

[0033] [Beneficial Effect]: The significance of the present invention lies in that it can provide a solution OPR-RL algorithm for solving the difficult problem of searching for two or more network paths with QoS guarantee for setting policy routing in large-scale communication networks. The algorithm has the advantages of convenient application, good performance and fast convergence. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 System Architecture

[0035] Figure 2 Prototype system with NSFNET topology

[0036] Figure 3 Comparison of average search time of different algorithms

[0037] Figure 4 Relationship between the number of nodes and the number of algorithm iterations

[0038] Figure 5 The impact of different parameters on iteration time DETAILED DESCRIPTION

[0039] The present invention is further described below in conjunction with the accompanying drawings and specific implementation methods.

[0040] We built a policy routing prototype system based on NFV on a HUAWEI 2288H V5 server. The server's CPU is an Intel(R) Xeon(R) Silver 4214 CPU (48 cores), the memory is 64GB, the storage is 4TB, the server runs the operating system version Ubuntu 20.04, and the kernel version is Linux 5.4.0-153-generic. To build a policy routing test scenario using NFV, you need to run the FRRouting suite in the LXD container to form virtual network devices, such as virtual routers, virtual hosts, and virtual servers, and then use bridges to connect them into the required network scenario. The topology of the test scenario is the NSFNET topology, and the OSPF routing protocol is used between routers. Figure 2 As shown, there are 14 virtual routers R0~R 13 There are 2 virtual hosts H1 and H2, and there are 19 links between nodes.

[0041] exist Figure 2 In the network prototype system shown, the links between virtual network devices are defined with performance parameters of 100-600Mbps bandwidth, 2-15ms latency, 0.01-0.09% packet loss rate, and 2-10 levels of security. Network fluctuations are simulated by resetting link attribute values ​​within a certain period of time. A server is set up as a monitor, which can read the information of all nodes and links and pass it to the learning engine. When the network information fluctuates greatly, the real-time data can be updated to the learning engine. A server is set up to run the learning engine software. The learning engine runs OPR based on the input network status information big data. - The RL algorithm trains a set of several optimal paths and passes them to the controller for matching settings. - The RL algorithm is implemented based on Python 3.11 and Pytorch 2.0.1.

[0042] Experiment 1: Application of OPR-RL algorithm in traffic engineering with QoS guarantee

[0043] Located in Figure 2 In the scenario, an application transmits large data between source node H1 and destination node H2, requiring bandwidth greater than 600Mbps and latency less than 100ms, with no other requirements. Since there is no end-to-end path in the network that can meet the above QoS performance indicators, we use traffic engineering methods to solve this problem.

[0044] Table 1 Algorithm parameters

[0045] parameter meaning value γ Discount Factor 0.9 <![CDATA[e initial ]]> Initial Probability 0.7 <![CDATA[α initial ]]> Learning initial efficiency 0.5 rate Rate of change 0.0001 episodes Iterations 1000 ∑{path} Number of search paths 3 condition Path QoS Requirements {600,100,1,0,999} coefficient Path attribute parameters {0.7,0.3,0,0,0}

[0046] First, RL runs the OPR-RL algorithm and adjusts the parameters as shown in Table 1. The reward function and path performance function are set according to equations (4) and (7). As a result, we obtain three paths with the largest reward value, as shown in Table 2. We can select two of these paths as traffic engineering solutions, so that the packet flow flows from two different ports of H1 to the destination node, thereby meeting the application requirements of large data transmission.

[0047] Table 2 Set of paths that meet QoS

[0048]

[0049] Secondly, the controller software defines the execution policy routing settings, drives the executors along the way to download the corresponding VNF tools, and completes the two paths that meet QoS output by the learning engine.

[0050] Next, we compare the OSPF routing algorithm using IP to obtain Figure 2 The path from H1 to H2 is: H1->R0->R1->R7->R9->R13->H2, with a bandwidth of only 197Mbps and a latency of 51ms. If path 1 and path 2 are selected for policy routing configuration, the bandwidth utilization rate is increased by about 325% compared with the default path. It can be seen that the traffic engineering solution provided by OPR-RL can guarantee the QoS required by the application.

[0051] Experiment 2: Comparison of average search time of different algorithms

[0052] Figure 3 The average search time of the OPR-RL algorithm, the Q-Learning algorithm, and the PSO multi-attribute optimization path algorithm in networks of different complexity levels was compared. Analysis shows that when searching for an optimization path, the search time of the OPR-RL algorithm is lower than that of the PSO algorithm. Compared with the Q-Learning algorithm, when the network topology is small, the search time of the OPR-RL algorithm is slightly higher than that of the Q-Learning algorithm, and when the network topology is large, the OPR-RL algorithm shows a shorter search time. When searching for two sets of optimization paths, the search time overhead of the OPR-RL algorithm increases slightly and can be completed within a few seconds. The algorithm's CPU and memory usage are both increased by less than 0.1%, and the resource consumption is not high, so the performance is guaranteed in the real environment. As the network environment becomes more complex, the average convergence time of the algorithm gradually increases. When the number of nodes is large, the growth rate of the algorithm convergence time overhead is greater.

[0053] Experiment 3: Relationship between the number of nodes and the number of algorithm iterations

[0054] Figure 4The number of iterations and convergence of the OPR-RL algorithm for network topologies of 30, 50, and 75 nodes are shown. The algorithm uses dynamic parameter adjustment, and the segmentation of the broken line indicates that the algorithm has searched a new path. Analysis shows that the algorithm gradually converges with the number of paths searched; secondly, the number of iterations of the algorithm first increases significantly with the increase in the number of nodes, and then increases slowly; finally, in a network with a certain number of nodes, the number of iterations spent by the algorithm to search for each new path increases significantly.

[0055] Experiment 4: The impact of the number of nodes on the algorithm convergence time

[0056] In order to investigate the influence of the selection probability e and learning rate α of the dynamic action selection strategy on the convergence time and search time of the OPR-RL algorithm, four control groups were set, including three groups of fixed values: e = 0.7, α = 0.5 and e = 0.8, α = 0.3 and e = 0.9, α = 0.1, and a group of dynamic values ​​of e and α. The other parameter settings are consistent with Table 2. Figure 5 The iteration time fitness of the algorithm searching for four optimization paths is shown. Analysis shows that when the OPR-RL algorithm uses dynamic values ​​of e and α, it has better fitness and shorter iteration time.

Claims

1. According to technical solution 1, a strategy routing optimization method OPR-RL based on reinforcement learning is proposed, which is characterized by A. Definition of P g = {B, D, L, S, H} represents the bandwidth set, delay set, packet loss rate set, security set and node set of all links in the network G respectively. Definition Represent link bandwidth, delay, packet loss rate, security and path node set respectively. In order to ensure that the path solved by the algorithm has QoS requirement guarantee, the QoS requirement guarantee is defined as Condition = {b min , d max , l max ,s min ,h max }, respectively represent the minimum bandwidth requirement, maximum delay limit, maximum packet loss rate limit, minimum security limit and maximum node hop limit. The link attributes of the searched path must satisfy formula (1): B. When the agent searches for a path, the set of optional nodes V and link parameters P observed when it is at the current node can be considered as its state S; the set of next-hop routes inferred by the agent based on the current state S can be considered as the action space A; when the agent moves, it is required to maximize future rewards, and its state transition probability from one node to an adjacent node is stable and only depends on the current position and has nothing to do with the previous position, searching for a multi-attribute optimization path with QoS guarantee k The process is a Markov decision process (MDP). For different application requirements, the algorithm needs to consider the impact of path attributes separately, and define θ, β, γ, δ, ε, a total of 5 parameters, representing P g = the importance ratio of {B, D, L, S, H} in path selection, and the sum of these five parameters is 1, as shown in formula (2): Coefficient={θ, β, γ, δ, ε|θ+β+γ+δ+ε=1} (2) The maximum and minimum normalization method of formula (3) is used for processing, where x = {b kj , d kj , l kj ,s kj }, X = {B, D, L, S}: M x =(x-min{X}) / (max{X}-min{X}) (3) In the multi-attribute optimization path search process, the reward function determines the reward that the agent can obtain when selecting a node, and indirectly affects the agent's next choice of node. The reward function is expressed by formula (4), and the reward value it returns represents P g The overall benefit of the search path. The larger the value, the more the link meets the path requirements of the current application. Reward=αM b +βM d +γM l +δM s (4) C. Optimized Policy-based Routing based on Reinforcement Learning (OPR-RL), first initialize the Q-Table based on the state space S and action space A obtained from the knowledge plane. Initialize the current state to the source node s = v s , use the action selection strategy to select the next hop node. If the next hop node exists, use formula (5) to update the Q-Table and continue to iterate until the destination node, where R is the reward value of the reward function formula (4) and γ is the discount factor. If the next hop node does not exist or passes through a repeated node, R is a large negative reward to avoid the node being selected again. If the next hop node is the destination node v d , storing the ordered nodes that the path passes through to the pathList collection and execute the next step. After that, the iterative difference is calculated. If the difference reaches the convergence threshold, the iteration is exited, otherwise, the next iteration is continued. Q(S t ,A t )←Q(S t ,A t )+α[R+γ*MaxQ(S t+1 ,A t+1 )-Q(S t ,A t )] (5) In addition, during the iteration process, we dynamically adjust the selection probability e and learning efficiency α of the action strategy based on the number of paths searched, as shown in Equation (6). Where ∑{path} represents the number of paths currently searched, rate represents the change rate, step represents the number of steps the current agent moves, and e initial and α initial Represents the initialization value of probability and efficiency. This dynamic strategy indicates that as the number of searched paths and the number of steps the agent moves increases, the e value gradually increases and the α value gradually decreases. In the early iteration of the algorithm, there is a greater possibility to randomly explore links, which helps to find the destination node faster; in the later iteration of the algorithm, it will rely more on the knowledge that has been learned, reducing random and meaningless searches. After the learning engine determines a set of two or more end-to-end paths, it first uses formula (1) to determine the QoS requirements of the paths. If the requirements are not met, the path is directly discarded. If the requirements are met, the comprehensive performance function shown in formula (7) is used to calculate the performance of all paths: Fitness(path k )=θN b +βN d +γN l +δN S +εN h (7) Among them, in order to make Fitness(path k ) values ​​are meaningful, and the parameters with different dimensions need to be normalized, as shown in formula (8). After the paths are comprehensively sorted, several better paths ranked in the top are returned. If all paths do not meet the defined QoS requirements, several paths closest to constraint (1) and the attribute information of the several paths are returned to the controller for backup selection. D. The pseudo code description of the OPR-RL algorithm is as follows: