Network routing system and method based on kdn architecture and multi-agent reinforcement learning

Abstract

The application provides a network routing system and method based on KDN architecture and multi-agent reinforcement learning, a data plane is responsible for storing, forwarding and processing data packets, a control plane constructs original data of a network by interacting with the data plane, fills all traffic matching switch flow tables according to best path information obtained by a knowledge plane, and a management plane stores and aggregates data of a database and original data of the network constructed by the control plane, performs data processing and analysis, judges key nodes, deploys agents on the key nodes by the knowledge plane, calculates the best path between any two nodes, and transmits the best path information to the control plane; the added knowledge plane solves the problem that the prior art center does not perform network automatic exploration, the proposed multi-agent intelligent routing system can reduce network delay and packet loss rate and improve throughput, and a multi-agent selection module of the management plane can better balance performance and overhead by selecting key nodes.

Description

Technical Field

[0001] This invention relates to the field of network communication technology, specifically to a network routing system and method based on KDN architecture and multi-agent reinforcement learning. Background Technology

[0002] With the development of network communication technology and the significant increase in user demand, new requirements have been put forward for research on computer network routing optimization in order to ensure the high performance, stability, robustness and rapid convergence of the network environment.

[0003] Traditional routing schemes (such as the shortest path algorithm Dijkstra and the multi-path routing algorithm ECMP) rely on manual configuration by network administrators, cannot learn from historical network conditions, and their routing optimization metrics are singular, targeting only a single network indicator, such as hop count, remaining bandwidth, or overhead, which is not conducive to improving overall network performance. In one of the retrieved patent documents (application number: CN202010067425.X), a genetic algorithm is used to uniformly plan routes to address flow conflicts caused by limited network bandwidth in high-load networks, improving overall network throughput and alleviating network congestion from a global perspective. However, heuristic algorithms have strict applicable scenarios, incur huge computational overhead, and are also difficult to handle real-time, highly dynamic network environments.

[0004] Machine learning-based traffic routing optimization has become a mainstream trend in SDN. Since network traffic path selection can be viewed as a decision problem, the emerging intelligent optimization methods mainly involve supervised learning and reinforcement learning. Supervised learning requires labeled datasets and is not suitable for variable network environments, so reinforcement learning (RL) has gradually become the main method for solving routing problems.

[0005] Deep reinforcement learning applied to route optimization can be divided into two categories based on the continuity of the action space. One category involves obtaining the Q-value of a path and selecting the path with the highest Q-value as the next-hop path. One paper that can be found describes a deterministic policy gradient (STDPG) agent that utilizes spatial and temporal features to select the optimal path from all source and destination paths, reducing network latency. However, since the candidate path set is the operational space, routing quality is affected by the path set itself. Patent document 2 (application number: CN202011635365.3) proposes a Q-table-based QoS routing optimization algorithm for SDN core networks. It obtains the optimal forwarding path based on the iteratively updated Q-table, but the Q-table incurs significant storage overhead as the topology increases. Another category of research focuses on outputting link weights or traffic splitting ratios, and then using these ratios to influence routing decisions. One study proposed DROM, which uses neural networks instead of Q-tables to save the storage overhead of maintaining Q-tables. It optimizes network latency and throughput by changing the weights of links. However, this indirect way of obtaining node routing paths has a certain path calculation delay, and the optimization effect is not as good as the way of directly obtaining the path.

[0006] Most current research deploys a unified agent within the controller. This approach is only suitable for small-scale networks and is difficult to apply to real-world systems because its computational complexity increases exponentially with network size. In complex networks, the increased state and action spaces further complicate neural networks. Recent work has focused on deploying multiple agents across multiple nodes in the network environment for path selection. One paper in this direction proposes Deep Graph Attention Network Routing (DGA TR), extending RL algorithms to multi-agent models to solve routing problems. Each router is treated as an independent agent, capable of making its own routing decisions. While this method avoids the impact of network size on the model, deploying agents on each node incurs significant deployment overhead.

[0007] In summary, existing intelligent routing methods based on deep reinforcement learning still have some problems:

[0008] 1. In a network topology, if the current network node experiences congestion, nearby nodes will also experience varying degrees of congestion to a large extent. Current methods do not consider the impact of nearby nodes on the current node.

[0009] 2. Most studies typically use an agent for training, but when the network becomes complex, the increased state space and action space make the neural network more complicated. Some studies use alternative paths to reduce the action space, but the quality of routing is affected by the alternative paths.

[0010] 3. The dataset was used for training, but no exploration of network automation was conducted. Summary of the Invention

[0011] The purpose of this invention is to provide a network routing system and method based on KDN architecture and multi-agent reinforcement learning, so as to overcome the problems in the prior art that do not consider the influence of nearby nodes on the current node, the routing quality is affected by alternative paths, and there is no automated network exploration.

[0012] To achieve the above objectives, the present invention provides the following technical solution:

[0013] A network routing system based on KDN architecture and multi-agent reinforcement learning, including a knowledge plane, management plane, data plane and control plane;

[0014] The data plane is responsible for storing, forwarding, and processing data packets;

[0015] The control plane constructs the network's raw data by interacting with the data plane and stores the data in the database. At the same time, it populates the flow tables of all matching traffic based on the best path information obtained from the knowledge plane.

[0016] The management plane stores and aggregates the data stored in the database and the raw network data constructed by the control plane, and performs data processing and analysis to identify key nodes in the knowledge plane;

[0017] The knowledge plane deploys agents for key nodes, calculates the optimal path between any two nodes, and transmits the optimal path information to the control plane.

[0018] Preferably, the control plane includes an information statistics module, a routing calculation module, and a flow table installation module. The information statistics module collects node and link information of the data plane topology. The routing calculation module is responsible for calculating the alternative path between any two points in the network device. The flow table installation module fills the flow tables of all traffic-matching switches with the best path information obtained from the knowledge plane through active installation.

[0019] Preferably, the knowledge plane includes a multi-agent routing module and a path detection module. The multi-agent routing module is used to obtain the best path information, and the path detection module is used to avoid routing loop problems.

[0020] Preferably, the multi-agent routing module is based on the MAPPO model and uses a centralized training and decentralized decision-making approach to iterate and update the algorithm.

[0021] Preferably, the management plane includes a data processing module and a multi-agent selection module. The multi-agent selection module is used to select key nodes, and the data processing module uses the raw data collected by the control plane and the information in the database to update and calculate the status and reward information.

[0022] Preferably, the multi-agent selection module selects key nodes in the network based on the global network topology information obtained by the RYU controller.

[0023] Preferably, the specific method for selecting key nodes in the network is as follows:

[0024] Obtain the degree centrality, eigenvector centrality, and betweenness centrality of each node at the topology level. Obtain the node path metrics based on link weight and link bandwidth at the business level. Then, apply the entropy weight method + TOPSIS ranking method to the metrics at the topology level and the business level respectively to obtain the Z value of each topology level and the business level. Finally, obtain the node score.

[0025] Preferably, the formula for obtaining the node's score is:

[0026] Z i =W topo ZTI i +W routing ZRI i

[0027] Among them, W topo ZTI is the weight of the Z-value of a node at the topology level. i ZRI is the Z-value at the node topology level. i For the Z-value at the node business level, W routing The weight of the Z-value of the node at the business level.

[0028] Preferably, the degree centrality is calculated as follows:

[0029]

[0030] Where, d i This represents the number of edges currently connected to the node, where N-1 represents the number of edges connecting the node to all other nodes.

[0031] The eigenvector centrality is calculated as follows:

[0032]

[0033] Where x i The importance of node i is represented by λ, which is a constant and λ ≠ 0. ij Let x be the adjacency matrix. j The importance of node j

[0034] The method for calculating betweenness centrality is as follows:

[0035]

[0036] in, This represents the number of paths that pass through the node and are the shortest path, where s is the source node, t is the destination node, and n is the destination node. st This represents the number of shortest paths connecting s and t.

[0037] A network routing method based on KDN architecture and multi-agent reinforcement learning, based on the aforementioned system, includes: acquiring global network topology information; selecting key nodes based on the global network topology information; deploying agents on the key nodes; acquiring the traffic demand between a random pair of source and target nodes; selecting the optimal path between the source and target nodes; checking the validity of the path; if the path is invalid, it is not installed; if the path is valid, the flow table of all matching switches is filled; and the process of acquiring the traffic demand between a random pair of source and target nodes is repeated until 300,000 pairs of source and target node path information are acquired.

[0038] Compared with existing technologies, this invention has the following advantages: This invention provides a network routing system based on KDN architecture and multi-agent reinforcement learning, including a knowledge plane, a management plane, a data plane, and a control plane. The data plane is responsible for storing, forwarding, and processing data packets. The control plane interacts with the data plane to construct the original network data and stores it in a database. Simultaneously, based on the optimal path information obtained from the knowledge plane, it populates the flow tables of all matching traffic switches. The management plane stores and aggregates the data stored in the database and the original network data constructed by the control plane, performs data processing and analysis, identifies key nodes in the knowledge plane, deploys agents for key nodes, calculates the optimal path between any two key nodes, and transmits the optimal path information to the control plane. Based on the KDN closed-loop control mechanism, a knowledge plane is added. The KDN architecture design promotes network automation and intelligence. The proposed multi-agent intelligent routing system can reduce network latency and packet loss rate, and improve throughput. The multi-agent selection module of the management plane can better balance performance and overhead by selecting key nodes.

[0039] Furthermore, by using the entropy weight method combined with the TOPSIS ranking method to select important nodes, the node evaluation method becomes more comprehensive, objective, and accurate.

[0040] Furthermore, by adopting the MAPPO multi-agent centralized training and decentralized decision-making implementation architecture, communication overhead can be reduced while still achieving agent communication. Attached Figure Description

[0041] Figure 1A comparison chart of unified deployment and multi-agent deployment.

[0042] Figure 2 A diagram of the centralized training and decentralized decision-making structure for MAPPO.

[0043] Figure 3 This invention presents an intelligent routing system framework based on KDN architecture and multi-agent reinforcement learning.

[0044] Figure 4 These are three network topologies tested in this invention. Detailed Implementation

[0045] The present invention will be further described in detail below with reference to specific embodiments. These descriptions are for explanation purposes only and are not intended to limit the scope of the invention.

[0046] like Figure 2 As shown, this invention provides a network routing system based on KDN architecture and multi-agent reinforcement learning, including a knowledge plane, a management plane, a data plane, and a control plane. The data plane is responsible for storing, forwarding, and processing data packets. The control plane interacts with the data plane to construct the original network data and stores it in a database. Simultaneously, based on the optimal path information obtained from the knowledge plane, it populates the flow tables of all matching traffic switches. The management plane aggregates the data stored in the database and the original network data constructed by the control plane, performs data processing and analysis, and identifies key nodes in the knowledge plane. The knowledge plane deploys agents for key nodes, calculates the optimal path between any two nodes, and transmits the optimal path information to the control plane.

[0047] 1. Data plane

[0048] The data plane is responsible for storing, forwarding, and processing data packets, including those from forwardable network devices and the links connecting those devices. Traffic in the data plane is forwarded according to the path information installed in the flow tables of the forwarding devices. The flow tables rely on knowledge acquired from the control plane, management plane, and knowledge plane for population and updating. The topology types constructed in the data plane are shown in the appendix. Figure 4 As shown.

[0049] 2. Control plane

[0050] The control plane interacts with the data plane to construct raw network data and stores it in a database for easy statistics and management. The control plane mainly consists of three modules: an information statistics module, a route calculation module, and a flow table installation module. The information statistics module collects node and link information from the data plane topology and provides a global view of network updates. The route calculation module calculates alternative paths between any two points in the network, avoiding the impact of multi-agent routing loops on network security. The flow table installation module, based on the optimal path information obtained from the knowledge plane, actively populates the flow tables of all matching switches.

[0051] 3. Management plane

[0052] The management plane stores and aggregates fine-grained information from the data plane stored in the database, as well as information from the global network view formed by the control plane, providing statistical analysis of the underlying forwarding device information. This plane includes a multi-agent selection module and a data processing module. The multi-agent selection module obtains the comprehensive importance of network nodes based on network topology layer evaluation indicators and service evaluation indicators related to routing optimization, thereby selecting important nodes in the network. Multiple agents in the knowledge plane are deployed only on important nodes. The data processing module uses the raw data collected by the control plane and information from the database to update and calculate status and reward information.

[0053] Furthermore, the multi-agent selection module selects key nodes in the network based on the global network topology information obtained by the RYU controller. This module, starting from the topology level of network attributes and the service level of route optimization, combines the entropy weight method and the TOPSIS ranking algorithm to obtain the comprehensive importance of nodes in the network. It mainly includes the following steps:

[0054] (1) First, we describe the evaluation metrics at the topological level, using three commonly used metrics: degree centrality, eigenvector centrality, and betweenness centrality.

[0055] Degree centrality (DC): The most direct measure of node centrality. Degree centrality describes the local characteristics of a node; the higher the value, the more important the node. For a given node v... i The degree centrality is calculated as follows:

[0056]

[0057] Where, d i Indicates the existing node v i The number of connected edges, N-1 represents the number of nodes v. i The number of edges that are connected to all other nodes.

[0058] Eigenvector centrality (EC): If the access path of a node from v iTo vj, then v i A malfunction occurred, v j It may be due to v i A failure results in the loss of access. In other words, the more important the nodes a node is connected to, the more important that node becomes. Eigenvector centrality assigns high eigenvector centrality to nodes with few but important connections. Assume x... i Indicate the importance of node i:

[0059]

[0060] Where λ≠0, x=[x1,x2,x3,......,x n ] T When a steady state is reached after multiple iterations, it can be written as: Ax = λx

[0061] Betweenness Centrality (BC): Node betweenness refers to the number of nodes in a network that can be accessed through nodes v. i The number of shortest paths through a node. Betweenness centrality characterizes the global properties of a node. If the number of shortest paths through a node is large, then the node is influential. The betweenness centrality of a node is calculated as follows:

[0062]

[0063] in, n represents the number of paths that pass through node i and are the shortest paths. st This represents the number of shortest paths connecting s and t.

[0064] (2) Secondly, in services related to route optimization, link weight and link bandwidth are two elements with a significant impact, and therefore they are used as our service evaluation indicators.

[0065] Link-weighted node path metric (WI): refers to the shortest path through node v in the topology obtained using link weights. i The number of shortest paths. Edge weights in a network represent the strength of a certain association between nodes, such as distance between two nodes, cost, or a certain level of difficulty. The more times a node is visited, the greater its importance.

[0066] Link bandwidth-based node path metric (BI): refers to the shortest path through node v in the topology obtained using link bandwidth. i The number of shortest paths. Network link bandwidth represents the link capacity that can be effectively utilized in the entire transmission entity. The smaller the bandwidth, the smaller the data throughput that can be accommodated, and the less traffic that can be transmitted in the same time period. Therefore, links with larger bandwidth are usually less prone to congestion.

[0067] The above two metrics are calculated using the formulas shown below. In the shortest path calculated using link weight or bandwidth, y represents the number of paths that pass through node i and are the shortest paths. st This represents the number of shortest paths connecting s and t.

[0068]

[0069] (3) Finally, we applied the entropy weight method + TOPSIS ranking method, which are common evaluation methods in current mathematical modeling, to the topology-level and business-level indicators to obtain the Z-values ​​of node i at the topology-level and business-level, namely ZTI. i ZRI i .

[0070] Assuming N represents the number of samples to be evaluated, i.e. the number of nodes in the network, and M represents the number of evaluation indicators involved, then the original indicator data matrix can be constructed as follows:

[0071]

[0072] In this context, the topology layer has M=3, and the business layer has M=2. X ij Let represent the value of the j-th evaluation indicator for the i-th sample. For the j-th indicator, its information entropy is calculated using the following formula: the smaller the information entropy, the more information the indicator can provide.

[0073]

[0074]

[0075] The entropy weight for each network metric is obtained using the following formula:

[0076]

[0077] Based on the TOPSIS method, we construct a weighted data matrix Y, where each element Y... ij The calculation is as follows:

[0078] Y ij =W j ×X ij

[0079] Find the optimal and worst solutions and calculate the Euclidean distance between each network metric and its corresponding metric using the following method.

[0080]

[0081]

[0082]

[0083] After performing the same steps at the topology or business level, the node score is obtained as follows:

[0084]

[0085] The overall importance of a node is calculated as shown below, W topo and W routing In this paper, all values ​​are set to 0.5. We believe that the topology layer and the business layer are equally important, and we select the nodes with larger Z values ​​as the key nodes in the network.

[0086] Z i =W topo ZTI i +W routing ZRI i

[0087] 4. Knowledge Horizon

[0088] The knowledge plane learns network behavior from information within the network to determine the optimal path between two points. This path is obtained using a MAPPO-based intelligent routing algorithm, and an example entry in the repository is shown below: (src=1, dst=5, route=[1,3,4,5]). The SDN controller installs flow table entries based on this path, thereby changing the path for traffic forwarding. Simultaneously, this plane includes a path detection module to avoid routing loops. If duplicate nodes are found in the existing path set, alternative paths stored in the database are used for path optimization, increasing the robustness and feasibility of the entire system.

[0089] Furthermore, the MAPPO-based multi-agent routing module employs additional... Figure 2 The algorithm iterates and updates using a centralized training and decentralized decision-making approach. We deploy multiple agents on key nodes selected by the multi-agent selection module. These agents deployed on the switch can be considered fully cooperative, working together to achieve high returns and overall network performance. The specific operation steps are as follows:

[0090] Initially, the agent transmits its state observation data to the central controller. After receiving the global state S, the central controller trains the model, obtains the optimal distributed control policy, and then transmits it to the agent. Once training is complete, the agent no longer needs to communicate with the central controller and can generate the optimal action based on its own local observation state and its own action policy function. Specifically, θ = [θ1, θ2, ..., θ...] n ] represents the parameters of n agents, π = [π1, π2, ..., π] n ] represents the policy of n agents, where each agent i in MAPPO is based on local observations oi and independent policy actor network π θ (a i |o i Generate an action a i To maximize cumulative rewards from discounts Learn a central value function V based on the global state s. φ (s). The optimization objective of the actor network is:

[0091]

[0092] The optimization objective of the critic network is:

[0093]

[0094] Specifically, the MAPPO agent interacts with the network environment through the three elements of reinforcement learning: state space, action space, and reward function. By inputting the corresponding state, the agent can make its own decisions and update the model parameters based on the reward of the current environment.

[0095] State space: The state space is the set of states that the agent can observe. The local state space of agent i includes: a randomly selected pair of source and destination nodes for communication, the remaining capacity of neighboring links, the packet loss rate of neighboring links, and the bandwidth involved in the communication. The global state is the sum of the local states observed by agents in the topology.

[0096] Action Space: Each MAPPO agent can obtain the probability distribution of the next hop of its neighboring nodes based on the current state observed by the agent. Finally, the agent samples their actions to select a neighboring node as the next hop node. The size of the action space for each agent is the number of neighboring nodes, which can significantly reduce the size of the action space compared to a unified deployment scenario.

[0097] Reward Function: Considering that latency, throughput, and packet loss rate are important factors affecting network routing performance, we use these three factors as our evaluation metrics. The reward function is designed to be inversely proportional to latency and packet loss rate, and directly proportional to throughput. We also use normalization techniques to avoid the influence of different metric scales, as shown in the following formula:

[0098]

[0099] Where β1, β2 and β3 represent the weights of different performance indicators. These represent the normalized values ​​of latency, throughput, and packet loss rate in the network, respectively.

[0100] Furthermore, the implementation environment is described below:

[0101] We used Python scripts to build and deploy the test topology in Mininet, and employed the iperf traffic simulation tool to generate traffic. A Python-based Ryu controller was used in the control plane. For the management and knowledge planes, multi-agent routing and data processing modules were developed using Python 3.6 and PyTorch 1.8. We considered three network topologies for training and evaluating the model: a 14-node NSFNet, a 23-node GEANT2, and a 36-node mesh topology to test network performance variations.

[0102] The intelligent routing optimization system based on the KDN architecture described in this invention can collect data plane topology information in real time. Our multi-agent scheme can select the optimal network path for each pair of source and destination nodes in the network based on the global view information processed by the management plane. This scheme can reduce network latency and packet loss rate, and improve throughput, especially in more complex topology scenarios. Through experimental testing, we found that the combination of the MAPPO model and the multi-agent selection module can achieve excellent routing performance in different network environments. Furthermore, our experimental design and evaluation metrics have a certain degree of scalability and applicability, providing a reference for future related research.

[0103] Meanwhile, this invention provides a network routing method based on KDN architecture and multi-agent reinforcement learning, including obtaining global network topology information, selecting key nodes based on the global network topology information, deploying agents on the key nodes, obtaining the traffic demand between a random pair of source and target nodes, selecting the optimal path between the source and target nodes, checking the legality of the path, and if it is invalid, not installing it; if it is valid, filling the flow table of all matching switches, and repeatedly obtaining the traffic demand between a random pair of source and target nodes until 300,000 pairs of source and target node path information are obtained.

[0104] Although embodiments of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the specific embodiments and application fields described above. The specific embodiments described above are merely illustrative and instructive, and not restrictive. Those skilled in the art, guided by the specification, can make many other modifications without departing from the scope of the claims of the present invention, and all of these modifications are within the scope of protection of the present invention.

Claims

1. A network routing system based on KDN architecture and multi-agent reinforcement learning, characterized in that, It includes the knowledge plane, management plane, data plane, and control plane; The data plane is responsible for storing, forwarding, and processing data packets; The control plane constructs the network's raw data by interacting with the data plane and stores the data in the database. At the same time, it populates the flow tables of all traffic-matching switches based on the optimal path information obtained from the knowledge plane. The management plane stores and aggregates the data stored in the database and the original network data constructed by the control plane, and performs data processing and analysis to determine the key nodes in the knowledge plane. The knowledge plane deploys agents for key nodes, calculates the optimal path between any two nodes, and transmits the optimal path information to the control plane. The management plane includes a data processing module and a multi-agent selection module. The multi-agent selection module is used to select key nodes. The data processing module uses the raw data collected by the control plane and the information in the database to update and calculate the status and reward information. The multi-agent selection module selects key nodes in the network based on the global network topology information obtained by the RYU controller. The selection of key nodes involves choosing nodes with higher node scores as key nodes in the network. The specific method for selecting key nodes in the network is as follows: Obtain the degree centrality, eigenvector centrality, and betweenness centrality of each node at the topology level. Obtain the node path indicators based on link weight and link bandwidth at the business level. Then, apply the entropy weight method + TOPSIS ranking method to the topology level and business level indicators respectively to obtain the Z value of each topology level and business level. Finally, obtain the node score. The formula for obtaining a node's score is: in, The weights of the Z-values ​​of nodes at the topology level. The Z value represents the node topology level. The Z-value represents the node's business layer. The weight of the Z-value of the node at the business level.

2. The network routing system based on KDN architecture and multi-agent reinforcement learning according to claim 1, characterized in that, The control plane includes an information statistics module, a routing calculation module, and a flow table installation module. The information statistics module collects node and link information of the data plane topology. The routing calculation module is responsible for calculating the alternative path between any two points in the network device. The flow table installation module fills the flow tables of all traffic-matching switches with the best path information obtained from the knowledge plane through active installation.

3. The network routing system based on KDN architecture and multi-agent reinforcement learning according to claim 1, characterized in that, The knowledge plane includes a multi-agent routing module and a path detection module. The multi-agent routing module is used to obtain the best path information, and the path detection module is used to avoid routing loop problems.

4. The network routing system based on KDN architecture and multi-agent reinforcement learning according to claim 3, characterized in that, The multi-agent routing module is based on the MAPPO model and uses a centralized training and decentralized decision-making approach to iterate and update the algorithm.

5. The network routing system based on KDN architecture and multi-agent reinforcement learning according to claim 1, characterized in that, Degree centrality is calculated as follows: in, This represents the number of edges currently connected to node i. This represents the number of edges connecting a node to all other nodes. The eigenvector centrality is calculated as follows: in Indicate the importance of node i. It is a constant. , It is an adjacency matrix. The importance of node j The method for calculating betweenness centrality is as follows: in, This represents the number of paths that pass through node i and are the shortest path, where s is the source node and t is the destination node. This represents the number of shortest paths connecting s and t.

Images