A key node based distributed scalable intelligent routing method

Abstract

The application provides a kind of distributed scalable intelligent routing method based on key node, belongs to network resource scheduling optimization field.The method according to network topology scale, network density and the centrality of various nodes calculates network average aggregation coefficient, key node proportion and node key degree, determines the position of agent deployment by comprehensively considering key node proportion and node key degree;Agent generates routing strategy according to service flow demand and network state, network feedback utility function after executing strategy, and agent adjusts parameters through feedback to optimize strategy;After the training of agent is completed, routing strategy is continuously adjusted with the change of network state, and the network performance is continuously optimized.The application realizes complex network control through a small number of agent nodes, effectively improves network throughput, reduces average service delay and deployment cost on the basis of not greatly changing existing network environment, has good scalability, can be flexibly deployed under different network scales, and is suitable for engineering implementation.

Description

Technical Field

[0001] This invention relates to the field of distributed intelligent routing optimization, and in particular to a distributed scalable intelligent routing method based on key nodes. Background Technology

[0002] With the development of Internet technology, the complexity and dynamism of communication networks continue to increase. Breakthroughs in materials science and manufacturing processes have driven the continuous improvement of the processing capabilities of terminal devices. However, the development speed of network equipment and transmission channels is relatively slow. A series of new services such as live video streaming, e-sports, telemedicine, and financial data have placed higher demands on the performance of communication networks. "High reliability, low latency, and large bandwidth" have become the development goals of future network control technology.

[0003] Traditional routing methods typically employ shortest path algorithms to plan routes, designing fixed network traffic models and solving for the objective function. Common examples include distance-vector routing algorithms such as RIP, IGRP, and EIGRP, and link-state routing algorithms such as OSPF and IS-IS. These algorithms are widely deployed in various environments and are mature in use. However, the best-effort forwarding approach cannot formulate differentiated routing strategies for network data flows with different characteristics, thus failing to meet the diverse service requirements in complex networks. Finding an efficient and adaptive network service routing control scheme to ensure network QoS, reduce unnecessary network resource overhead, and improve network resource utilization, while providing efficient and stable service support for upper-layer applications under limited computing, storage, and communication resources, is a pressing problem in current communication networks.

[0004] To address the problems of slow convergence speed in complex environments and inability to adapt to dynamic network environments in traditional routing methods, recent research has applied new technologies and architectures such as machine learning and SDN to routing planning, achieving some progress in areas such as congestion control and load balancing. Routing algorithms can be broadly categorized into two types based on the type of machine learning method they employ: supervised learning-based intelligent routing algorithms and reinforcement learning (RL)-based intelligent routing algorithms. Supervised learning-based intelligent routing algorithms can accurately calculate appropriate routing schemes using network state and topology information, while also demonstrating advantages over traditional methods in terms of improved convergence speed and reduced signaling overhead. However, using supervised learning algorithms always faces a problem: deep neural networks use millions or even tens of millions of parameters, which can only be treated as a black box. This makes intelligent algorithms difficult to debug, and their deployment scale is large, making detailed adjustments difficult.

[0005] Intelligent routing methods based on Deep Reinforcement Learning (DRL) typically require only a single computation to derive a near-optimal network configuration. They learn from real-world network data, continuously interacting with the environment without requiring any simplifications. By performing computations based on actual information, they can adapt to complex, nonlinear systems. However, the convergence of DRL models is strongly correlated with the output dimension. To circumvent this issue, most algorithms calculate routes indirectly, such as using DRL to calculate link weights and then employing other traditional algorithms to make routing decisions. This approach fails to achieve truly intelligent routing.

[0006] In recent years, research on intelligent routing has focused on improving network performance in specific scenarios. However, in real-world applications, due to factors such as large network scale and variable environments, the robustness and reliability of existing methods are insufficient, and these algorithms are far from meeting the requirements for daily network management. Therefore, designing a simple, effective, and low-cost routing scheme has become a research challenge and hot topic in the field of intelligent routing planning. Summary of the Invention

[0007] In view of this, the present invention proposes a distributed scalable intelligent routing method based on key nodes. This method uses the RL algorithm to construct distributed intelligent agents, determines the deployment location of intelligent agents according to the network topology scale and node criticality, and realizes complex network control through a small number of intelligent agent nodes. In the existing network environment, it effectively improves network throughput, reduces average service latency and deployment cost, has good scalability, can be flexibly deployed under different network scales, and is suitable for engineering implementation.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A distributed, scalable, intelligent routing method based on key nodes includes the following steps:

[0010] (1) Collect network link status information in real time, and calculate the v of each node in network G=(V,E) based on the network status using the centrality and criticality method. i Centrality criticality ρ i Where, V = {v1, v2, ... v} N} represents the set of nodes, N represents the total number of nodes, and E represents the set of all edges in the network topology;

[0011] (2) Arrange the nodes from largest to smallest according to their centrality criticality values, and take the nodes with the highest ranking as the key nodes of the network topology;

[0012] (3) Calculate the shortest path from the current node to all other target nodes using the Dijkstra method at the node, obtain the next-hop neighbor node, and form a routing table;

[0013] (4) Deploy reinforcement learning-based agents at the selected key nodes, start agent pre-training, randomly generate business flow requirements, recalculate the next-hop route for the business flow passing through the key nodes through the agent, standardize the performance indicators such as business flow latency and packet loss, determine the policy reward value according to the utility function and adjust the agent model parameters.

[0014] (5) In real business flow, the trained agent is used to perform routing calculations at each key node and learns continuously during the routing process to achieve intelligent routing.

[0015] Furthermore, the specific method of step (1) is as follows:

[0016] (101) Calculate the degree centrality (DC) of each node in the network. i Betweenness centrality BC i Proximity centrality CC i :

[0017]

[0018]

[0019]

[0020] Where, k i Indicates the existing node v i The number of connected edges, g st Indicates connection v s and v t The number of shortest paths, This indicates that the connection passes through node v. i And connect v s and v t The number of shortest paths, d ij Represents node v i to node v j The distance;

[0021] (102) Calculate the centrality and criticality of all nodes according to the formula for calculating node centrality and criticality:

[0022]

[0023] Furthermore, the specific method of step (2) is as follows:

[0024] (201) Calculate the network aggregation coefficient:

[0025]

[0026] Where, k i Represents node v i The number of all neighbors, E i Represents node v i The number of edges between all neighbors, C i Represents node v i The clustering coefficient, C represents the average clustering coefficient of the network;

[0027] (202) Determine the proportion m of key nodes based on network size and clustering coefficient:

[0028]

[0029] (203) Arrange the nodes from largest to smallest according to their centrality criticality values, and determine the critical nodes according to the proportion m.

[0030] Furthermore, the specific method of step (3) is as follows:

[0031] (301) All nodes spread their own topological connections and establish a topology table;

[0032] (302) The routing table is formed by calculating the topology table using the Dijkstra method, and the network state awareness data is exchanged.

[0033] Furthermore, the specific method of step (4) is as follows:

[0034] (401) Set the service type, service requirements and service size of the service flow, randomly generate service flow data, and adjust the duration of the service flow according to the network load level to simulate the real service flow status.

[0035] (402) When an agent exists in a certain business flow routing path, the agent decides the next hop action. All agents share the global state space (S, A, P, R), where S is the set of current states of all agents, and A = ∏ i∈N A i Denotes the joint action space of all intelligent agents, {A i} i∈N Let P be the action space of agent i, where P: S×A×S→[0,1] is the state transition probability, and R: It is a global reward calculation function. Each agent makes routing decisions based on the current network situation and service flow information, thereby increasing the global reward.

[0036] (403) Construct a state indicator variable K for a node relative to a business flow. If a business flow passes through node i, then K... i =1, otherwise K i =0; when Ki When the value is 1, node i will not be used to plan the next-hop route; the decision of the agent at node i is represented as follows. In the formula θ i To reinforce the learning of network parameters, s i This represents a partial state of the agent at node i, including service flow information, partial neighbor routing tables, and available link bandwidth. i Let K be the conditional state of the agent's decision at node i, consisting of the state indicator variable K; in order to maximize network transmission capacity, The goal is to maximize the global reward of the network, where J(θ) is the global objective function and θ is the parameter of each agent in the reinforcement learning network. i The set of values ​​is used to update θ by calculating the gradient of J(θ) using the following formula:

[0037]

[0038] Where τ represents the path through τ~p θ (τ) for π θ The sampled state-action pair (s, a) is used to estimate the policy π in state s. θ The degree of advantage of taking action 'a' relative to taking random action, when A(s,a)>0, is the policy π. θ Action 'a' will yield better results than random action 'π'. θ (a|s) represents the strategy of taking action a given network parameters θ and agent state S;

[0039] (404) Update the reinforcement learning network parameters by calculating the gradient function of the agent's policy parameters at node i. The calculation formula is as follows:

[0040]

[0041] Among them, A i (s i a) is the local unbiased estimate of the generalized dominance value A(s, a), when agent i is in the routing path. At this point, the agent will not update its policy parameters;

[0042] (405) The partial state s of the agent at node i i Conditional state c i Under the given conditions, the reward function for the action is calculated by using the output action of the reinforcement learning network, which is the probability distribution of the next-hop neighbor selection in the business flow:

[0043]

[0044] Where γ∈[0,1] is the reward discount factor, r tLet the utility value of the t-th business flow be expressed as:

[0045]

[0046] in Let α1, α2, and α3 represent the normalized throughput, latency, and packet loss of the t-th service flow, respectively. Normalization is calculated by dividing the parameters of the t-th service flow by the average value of the parameters of service flows of the same type and origin / target node. α1, α2, and α3 are non-negative scalars, representing the importance weights of performance indicators such as throughput, latency, and packet loss.

[0047] (406) After the path decision is completed by the agent, check whether there are loops or unsafe paths in the path whose link status does not meet the business flow requirements. If there are unsafe paths, the agent will give a negative scalar as a penalty for the decision reward feedback and recalculate the path using the Dijkstra method.

[0048] The beneficial effects of this invention are as follows:

[0049] 1. This invention introduces a reinforcement learning-based intelligent routing algorithm, which significantly improves network throughput and average service latency compared to traditional routing protocols.

[0050] 2. This invention adopts a distributed routing scheme, which determines the deployment of intelligent agents at key nodes based on network size and aggregation. This method has good scalability, can be flexibly deployed under different network sizes, and is easy to implement in engineering. Attached Figure Description

[0051] Figure 1 This is an overall flowchart of a distributed, scalable, intelligent routing method based on key nodes in an embodiment of the present invention.

[0052] Figure 2 This is a flowchart of the calculation of key nodes in an embodiment of the present invention.

[0053] Figure 3 This is a flowchart of the distributed multi-agent routing scheme in an embodiment of the present invention.

[0054] Figure 4 This is a simulation experiment network topology diagram in an embodiment of the present invention.

[0055] Figure 5 This is a schematic diagram comparing the average service latency under different numbers of intelligent agents in the simulation experiment of this invention. Detailed Implementation

[0056] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings from the simulation experiment.

[0057] A distributed, scalable, intelligent routing method based on key nodes, the main process of which is as follows: Figure 1 As shown, the method includes the following steps:

[0058] (1) Collect network link status information in real time, and calculate the v of each node in network G=(V,E) based on the network status using the centrality and criticality method. i Centrality criticality ρ i Where V represents the set of nodes, the total number of nodes is N, and E represents the set of all edges in the network topology;

[0059] (2) Arrange the nodes from largest to smallest according to their centrality criticality values, and take the nodes with the highest ranking as the key nodes of the network topology;

[0060] (3) Calculate the shortest path from the current node to all other target nodes using the Dijkstra method at the node, and calculate the next-hop neighbor node to form a routing table;

[0061] (4) Deploy reinforcement learning-based agents at the selected key nodes, start agent pre-training, randomly generate business flow requirements, recalculate the next-hop route for the business flow passing through the key nodes through the agents, standardize the performance indicators such as business flow latency and packet loss, determine the policy reward value according to the utility function and adjust the agent model parameters.

[0062] (5) In real business flow, the trained agent is used to perform routing calculations at each key node and learns continuously during the routing process to achieve intelligent routing.

[0063] This method reduces the number of agents required to control the network by introducing key network nodes. By deploying agents in conjunction with the locations of key network nodes, it improves the network control efficiency of a single agent, thereby driving the overall network with local intelligence and forming an intelligent network that can evolve autonomously.

[0064] Furthermore, such as Figure 2 As shown, in step (1), the centrality of a node is calculated using degree centrality, betweenness centrality, and proximity centrality. Degree centrality determines a node's importance by measuring the number of its neighbors; betweenness centrality determines the dependencies between nodes by dividing the shortest path through a node by the total number of shortest paths in the entire network, with a higher betweenness centrality indicating a greater influence of the node on other nodes in the network; proximity centrality is similar to betweenness centrality, also using network-wide characteristics to determine node centrality by measuring its centrality by the negative of the average distance from a node to other nodes in the network. The three centrality calculation methods are shown in the following formulas:

[0065]

[0066] Among them, DCi For node v i Degree centrality, BC i For node v i Betweenness centrality, CC i For node v i The proximity centrality of k i Indicates the existing node v i The number of connected edges, where N is the total number of nodes, g st Indicates connection v s and v t The number of shortest paths, This indicates that the connection passes through node v. i And connect v s and v t The number of shortest paths, d ij Represents node v i to node v j The distance.

[0067] Different centrality calculation methods measure the influence of nodes on the entire network from different perspectives. This method considers three different centrality parameters and proposes a formula for calculating the centrality criticality of nodes. The formulas for calculating the centrality criticality of all nodes are as follows:

[0068]

[0069] In the formula ρ i Represents node k i Centrality and criticality.

[0070] Furthermore, in step (2), the parameter m needs to be determined based on the network size and the clustering coefficient. The clustering coefficient measures the density of the network. For networks of the same size, a higher clustering coefficient means that the network can be controlled by fewer intelligent nodes. The formula for calculating the network clustering coefficient is as follows:

[0071]

[0072] In the formula, k i Represents node v i The number of all neighbors, E i Represents node v i The number of edges between all neighbors, C i Represents node v i The clustering coefficient, C represents the average clustering coefficient of the network.

[0073] The calculation method for the critical node selection scale parameter m is as follows:

[0074]

[0075] Furthermore, the distributed intelligent agent routing scheme process is as follows: Figure 3 As shown, in step (3), all nodes first spread their own topology connection relationship and establish a topology table. After the network converges and stabilizes, each node uses the Dijkstra method to calculate the distance to other target nodes and the next-hop neighbor number through the topology table to form a routing table. At the same time, they exchange network status awareness data. During network operation, each node can perceive and calculate the current network bandwidth, link load and service flow information based on the default value, set value or current real data.

[0076] Furthermore, in step (4), during the agent pre-training stage, the service type, service requirements and service size of the service flow are set, the service flow data is randomly generated, and the duration of the service flow is adjusted according to the network load level. For a certain service flow, when the starting point (current node) of the service flow is a non-agent, the next hop node is determined directly according to the routing table until the node is an agent.

[0077] When an agent exists in a certain business flow routing path, the agent decides the next hop action. All agents share a global state space (S, A, P, R), where S is the set of current states of all agents, and A = Π. i∈N A i Denotes the joint action space of all intelligent agents, {A i} i∈N Let P:S×A×S→[0,1] be the action space of agent i, where P:S×A×S→[0,1] is the state transition probability, and R: It is a global reward calculation function. The agent at each node makes routing decisions based on the current network situation and business flow information, while increasing the global reward.

[0078] Construct a state indicator variable K for a node relative to a business flow. If a business flow passes through node i, then K... i =1, otherwise K i =0; when K i When the value is 1, node i will not be used to plan the next-hop route; the agent's decision at node i can be represented as In the formula θ i Reinforcement learning network parameters, s i This represents a partial state of agent i, including service flow information, partial neighbor routing tables, and available link bandwidth. i The conditional state of the current agent's decision is represented by the state indicator variable K; to maximize network transmission capacity, The goal is to maximize the global reward of the network, where J(θ) is the global objective function and θ is the parameter of each agent in the reinforcement learning network. i The set of values ​​can be updated by calculating the gradient of J(θ), as shown in the following formula:

[0079]

[0080] In the formula, τ represents the passage through τ~p θ (τ) for π θ The sampled state-action pair (s,a) is used to estimate the policy π in state s. θ The degree of advantage of taking action 'a' relative to taking random action, when A(s,a)>0, is the policy π. θ Action 'a' will yield better results than random action 'π'. θ (a|s) represents the strategy of taking action a given network parameters θ and agent state S.

[0081] Furthermore, the gradient function of the agent's policy parameters at node i is:

[0082]

[0083] In the formula, A i (s i a) is the local unbiased estimate of the generalized dominance value A(s,a), when agent i is in the routing path. At this point, the agent will not update its policy parameters.

[0084] The partial state s of the agent at node i i Conditional state c i Under the given conditions, the reward function for the action is calculated by using the output action of the reinforcement learning network, which is the probability distribution of the next-hop neighbor selection in the business flow:

[0085]

[0086] In the formula, γ∈[0,1] is the reward discount factor, and r t The utility value of the t-th business flow can be expressed as:

[0087]

[0088] in Let α1, α2, and α3 represent the normalized throughput, latency, and packet loss of the t-th service flow, respectively. Normalization is calculated by dividing the parameters of the t-th service flow by the average value of the parameters of service flows of the same type and origin / target node. α1, α2, and α3 are non-negative scalars, representing the importance weights of performance indicators such as throughput, latency, and packet loss.

[0089] To mitigate insecure routing issues caused by random exploration during agent learning, this method introduces an intelligent decision path detection mechanism. After the agent completes a path decision, it should detect whether there are loops or insecure paths where the link status does not meet the business flow requirements. If an insecure path exists, the agent will use a negative scalar as a penalty for the decision reward and recalculate the path using the Dijkstra method.

[0090] The following embodiments simulate, for example Figure 4 The network shown further illustrates the above method.

[0091] (1) Read the simulated network topology information and the parameters of each node, calculate the degree centrality, betweenness centrality, and proximity centrality of each node, and use the formula The centrality criticality of each node was determined, and the calculation results are shown in Table 1:

[0092] Table 1. Calculation of Node Centrality in All Network Instances

[0093]

[0094] As shown in Table 1, the centrality criticality of each node in the network, from largest to smallest, is as follows: {V4, V6, V8, V7, V1, V3, V5, V9, V2, V10}.

[0095] (2) Calculate the network clustering coefficient based on the network topology information (number of nodes, number of edges). The calculation results of the clustering coefficient of each node in the network are shown in Table 2:

[0096] Table 2 Clustering coefficients of network nodes

[0097]

[0098] The table above shows that the average data aggregation coefficient for the entire network is 0.42. Using the formula... The critical node ratio is calculated, so the critical node ratio of the entire network is 0.6. The total number of nodes in the network is 10, so the number of critical nodes is 6. Based on the centrality and criticality, the 6 nodes with the higher criticality are selected as {V4, V6, V8, V7, V1, V3} as critical nodes, which are the positions where the agents are deployed in the network.

[0099] (3) Distributed intelligent agents are deployed at key node locations. The network randomly generates network-wide service traffic. In this embodiment, the network load is registered as 0.8, meaning the total network service traffic is 0.8 times the total network bandwidth. The average network service transmission latency is calculated for three scenarios: no intelligent agents deployed, intelligent agents deployed at key nodes, and intelligent agents deployed across the entire network. The latency is averaged every 3000 service flows. The latency curves for the three scenarios are shown below. Figure 5 As shown. By Figure 5 As can be seen, initially, the agent is in the learning phase, and the network with the agent deployed has a higher latency compared to the network without the agent deployed. However, after learning, the latency gradually decreases, eventually stabilizing and becoming lower than the average latency of the network without the agent deployed. The effects of deploying agents across the entire network and deploying agents at key nodes are similar. This embodiment demonstrates that the method of deploying agents at key nodes can effectively reduce the average network latency, improve network transmission performance, has good scalability, can be flexibly deployed under different network scales, is easy to implement in engineering, and reduces costs, thus achieving the expected goals.

[0100] In summary, this invention achieves complex network control with a small number of intelligent agent nodes, effectively improving network throughput, reducing average service latency and deployment costs without significantly altering the existing network environment. It also exhibits good scalability, can be flexibly deployed under different network scales, and is suitable for engineering implementation.

Claims

1. A distributed, scalable, intelligent routing method based on key nodes, characterized in that: Includes the following steps: (1) Collect network link status information in real time, and calculate the network centrality and criticality based on the network status. Each node Centrality Keyness ;in, Represents a set of nodes. N represents the total number of nodes, and E represents the set of all edges in the network topology; the specific method of step (1) is as follows: (101) Calculate the degree centrality of each node in the network. Betweenness centrality Proximity centrality : in, Represents a node The number of all neighbors, Indicates connection and The number of shortest paths, Indicates that the connection passes through nodes. And connection and The number of shortest paths, Represents a node To the node The distance; (102) Calculate the centrality and criticality of all nodes according to the formula for calculating node centrality and criticality: ； (2) Arrange the nodes from largest to smallest according to their centrality criticality values, and take the nodes with the highest ranking as the key nodes of the network topology; (3) Calculate the shortest path from the current node to all other target nodes using the Dijkstra method at the node, obtain the next-hop neighbor node, and form a routing table; (4) Deploy reinforcement learning-based agents at the selected key nodes of the network topology, start agent pre-training, randomly generate service flow requirements, recalculate the next-hop route for the service flow passing through the key nodes of the network topology through the agent, standardize the performance indicators of service flow latency and packet loss, determine the policy reward value according to the utility function and adjust the agent model parameters. (5) In real business flow, the trained agent performs routing calculations at key nodes of each network topology and learns continuously during the routing process to achieve intelligent routing.

2. The distributed, scalable, intelligent routing method based on key nodes according to claim 1, characterized in that, The specific method for step (2) is as follows: (201) Calculate the network aggregation coefficient: in, Represents a node The number of edges between all neighbors Represents a node The aggregation coefficient, This represents the average clustering coefficient of the network; (202) Determine the proportion m of critical nodes in the network topology based on the network size and clustering coefficient: (203) Arrange the nodes from largest to smallest according to the centrality criticality value, and determine the key nodes of the network topology according to the proportion m.

3. The distributed, scalable, intelligent routing method based on key nodes according to claim 1, characterized in that, The specific method for step (3) is as follows: (301) All nodes spread their own topological connections and establish a topology table; (302) The routing table is formed by calculating the topology table using the Dijkstra method and the network state awareness data is exchanged.

4. The distributed, scalable, intelligent routing method based on key nodes according to claim 1, characterized in that, The specific method of step (4) is as follows: (401) Set the service type, service requirements and service size of the service flow, randomly generate service flow data, and adjust the duration of the service flow according to the network load level to simulate the real service flow status; (402) When there is an agent in a certain business flow routing path, the agent decides the next hop action, and all agents share the global state space. In the formula, S is the set of current states of all agents. Represents the joint action space of all intelligent agents. For intelligent agents The space of motion Let be the state transition probability. It is a global reward calculation function. Each agent makes routing decisions based on the current network situation and service flow information, thereby increasing the global reward. (403) Construct a state indicator variable for a node relative to the business flow. If a certain business flow passes through a node ,but ,otherwise ;when When it is 1, the node It will not be used to plan the next-hop route; node The decision-making representation of the agent at the location is as follows In the formula To enhance the learning of network parameters, For nodes The system displays partial states of the agent, including service flow information, partial neighbor routing tables, and available link bandwidth. For nodes The conditional state of the agent's decision-making is constituted by the state indicator variable K; in order to maximize network transmission capacity, The global network reward should be maximized. The global objective function is... Reinforcement learning network parameters for each agent The set is calculated using the following formula. gradient update : in, Indicates passage right Sampled state-action pairs estimator Used to estimate in state Next strategy Actions taken The degree of advantage relative to taking random actions, when At that time, strategy Actions taken Better results will be achieved compared to random action. Indicates network parameters Actions are adopted under the condition of agent state s. Strategies; (404) By computing nodes The gradient function of the agent's policy parameters updates the parameters of the reinforcement learning network, and the calculation formula is as follows: in, Generalized dominance value The local unbiased estimate, when the agent When located in the routing path, In this case, the agent will not update its policy parameters; (405) at node Partial state of the agent Conditions and states Under the given conditions, the reward function for the action is calculated by using the output action of the reinforcement learning network, which is the probability distribution of the next-hop neighbor selection in the business flow: in As a reward discount factor, For the first The utility value of each business flow is expressed as: in , , They represent the first Normalized throughput, latency, and packet loss for each service flow; normalization adopts the first... Each service flow parameter is calculated by dividing the average value of service flow parameters of the same type and origin / target node. , , It is a non-negative scalar, representing the importance weight of performance metrics such as throughput, latency, and packet loss; (406) After the path decision-making by the agent is completed, check whether there are loops or unsafe paths in the path whose link states do not meet the business flow requirements. If there are unsafe paths, the agent is rewarded with a negative scalar as a penalty and the path is recalculated using the Dijkstra method.

Images