Dynamic packet routing algorithm based on multi-dimensional information and reinforcement learning

Through dynamic group routing algorithms based on multi-dimensional information and reinforcement learning, the problem of low efficiency of traditional routing algorithms in high load and high dynamic networks is solved, and the independent optimization and stability of routing strategies are achieved, and the performance is better than that of traditional algorithms.

CN118972309APending Publication Date: 2024-11-15SHANXI FINANCE VOCATIONAL COLLEGE +2
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410937004.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-12
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

In a high-load and highly dynamic network environment, traditional routing algorithms are inefficient and difficult to cope with network traffic fluctuations and topological changes. How to design a routing algorithm based on machine learning to determine the mapping relationship between network state and routing actions and maximize the algorithm benefits.

Method used

Using dynamic group routing algorithm based on multidimensional information and reinforcement learning, through Markov process modeling and Double DQN algorithm framework, a deep learning network is used to build a comment network, and the routing strategy optimization is performed based on current status, actions and reward returns to determine data packets. The next hop routing action.

Benefits of technology

This algorithm improves the stability and generalization of routing decisions, has the ability to update independently, and can determine the best action from a huge action space in static and dynamic scenarios, and has better performance than traditional algorithms.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118972309A_ABST
    Figure CN118972309A_ABST
Patent Text Reader

Abstract

The invention discloses a dynamic packet routing algorithm based on multi-dimensional information and reinforcement learning, and belongs to the technical field of routing networks. For the problems of how to determine a mapping relationship between a network state and a routing action in a machine learning algorithm, how to design an online optimization algorithm strategy and how to maximize algorithm benefits, modeling is performed on a routing process based on a Markov process; constructing a comment network Q (.) and a target comment network Q-(.) based on a Double DQN algorithm framework and by adopting a DNN network; and performing update training and reinforcement learning on the established model by using a deep learning network to find an optimal routing grouping strategy. According to the method, each router node is abstracted as an independent agent, multi-dimensional data such as a current position node of a data packet, the number of nodes in the network, the data packet receiving length of the current position node, a neighbor node set and the like are used as input of a neural network, and the optimal action of routing is determined in combination with a segmented reward function; and finally, the efficiency maximization of the routing process is realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of routing networks, and in particular relates to a dynamic grouping routing algorithm based on multi-dimensional information and reinforcement learning. Background Art

[0002] Packet routing is one of the key functions of communication networks. It usually takes minimizing the transmission time of packets from source nodes to destination nodes (i.e., end-to-end delay) as the routing goal. However, with the explosive growth of current network scale and the traffic it carries, the network structure is distributed and dynamic, and the network traffic is bursty and unpredictable. In this case, how to ensure the connectivity of the network and guarantee the quality of service (QoS) of various network applications has brought huge challenges to traditional routing algorithms and routing protocols. For example, the classic Bellman-Ford algorithm finds the shortest path in the network topology and uses the shortest path to minimize the transmission delay of packet routing. It lacks consideration of other factors such as network traffic fluctuations, load balancing, and network topology changes; while the DSDV routing protocol needs to use a flooding method to maintain a globally consistent routing table. However, in a distributed network that lacks a global coordination method and mechanism, the router needs to select the next-hop routing node of the packet on its own. This requires the routing algorithm to have the ability to autonomously learn the local characteristics of the network and then select the appropriate next-hop route, which is usually a challenging task.

[0003] At present, routing algorithms can be roughly divided into two categories. One is the traditional shortest path algorithm, such as the Bellman-Ford algorithm and the Dijkstra algorithm. Such algorithms are suitable for static networks where the complete state of the network can be obtained. Such algorithms usually take minimizing transmission delay or the minimum number of hops as the routing goal. When the network load is low, traditional routing algorithms can efficiently transmit data packets, but such algorithms are usually extremely inefficient in high-load, highly dynamic networks; the other is the routing algorithm based on machine learning, among which the routing algorithms based on reinforcement learning and deep learning are the most representative. The reinforcement learning-based method acquires knowledge through interaction with the local environment, without making strong assumptions or prior knowledge about the entire network model, so it can cope with the uncertainty and dynamics of the network environment and update the algorithm strategy in time. In recent years, the continuous development of network technology (the complete separation of the control plane and the data plane in software-defined networks (SDN)) has also provided the basic conditions for the actual deployment of machine learning-based routing algorithms, so routing strategies based on machine learning methods have become a hot topic of research. However, how to determine the mapping relationship between network status and routing actions in machine learning algorithms, how to design online optimization algorithm strategies, and how to maximize algorithm benefits are the difficulties in designing routing algorithms based on machine learning. Summary of the invention

[0004] In order to determine the mapping relationship between network status and routing actions in machine learning algorithms, design online optimization algorithm strategies, and maximize algorithm benefits, the present invention provides a dynamic packet routing algorithm based on multidimensional information and reinforcement learning.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A dynamic packet routing algorithm based on multi-dimensional information and reinforcement learning, the algorithm comprising the following steps:

[0007] Step 1: Model the routing process based on Markov process;

[0008] Furthermore, the routing process is modeled as M = <S, A, F, R, γ>, where S is the state space of the agent; A is the action space of the agent; and F is the agent's action space from the current state s t Transition to next state s t+1 The state transition probability; R is the reward returned by the agent; γ is the discount factor, which indicates the influence of future rewards on current rewards;

[0009] Furthermore, the state space S of the agent is expressed as follows:

[0010]

[0011] In the formula, poi s' represents the location information of node s' at time slot t, that is, the index of node s'; N represents the number of nodes in the network at time slot t, represents the number of packets received by node s' in time slot t, represents the length of the queue of the sent data packet of node s', s' Represents the set of neighbor nodes of node s';

[0012] Furthermore, the action space A of the agent is expressed as follows:

[0013] A={a t =ο s'}

[0014] Wherein, s' Represents the set of neighbor nodes of node s'; a t Represents the decision action. For each agent, its action space is equal to the neighbor nodes of the current node.

[0015] Furthermore, the agent changes from the current state s t Transition to next state s t+1 The state transition probability F is expressed as follows:

[0016] F:S×A×S→[0,1]

[0017] In the formula, S is the state space of the agent, A is the action space of the agent, and F represents the state transition probability;

[0018] Furthermore, the reward R obtained by the agent is specifically embodied as follows:

[0019] Case 1: Data packet p is successfully delivered from source node o to destination node d:

[0020] r o =α*N

[0021] Among them, r o Indicates the reward obtained under Case 1; α represents a parameter value, and N is the total number of nodes;

[0022] Case 2: Data packet p is only sent from the previous node o' to the next node o",

[0023] r w =-(r n+1 -r n +r g )

[0024]

[0025] r n =σ n

[0026] r g =ω*φ

[0027] Among them, r w Indicates the reward received under Case 2; r n+1 Represents the comprehensive reward for the next node sending and receiving data packets; r n Indicates the reward for the maximum number of packets received by the previous node; r g Indicates the reward for the previous node receiving the data packet; ψ n+1 They represent the length of the queue of the next node o" sending data packets. and receive packet queue length σ n Indicates the maximum receiving capacity of the data packet of node o' ω represents a parameter value, φ represents the length of the packet receiving queue of node o'

[0028] Case 3: When there is no available path to send the data packet p to the destination node d, the reward obtained is: r t = -ξ;

[0029] Among them, r t Represents the reward obtained under Case 3; ζ represents a parameter value.

[0030] Step 2: Based on the Double DQN algorithm framework and using the DNN network to build the comment network Q() and the target comment network Q - ();

[0031] Furthermore, the review network Q() and the target review network Q - () is a fully connected network consisting of two hidden layers; at time slot t, each node obtains the current state S of the surrounding t , generates corresponding decision actions a through its own neural network t , thus reaching the next state S t+1 , and obtain an immediate reward value r t ; Change the current state S t , decision action a t , next state S t+1 , immediate reward value r t As an experience set (s t ,a t ,r t ,s t+1 ), and the experience set (s t ,a t ,r t ,s t+1 ) is stored in the buffer pool H; when the network is updated, B is taken out by priority sampling S Experience group to learn, B S Indicates the number of experience groups.

[0032] Step 3: Use the deep learning network to update and train the established model and strengthen learning to find the best routing grouping strategy. Update the review network constructed in step 2;

[0033] Decision Action a t Based on the following formula:

[0034]

[0035] Among them, Q(s,a;θ) represents the comment network, a represents the action corresponding to the state s, θ represents the parameter of the comment network; ε represents a probability value;

[0036] The comment value y of the next state is obtained based on the following formula:

[0037]

[0038] Among them, r represents the reward value, Q represents the output value of the comment network, and a max represents the action corresponding to the maximum comment network value in the next state s', θ represents the parameters of the comment network, and θ' represents the parameters of the prediction network;

[0039] The following loss function is updated with gradient:

[0040] L=||yQ(s,a;θ)|| 2

[0041] || || 2 represents the mean square error.

[0042] Compared with the prior art, the present invention has the following advantages:

[0043] (1) The algorithm uses multi-dimensional information such as the current location node of the data packet, the number of nodes in the network, the length of the data packet received at the current location node, and the status of the current node's neighbor nodes as input to determine the next hop routing action of the data packet. The introduction of multi-dimensional network information enhances the stability and generalization of the algorithm.

[0044] (2) The value of the algorithm’s reward function is determined based on the next-hop routing selection effect, which enables the algorithm to have the ability to update itself and also gives the algorithm the characteristics of fully distributed routing;

[0045] (3) In static and dynamic scenarios, the present invention conducts a multi-index comprehensive comparative analysis of the algorithm performance. Experiments show that the algorithm can determine the optimal action from a huge action space. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A diagram for modeling packet transmission of the intelligent agent of the present invention during the routing process;

[0047] Figure 2 A framework diagram of a routing algorithm in an embodiment of the present invention;

[0048] Figure 3 It is a graph of algorithm convergence of the algorithm in a dynamic network in an embodiment of the present invention;

[0049] Figure 4 It is a performance diagram of the “percentage of full capacity nodes” indicator of the algorithm in a dynamic network in an embodiment of the present invention;

[0050] Figure 5 This is a performance diagram of the indicator “maximum number of packets sent by a node” of the algorithm in a static network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to facilitate the understanding of the present invention, the present invention will be described more fully below. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0052] Example 1

[0053] A dynamic group routing algorithm based on multi-dimensional information and multi-agent reinforcement learning, the algorithm comprising the following steps:

[0054] Step 1: Model the routing process based on Markov process;

[0055] Furthermore, the routing process is modeled as M = <S, A, F, R, γ>, where S is the state space of the agent; A is the action space of the agent; and F is the agent's action space from the current state s t Transition to next state s t+1 The state transition probability; R is the reward returned by the agent; γ is the discount factor, which indicates the influence of future rewards on current rewards;

[0056] Furthermore, the state space S of the agent is expressed as follows:

[0057]

[0058] In the formula, poi s' represents the location information of node s' at time slot t, that is, the index of node s'; N represents the number of nodes in the network at time slot t, represents the number of packets received by node s' in time slot t, represents the length of the queue of the sent data packet of node s', s' Represents the set of neighbor nodes of node s';

[0059] Furthermore, the action space A of the agent is expressed as follows:

[0060] A={a t =ο s'}

[0061] Wherein, s' Represents the set of neighbor nodes of node s'; a t Represents the decision action. For each agent, its action space is equal to the neighbor nodes of the current node.

[0062] Furthermore, the agent changes from the current state s t Transition to next state s t+1 The state transition probability F is expressed as follows:

[0063] F:S×A×S→[0,1]

[0064] In the formula, S is the state space of the agent, A is the action space of the agent, and F represents the state transition probability;

[0065] Furthermore, the reward R obtained by the agent is specifically embodied as follows:

[0066] Case 1: Data packet p is successfully delivered from source node o to destination node d:

[0067] r o =α*N

[0068] Among them, r o Indicates the reward obtained under Case 1; α represents a parameter value, and N is the total number of nodes;

[0069] Case 2: Data packet p is only sent from the previous node o' to the next node o",

[0070] r w =-(r n+1 -r n +r g )

[0071]

[0072] r n =σ n

[0073] r g =ω*φ

[0074] Among them, r w Indicates the reward received under Case 2; r n+1 Represents the comprehensive reward for the next node sending and receiving data packets; r n Indicates the reward for the maximum number of packets received by the previous node; r g Indicates the reward for the previous node receiving the data packet; ψ n+1 They represent the length of the queue of the next node o" sending data packets. and receive packet queue length σ n Indicates the maximum receiving capacity of the data packet of node o' ω represents a parameter value, φ represents the length of the packet receiving queue of node o'

[0075] Case 3: When there is no available path to send the data packet p to the destination node d, the reward obtained is: r t = -ξ;

[0076] Among them, r t Represents the reward obtained under Case 3; ζ represents a parameter value.

[0077] Step 2: Based on the Double DQN algorithm framework and using the DNN network to build the review network Q(·) and the target review network Q - ();

[0078] Furthermore, the review network Q() and the target review network Q - () is a fully connected network consisting of two hidden layers; at time slot t, each node obtains the current state S of the surrounding t , generates corresponding decision actions a through its own neural network t , thus reaching the next state S t+1 , and obtain an immediate reward value r t ; Change the current state S t , decision action a t , next state S t+1 , immediate reward value r t As an experience set (s t ,a t ,r t ,s t+1 ), and the experience set (s t ,a t ,r t ,s t+1 ) is stored in the buffer pool H; when the network is updated, B is taken out by priority sampling S Experience group to learn, B S Indicates the number of experience groups.

[0079] Step 3: Use the deep learning network to update and train the established model and strengthen learning to find the best routing grouping strategy. Update the review network constructed in step 2;

[0080] Decision Action a t Based on the following formula:

[0081]

[0082] Among them, Q(s,a;θ) represents the comment network, a represents the action corresponding to the state s, θ represents the parameter of the comment network; ε represents a probability value;

[0083] The comment value y of the next state is obtained based on the following formula:

[0084]

[0085] Among them, r represents the reward value, Q represents the output value of the comment network, and a max represents the action corresponding to the maximum comment network value in the next state s', θ represents the parameters of the comment network, and θ' represents the parameters of the prediction network;

[0086] The following loss function is updated with gradient:

[0087] L=||yQ(s,a;θ)|| 2

[0088] || || 2 represents the mean square error.

[0089] In the above technical solution, the network is modeled as an undirected graph Each vertex n∈N represents a network node (router), and each edge e∈E represents a link between nodes. nodes, edges; each node has the following attributes: in It indicates the capacity of a node to receive data packets at a certain moment. It indicates the capacity of a node to send data packets at a certain moment. Indicates the maximum capacity of a node to send data packets. Represents the maximum capacity of a node to receive data packets; each edge has the following attributes: represents the number of edge traversals, Indicates the weight value of the edge (edg v <edg max ), Represents the sine state value of the edge. Data packets are transmitted between adjacent nodes, where the source node and the target node form an "action-guidance" relationship, that is, the target node takes corresponding actions based on the current state information of the source node.

[0090] This example is divided into static network scenarios and dynamic network scenarios. For static network scenarios, during the packet routing process, the network is in a "static" state. During the sampling period, the links and data packets in the network are in a relatively stable state. When a data packet arrives at the destination node, its source node sends another identical data packet, and so on. For dynamic network scenarios, the network is in a "non-static" state, and some links are randomly deleted, and the network topology changes.

[0091] This example has 50 agents. In the initial state, the network has 2500 data packets, and an additional 500 data packets are generated during operation. The data packet sending capacity of each node is the same, which is 75; the data packet receiving capacity of each node is also the same, which is 20. Consider the network after the algorithm executes 500 rounds, each round passes 1000 time slots, and not all data packets reach the target node.

[0092] The performance of the routing process can be seen in Figure 3 , 4 As shown in Figure 5, the time required for the routing process in the present invention is gradually reduced and convergence is achieved. Regardless of whether it is in a static network scenario or a dynamic network scenario, the performance is better than that of the traditional algorithm.

[0093] The technical features of the above-mentioned embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0094] The contents not described in detail in the specification of the present invention belong to the prior art known to the professional and technical personnel in the field. Although the illustrative specific embodiments of the present invention are described above to facilitate the understanding of the present invention by the technical personnel in the field, it should be clear that the present invention is not limited to the scope of the specific embodiments. For the ordinary technical personnel in the field, as long as various changes are within the spirit and scope of the present invention defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A dynamic packet routing algorithm based on multi-dimensional information and reinforcement learning, It is characterized in that The algorithm comprises the following steps: Step 1: Model the routing process based on Markov process; Step 2: Based on the DoubleDQN algorithm framework and using the DNN network to build the review network Q(·) and the target review network Q - (·); Step 3: Use the deep learning network to update the established model and perform reinforcement learning to find the best routing grouping strategy.

2. A dynamic packet routing algorithm based on multidimensional information and reinforcement learning according to claim 1, It is characterized in that The specific operations of step 1 are: The routing process is modeled as M = <S, A, F, R, γ>, where S is the state space of the agent; A is the action space of the agent; and F is the agent's movement from the current state s t Transition to next state s t+1 The state transition probability; R is the reward returned by the agent; γ is the discount factor, which indicates the influence of future rewards on current rewards; The state space S of the agent is expressed as follows: In the formula, poi s' represents the location information of node s' at time slot t, that is, the index of node s'; N represents the number of nodes in the network at time slot t, represents the number of packets received by node s' in time slot t, represents the length of the queue of the sent data packet of node s', s' Represents the set of neighbor nodes of node s'; The action space A of the agent is expressed as follows: A={a t =o s' } Wherein, s' Represents the set of neighbor nodes of node s'; a t Represents the decision action. For each agent, its action space is equal to the neighbor nodes of the current node. The agent starts from the current state s t Transition to next state s t+1 The state transition probability F is expressed as follows: F:S×A×S→[0,1] In the formula, S is the state space of the agent, A is the action space of the agent, and F represents the state transition probability; The reward R obtained by the agent is specifically embodied as follows: Case 1: Data packet p is successfully delivered from source node o to destination node d: r o =α*N Among them, r o Indicates the reward obtained under Case 1; α represents a parameter value, and N is the total number of nodes; Case 2: Data packet p is only sent from the previous node o' to the next node o", r w =-(r n+1 -r n +r g ) r n =s n r g =ω*φ Among them, r w Indicates the reward received under Case 2; r n+1 Represents the comprehensive reward for the next node sending and receiving data packets; r n Indicates the reward for the maximum number of packets received by the previous node; r g Indicates the reward for the previous node receiving the data packet; ψ n+1 They represent the length of the queue of the next node o" sending data packets. and receive packet queue length σ n Indicates the maximum receiving capacity of the data packet of node o' ω represents a parameter value, φ represents the length of the packet receiving queue of node o' Case 3: When there is no available path to send the data packet p to the destination node d, the reward obtained is: r t = -ξ; Among them, r t Represents the reward obtained under Case 3; ζ represents a parameter value.

3. A dynamic packet routing algorithm based on multidimensional information and reinforcement learning according to claim 2, It is characterized in that Step 2: Based on the DoubleDQN algorithm framework and using the DNN network to construct the review network Q(·) and the target review network Q - (·);in: The review network Q(·), the target review network Q - (·) is a fully connected network consisting of two hidden layers; at time slot t, each node obtains the current state S of the surrounding t , generates corresponding decision actions a through its own neural network t , thus reaching the next state S t+1 , and obtain an immediate reward value r t ; Change the current state S t , decision action a t , next state S t+1 , immediate reward value r t As an experience set (s t ,a t ,r t ,s t+1 ), and the experience set (s t ,a t ,r t ,s t+1 ) is stored in the buffer pool H; when the network is updated, B is taken out by priority sampling S Experience group to learn, B S Indicates the number of experience groups.

4. A dynamic packet routing algorithm based on multidimensional information and reinforcement learning according to claim 3, It is characterized in that The specific operations of step 3 are: Update the review network constructed in training step 2; Decision Action a t Based on the following formula: Among them, Q(s,a;θ) represents the comment network, a represents the action corresponding to the state s, θ represents the parameter of the comment network; ε represents a probability value; The comment value y of the next state is obtained based on the following formula: Among them, r represents the reward value, Q represents the output value of the comment network, and a max represents the action corresponding to the maximum comment network value in the next state s', θ represents the parameters of the comment network, and θ' represents the parameters of the prediction network; The following loss function is updated with gradient: L=||y-Q(s,a;θ)|| 2 || || 2 represents the mean square error.