Multi-path routing method and device, medium and equipment

Abstract

The invention discloses a multi-path routing method and device, a medium and equipment, and relates to the technical field of network communication. The method comprises the following steps: firstly, acquiring topological information of each network node, state information of each network node and information of each data stream to be forwarded at the current moment to form a state space at the current moment, and then determining the data stream to be forwarded by the network node at the current moment for each network node to be subjected to data stream forwarding. And determining the action space of the forwarding action of the network node according to the degree of the network node at the current moment, thereby determining the forwarding action of the network node according to the state space and the action space, namely the next path network node of the to-be-forwarded data flow, and finally determining the routing path of each to-be-forwarded data flow through multi-round iteration moment by moment. By splitting the route determination process of the data flow to be forwarded into the next hop determination process of the single network node moment by moment, the resource consumption is reduced, the convergence efficiency is improved, and the flexibility and the load balancing capability are improved.

Description

Technical Field

[0001] The present invention relates to the field of network communication technology, and in particular to a multi-path routing method, device, medium and equipment. Background Art

[0002] In recent years, with the continuous development of Internet technology, emerging industries such as cloud computing and big data have risen rapidly, resulting in a large amount of data traffic being concentrated in data centers for storage and management. As an important part of the data center infrastructure, the Data Center Network (DCN) is responsible for enabling communication between a large number of computing resources within the data center. Therefore, data center networks require efficient and flexible routing algorithms to cope with large-scale traffic transmission. However, traditional static routing algorithms, such as Shortest Path Routing (SPR) or Equal-Cost Multi-Path Routing (ECMP), usually fail to fully consider the actual traffic characteristics. The routing rules of these algorithms are established independently of the traffic distribution in the network and cannot achieve the best performance level.

[0003] Software-Defined Networking (SDN) expands the horizon of solving related problems. By decoupling the control plane and data plane in the network, it makes it possible to effectively monitor the network status and dynamically deploy network policies. Therefore, SDN is very suitable for optimizing flow routing. However, flow routing algorithms are relatively complex. Despite significant progress in recent decades, flow routing is still a complex routing method that requires detailed communication network and traffic models. In recent years, model-free artificial intelligence techniques based on reinforcement learning have been successfully applied to various complex control and optimization problems. Therefore, a large number of recent studies have applied reinforcement learning methods to routing optimization problems. These studies have demonstrated the potential of reinforcement learning for adaptive and optimized decision-making in dynamic environments, providing new solutions for flow routing algorithms.

[0004] However, when the network scale expands, that is, the network topology is complex or the number of nodes is large, traditional routing methods are difficult to handle large-scale state and action spaces, consume huge computing resources and are difficult to converge. At the same time, in traffic routing, the same flow path is often selected for the same source-destination network node pair, lacking the flexibility of multiple routing paths, which limits the network's load balancing capabilities and easily leads to local path overload. Summary of the invention

[0005] Based on this, it is necessary to provide a multi-path routing method, device, medium and equipment to address the above technical problems.

[0006] The present invention adopts the following technical solutions:

[0007] The present invention provides a multi-path routing method, comprising:

[0008] Obtain the topology information of each network node at the current moment, the state information of each network node, and the source network node and the target network node of each data flow to be forwarded, to form the state space at the current moment;

[0009] For each network node to forward data streams, determine the action space of the forwarding action of the network node according to the degree of the network node at the current moment;

[0010] According to the state space at the current moment and the action space of the network node, the next path network node of the data flow to be forwarded by the network node at the current moment is determined by a reinforcement learning model that is pre-trained and deployed on the network node, so as to form a forwarding action of the network node;

[0011] The state information of each network node at the next moment is determined according to the forwarding action of each network node to obtain the state space at the next moment, and the network node to be used for data stream forwarding at the next moment is re-determined. The routing path of each data stream to be forwarded from the source network node to the target network node is determined through multiple rounds of iterations at each moment.

[0012] Optionally, training the reinforcement learning model specifically includes:

[0013] For each network node, the reward for the network node to execute each forwarding action in the action space and then transition to the state space at the next moment is determined by the following formula based on the state space at the historical moment and the action space of the network node:

[0014]

[0015] According to the state space at the historical moment, the action space of the network node, and the reward of the network node executing each forwarding action in the action space and converting to the state space at the next moment, the value of each forwarding action in the action space executed by the network node is determined by the following formula:

[0016]

[0017] Through multiple rounds of iterations, the value table corresponding to the state transition achieved by each forwarding action in the action space of each network node in different state spaces is obtained;

[0018] in, Perform forwarding action for the nth network node at time t The reward is α1, α2, β1, β2 and β3 are different adjustment parameters. is the global perception reward at time t, is the local perception reward at time t, thruput flow is the average transmission rate of each data stream, latency flow is the average delay of each data flow, loss flow is the average packet loss rate of each data flow, dist[dpid][dst] is the shortest distance from the nth network node to the target network node of the data flow it forwards, and dist[nextdpid][dst] is the shortest distance from the next node to the target network node of the data flow it forwards after the nth network node performs the forwarding action. is the nth network node in the state space s at time t t Execute action when The value of , α is the learning rate, γ is the discount factor, is the nth network node in the state space s at time t+1 t+1 Execute the best action The maximum estimated value of .

[0019] Optionally, training the reinforcement learning model specifically includes:

[0020] For each network node, a random value is generated. When the random value is greater than or equal to the boundary value of the preset exploration or utilization strategy, a forwarding action corresponding to the maximum value in the preset value table is determined as the forwarding action to be performed by the network node; the value table includes the value corresponding to the state transition achieved by each forwarding action in the action space of each network node through different state spaces;

[0021] When the random value is less than the preset boundary value of the exploration or utilization strategy, the probability of the network node executing each forwarding action in its action space is determined according to the value table, and a forwarding action is selected as the forwarding action to be executed by the network node based on the probability of each forwarding action through a random algorithm;

[0022] Determine the reward of the network node executing the determined forwarding action and converting to the state space at the next moment; determine the value of the network node executing the determined forwarding action based on the state space at the historical moment, the action space of the network node, and the reward of the network node executing the determined forwarding action and converting to the state space at the next moment; update the value table through multiple rounds of iterations.

[0023] Optionally, the determining, based on the state space at the current moment and the action space of the network node, through a reinforcement learning model pre-trained and deployed on the network node, the next path network node of the data flow to be forwarded by the network node at the current moment specifically includes:

[0024] According to the current state space and the action space of the network node, by querying the value table corresponding to the state transition achieved by each forwarding action in the action space of each network node in different state spaces, the forwarding action corresponding to the maximum value in the current state space is determined;

[0025] According to the forwarding action, the next network node of the data flow to be forwarded by the network node at the current moment is determined.

[0026] Optionally, determining the routing path of each data stream to be forwarded from the source network node to the target network node by multiple rounds of iterations at each moment specifically includes:

[0027] Determine the candidate routing path of each data stream to be forwarded through multiple rounds of iterations at each moment, and judge whether each candidate routing path is a valid path; wherein, a routing path is a valid path when it has no loops and no black holes;

[0028] If yes, each of the candidate paths is used as a routing path corresponding to each of the data flows to be forwarded;

[0029] If not, then determine the maximum available bandwidth path from the source network node to the target network node according to the source network node and the target network node of the data flow to be forwarded corresponding to the invalid path, and use it as the routing path of the data flow to be forwarded corresponding to the invalid path.

[0030] Optionally, the method further comprises:

[0031] For each network node, save the current state space, the forwarding action of the network node, the reward corresponding to the forwarding action of the network node, and the state space of the next moment after each network node executes the corresponding forwarding action in each round of iteration to form the training data corresponding to the network node;

[0032] Based on the number and topology information of network nodes, multiple reinforcement learning models to be trained are instantiated through the cloud server, and according to the corresponding relationship between each reinforcement learning model to be trained and each network node, each reinforcement learning model to be trained is trained separately through the training data of the corresponding network node, and the value tables corresponding to the state transition achieved by each forwarding action in the action space of each network node corresponding to different state spaces of each network node are obtained, and they are deployed in the corresponding network nodes respectively.

[0033] Optionally, the remaining bandwidth, transmission delay and packet loss rate of the links between the network nodes.

[0034] The present invention provides a multi-path routing method and device, comprising:

[0035] An acquisition module is used to acquire the topology information of each network node at the current moment, the state information of each network node, and the source network node and the target network node of each data flow to be forwarded, so as to form a state space at the current moment;

[0036] A determination module, for determining, for each network node to be forwarding a data stream, an action space of the network node forwarding action according to the degree of the network node at a current moment;

[0037] A routing module is used to determine the next path network node of the data flow to be forwarded by the network node at the current moment according to the state space at the current moment and the action space of the network node, through a reinforcement learning model that is pre-trained and deployed on the network node, so as to form a forwarding action of the network node;

[0038] The iteration module is used to determine the state information of each network node at the next moment according to the forwarding action of each network node, so as to obtain the state space at the next moment, and re-determine the network node to forward the data flow at the next moment, and determine the routing path of each data flow to be forwarded from the source network node to the target network node through multiple rounds of iterations at each moment.

[0039] The present invention provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the multi-path routing method is implemented.

[0040] The present invention provides a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the multipath routing method when executing the program.

[0041] At least one of the above technical solutions adopted by the present invention can achieve the following beneficial effects:

[0042] First, the topological information of each network node, the status information of each network node and the information of each data flow to be forwarded at the current moment are obtained to form the state space at the current moment. Then, for each network node to forward the data flow, the data flow to be forwarded by the network node at the current moment is determined, and the action space of the forwarding action of the network node is determined according to the degree of the network node at the current moment. Thus, the forwarding action of the network node is determined according to the state space and the action space, that is, the next network node of the data flow to be forwarded. Finally, the routing path of each data flow to be forwarded is determined through multiple rounds of iterations at each moment.

[0043] The present invention splits the routing determination process of the data stream to be forwarded into the next hop determination process of a single network node at each moment, thereby splitting the large-scale state space into the state space at each moment, and splitting the overall action space into the action space of multiple network nodes, and processing them separately at each moment through the reinforcement learning model deployed on each network node, thereby reducing resource consumption and improving convergence efficiency. At the same time, the forwarding action of each network node at the previous moment in the routing determination process at each moment after the split will affect the state space at the next moment. Therefore, even for the same source-target network node pair, different routing paths will be obtained due to the change of the state space, which improves flexibility and load balancing capabilities and avoids local path overload. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0045] Figure 1 A schematic diagram of a multi-path routing method provided by the present invention;

[0046] Figure 2 A schematic diagram of a KDN architecture provided by the present invention;

[0047] Figure 3 A schematic diagram of a routing system architecture provided by the present invention;

[0048] Figure 4 A schematic diagram of a multi-path routing method provided by the present invention;

[0049] Figure 5 A schematic diagram of a parallel coordination mechanism of intelligent agents provided by the present invention;

[0050] Figure 6 A schematic diagram of a multi-path routing method device provided by the present invention;

[0051] Figure 7 A schematic diagram of a computer device for implementing a multi-path routing method provided by the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0053] At present, the traditional routing method has the following problems:

[0054] (1) Bandwidth fragmentation problem: Some studies use equal-cost multi-path routing (ECMP) and linear search methods in traffic routing. Although these methods can optimize network performance, they are prone to bandwidth fragmentation problems. This can lead to under-utilization of network resources and affect overall network efficiency.

[0055] (2) Limitations of static models: Most algorithms rely on complex analysis of network traffic and building static models for routing selection. Such static models perform poorly in the face of complex and dynamically changing network environments and are difficult to adapt to different network scenarios because traffic characteristics are often inconsistent.

[0056] (3) The curse of dimensionality: In some studies based on reinforcement learning, the curse of dimensionality is prone to occur when the network topology is complex or the number of nodes is large. Specifically, in complex network topologies (such as the fat-tree topology with K=4), the algorithm is difficult to effectively handle large-scale state spaces and action spaces, resulting in huge consumption of computing resources and difficulty in convergence.

[0057] (4) Uniqueness of path selection: Although some studies based on deep reinforcement learning have alleviated the problem of the curse of dimensionality, they often select the same path for the same source-destination switch pair in traffic routing, lacking the flexibility of multi-path routing. This limits the load balancing capability of the network and may also cause some paths to be overloaded.

[0058] (5) Considering only a single performance indicator: Some studies only focus on link delay and ignore other important performance indicators such as available bandwidth and packet loss rate. This may result in the selected path having poor overall performance and failing to meet the diverse needs of practical applications.

[0059] (6) Limitations of static environment interaction: Some reinforcement learning-based methods select routing paths based on static environment interaction and cannot dynamically respond to changes in the network environment. Such static models may have difficulty coping with real-time changes in network status, resulting in insufficient flexibility and adaptability in routing selection.

[0060] (7) Application limitations of Multi-Agent Reinforcement Learning (MARL): Although some studies have demonstrated the potential of MARL-based routing algorithms in scalability and traffic optimization, these methods have not yet been widely used in data center network scenarios, and their actual effectiveness and applicability may still need further verification.

[0061] In general, existing technologies still face challenges such as bandwidth fragmentation, dimensionality disaster, single path selection, single performance index, and lack of dynamic interaction when dealing with complex and dynamic network environments. The method proposed in this invention aims to overcome these shortcomings and use centralized control capabilities and the adaptability of reinforcement learning to achieve more flexible and efficient flow routing selection.

[0062] The technical solutions provided by various embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0063] Figure 1 The present invention is a schematic diagram of a multi-path routing method flow chart, which specifically includes the following steps:

[0064] S101: Acquire the topology information of each network node at the current moment, the state information of each network node, and the source network node and the target network node of each data flow to be forwarded, and form a state space at the current moment.

[0065] S102: For each network node to forward a data stream, determine the action space of the forwarding action of the network node according to the degree of the network node at the current moment.

[0066] S103: According to the state space at the current moment and the action space of the network node, the next path network node of the data flow to be forwarded by the network node at the current moment is determined through a reinforcement learning model that is pre-trained and deployed on the network node, so as to form a forwarding action of the network node.

[0067] S104: Determine the state information of each network node at the next moment according to the forwarding action of each network node to obtain the state space at the next moment, and re-determine the network node to forward the data stream at the next moment, and determine the routing path of each data stream to be forwarded from the source network node to the target network node through multiple rounds of iterations at each moment.

[0068] Knowledge Defined Networking (KDN) has a highly programmable knowledge plane that can use machine learning technology to provide more efficient routing solutions. The main advantage of KDN is that it can achieve autonomous network operations based on learned knowledge. Depending on where the knowledge is deployed, the KDN architecture can be divided into centralized, distributed, and hybrid. Figure 2 As shown, Figure 2 Schematic diagram of a KDN architecture in the present invention.

[0069] In the centralized KDN architecture, Figure 2 As shown in the part marked (a) in Figure 2, knowledge is only deployed on the controller side, which may face scalability issues in large-scale networks. In contrast, the distributed KDN architecture, such as Figure 2As shown in the marked part (b), deploying knowledge on the device side provides better scalability, but may lack a global perspective. Figure 2 As shown in the part marked (c), a balance is achieved between the two. By deploying knowledge on both the controller and the device, the controller and the device are responsible for knowledge acquisition and deployment respectively, thereby enhancing the practicality of the flow routing solution. For the multi-path routing problem based on MARL, the present invention is based on a hybrid KDN architecture, where knowledge refers to the strategy learned by the agent. The specific architecture is as follows Figure 3 As shown, Figure 3 A schematic diagram of a routing system architecture in the present invention.

[0070] The system architecture designed by the present invention consists of four planes: data plane, control plane, management plane and knowledge plane. Specifically:

[0071] (1) Data plane: The data plane is based on a single autonomous system (AS) network model consisting of multiple physical or virtual switches (OpenvSwitch, OVS).

[0072] A traditional virtual switch (vSwitch) is a software-implemented network switch used in a virtualized environment to support data flow between virtual machines (VMs) and between virtual machines and physical networks. A virtual switch can establish network connections between virtual machines on the same physical host and manage packet forwarding between them. Its core function is similar to that of a physical switch, which is based on MAC address forwarding, but it runs in a virtualized environment and is directly integrated into virtualization hypervisors (such as KVM, Xen, VMware, etc.).

[0073] The virtual switch mentioned in the present invention refers to a virtual switch that mainly controls forwarding through the OpenFlow protocol. It is an open source, feature-rich virtual switch implementation designed for multi-tenant data center environments, network virtualization, and large-scale cloud infrastructure. OpenvSwitch provides more advanced functions than simple virtual switches, allowing the creation of complex network topologies, supporting encapsulation protocols such as VLAN, GRE, VXLAN, and integration into various network controllers (such as OpenFlow), supporting software-defined networks (SDN). For the convenience of explanation, it is hereinafter referred to as a switch.

[0074] In routing selection, each switch can be regarded as a network node. The data plane uses the programmability and real-time response capabilities of the flow table rules to implement routing strategies according to the instructions of the control plane, is responsible for actual data flow processing and forwarding, and also responds to the control plane's request to periodically transmit network node status information.

[0075] (2) Control plane: The control plane communicates with the knowledge plane through the northbound API to obtain the operations of the intelligent agent reinforcement learning model deployed in each switch. The control plane communicates with the data plane through the southbound API to install and update flow table rules on the switch and collect information from the switch to build a global view of the data plane.

[0076] It should be noted that in one or more embodiments of the present invention, the controller designed by the present invention is a local centralized controller for the AS. Therefore, the designed distributed control plane can be extended to a large-scale network composed of multiple ASs. In this case, the same KDN architecture can be applied to each AS to achieve independent local management. In addition, the scheduler of the control plane can also be used to coordinate operations between multiple ASs.

[0077] (3) Management plane: The management plane processes the collected network information. The information collection module uses the control plane (through the network resource sensing unit) to collect the original data of the data stream transmission. The information processing module calculates the data stream delay, data stream transmission rate, data stream packet loss rate and other information based on the original data. This information is an important basis for the intelligent agent in the knowledge plane to learn and make routing decisions.

[0078] (4) Knowledge plane: The knowledge plane is the core component responsible for learning strategies and calculating paths. It interacts with the management plane and the control plane. The input scheduler and output aggregator are used to synchronize the decisions of multiple agents; the execution agent collects experience and infers routing decisions during the interaction with the environment; the training agent learns from the collected experience and executes the agent to update the strategy; and the auxiliary learning mechanism is used to judge the effectiveness of the generated path.

[0079] Based on the above system architecture, the multi-path routing method provided by the present invention can be executed. Figure 4 This is a schematic diagram of a multi-path routing method in the present invention. The present invention pre-deploys an agent based on a trained reinforcement learning (RL) model at each network node. These agents only need to determine the next hop of the incoming data flow to complete the routing selection. In particular, different paths can be selected for different flows of the same source-destination switch pair, thereby realizing multi-path routing. Specifically, when a new flow request arrives, the agent can quickly make a routing decision and send the result to the local data plane of the switch, thereby realizing the corresponding data flow forwarding.

[0080] The agent is the subject that interacts with the environment. In order to ensure the scalability of MARL in multi-path routing scenarios, the present invention instantiates an agent for each switch in the AS and proposes an agent parallel collaboration mechanism, such as Figure 5 As shown, Figure 5 Schematic diagram of a parallel coordination mechanism of an agent in the present invention. Specifically, for an AS composed of N switches, in one or more embodiments of the present invention, N agent instances can be instantiated in parallel on the device side (such as an edge computing unit) and the controller side (such as a cloud computing server). The two groups of agent instances run independently, so the execution process will not be interrupted and the training process can be executed flexibly. In addition, the structure of the execution agent instance is copied from the training agent instance, so the strategies on the device side and the controller side can be transferred.

[0081] As mentioned above, the present invention calculates the route for each flow in a hop-by-hop manner, thereby decomposing the routing problem into multiple sub-problems with smaller state spaces and action spaces. When a data flow forwarding request arrives at a certain network node, the network node (also an intelligent agent) will take action to select the next hop for the data flow. The next network node will also take a similar approach until the data flow reaches the target network node. Finally, the complete path of the data flow can be obtained, that is, an ordered sequence of network nodes.

[0082] Therefore, the present invention models the route generation process as a typical multi-agent reinforcement learning (MARL) problem, in which each network node is an agent. The decision-making process of MARL is usually expressed as a Markov Game (MG), which is characterized by a tuple (N, S, A, P, R, γ). The specific definition is as follows: N = {1, ..., n, ..., |N|} represents the index set of |N| agents. S represents the global state space shared by all agents n∈N. A = A1× ... ×A n ×…×A |N| represents the joint action space of all agents, where A n represents the action space of agent n, a t ∈A and They represent the joint action of all agents at time t and the action of agent n at time t. P∶S×A×S→[0,1] represents the joint action a t ∈A, from state s t ∈S to the next time step s t+1 ∈s state transition probability. R={R1,…,R n ,…,R |N|} represents a set of reward functions, where Denotes the reward function of agent n, and represents the reward obtained by agent n at time t, that is γ is a discount factor that controls the importance of future rewards.

[0083] For the knowledge plane, it can first obtain the topological information of each network node at the current moment (such as the global directed graph of the network), the status information of each network node and the information of each data stream to be forwarded, forming the state space at the current moment. The status information of each network node includes: the remaining bandwidth, transmission delay and packet loss rate of the link between each network node. The information of each data stream to be forwarded includes: the source network node and the target network node of each data stream. Of course, in one or more embodiments of the present invention, during the implementation of the routing method, the reward for each network node to perform the forwarding action can be given in combination with the transmission situation of each data stream. Therefore, in the process of determining the route hop by hop, the transmission rate, delay and packet loss rate of each data stream under the transmission of the corresponding route can also be determined at the same time, that is, the information of each data stream to be forwarded can also include the transmission rate, delay and packet loss rate of each data stream.

[0084] Then, for each network node to forward data flow, determine the data flow to be forwarded by the network node at the current moment, and determine the action space of the network node forwarding action according to the degree of the network node at the current moment. Usually, for each network node, when it forwards data flow, the action space is the probability distribution of selecting an adjacent network node as the next hop. Therefore, the number of actions of the network node is equal to the degree of the network node dr (node n ), that is, the number of switches connected to the corresponding switch of the network node. Action space A n The size of can be defined as |A n |=dr(node n ).

[0085] After that, the reinforcement learning model that has been pre-trained and deployed on the network node can be used to determine the next network node of the data flow to be forwarded by the network node at the current moment according to the current state space and the action space of the network node. Here, according to the current state space and the action space of the network node, the forwarding action corresponding to the maximum value under the current state space can be determined by querying the value table corresponding to the state transition of each forwarding action in the action space of each network node through different state spaces. According to the forwarding action, the next network node of the data flow to be forwarded by the network node at the current moment is determined.

[0086] The value table can be obtained through the training process of the reinforcement learning model. Specifically, for each network node, the reward of the network node after executing each forwarding action in the action space and converting to the state space at the next moment is determined according to the state space at the historical moment and the action space of the network node through the following formula: Of course, the reward can also be calculated and saved in real time by the executing agent during the actual application process, and can be directly used during training.

[0087] In the formula, Perform forwarding action for the nth network node at time t The reward is α1, α2, β1, β2 and β3 are different adjustment parameters. is the global perception reward at time t, is the local perceived reward at time t, threuput flow is the average transmission rate of each data stream, latency flow is the average delay of each data flow, loss flow is the average packet loss rate of each data flow, dist[dpid][dst] is the shortest distance from the nth network node to the target network node of the data flow it forwards, and dist[nextdpid][dst] is the shortest distance from the next node to the target network node of the data flow it forwards after the nth network node performs the forwarding action.

[0088] From the above reward calculation process, it can be seen that the global perception reward is related to the stream transmission rate thruput flow Inversely proportional to the flow latency flow and flow packet loss rate loss flow Since the transmission rate, delay and packet loss rate have different dimensions, they can be normalized by the following formula:

[0089] In the formula, is the normalized ith data, x i is the i-th data to be normalized, min(X) is the minimum value of the data of this type to be normalized, max(X) is the maximum value of the data of this type to be normalized, a is the lower bound of the interval mapped to the normalization, and b is the upper bound of the interval mapped to the normalization.

[0090] The local perception reward is used to evaluate the change in the shortest path length from the current node agent to the target node after taking an action. If the current action causes the path length to the target node to increase, a penalty will be given, thereby prompting the agent to learn to choose the shortest possible path.

[0091] In one or more embodiments of the present invention, the agent can use Q-Learning technology to select the next hop for the data flow. That is, a Q table can be formulated for each agent. Q-Learning is a classic reinforcement learning algorithm, and its core idea is to learn the state-action value function. This function estimates the long-term expected benefit of taking a specific action in a given state.

[0092] Therefore, according to the state space at the historical moment, the action space of the network node, and the reward of the network node executing each forwarding action in the action space and converting to the state space at the next moment, the value of each forwarding action in the action space executed by the network node can be determined by the following formula:

[0093] In the formula, is the nth network node in the state space s at time t t Execute action when The value of , α is the learning rate, γ is the discount factor, is the nth network node in the state space s at time t+1 t+1 Execute the best action The maximum estimated value of .

[0094] Finally, through multiple rounds of iterations, the value table corresponding to the state transition achieved by each forwarding action in the action space of each network node in different state spaces is obtained.

[0095] Furthermore, when determining the reward for a network node to convert to the state space at the next moment after executing each forwarding action in the action space, a trade-off can be made between selecting the expected optimal action (utilization) and selecting different actions in the hope of obtaining greater rewards in the future (exploration). The most commonly used strategies for the Q-Learning algorithm are ε-greedy and softmax strategies.

[0096] In the ε-greedy strategy, exploration is achieved by selecting random actions with equal probability, which ignores the high and low Q values. In contrast, softmax takes a more targeted approach and assigns probabilities to actions based on the estimated action value. Therefore, softmax may over-prefer actions with high estimated values, thereby limiting the exploration of other actions.

[0097] In one or more embodiments of the present invention, a good balance between exploration and utilization can be achieved by alleviating the shortcomings of traditional strategies (i.e., ε-greedy and softmax). Specifically, for each network node, a random value can be generated first. When the random value is greater than or equal to the boundary value of the preset exploration or utilization strategy, the forwarding action corresponding to the maximum value in the preset value table is determined as the forwarding action to be performed by the network node; the value table includes the value corresponding to the state transition achieved by each forwarding action in the action space of each network node through different state spaces. When the random value is less than the boundary value of the preset exploration or utilization strategy, the probability of the network node executing each forwarding action in its action space is determined according to the value table, and a forwarding action is selected as the forwarding action to be performed by the network node based on the probability of each forwarding action through a random algorithm.

[0098] Afterwards, the reward for the network node to convert to the state space at the next moment after executing the determined forwarding action can be determined; based on the state space at the historical moment, the action space of the network node, and the reward for the network node to convert to the state space at the next moment after executing the determined forwarding action, the value of the network node executing the determined forwarding action is determined, and the value table is updated through multiple rounds of iterations.

[0099] For example, the Max-Boltzman Exploration (MBE) method can be used, which actually combines the advantages of the ε-greedy and softmax strategies. Among them, the ε-greedy strategy allows a certain degree of random exploration, while the softmax strategy assigns probabilities according to the estimated values ​​of actions, making the exploration of the environment more targeted. Therefore, the MBE strategy can more effectively balance exploration and utilization, improve exploration efficiency, and avoid falling into local optimality.

[0100] Therefore, the agent can first generate a random value x∈[0,1], and determine the action to be performed by the network node from the action space according to the random value by the following formula:

[0101] Where ∈ is the boundary value of the preset exploration or exploitation strategy, that is, if x < ∈, the agent will explore; otherwise, it will exploit the current knowledge. τ is the temperature parameter. If the τ value is high, the agent tends to explore more to acquire new knowledge; if the τ value is low, the agent tends to exploit existing knowledge and choose the action with the highest Q value more frequently.

[0102] When x ≥ ∈, the agent can To determine the forwarding action corresponding to the highest value, when x<∈, the agent can The probability of executing forwarding action b is determined, and an existing random algorithm (such as a roulette algorithm) is used to select a forwarding action according to the probability of executing each forwarding action to achieve the purpose of exploration.

[0103] In addition, in one or more embodiments of the present invention, reinforcement learning is a method that combines exploration and exploitation in the learning phase. However, exploration in a real network may lead to unsafe routing decisions (eg, routing loops), resulting in poor routing performance.

[0104] Therefore, in order to improve the reliability of the system and accelerate the learning process, the present invention also proposes an auxiliary learning mechanism. Specifically, the intelligent agent can determine the candidate routing path of each data stream to be forwarded through multiple rounds of iterations at each moment, and judge whether each candidate routing path is a valid path; wherein, the routing path can be used as a valid path when there is no loop or black hole, which is only used as an example here. If it is a valid path, each candidate path is used as the routing path corresponding to each data stream to be forwarded. Otherwise, according to the source network node and the target network node of the data stream to be forwarded corresponding to the invalid path, the maximum available bandwidth path from the source network node to the target network node is determined, and it is used as the routing path of the data stream to be forwarded corresponding to the invalid path.

[0105] This mechanism checks whether the routing path generated by the agent is a valid path (i.e., no loops, no black holes, non-severely congested paths, etc.). If it is a valid path, it is directly applied to the data plane. If it is not a valid path, the auxiliary learning mechanism will select the path with the maximum available bandwidth. Once the auxiliary learning mechanism is activated, the path generated by it will replace the path generated by the agent and be applied to the data plane. Furthermore, in order to reduce the possibility of the agent generating invalid paths in subsequent decision-making processes, a penalty term (local perception reward) is added to the reward function. As the training progresses, the participation of this mechanism gradually decreases, allowing the agent to eventually obtain the optimal strategy.

[0106] Furthermore, for the execution agent and the training agent, the execution agent can be deployed in the network node, and the training agent can be deployed and trained in parallel through the cloud server, etc. Therefore, the execution agent can continuously collect experience during the actual application process, the training agent can be trained based on the collected experience, and the execution agent can be quickly updated based on the parameters obtained from the training.

[0107] Specifically, for each network node, the state space at the current moment, the forwarding action of the network node, the reward corresponding to the forwarding action of the network node, and the state space converted to the next moment after each network node executes the corresponding forwarding action in each round of iteration can be saved to form the training data corresponding to the network node.

[0108] Based on the number and topology information of network nodes, multiple reinforcement learning models to be trained are instantiated through the cloud server, and according to the corresponding relationship between each reinforcement learning model to be trained and each network node, each reinforcement learning model to be trained is trained separately through the training data of the corresponding network node, and the value tables corresponding to the state transition achieved by each forwarding action in the action space of each network node corresponding to different state spaces of each network node are obtained, and they are deployed in the corresponding network nodes respectively.

[0109] Thereby, the execution process will not be interrupted, the training process can be flexibly executed, and the execution agent can be flexibly updated. There are two main stages of reinforcement learning: the execution stage and the training stage. In the execution stage, the method of the present invention collects experience from the interaction with the environment, but does not perform training. In the training stage, the method of the present invention uses the experience collected in the execution stage to optimize the strategy of the training agent. Through the collaborative work of these two stages, the method of the present invention can efficiently learn the optimal strategy and find the optimal path for all data flows in the data plane.

[0110] Table 1 Pseudocode of multi-agent Q-learning routing algorithm

[0111]

[0112] Table 1 is a pseudo code diagram of a multi-agent Q-learning routing algorithm in the present invention. As can be seen from Table 1, the input of Algorithm 1 includes the learning rate α, parameter ∈, temperature τ, the source-destination network node pairs of all data flows nodeList and the network topology information linkLStatus. The output is the new path of all flows and the latest strategy of the executing agent. The routing algorithm of the present invention can perform routing learning according to the following steps: ① Clear the experience buffer (line 1). ② Observe the current state, select the next hop for the data flow passed by all switches where the executing agents are deployed, and Store in the experience buffer until all data flows find a path from the source to the destination network node (lines 2-6). ③ Determine whether all generated paths are safe. If not, trigger the auxiliary learning mechanism, update the buffer, and generate new paths (lines 8-11). These paths can be retrieved and installed in the switch's routing table. ④ Send the new path to the control plane, and the control plane installs the new path to the switch (line 12). ⑤ The training agent obtains experience from the experience buffer and updates the strategy of each training agent (lines 13-16). ⑥ After training is completed, the new strategy is updated to the execution agent (line 17).

[0113] based on Figure 1The multi-path routing method shown in the figure first obtains the topological information of each network node, the state information of each network node and the information of each data flow to be forwarded at the current moment to form the state space at the current moment, and then for each network node to forward the data flow, determines the data flow to be forwarded by the network node at the current moment, and determines the action space of the forwarding action of the network node according to the degree of the network node at the current moment, thereby determining the forwarding action of the network node according to the state space and the action space, that is, the next network node of the data flow to be forwarded, and finally determines the routing path of each data flow to be forwarded through multiple rounds of iterations at each moment.

[0114] The present invention splits the routing determination process of the data stream to be forwarded into the next hop determination process of a single network node at each moment, thereby splitting the large-scale state space into the state space at each moment, and splitting the overall action space into the action space of multiple network nodes, and processing them separately at each moment through the reinforcement learning model deployed on each network node, thereby reducing resource consumption and improving convergence efficiency. At the same time, the forwarding action of each network node at the previous moment in the routing determination process at each moment after the split will affect the state space at the next moment. Therefore, even for the same source-target network node pair, different routing paths will be obtained due to the change of the state space, which improves flexibility and load balancing capabilities and avoids local path overload.

[0115] The method proposed in the present invention aims to solve multiple problems in traditional routing methods:

[0116] (1) Bandwidth fragmentation problem: Traditional algorithms such as ECMP methods easily lead to bandwidth fragmentation. The method of the present invention intelligently selects paths for different data flows by comprehensively considering flow delay, flow transmission rate and packet loss rate, thereby reducing the generation of bandwidth fragmentation.

[0117] (2) Limitations of static models: Traditional routing methods rely on static models and cannot adapt to dynamically changing network environments. The method of the present invention utilizes multi-agent reinforcement learning (MARL) and the centralized control capabilities of SDN to dynamically adapt to changes in network traffic, thereby achieving more flexible and efficient routing selection.

[0118] (3) Dimensional curse and single path selection: Methods based on Q-Learning and single-agent DRL are prone to dimensional curse and single path selection when facing complex topologies. The method of the present invention overcomes the dimensional curse problem through the collaborative work of multiple agents, realizes multi-path routing, and optimizes traffic distribution.

[0119] (4) Simplification of performance indicators: Some existing studies only focus on link delay, while the method of the present invention comprehensively considers multiple key performance indicators (such as delay, transmission rate, and packet loss rate) to formulate a more comprehensive routing strategy, thereby improving the overall performance of the network.

[0120] (5) Lack of dynamic interaction capability: Traditional routing methods often interact in a static environment. The method of the present invention not only improves the accuracy of routing decisions, but also enhances the adaptability and scalability of the algorithm through dynamic interaction with the actual network environment.

[0121] The method of the present invention has significant advantages in the following aspects:

[0122] (1) Efficient multi-path routing selection: Through the collaborative work of multiple agents and the adaptability of reinforcement learning, RL-MR can intelligently select the optimal path under different network traffic characteristics, thereby improving network throughput and link bandwidth utilization.

[0123] (2) Good adaptability and reliability: The method proposed in this paper introduces an auxiliary learning mechanism, which accelerates the online learning process, enhances the adaptability and reliability of the algorithm when facing unknown large-scale topologies and partial deployment scenarios, and ensures the stability and continuous performance improvement of the algorithm in a dynamic network environment.

[0124] (3) Scalability: Due to the use of multi-agent reinforcement learning, the method proposed in the present invention can be effectively expanded to a larger-scale network topology, solving the problem that the traditional single-agent method is difficult to expand, and showing broad prospects in real-world applications.

[0125] (4) Comprehensive performance advantages: The method proposed in the present invention formulates an intelligent routing strategy based on comprehensive consideration of latency, transmission rate and packet loss rate, which is significantly better than traditional routing methods and improves the overall performance of the network.

[0126] When applying the multipath routing method provided by the present invention, it is not necessary to Figure 1 The steps are executed in the order shown. The specific execution order of the steps can be determined according to needs, and the present invention does not limit this.

[0127] The above is a multi-path routing method provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding multi-path routing method device, such as Figure 6 shown.

[0128] Figure 6 A schematic diagram of a multi-path routing method device provided by the present invention includes:

[0129] The acquisition module 201 is used to acquire the topology information of each network node at the current moment, the state information of each network node, and the source network node and the target network node of each data flow to be forwarded, so as to form the state space at the current moment;

[0130] A determination module 202 is used to determine, for each network node to forward a data stream, an action space of the network node forwarding action according to the degree of the network node at a current moment;

[0131] The routing module 203 is used to determine the next path network node of the data flow to be forwarded by the network node at the current moment according to the state space at the current moment and the action space of the network node, through the reinforcement learning model that is pre-trained and deployed on the network node, so as to form the forwarding action of the network node;

[0132] The iteration module 204 is used to determine the state information of each network node at the next moment according to the forwarding action of each network node, so as to obtain the state space at the next moment, and re-determine the network node to forward the data stream at the next moment, and determine the routing path of each data stream to be forwarded from the source network node to the target network node through multiple rounds of iterations at each moment.

[0133] For the specific definition of the multi-path routing method device, please refer to the definition of the multi-path routing method above, which will not be repeated here. Each module in the above multi-path routing method device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0134] The present invention also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 Provides multipath routing methods.

[0135] The present invention also provides Figure 7 The structural diagram of the computer device shown in FIG. Figure 7 As shown in the figure, at the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Provides multipath routing methods.

[0136] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0137] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present invention.

Claims

1. A multipath routing method, characterized in that: include: Obtain the topology information of each network node at the current moment, the state information of each network node, and the source network node and the target network node of each data flow to be forwarded, to form the state space at the current moment; For each network node to forward data streams, determine the action space of the forwarding action of the network node according to the degree of the network node at the current moment; According to the state space at the current moment and the action space of the network node, the next path network node of the data flow to be forwarded by the network node at the current moment is determined by a reinforcement learning model that is pre-trained and deployed on the network node, so as to form a forwarding action of the network node; The state information of each network node at the next moment is determined according to the forwarding action of each network node to obtain the state space at the next moment, and the network node to be used for data stream forwarding at the next moment is re-determined. The routing path of each data stream to be forwarded from the source network node to the target network node is determined through multiple rounds of iterations at each moment.

2. The multipath routing method according to claim 1, wherein: Training the reinforcement learning model specifically includes: For each network node, the reward for the network node to execute each forwarding action in the action space and then transition to the state space at the next moment is determined by the following formula based on the state space at the historical moment and the action space of the network node: According to the state space at the historical moment, the action space of the network node, and the reward of the network node executing each forwarding action in the action space and converting to the state space at the next moment, the value of each forwarding action in the action space executed by the network node is determined by the following formula: Through multiple rounds of iterations, the value table corresponding to the state transition achieved by each forwarding action in the action space of each network node in different state spaces is obtained; in, Perform forwarding action for the nth network node at time t The reward is α1, α2, β1, β2 and β3 are different adjustment parameters. is the global perception reward at time t, is the local perception reward at time t, thruput flow is the average transmission rate of each data stream, latency flow is the average delay of each data flow, loss flow is the average packet loss rate of each data flow, dist[dpid][dst] is the shortest distance from the nth network node to the target network node of the data flow it forwards, and dist[nextdpid][dst] is the shortest distance from the next node to the target network node of the data flow it forwards after the nth network node performs the forwarding action. is the nth network node in the state space s at time t t Execute actions when The value of , α is the learning rate, γ is the discount factor, is the nth network node in the state space s at time t+1 t+1 Execute the best action The maximum estimated value of .

3. The multipath routing method according to claim 2, wherein: Training the reinforcement learning model specifically includes: For each network node, a random value is generated. When the random value is greater than or equal to the boundary value of the preset exploration or utilization strategy, a forwarding action corresponding to the maximum value in the preset value table is determined as the forwarding action to be performed by the network node; the value table includes the value corresponding to the state transition achieved by each forwarding action in the action space of each network node through different state spaces; When the random value is less than the preset boundary value of the exploration or utilization strategy, the probability of the network node executing each forwarding action in its action space is determined according to the value table, and a forwarding action is selected as the forwarding action to be executed by the network node based on the probability of each forwarding action through a random algorithm; Determine the reward of the network node executing the determined forwarding action and converting to the state space at the next moment; determine the value of the network node executing the determined forwarding action based on the state space at the historical moment, the action space of the network node, and the reward of the network node executing the determined forwarding action and converting to the state space at the next moment; update the value table through multiple rounds of iterations.

4. The multipath routing method according to claim 2, wherein: The method of determining the next path network node of the data flow to be forwarded by the network node at the current moment through a reinforcement learning model that has been pre-trained and deployed on the network node according to the current state space and the action space of the network node specifically includes: According to the current state space and the action space of the network node, by querying the value table corresponding to the state transition achieved by each forwarding action in the action space of each network node in different state spaces, the forwarding action corresponding to the maximum value in the current state space is determined; According to the forwarding action, the next network node of the data flow to be forwarded by the network node at the current moment is determined.

5. The multipath routing method according to claim 1, wherein: The step of determining the routing path of each data stream to be forwarded from the source network node to the target network node through multiple rounds of iterations at each moment specifically includes: Determine the candidate routing path of each data stream to be forwarded through multiple rounds of iterations at each moment, and judge whether each candidate routing path is a valid path; wherein, a routing path is a valid path when it has no loops and no black holes; If yes, each of the candidate paths is used as a routing path corresponding to each of the data flows to be forwarded; If not, then determine the maximum available bandwidth path from the source network node to the target network node according to the source network node and the target network node of the data flow to be forwarded corresponding to the invalid path, and use it as the routing path of the data flow to be forwarded corresponding to the invalid path.

6. The multipath routing method according to claim 2, wherein: The method further comprises: For each network node, save the current state space, the forwarding action of the network node, the reward corresponding to the forwarding action of the network node, and the state space of the next moment after each network node executes the corresponding forwarding action in each round of iteration to form the training data corresponding to the network node; Based on the number and topology information of network nodes, multiple reinforcement learning models to be trained are instantiated through the cloud server, and according to the corresponding relationship between each reinforcement learning model to be trained and each network node, each reinforcement learning model to be trained is trained separately through the training data of the corresponding network node, and the value tables corresponding to the state transition achieved by each forwarding action in the action space of each network node corresponding to different state spaces of each network node are obtained, and they are deployed in the corresponding network nodes respectively.

7. The multipath routing method according to claim 1, wherein: The status information of each network node includes: remaining bandwidth, transmission delay and packet loss rate of the link between each network node.

8. A multi-path routing method and device, characterized in that: include: An acquisition module is used to acquire the topology information of each network node at the current moment, the state information of each network node, and the source network node and the target network node of each data flow to be forwarded, so as to form a state space at the current moment; A determination module, for determining, for each network node to be forwarding a data stream, an action space of the network node forwarding action according to the degree of the network node at a current moment; A routing module is used to determine the next path network node of the data flow to be forwarded by the network node at the current moment according to the state space at the current moment and the action space of the network node, through a reinforcement learning model that is pre-trained and deployed on the network node, so as to form a forwarding action of the network node; The iteration module is used to determine the state information of each network node at the next moment according to the forwarding action of each network node, so as to obtain the state space at the next moment, and re-determine the network node to forward the data flow at the next moment, and determine the routing path of each data flow to be forwarded from the source network node to the target network node through multiple rounds of iterations at each moment.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

Images