Network traffic load balancing method based on reinforcement learning

Abstract

The application discloses a network traffic load balancing method based on reinforcement learning, comprising the following steps: 1) a software defined network (SDN) controller acquires switch flow table information, constructs an end-to-end traffic matrix and a network topology; 2) based on the network topology and the traffic matrix, a linear programming model is used to solve an optimization problem with the minimum maximum link bandwidth utilization as the target, and the result is stored in a database in the form of data pairs; 3) a reinforcement learning model with the optimal bandwidth utilization and the minimum service disturbance as the target is established, and data in the database is extracted for training; 4) actual traffic forwarding requirements are collected and sent into the reinforcement learning model to obtain a key data flow set, and a linear programming model is used to solve a key data flow rerouting scheme; 5) the rerouting scheme is parsed into a transmission path; and 6) the network configuration is updated according to the transmission path, and the link load is optimized. The application has good application prospects while realizing load balancing and taking the influence on services into account.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network traffic scheduling, in particular to a network traffic load balancing method based on reinforcement learning. BACKGROUND

[0002] With the rapid development of the Internet, the traffic of data centers is growing explosively, and the business traffic demand is unevenly distributed. In order to improve the processing capacity of the data center, it is a feasible solution to realize network link load balancing by optimizing routing through traffic engineering. The traditional traffic engineering scheme realizes the best or nearly best performance by rerouting as many data streams as possible; but the packet disorder problem and delay jitter caused by rerouting have an impact on the business. In order to balance the link load while reducing the impact on the business, a partial rerouting scheme can be used, which only reroutes a small part of the key data streams.

[0003] In the partial rerouting scheme, the selection strategy of the key data streams needs to be dynamically adjusted with the changes of the network structure and the business types. However, the mainstream heuristic traffic load balancing method cannot adaptively adjust the selection of the key data streams with the changes of the network structure. SUMMARY

[0004] The present application aims to overcome the shortcomings and deficiencies of the prior art, and provides a network traffic load balancing method based on reinforcement learning, which realizes load balancing while taking into account the impact on the business, and realizes better balancing effect by rerouting as few business streams as possible and rerouting delay-insensitive business streams, thereby improving the processing capacity and efficiency of the network.

[0005] To achieve the above-mentioned purpose, the technical scheme provided by the present application is as follows: a network traffic load balancing method based on reinforcement learning, comprising the following steps:

[0006] 1) The software defined network SDN controller acquires the switch flow table information, constructs the end-to-end traffic matrix and the network topology;

[0007] 2) Based on the network topology and the traffic matrix, a linear programming model is used to solve the optimization problem with the objective of minimizing the maximum link bandwidth utilization, and the result is stored in the database in the form of data pairs;

[0008] 3) A reinforcement learning model is established with the optimization objectives of the optimal bandwidth utilization and the minimum business disturbance, and the data in the database is extracted for training;

[0009] 4) The actual traffic forwarding demand is collected and sent into the reinforcement learning model to obtain a set of key data streams, and a rerouting scheme for the set of key data streams is obtained by using a linear programming model;

[0010] 5) parse the rerouting scheme into transmission paths;

[0011] 6) update network configuration according to transmission paths, optimize link load.

[0012] Further, in step 1), the software defined network (SDN) controller acquires switch flow table information, constructs an end-to-end traffic matrix and a network topology, and the process is as follows:

[0013] The SDN controller traverses the flow table entries of each switch, receives a FlowStatsReply packet in reply from the switch, extracts the IP field and ByteCount field of the packet, aggregates the traffic according to the access switch id mapped by the IP field, and obtains the cumulative total amount of data sent by the current switch to other switches; the total amount of data ByteCountTotal1 is obtained at time t1, the total amount of data ByteCountTotal2 is obtained at time t2, the flow rate of the current switch sent to other switches is calculated through the data amount difference ByteDiff = ByteCountTotal2 - ByteCountTotal1 and the statistical time difference TimeDiff = t2 - t1, and the traffic matrix of the whole network is obtained after processing all switch flow tables; for the network topology, the SDN controller constructs a network topology graph by listening to the access and exit events of switches and their ports.

[0014] Further, in step 2), the linear programming model is as follows:

[0015] G(V,E) is a directed graph representing the network topology, V represents the switch nodes, E represents the links between switches, i, j, k represent the numbers of switch nodes;

[0016] c i,j represents the capacity bandwidth of link <i,j>, <i,j> ∈ E;

[0017] l i,j represents the total load of link <i,j>, <i,j> ∈ E;

[0018] represents the initial load of link <i,j>, <i,j> ∈ E;

[0019] D s,d represents the bandwidth requirement from source switch node s to destination switch node d in the traffic matrix, s, d ∈ V, s ≠ d;

[0020] represents the probability of s to d traffic transmission through link <i,j>, s, d ∈ V, s ≠ d, <i,j> ∈ E;

[0021] The problem of minimizing the maximum link bandwidth utilization is modeled as follows:

[0022] Goal: Minimize the maximum bandwidth utilization U;

[0023] Constraint: Link load i,j:<i,j> ∈E;

[0024] Load constraint i,j ≤c i,j ·U;i,j:<i,j> ∈E;

[0025] Path Constraints

[0026] Indicates that the traffic from s to d passes through the link starting from node k<k,i> The probability of transmission, Indicates that traffic from s to d passes through the link ending at node k<i,k> The probability of transmission.

[0027] Furthermore, in step 3), the key flow selection problem is modeled as a Markov decision process, which includes executing actions, updating states, and obtaining rewards. The corresponding action space A, state space S′, and reward function R are specifically defined as follows:

[0028] Action space A: node pairs<s,d> ,s≠d is used as the action space. In a network with a given N nodes, the size of the action space is N*(N-1). Each interaction selects M elements from it, and the set of these M elements is used as the selected action;

[0029] State space S′: The traffic matrix is ​​used as the state space, and each item represents the traffic transmission demand from the source node to the destination node. In a given network of N nodes, the state space size is N*N;

[0030] Reward function R: the data flow F corresponding to the selected M node pairs in the traffic matrix m =(f1,f2,...,f m ), f m For the mth node pair, the linear programming model of step 2) is used to solve. To remove the data stream (f1,f2,...,f m ) and obtain the optimized bandwidth utilization index U; at the same time, the data flow (f1,f2,...,f m ) is evaluated and the delay-sensitive data flow ratio P1 is obtained = the number of delay-sensitive data flows C / the total number of data flows M; the reward value ε is a small constant used to eliminate the anomaly caused by P1 being 0;

[0031] The deep neural network is used as a reinforcement learning model to approximate the optimal strategy. The probability values of the action space elements are randomly set initially. In order to improve the overall performance of the network, the maximum reward is taken as the training target, and the gradient ascent method is used to update the neural network parameters. At the same time, Baseline and information entropy are introduced in the training process to optimize the model and accelerate the convergence of the model.

[0032] The deep neural network is used as a reinforcement learning model to approximate the optimal strategy. The probability values of the action space elements are randomly set initially. In order to improve the overall performance of the network, the maximum reward is taken as the training target, and the gradient ascent method is used to update the neural network parameters. At the same time, Baseline and information entropy are introduced in the training process to optimize the model and accelerate the convergence of the model. m .

[0033] Further, the step 4) comprises the following steps:

[0034] 4.1) Collect the traffic forwarding requirements in the current network by the method of step 1);

[0035] 4.2) Filter out the key data flow set F" by the reinforcement learning model of step 3) m ;

[0036] 4.3) Solve the re-routing scheme of the key data flow set by the linear programming model of step 2) <s,d>∈F″ m .

[0037] Further, the step 5) comprises the following steps:

[0038] 5.1) Remove the elements with value 0 in solu;

[0039] 5.2) Select a data flow f m from the key data flow set F" y , determine its source node and destination node <s y ,d y >, and filter out the routing scheme subset corresponding to the data flow f y from solu At the same time, create three empty queues q1, q2, q3, add the source node s y to the queue q1, and record the current node value s t =s y ;

[0040] 5.3) Select an element from Add the node s j to the queue q1, and the node probability value to q2, and record the current node as s t =sj ;

[0041] 5.4) If the current node s t is the destination node d y , add the path in q1 to q3 and record the path probability Pop the last element of q1 and q2, return to step 5.3);

[0042] 5.5) According to the path probability Select the rerouting path Path of the data stream f y from the q3 queue;

[0043] 5.6) Select the next data stream in turn, process according to steps 5.2) to 5.5) to obtain the transmission path Paths of the key data stream set F′ m after rerouting;

[0044] Further, in step 6), the process of updating the network configuration according to the obtained transmission path is as follows:

[0045] The transmission path Paths obtained is converted into the corresponding flow table modification instruction FlowMod in the Ryu controller, and is issued to the switch through the southbound interface, so as to achieve the effect of traffic load balancing.

[0046] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0047] 1. The present application improves the efficiency of traffic load balancing, and a small amount of key data streams are rerouted to achieve near-optimal network load balancing effect, while the time for calculating the optimization scheme is shortened, which is more suitable for real-time optimization of network data stream transmission scenarios.

[0048] 2. The present application reduces the disturbance of the traffic load balancing process to the service. In the selection process of rerouting the key flow, the concept of data stream importance is introduced, and through the classification and evaluation of the data stream, the delay-insensitive data stream is preferentially selected for rerouting, which optimizes the load balancing effect while minimizing the impact on the service.

[0049] 3. The method of the present application completes the exploration from theoretical optimization to routing deployment, and has good application prospect in the field of computer network traffic scheduling. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 is the logical flow diagram of the present application.

[0051] Figure 2 is the structure diagram of the deep neural network. DETAILED DESCRIPTION

[0052] The application will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the application are not limited thereto.

[0053] As shown in the embodiment, the network traffic load balancing method based on reinforcement learning comprises the following steps: Figure 1

[0054] 1) Collecting traffic matrix and network topology information: in a network with N nodes, the traffic matrix is a two-dimensional matrix T of size N*N, and the element T [i][j] represents the bandwidth requirement of the data flow with i as the source node and j as the destination node. When collecting, the SDN controller periodically traverses the flow table items of each switch through the Openflow protocol, extracts the IP field and ByteCount field of the FlowStatsRequest message returned by the switch, aggregates the traffic according to the access switch id mapped by the IP field, and obtains the cumulative data total amount sent by the current switch to other switches. The data total amount ByteCountTotal1 is obtained at t1, and the data total amount ByteCountTotal2 is obtained at t2. The traffic rate (ByteDiff / TimeDiff) of the current switch sent to other switches is calculated through the data amount difference ByteDiff=ByteCountTotal2-ByteCountTotal1 and the statistical time difference TimeDiff=t2-t1. After processing all the switch flow tables, the traffic matrix of the whole network can be obtained. At the same time, the SDN controller constructs the network topology graph by listening to the access and exit events of the switch and its port.

[0055] 2) Obtaining the minimum maximum bandwidth utilization: according to the traffic matrix and network topology obtained in step 1), a linear programming model is used to solve the optimization problem with the goal of minimizing the maximum bandwidth utilization, and the linear programming model is as follows:

[0056] G(V,E) is a directed graph representing the network topology, V represents the switch node, E represents the link between the switches, i, j, k represent the number of switch nodes;

[0057] c i,j represents the capacity bandwidth of the link <i,j>, <i,j>∈E;

[0058] l i,j represents the total load of the link <i,j>, <i,j>∈E;

[0059] represents the initial load of the link <i,j>, <i,j>∈E;

[0060] D s,d ​represents the bandwidth requirement from source node s to destination node d in the traffic matrix, s,d∈V,s≠d;

[0061] Indicates that traffic from s to d passes through link<i,j> The probability of transmission s,d∈V,s≠d,<i,j> ∈E;

[0062] The problem of minimizing the maximum link bandwidth utilization is modeled as follows:

[0063] Goal: Minimize the maximum bandwidth utilization U;

[0064] Constraint: Link load i,j:<i,j> ∈E;

[0065] Load constraint i,j ≤c i,j ·U i,j:<i,j> ∈E;

[0066] Path Constraints

[0067] Indicates that the traffic from s to d passes through the link starting from node k<k,i> The probability of transmission, Indicates that traffic from s to d passes through the link ending at node k<i,k> the probability of transmission;

[0068] After solving the problem of minimizing the maximum bandwidth utilization U, it and the traffic matrix are stored in the database in the form of data pairs.

[0069] 3) The key flow selection problem is modeled as a Markov decision process, including executing actions, updating states, and obtaining rewards. The corresponding action space A, state space S′, and reward function R are specifically defined as:

[0070] Action space A: node pairs<s,d> ,s≠d is used as the action space. In a network with a given N nodes, the size of the action space is N*(N-1). Each interaction selects M elements from it, and the set of these M elements is used as the selected action;

[0071] State space S′: The traffic matrix is ​​used as the state space, and each item represents the traffic transmission demand from the source node to the destination node. In a given network of N nodes, the state space size is N*N;

[0072] Reward function R: the data flow F corresponding to the selected M node pairs in the traffic matrix m =(f1,f2,...,f m ), f m For the mth node pair, the linear programming model of step 2) is used to solve. To remove the link load after the data stream (f1, f2,..., f m ), the optimized bandwidth utilization index U is obtained; at the same time, the data stream (f1, f2,..., f m ) is evaluated, and the proportion of delay-sensitive data stream P1 = number of delay-sensitive data stream C / total number of data stream M is obtained; the reward value ε is a small constant, which is used to eliminate the abnormality when P1 is 0;

[0073] The deep neural network is used as the reinforcement learning model to approximate the optimal strategy, and the probability value of the action space element is randomly set at the beginning; in order to improve the overall performance of the network, the maximum reward is taken as the training target, the gradient ascent method is used to update the neural network parameters, and Baseline and information entropy are introduced in the training process to optimize the model and accelerate the convergence of the model;

[0074] The traffic matrix obtained in step 2) and the data pair set composed of the minimum maximum bandwidth utilization are taken out from the database and input into the deep neural network, the structure of the network is as shown in Figure 2 , the feature information of the traffic matrix T is extracted through a convolutional layer, and the expression ability of the model is realized through a fully connected layer, and the obtained strategy is the probability value P" of the node pair in the traffic matrix, and the first M items with the maximum probability are selected as the key data stream set F' m .

[0075] 4) Solve the rerouting scheme of the key data stream set: the actual traffic transmission demand collected is input into the deep neural network, the key data stream is obtained, and the rerouting scheme of the key data stream set is obtained using a linear programming model <s,d>∈F′ m .

[0076] 5) Analyze the rerouting scheme: according to the rerouting scheme obtained in step 4), the backtracking algorithm is used to analyze the transmission path, which is specifically:

[0077] 5.1) Remove the elements with value 0 in solu;

[0078] 5.2) Select a data stream f m from the key data stream set F" y , determine its source node and destination node <s y ,d y >, and select the routing scheme subset corresponding to the data stream f y from solu ; simultaneously create three empty queues q1, q2, q3, and add the source node s y to the queue q1, and record the current node value s t =s y;

[0079] 5.3) from select an element s j is added to the queue q1, and the node probability value is added to q2, and the current node is recorded as s t =s j ;

[0080] 5.4) if the current node s t is the destination node d y , the path in q1 is added to q3, and the path probability is recorded The end elements of q1 and q2 are popped out, and step 5.3) is returned to;

[0081] 5.5) according to the path probability , the rerouting path Path of the data stream f y is selected from the q3 queue;

[0082] 5.6) the next data stream is selected in turn, and the steps 5.2) to 5.5) are processed to obtain the transmission path Paths of the key data stream set F′ m after rerouting.

[0083] 6) Configuration is issued: according to the transmission path Paths parsed out in step 5), the corresponding flow table modification instruction FlowMod is converted in the Ryu controller, and is issued to the switch through the southbound interface to complete the configuration update, and a better load balancing effect is achieved.

[0084] The above embodiment is a preferred embodiment of the present application, but the embodiments of the present application are not limited by the above embodiment, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application should be equivalent replacement methods, and are all included in the protection scope of the present application.

Claims

1. A network traffic load balancing method based on reinforcement learning, characterized in that: The following steps are involved: 1) The software-defined network (SDN) controller obtains switch flow table information and builds an end-to-end traffic matrix and network topology; 2) Based on the network topology and traffic matrix, a linear programming model is used to solve the optimization problem with the goal of minimizing the maximum link bandwidth utilization, and the results are stored in the database in the form of data pairs; The linear programming model is as follows: The G(V,E) directed graph represents the network topology, where V represents the switch node, E represents the link between switches, and i, j, k represent the numbers of the switch nodes; c i,j represents the capacity bandwidth of link <i,j>,<i,j> ∈E; l i,j Indicates a link<i,j> The total load,<i,j> ∈E; Denote the initial load of link <i,j>, <i,j> ∈ E; D s,d represents the bandwidth requirement from the source switch node s to the destination switch node d in the traffic matrix, s,d∈V,s≠d; represents the probability that the traffic from s to d is transmitted through link <i,j>, s,d∈V,s≠d, <i,j>∈E; The problem of minimizing the maximum link bandwidth utilization is modeled as follows: Goal: Minimize the maximum bandwidth utilization U; Constraint: Link load Load constraint i,j ≤c i,j ·U;i,j:<i,j>∈E; Path Constraints represents the probability that the traffic from s to d is transmitted through the link <k,i> starting from node k, represents the probability that traffic from s to d is transmitted through the link 〈i,k〉 with node k as the terminal; 3) Establish a reinforcement learning model with optimal bandwidth utilization and minimal service disruption as optimization goals, and extract data from the database for training; The key flow selection problem is modeled as a Markov decision process, which includes executing actions, updating states, and obtaining rewards. The corresponding action space A, state space S′, and reward function R are specifically defined as follows: Action space A: The node pair <s, d>, s≠d is used as the action space. In a network with N nodes, the size of the action space is N*(N-1). Each interaction selects M elements from it, and the set of these M elements is used as the selected action. State space S′: The traffic matrix is ​​used as the state space, and each item represents the traffic transmission demand from the source node to the destination node. In a given network of N nodes, the state space size is N*N; Reward function R: the data flow F corresponding to the selected M node pairs in the traffic matrix m =(f1,f2,...,f m ), f m For the mth node pair, the linear programming model of step 2) is used to solve. To remove the data stream (f1,f2,...,f m ) and obtain the optimized bandwidth utilization index U; at the same time, the data flow (f1,f2,...,f m ) is evaluated and the delay-sensitive data flow ratio P1 is obtained = the number of delay-sensitive data flows C / the total number of data flows M; the reward value ε is a small constant used to eliminate the anomaly caused by P1 being 0; A deep neural network is used as a reinforcement learning model to approximate the optimal policy. Initially, the probability values ​​of the action space elements are randomly set. To improve the overall performance of the network, maximizing the reward is set as the training goal. The gradient ascent method is used to update the neural network parameters. At the same time, baseline and information entropy are introduced during the training process to optimize the model and accelerate the convergence of the model. The deep neural network extracts the characteristic information of the traffic matrix through a convolutional layer, and then realizes the expression ability of the network with a fully connected layer. The obtained strategy is the probability value of the node pair in the traffic matrix, and the top M items with the largest probability are selected as the key data flow set F′ that needs to be rerouted. m ; 4) Collect actual traffic forwarding requirements and feed them into a reinforcement learning model to obtain a set of key data flows. A linear programming model is then used to solve a rerouting solution for this set of key data flows. 5) Parsing the rerouting solution into a transmission path; 6) Update network configuration according to the transmission path and optimize link load.

2. The network traffic load balancing method based on reinforcement learning according to claim 1 is characterized in that: In step 1), the SDN controller obtains the switch flow table information and builds the end-to-end traffic matrix and network topology. The process is as follows: The SDN controller traverses the flow table entries of each switch. After receiving the FlowStatsReply message from the switch, it extracts the IP field and ByteCount field of the message, aggregates the traffic based on the access switch ID mapped to the IP field, and obtains the total amount of data sent by the current switch to other switches. The total data volume ByteCountTotal1 is obtained at time t1, and the total data volume ByteCountTotal2 is obtained at time t2. The traffic rate sent by the current switch to other switches is calculated through the data volume difference ByteDiff = ByteCountTotal2 - ByteCountTotal1 and the statistical time difference TimeDiff = t2 - t1. After processing all switch flow tables, the traffic matrix of the entire network can be obtained; for the network topology, the SDN controller builds a network topology map by monitoring the access and exit events of the switches and their ports.

3. The network traffic load balancing method based on reinforcement learning according to claim 2 is characterized in that: The step 4) comprises the following steps: 4.1) Collect traffic forwarding requirements in the current network using the same method as step 1); 4.2) Filter out the key data flow set F′ through the reinforcement learning model in step 3) m ; 4.3) Solve the rerouting solution of the key data flow set through the linear programming model of step 2) <s,d>∈F′ m .

4. The network traffic load balancing method based on reinforcement learning according to claim 3 is characterized in that: The step 5) comprises the following steps: 5.1) Remove the elements with value 0 in solu; 5.2) Select the key data flow set F′ m A data stream f in y , determine its source node and destination node y ,d y >, and filter out the data stream f from solu y The corresponding routing solution subset At the same time, create three empty queues q1, q2, and q3, and set the source node s y Join queue q1 and record the current node value s t =s y ;​ 5.3) From Select an element The node s j Join queue q1, node probability value Join q2 and record the current node as s t =s j ; 5.4) If the current node s t Destination node d y , add the path in q1 to q3 and record the path probability Pop the last elements of q1 and q2 and return to step 5.3); 5.5) According to the path probability Select data flow f from queue q3 y The rerouting path Path; 5.6) Select the next data stream in turn and process it according to steps 5.2) to 5.5) to obtain the key data stream set F m Paths after rerouting; 5. The network traffic load balancing method based on reinforcement learning according to claim 4 is characterized in that: In step 6), the process of updating the network configuration according to the obtained transmission path is as follows: The obtained transmission path Paths is converted into the corresponding flow table modification instruction FlowMod in the Ryu controller and sent to the switch through the southbound interface to achieve traffic load balancing.

Images