Flow routing method and device in segment routing network, equipment and storage medium

Abstract

The invention discloses a traffic routing method and device in a segment routing network, equipment and a storage medium. The method comprises the steps that an initial traffic matrix corresponding to a first time period and an initial routing configuration strategy corresponding to the initial traffic matrix are acquired, and the initial traffic matrix is a traffic matrix formed by traffic requirements in a segmented routing network in the first time period; if it is detected that there is a difference between the target traffic matrix corresponding to the second time period and the initial traffic matrix, calculating an initial link utilization rate of the target traffic matrix under an initial routing configuration strategy; determining a target routing configuration strategy from a plurality of routing configuration strategies based on the initial link utilization rate through a routing decision model, wherein the target routing configuration strategy is a routing configuration strategy with the minimum maximum link utilization rate; and routing the target traffic matrix based on the target routing configuration strategy. According to the scheme provided by the embodiment of the invention, network congestion in the segmented routing network can be avoided, and balanced distribution of network traffic is realized.

Description

Technical Field

[0001] The present application belongs to the field of network traffic engineering technology, and in particular, relates to a traffic routing method, device, equipment and storage medium in a segmented routing network. Background Art

[0002] With the rapid development of the Internet, the Internet has encountered problems such as explosive growth in network traffic. In addition, the continuous development of audio and video services has also put forward service quality requirements for the Internet. Limited by routing algorithms and scheduling strategies, network traffic is very likely to be unevenly distributed on the link, resulting in network congestion and reduced network service quality. Traffic engineering is a technology that optimizes network traffic distribution. It can optimize the scheduling of network traffic to achieve network traffic load balancing, reduce congestion, and improve the utilization of network resources. The traffic in the network has been continuously changing over time. It is of great significance to consider traffic changes in network traffic engineering.

[0003] However, in the related art, when the network traffic changes, the original routing configuration strategy in the network traffic engineering is still used, or the new routing configuration strategy used is not compatible with the changed network traffic, resulting in uneven distribution of network traffic on the link, causing network congestion. Summary of the invention

[0004] The embodiments of the present application provide a traffic routing method, apparatus, device and storage medium in a segmented routing network, which can avoid network congestion in the segmented routing network and achieve balanced distribution of network traffic.

[0005] In a first aspect, an embodiment of the present application provides a traffic routing method in a segmented routing network, the method comprising: obtaining an initial traffic matrix corresponding to a first time period and an initial routing configuration policy corresponding to the initial traffic matrix, the initial traffic matrix being a traffic matrix formed by traffic demand in the segmented routing network during the first time period; when a difference is detected between a target traffic matrix corresponding to a second time period and the initial traffic matrix, calculating an initial link utilization of the target traffic matrix under the initial routing configuration policy, wherein the second time period is a next time period corresponding to the first time period; inputting the initial link utilization into a routing decision model, determining a target routing configuration policy from a plurality of routing configuration policies based on the initial link utilization through the routing decision model, wherein the target routing configuration policy is a routing configuration policy with the smallest maximum link utilization; and routing the target traffic matrix based on the target routing configuration policy.

[0006] In a second aspect, an embodiment of the present application provides a traffic routing device in a segmented routing network, the device comprising: a strategy acquisition module, used to obtain an initial traffic matrix corresponding to a first time period and an initial routing configuration policy corresponding to the initial traffic matrix, the initial traffic matrix being a traffic matrix formed by traffic demand in the segmented routing network during the first time period; a link calculation module, used to calculate the initial link utilization of the target traffic matrix under the initial routing configuration policy when a difference is detected between the target traffic matrix corresponding to the second time period and the initial traffic matrix, wherein the second time period is the next time period corresponding to the first time period; a routing decision module, used to input the initial link utilization into a routing decision model, and determine a target routing configuration policy from multiple routing configuration policies based on the initial link utilization through the routing decision model, wherein the target routing configuration policy is a routing configuration policy with the smallest maximum link utilization; a traffic routing module, used to route the target traffic matrix based on the target routing configuration policy.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the traffic routing method in the segmented routing network as described in the first aspect is implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the traffic routing method in the segmented routing network as described in the first aspect is implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the traffic routing method in a segmented routing network as described in the first aspect.

[0010] It can be seen from the above content that when a change in the traffic matrix is ​​detected, the link utilization of the traffic matrix after the change under the initial routing configuration strategy corresponding to the traffic matrix before the change is first calculated, and then the link utilization is processed using the routing decision model to determine the routing configuration strategy with the minimum maximum link utilization, thereby obtaining the target routing configuration strategy corresponding to the traffic matrix after the change. That is, in the embodiment of the present application, when a change in the traffic matrix is ​​detected, the target routing configuration strategy corresponding to the traffic matrix after the change can be determined in time, thereby avoiding the problem of network congestion caused by untimely updating of the routing configuration strategy.

[0011] In addition, in the embodiment of the present application, the maximum link utilization can characterize the balance of traffic distribution in the network. The smaller the maximum link utilization, the more balanced the traffic distribution in the network after executing its corresponding routing configuration strategy. In the embodiment of the present application, the routing configuration strategy with the smallest maximum link utilization is used as the routing strategy of the changed traffic matrix, which can avoid the problem of network congestion caused by the incompatibility of the routing configuration strategy with the traffic matrix, and achieve balanced distribution of network traffic. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0013] Figure 1 This is a schematic diagram of traffic routing corresponding to SR provided by an embodiment of the present application;

[0014] Figure 2 It is a flow chart of a traffic routing method in a segmented routing network provided by an embodiment of the present application;

[0015] Figure 3 This is a traffic routing diagram provided by an embodiment of the present application;

[0016] Figure 4 This is a routing decision diagram provided by an embodiment of the present application;

[0017] Figure 5 is a schematic diagram of the structure of an action network provided by an embodiment of the present application;

[0018] Figure 6 (a) is a schematic diagram of the cumulative probability distribution of the performance ratio obtained by the Abilene method provided by an embodiment of the present application on the traffic matrix of a small test set;

[0019] Figure 6 (b) is a schematic diagram of the cumulative probability distribution of the performance ratio obtained by the CERNET method provided by an embodiment of the present application on the traffic matrix of a small test set;

[0020] Figure 7 (a) is a schematic diagram of the cumulative probability distribution of the performance ratio obtained by the Abilene method provided by an embodiment of the present application on the traffic matrix of a large test set;

[0021] Figure 7 (b) is a schematic diagram of the cumulative probability distribution of the performance ratio obtained by the CERNET method provided by an embodiment of the present application on the traffic matrix of a large test set;

[0022] Figure 8 is a structural schematic diagram of a traffic routing device in a segmented routing network provided by another embodiment of the present application;

[0023] Fig. 9 It is a structural schematic diagram of an electronic device provided in yet another embodiment of the present application. DETAILED DESCRIPTION

[0024] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0025] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0026] For ease of understanding, before describing the solution provided in the embodiments of the present application, the background of the solution provided in the embodiments of the present application is first described.

[0027] Segment Routing (SR) is a source routing mechanism that can be applied to IP (Internet Protocol) / MPLS (Multi-Protocol Label Switching) or IPv6 (Internet Protocol Version 6) networks. In SR, a segment is associated with a topology- or service-based instruction and is represented in the form of a segment identifier (SID). There are various types of segments. The commonly used segment in traffic engineering is the node segment, which can be used to represent the shortest path for a routing node to forward a data packet to a specific node in ECMP (Equal-Cost Multi-Path Routing), that is, it stipulates that the route of the data packet must pass through a certain routing node. When using SR routing, a segment list is encapsulated in the header of the data packet to indicate the route of the data packet. For example, in Figure 1 In the traffic routing diagram corresponding to the SR shown in the figure, the routing network includes 8 routing nodes, namely A, B, C, D, E, F, G, and H. Routing node A is the initiating node of the data packet, and routing node H is the destination node of the data packet. There are multiple links between routing node A and routing node H, and these multiple links have the same IGP (Interior Gateway Protocol) link weight. Figure 1 In the figure, the header of the data packet (Packet) starting from routing node A is encapsulated with a segment list [E, H], which indicates that the data packet needs to pass through routing node E and routing node H in sequence during the routing process. The data packet is first transmitted along the shortest routing path ACE from routing node A to routing node E. When the data packet reaches routing node E, routing node E in the segment list is popped up. Then, the data packet is routed along the shortest routing paths EFH and EGH from routing node E to routing node H until it reaches the destination node H, and routing node H in the segment list is popped up, so that routing node H can obtain the complete data packet.

[0028] From the above description, we can see that Figure 1 The routing process shown uses two segments to indicate an intermediate routing node (i.e. Figure 1 The routing node E in the example is called 2-SR routing. Figure 1It can be seen that the SR protocol controls the traffic path through the segment list encapsulated in the packet header, and the intermediate forwarding nodes do not need to store path status information, thereby reducing the routing configuration strategy and update overhead of the SR protocol. It can be seen that SR can be applied to online traffic engineering where the routing configuration strategy changes dynamically with traffic changes.

[0029] It should be noted that in the embodiment of the present application, the network topology is G = (V, E), where V is a set of nodes, E is a set of unidirectional links, and each routing node v∈V supports the SR protocol. For each link e∈E, its weight w e and bandwidth (i.e. link capacity c e ) are all known numbers. In the embodiment of the present application, the traffic demand in the network changes with time, and the set time is divided into continuous time slots τ = 1, 2, ..., each time slot can represent a short period of time, for example, 5 minutes. In each time slot τ, the traffic demand in the network constitutes the traffic matrix TM (τ) , the traffic matrix contains a total of |V|(|V|-1) traffic demands between each pair of routing nodes in the network. It represents the traffic demand between routing node i and routing node j at time slot τ.

[0030] The embodiment of the present application aims to achieve that, in the scenario of the network topology and the network traffic demand that changes over time, in each time slot τ, the traffic matrix TM is (τ) Find the best routing configuration strategy RC (τ) , thereby minimizing the current maximum link utilization MLU (τ) .

[0031] To achieve the above objectives, the embodiment of the present application uses a 2-SR method to route traffic and restricts each traffic to use only one SR path for routing, rather than splitting traffic on multiple traffic. The SR routing configuration strategy for all traffic in the network can be expressed as RC = {m i,j |i,j∈V,i≠j}, where variable m i,j Indicates the intermediate routing nodes that the traffic route from routing node i to routing node j passes through.

[0032] It should be noted that, in the embodiment of the present application, any routing node in the network can be selected as the intermediate routing node. i,j It can be any routing node in V. i,j =i and m i,j = j corresponds to the same routing path, that is, the traffic route from routing node i to routing node j does not pass through any intermediate routing nodes, but is routed on the IGP shortest routing path between routing node i and routing node j. Therefore, in the embodiment of the present application, m is set i,j≠i. In addition, the variable represents the intermediate routing node between routing node i and routing node j at time slot τ; It represents the proportion of traffic passing through link e when the traffic is routed on the IGP shortest routing path between routing node i and routing node k, and routing node k and routing node j. e When known It can be calculated by the shortest path algorithm such as Floyd's algorithm, and the SR-TE problem to be solved in each time slot τ can be modeled as an integer linear programming problem as shown below, namely the 2SR-TE-ILP problem:

[0033] min MLU (τ) (1)

[0034]

[0035]

[0036]

[0037] In the above formulas (1) to (4), is a binary variable, indicating whether the traffic from routing node i to routing node j is routed through routing node k as the intermediate routing node at time slot τ; if but

[0038] In addition, in the above 2SR-TE-ILP problem, formula (1) represents the optimization goal of the 2SR-TE-ILP problem, that is, minimizing the maximum link utilization; formula (2) is used to constrain the traffic size routed on each link not to exceed the bandwidth of the link; formula (3) is used to constrain each traffic demand to select only one 2-SR path when routing; formula (4) is used to constrain the variable The value is 0 or 1.

[0039] In real-time traffic matrix MLU (τ) When the time slot τ is known, the optimal routing configuration strategy for each time slot τ can be obtained by solving the 2SR-TE-ILP problem using linear programming software such as Gurobi. However, in practical applications, it is difficult to measure the real-time traffic matrix, and the prediction of the traffic matrix is ​​not accurate, but the network link utilization is relatively easy to measure. e (TM,RC) represents the traffic size of the route on link e when the traffic matrix TM in the network is routed according to the routing configuration policy RC, that is:

[0040]

[0041] In formula (5), the link utilization u of link e is e It can be defined as the ratio of the link bandwidth occupied, that is:

[0042]

[0043] In formula (6), U = {u e |e∈E} is the set of utilizations of all links in the network, and the maximum link utilization

[0044] From equations (5) and (6), we can see that the real-time link utilization information U (τ) The measurement of U is relatively simple. (τ) It can be calculated based on the network link bandwidth and the traffic receiving and sending rate of the router port. Therefore, the target of the solution provided in the embodiment of the present application is the traffic matrix TM expected in the time slot τ (τ) Unknown and only the link utilization U (τ) Given the known situation, give the best routing configuration strategy RC (τ) . When the traffic matrix information is unknown, the relationship between the traffic routing configuration strategy and the network link utilization is implicit. When the SR routing configuration strategy is used as the decision variable of the problem and the link utilization is used as both the input and optimization target of the problem, it is impossible to establish a direct mathematical connection between the two through methods such as linear programming. In this regard, the embodiment of the present application uses deep reinforcement learning to learn the implicit relationship between the input (link utilization) and the output (routing configuration strategy) and solve the above problem.

[0045] In order to solve the above problems, the embodiments of the present application provide a method, apparatus, device and storage medium for traffic routing in a segmented routing network. The following first introduces the method for traffic routing in a segmented routing network provided by the embodiments of the present application.

[0046] Figure 2 FIG. 1 is a flow chart showing a flow chart of a traffic routing method in a segmented routing network provided by an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0047] Step S201, obtaining an initial traffic matrix corresponding to a first time period and an initial routing configuration strategy corresponding to the initial traffic matrix, wherein the initial traffic matrix is ​​a traffic matrix formed by traffic demands in a segment routing network in the first time period.

[0048] In step S201, the first time period is the above-mentioned time slot. In the embodiment of the present application, τ-1 is used to represent the first time period.

[0049] As an example, in Figure 3In the traffic routing diagram shown, 6 routing nodes are deployed in the segmented routing network, the straight lines between the routing nodes represent network links, and the curves with arrows represent network traffic demand and the paths they are routed in the network, where the curve width represents the size of the traffic demand. In the embodiment of the present application, U(TM,RC) represents the set of link utilizations of the network when the traffic matrix TM in the network is routed according to the routing configuration policy RC, and MLU(TM,RC) represents the maximum link utilization corresponding to the network when the traffic matrix TM in the network is routed according to the routing configuration policy RC.

[0050] exist Figure 3 In the time slot τ-1, the initial traffic matrix in the network is TM (τ-1) , the routing decision model can output the traffic matrix TM (τ-1) The corresponding initial routing configuration strategy RC (τ-1) , thereby obtaining the initial traffic matrix and the initial routing configuration strategy corresponding to the initial traffic matrix.

[0051] It should be noted that for each time slot τ, the routing decision model can interact with the network, wherein the routing decision model can determine the routing configuration strategy corresponding to the current traffic matrix according to the link utilization corresponding to the traffic matrix, and then use the routing configuration strategy to perform routing configuration on the current traffic matrix to minimize the maximum link utilization. The model structure and training process of the routing decision model are described below and will not be described here for the time being.

[0052] Step S202, when it is detected that there is a difference between the target traffic matrix corresponding to the second time period and the initial traffic matrix, the initial link utilization of the target traffic matrix under the initial routing configuration strategy is calculated.

[0053] In step S202, the second time period is the next time period corresponding to the first time period. In the embodiment of the present application, τ is used to represent the second time period. In addition, when there is a difference between the traffic matrices corresponding to the two time periods, it indicates that the traffic matrices corresponding to the two time periods have changed. At this time, the method provided by the embodiment of the present application can be used to route the traffic in the network. In addition, the network manager can also make routing decisions for the traffic in the network at other times according to actual needs. For example, the network manager can use the measurement platform to monitor the network link utilization in real time, and when the link utilization meets certain trigger conditions (for example, the maximum link utilization is greater than a threshold), the method provided by the embodiment of the present application is used to update the routing configuration strategy in the network.

[0054] As an example, in Figure 4 In the routing decision diagram shown in the figure, when the time comes to the time slot τ, the traffic matrix in the network changes, for example, from TM(τ-1) Change to TM (τ) , that is, the changed traffic matrix is ​​TM (τ) At the initial moment of time slot τ, the routing configuration strategy has not changed, and the routing configuration strategy in the network is still RC (τ-1) , but the routing configuration strategy corresponding to τ-1 may cause network congestion. In this regard, in the embodiment of the present application, in the routing configuration strategy RC (τ-1) Lower Convection MatrixTM (τ) Perform traffic routing and measure the network link utilization (i.e. initial link utilization) U(TM) through the measurement platform (τ) ,RC (τ-1) ).

[0055] It should be noted that the above-mentioned measurement platform can quickly calculate the current utilization of each link in the network according to the traffic receiving and sending rate of each router port and the bandwidth of the network link. The measurement platform can be any device that can realize link utilization measurement, and the embodiments of the present application do not specifically limit this.

[0056] Step S203, inputting the initial link utilization into the routing decision model, and determining the target routing configuration strategy from multiple routing configuration strategies based on the initial link utilization through the routing decision model, wherein the target routing configuration strategy is the routing configuration strategy with the minimum maximum link utilization.

[0057] In one example, if Figure 4 As shown, after obtaining the routing configuration policy RC (τ-1) Lower Convection MatrixTM (τ) After the initial link utilization is obtained by traffic routing, the initial link utilization U(TM (τ) ,RC (τ-1) ) as the input of the routing decision model, the routing decision model can output the target routing configuration strategy RC (τ) , that is, determine TM (τ) The 2-SR path for each traffic demand in .

[0058] Step S204: routing the target traffic matrix based on the target routing configuration policy.

[0059] It should be noted that the update of the routing configuration strategy in the network can be achieved through the above steps S201 to S203. After the routing configuration strategy in the network is updated, the traffic routing of the target traffic matrix can be performed based on the updated routing configuration strategy. For example, Figure 4 In the network, after the routing configuration strategy is updated, the traffic matrix TM in the time slot τ (τ) According to the new routing configuration policy RC (τ)Routing in the network, at this time, the link utilization changes to U(TM (τ) ,RC (τ) ), the corresponding maximum link utilization is MLU(TM (τ) ,RC (τ) At the same time, the maximum link utilization MLU (TM (τ) ,RC (τ) ) is also input into the routing decision model as feedback on traffic engineering performance.

[0060] In addition, as time goes by, the above steps S201 to S204 are repeated continuously, thus forming the workflow of the traffic engineering provided by the embodiment of the present application. In the above process, the traffic engineering solution measures the link utilization at the initial moment of each time slot (i.e., the end moment of the previous time slot), and uses it as the input of the routing decision model to decide the routing configuration strategy in the current time slot. The network administrator can also measure the link utilization and run the traffic engineering algorithm at other times according to the needs.

[0061] Based on the scheme defined by the above steps S201 to S204, it can be known that in the embodiment of the present application, when a change in the traffic matrix is ​​detected, the link utilization of the traffic matrix after the change under the initial routing configuration strategy corresponding to the traffic matrix before the change is first calculated, and then the link utilization is processed using the routing decision model to determine the routing configuration strategy with the minimum maximum link utilization, thereby obtaining the target routing configuration strategy corresponding to the traffic matrix after the change. That is, in the embodiment of the present application, when a change in the traffic matrix is ​​detected, the target routing configuration strategy corresponding to the traffic matrix after the change can be determined in time, thereby avoiding the problem of network congestion caused by untimely updating of the routing configuration strategy.

[0062] In addition, in the embodiment of the present application, the maximum link utilization can characterize the balance of traffic distribution in the network. The smaller the maximum link utilization, the more balanced the traffic distribution in the network after executing its corresponding routing configuration strategy. In the embodiment of the present application, the routing configuration strategy with the smallest maximum link utilization is used as the routing strategy of the changed traffic matrix, which can avoid the problem of network congestion caused by the incompatibility of the routing configuration strategy with the traffic matrix, and achieve balanced distribution of network traffic.

[0063] The following is an introduction to the routing decision model used to implement routing decisions in the embodiments of the present application.

[0064] The routing decision model provided in the embodiment of the present application combines the action branching technology on the basis of the reinforcement learning algorithm PPO (Proximal Policy Optimization), interacts with the network environment and learns how to determine the routing configuration strategy corresponding to each time slot.

[0065] In reinforcement learning, the agent can continuously interact with the environment through states, actions, and rewards. For each step t, the agent observes the state s from the environment t , and according to the state s t Select an action t Action a t is executed in the environment, and the state of the environment is transferred to s t+1 , while the reward value r t is sent to the agent, with state s t 、Action a t , reward value r t And the state after transfer s t+1 The formed quaternion (s t ,a t ,r t ,s t+1 ) is called state transfer, or simply transfer. Reinforcement learning can be modeled as a Markov decision process (MDP) Among them, S is the state set, is a collection of actions, is the transition probability matrix, represents the probability of the state transferring to s′ after taking action a in state s; r is the reward function, r(s t ,a t )(or abbreviated as r t ) means in state s t Take action a t The immediate reward obtained; γ∈[0,1] represents the discount factor.

[0066] The agent’s behavior is based on the policy π θ (·|s), the strategy π θ (·|s) is used to represent the probability distribution of choosing each action in a given state, where θ is the parameter corresponding to the strategy, The agent interacts with the environment for multiple rounds to collect experience, where experience is the state transition data. Each round consists of multiple steps, starting from the initial state and ending at the terminal state, or ending after a fixed number of steps. The goal of reinforcement learning is to learn the optimal strategy through the collected experience to maximize the expected cumulative discounted reward.

[0067] The proximal policy optimization algorithm PPO is a policy gradient reinforcement learning algorithm that can be applied to environments with continuous or discrete action spaces. PPO uses the action-evaluation method to train the policy π θ There are two neural networks in the action-criteria method, namely the action network and the critique network. The action network represents the strategy π θ , which takes state s as input and outputs the probability distribution π over actions θ (·|s); the evaluation network takes state s as input and outputs a state value function V with φ as parameter φ (s), the state value function is used to approximate the policy π after the state s. θ The expected cumulative reward that can be obtained by taking an action; Q value Q(s,a) represents the expected cumulative reward that can be obtained after taking action a in state s; advantage function A θ (s t ,a t ) is used to indicate that in state s t Take action a t How much better or worse is it than randomly choosing an action, which can be defined as:

[0068]

[0069] Set the PPO algorithm to interact with the environment for multiple rounds and collect a collection of trajectories Among them, each trajectory ρ={s1,a1,r1,…,s T ,a T ,r T} consists of a series of continuous state transitions. The PPO algorithm uses a generalized advantage estimation method to calculate the advantage value corresponding to each state and action in the trajectory through the following formula:

[0070]

[0071] δ t =r t +γV φ (s t+1 )-V φ (s t ) (9)

[0072] In formula (8), λ∈[0,1] is the discount coefficient. The gradient of the objective function J(θ) can be approximated as:

[0073]

[0074] In order to make full use of the collected data, in the PPO algorithm, formula (10) is not directly used as the gradient to update the parameters θ of the action network, but formula (11) is used as the gradient and the same data is used. Train the action network for multiple rounds to update the parameters θ:

[0075]

[0076] In formula (11), π θ′ Indicates data collection The old strategy used when θ Indicates the new strategy being updated. ∈∈(0,1) is used to control the update of strategy π θ With π θ′ The parameter of the degree of difference between Used to limit the extent of each update, that is, to ensure that each update is not too large. It can be determined by formula (12):

[0077]

[0078] At the beginning of each training, π θ =π θ′ For collecting data θ′ As the number of training times increases, π θ It is continuously updated during the training process and becomes similar to π θ′ After training, the action network of the PPO algorithm is based on the updated strategy π θ Interact with the environment again and collect trajectories; evaluate the parameters of the network by minimizing the state-value function V output by the network φ (s) and Usage Data The mean square error of the expected cumulative reward is calculated to update, and the mean square error can be calculated by formula (13):

[0079]

[0080] In formula (13), L(φ) is the objective function corresponding to the evaluation network, and φ is the target network parameter of the evaluation network; is the state transition data set, ρ is any state transition data set in the state transition data set, T is the number of training times of the action network; r t′is the reward value corresponding to the t′th step in each training; γ t′-t is the discount factor of the t′th step relative to the tth step in each training; V φ (s t ) is state s t The corresponding state value function.

[0081] Based on the above-mentioned reinforcement learning training method, the embodiment of the present application also trains the routing decision model according to the state, action and reward in reinforcement learning.

[0082] First, for the state, in the embodiment of the present application, the traffic matrix that needs to be routed at the tth step is limited to TM (t) , the old router in the network environment is configured as RC (t-1) The routing decision model is based on the new traffic matrix TM (t) Configure RC based on old routing (t-1) The link utilization U(TM) obtained by routing (t) ,RC (t-1) )={u e (TM (t) ,RC (t-1) )|e∈E} as input. In the embodiment of the present application, the elements in the link utilization set U are normalized, and the vector of 1×|E| shown in formula (14) is used as the state:

[0083]

[0084] For actions, the routing decision model takes the routing configuration strategy RC as the action, which can be expressed as a = [a i,j |i,j∈V,i≠j], the action is defined in discrete space. For problems with discrete action space, the dimension of the action network output layer, that is, the number of neurons, should be equal to the number of all possible actions, and each neuron in the output layer outputs the probability of selecting the corresponding action. In the SR-TE problem, there are L = |V| (|V|-1) traffic demands, and each traffic demand can choose one of K = |V|-1 routing nodes as an intermediate routing node for 2-SR routing. The number of all possible routing configuration strategies is L K =[|V|(|V|-1)] |V|-1 , even if the number of nodes in the network |V| is small, L K The corresponding value will also be very large. Therefore, a large number of action spaces is difficult for most reinforcement learning algorithms to handle. Moreover, the action space in the embodiment of the present application is not only discrete, but also high-dimensional. The action space has L dimensions, and each dimension a i,j =m i,jis a sub-action, corresponding to the intermediate routing node selected by the traffic from routing node i to routing node j. There are L possible choices in total. Therefore, the use of the existing solution will lead to the problem of explosion in the number of actions.

[0085] In order to solve the problem of the explosion of the number of actions, the embodiment of the present application adopts the action branching technology to solve the reinforcement learning problem with a high-dimensional discrete action space by changing the neural network structure of the action network.

[0086] Specifically, in an embodiment of the present application, the routing decision model includes at least an action network and an evaluation network, the action network is used to determine the probability distribution of the target state data on the routing strategy, and the target state data is used to characterize the link utilization of the target traffic matrix under the initial routing configuration strategy; the evaluation network is used to determine the expected cumulative reward obtained by routing the target traffic matrix according to the first routing configuration strategy, and the first routing configuration strategy is any one of multiple routing configuration strategies.

[0087] The evaluation network includes at least an input layer, multiple hidden layers and an output layer; the action network includes at least an input layer, multiple hidden layers and multiple branch output layers, the multiple branch output layers are separated from each other, each branch output layer corresponds to a routing configuration strategy, and each branch output layer is used to output the intermediate routing node through which the traffic flowing from the first routing node to the second routing node flows under the corresponding routing configuration strategy.

[0088] As an example, Figure 5 The schematic diagram of the action network is shown in Figure 1. Figure 5 As shown in the figure, the input layer and hidden layer of the changed neural network are the same as those of the ordinary neural network, but the output layer does not use a complete output layer, but multiple separate branch output layers. Figure 5 In the example, the input layer of the action network with a size of 1×|E| takes state s as input. After the input layer, there are multiple fully connected hidden layers. The input layer and the hidden layer together constitute the latent expression of the input state s, which is shared by L branch output layers. Each branch output layer has a size of 1×K, is fully connected to the neurons in the last hidden layer, and outputs the selection probability π of the K possible intermediate routing nodes of the flow from routing node i to routing node j. i,j (·|s), the probability distribution used to select the sub-action a i,j , that is, the intermediate routing node selected by the traffic demand from routing node i to routing node j.

[0089] During the model training process, in order to enable the algorithm to fully explore the entire action space and learn how to choose the best action, the sub-action a i,j According to the probability distribution π i,j(·|s) sampling selection. When running online, the routing decision model directly outputs the sub-action with the highest probability, that is, Select sub-action a according to the branch output layer i,j After that, all sub-actions a i,j Combined into a complete action a.

[0090] In the above way, the action representation is distributed on L branch output layers, and the total number of neurons in the output layer increases from L K Reduced to LK. Since all branch output layers share the same latent expression, the selection of sub-actions corresponding to each traffic demand is not completely independent, but coordinated by the shared neural network input layer and hidden layer, which not only makes the routing decision model output a more appropriate target routing configuration strategy, but also helps to improve the stability of model training.

[0091] For the evaluation network, since there is no problem of neuron explosion, in the embodiment of the present application, the evaluation network uses a common feedforward neural network structure. The evaluation network also takes the state as input, the input layer size is 1×|E|, there are multiple hidden layers, the output layer size is 1×1 and outputs the state value function V φ (s t ). Among them, the neurons in adjacent layers are connected by full connection.

[0092] For rewards, the routing decision model is based on MLU (TM (t) ,a t ) is used as feedback from the network environment. In the embodiment of the present application, MLU (TM (t) ,a t ) is abbreviated as MLU. In the offline training stage, the routing decision model is trained using the collected historical traffic matrix information, TM (t) For a known traffic matrix, the optimal maximum link utilization ILP of each traffic matrix under the 2-SR routing mode can be obtained by solving the 2SR-TE-ILP problem (i.e., formula (1) to formula (4)). In this scenario, the reward value can be determined by formula (15):

[0093]

[0094] In formula (15), Used to describe the gap between traffic engineering performance and the optimal integer linear programming problem.

[0095] From formula (15), we can see that when calculating the reward value, we first convert Subtract 1 and multiply by 2 to get Scale to a suitable range; then use an exponential function to give a larger penalty when traffic engineering performs extremely poorly; finally, add 1 to the obtained value to limit the possible range of the reward value to (-∞,0], where the closer the reward value is to 0, the better the performance of the routing decision model.

[0096] It is easy to notice that since the routing decision model uses a branched action network, the objective function used when updating the neural network parameters needs to be modified accordingly. According to formula (11), when calculating the objective function J for updating the action network parameters, PPO (θ), each state s in the trajectory needs to be known t and action a t Corresponding advantage A θ′ (s t ,a t ), probability π θ (a t |s t ) and probability π θ′ (a t |s t ). θ′ (s t ,a t ) is calculated using the r value in the trajectory t Value and evaluation network output V φ (s t ) to calculate, and has nothing to do with the action network. θ (a t |s t ) and π θ′ (a t |s t ) is the direct output of the action network.

[0097] It should be noted that the branch action network used in the embodiment of the present application outputs not the probability of a complete action π(·|s), but the probability of a sub-action π i,j (·|s). Although all branches share the same input layer and hidden layer, the sub-action a i,j The choice of depends only on the probability value output by the corresponding branch output layer. Therefore, when the selection of sub-actions is assumed to be an independent event, the probability of the complete action can be calculated by formula (16):

[0098]

[0099] In addition, in order to enhance the exploration efficiency of the routing decision model, the embodiment of the present application also adds an entropy reward on the basis of the original objective function, thereby enhancing the diversity of actions output by the neural network. For the branch action network used by the routing decision model, the embodiment of the present application uses the average value of the entropy of each branch output layer as the entropy of the entire action network. The objective function of the action network parameter update of the embodiment of the present application is shown in formula (17):

[0100]

[0101] In formula (17), is the entropy of the branch output layer, and its calculation formula is shown in formula (18):

[0102]

[0103] It should be noted that in formula (17), is the objective function of the action network, θ is the target network parameter of the action network, and θ′ is the network parameter before update; is the state transition data set, ρ is any state transition data set in the state transition data set, T is the number of training times of the action network, a i,j is the intermediate routing node that the traffic demand from routing node i to routing node j passes through; π θ (a i,j |s) is the intermediate routing node a under the target network parameter θ of the action network. i,j Routing configuration strategy in state s; π θ′ (a i,j |s) is the intermediate routing node a under the network parameter θ′ before the update. i,j In state a i,j Routing configuration policy under t is the state corresponding to the tth step in each training; a t is the action taken in the tth step in each training; A θ′ (s t ,a t ) is the network parameter θ′ before updating, at s t In state a t The corresponding advantage function; ∈ is a preset constant; β is a preset coefficient used to balance the proportion of entropy rewards in the objective function; L is the number of branch output layers; The entropy function corresponding to the branch output layer, is the target network parameter θ of the action network in state s t The probability that the traffic from routing node i to routing node j flows through the intermediate routing nodes is .

[0104] It should be noted that since the routing decision model provided in the embodiment of the present application does not use a special evaluation network structure, in the embodiment of the present application, the loss function of the evaluation network is as shown in formula (13).

[0105] In addition, in an embodiment of the present application, the training of the routing decision model includes two stages: offline training and network update. In the offline training stage, the embodiment of the present application uses historical traffic matrix data to simulate network traffic changes and train the routing decision model. In the offline training stage, the number of offline running rounds (i.e., the number of training times) M is determined by the network administrator, and the number of steps T in each round is the number of historical traffic matrices. At each step of the reinforcement learning algorithm, a historical traffic matrix is ​​used, and in one round, all historical traffic matrix data is used in sequence in chronological order. At the tth step, the traffic matrix TM is used in the network environment. (t) , reinforcement learning agent with MLU(TM (t) ,a t-1 ) as state s t And give the routing configuration a t In the above mode, the number of steps that each traffic matrix participates in is the number of rounds that the routing decision model runs.

[0106] It should be noted that the training of the above routing decision model usually only needs to run dozens or hundreds of rounds to train a reinforcement learning model with good traffic engineering performance. Therefore, thousands of historical traffic matrix data can be effectively utilized within an acceptable time. In addition, since the complete traffic matrix data set is used for training in each round of the routing decision model operation, each historical traffic matrix is ​​fully utilized.

[0107] In addition, in order to shorten the training time and improve the traffic engineering performance of the algorithm, the embodiment of the present application also uses a parallel environment during the offline training stage. The routing decision model does not interact with a simulated network environment during the offline training stage, but interacts with N independent parallel network environments simultaneously through multiple processes and collects trajectories. At each step, the routing decision model observes N different states from N parallel network environments, and selects N corresponding actions according to the strategy, which are executed in the corresponding network environment, and then receives N corresponding reward values. There are T steps in each round (that is, there are T historical traffic matrix data in total), and the routing decision model performs T steps every T steps. learn ≤T steps to conduct a learning (training and updating neural network parameters). At this time, a total of N lengths of T are collected. learn The trajectory (i.e., N*T learnIn this way, the routing decision model collects and learns unrelated experiences from independent network environments (i.e., multiple parallel network environments) at the same time, which helps the convergence of the neural network. In addition, due to the parallelism brought by multiple processes, the speed of data collection through interaction with the environment will also be accelerated.

[0108] Specifically, in the embodiment of the present application, the network parameters of the action network and the network parameters of the evaluation network can be determined by the following steps:

[0109] Step S301, obtaining initial network parameters of the action network and initial network parameters of the evaluation network;

[0110] Step S302, for each training, determining the state transition data of each historical traffic matrix in the historical traffic matrix set in multiple parallel network environments, and obtaining multiple state transition data groups corresponding to each historical traffic matrix, wherein the multiple parallel network environments have the same network topology result, and the multiple parallel network environments correspond to different routing configuration strategies respectively;

[0111] Step S303, storing multiple state transition data groups corresponding to each historical traffic matrix into the target buffer to obtain a state transition data set; wherein the state transition data group is a data group composed of state transition data, and the state transition data group may include state transition data before the strategy is updated, the routing configuration strategy before the update, the reward value, and the state transition data after the strategy is updated;

[0112] Step S304, based on the state transition data group in the state transition data set, the initial network parameters of the action network and the initial network parameters of the evaluation network are updated respectively to obtain the target network parameters of the action network and the target network parameters of the evaluation network.

[0113] In one example, in the process of determining a state transition data group, first, in a first parallel network environment, the maximum link utilization of a first historical traffic matrix under a previous routing configuration strategy is determined to obtain first state transition data, and the current routing configuration strategy is determined based on the first state transition data; then, the maximum link utilization of the first historical traffic matrix under the current routing configuration strategy is determined in the first parallel network environment to obtain second state transition data, and the reward value of the first historical traffic matrix under the current routing configuration strategy is calculated; finally, a state transition data group corresponding to the first historical traffic matrix is ​​formed based on the first state transition data, the current routing configuration strategy, the reward value, and the second state transition data.

[0114] It should be noted that the first parallel network environment mentioned above is any one of a plurality of parallel network environments, and the first historical traffic matrix is ​​any one of the traffic matrices in the historical traffic matrix set.

[0115] As an example, offline training of the routing decision model can be implemented through the following pseudo code:

[0116]

[0117]

[0118] It should be noted that the input of the offline training phase is the network topology G = (V, E), the historical traffic matrix data {TM (t) |t=1,…,T} and the maximum link utilization {ILP (t) |t=1,…,T}, the number of parallel network environments N, the number of rounds M and the number of training interval steps T when the algorithm is running learn .

[0119] From the pseudo code of the offline training stage above, we can see that in the offline training stage, first, the parameters θ of the action network and the parameters φ of the evaluation network are randomly initialized, and the trajectory buffer is cleared. Then, M rounds are run, each with T steps, where T is the number of historical traffic matrices. At each step, the routing decision model interacts with N simulated parallel network environments and collects state transitions. At step t, the traffic matrix of the routing in the network is TM (t) , the routing decision model observes the traffic matrix TM from each parallel network environment (t) In the previous step, the routing configuration a t-1 Maximum link utilization MLU (TM) when routing (t) ,a t-1 ), and calculate the state s t , select action a according to the state t Execute in the corresponding environment and receive the reward value r t . State transition data set t ,a t ,r t ,s t+1 ) is stored in the trajectory buffer In the algorithm implementation, the embodiment of the present application uses multiple processes to run each simulated parallel network environment in parallel and enables the intelligent agent to interact with it. The pseudo code uses a loop form for simplicity of expression.

[0120] It should be noted that although the traffic matrix TM is used in all network environments at step t (t) To calculate the state s t and reward r t However, due to the action a given by the agent t is based on the probability distribution πθ (·|s t ) sampling, the action a executed in each parallel network environment t are not the same, therefore, the state s of each parallel network environment t , Reward t and the trajectory experienced are different. When the trajectory buffer The number of transitions in is equal to N*T learn When , update the parameters of the action network and the evaluation network, and clear the trajectory buffer after the update

[0121] This completes the offline training of the network model.

[0122] In the network update phase, first, the advantage value corresponding to each state transition data group in the target buffer is calculated, and the network parameters when the function value of the objective function corresponding to the action network is maximized are calculated based on the advantage value to obtain the target network parameters of the action network; the network parameters when the function value of the objective function corresponding to the evaluation network is minimized are calculated based on the advantage value to obtain the target network parameters of the evaluation network.

[0123] As an example, the network update of the routing decision model can be implemented by the following pseudo code:

[0124]

[0125] It should be noted that the input of the network update phase is the network parameters θ and φ before the update, and the trajectory buffer Parameters γ, λ, ∈, β used to calculate the objective function and loss function, and the number of training rounds and the size of the mini-batch

[0126] From the pseudo code of the above network update phase, we can see that in the network update phase, first use The advantage value A corresponding to the action selected in each state is calculated according to formula (8) θ (s t ,a t ), and then update the parameters of the neural network In each update round, The transfers in are randomly divided into multiple small batches, each of which has state transition data. Using the state transition data in each mini-batch data set, the parameters θ of the action network are optimized by maximizing the objective function shown in formula (17): The parameter φ of the evaluation network is updated by minimizing the loss function L(φ) shown in formula (13).

[0127] In order to verify the effectiveness of the training method provided in the embodiment of the present application, the present application evaluates the performance of the online traffic engineering method Adpt-SRTE (i.e., the method provided in the embodiment of the present application) through simulation experiments. The process mainly includes three stages: the experimental setting stage, the traffic engineering performance comparison stage, and the time cost comparison stage.

[0128] In the experimental setup phase, two network topologies were used for the experiment: the Abilene topology has 12 nodes and 30 links, and the CERNET topology has 14 nodes and 32 links. The weights and bandwidths of the network links are provided by the topology data. This experiment used real traffic matrix data of the two topologies, and measured the traffic matrix every 5 minutes. In order to fully evaluate the performance of each method, this experiment used two traffic matrix data sets, smaller and larger. Among them, in the smaller traffic matrix data set, 10 traffic matrices are used for training, and 100 traffic matrices are used for testing; in the larger traffic matrix data set, 15 days of traffic matrix data are used for training, and the next 15 days of traffic matrix data are used for testing. There are 4320 traffic matrices in the training set and the test set.

[0129] This experiment uses Python language and PyTorch deep learning framework to implement the Adpt-SRTE method. The action network uses three hidden layers with 512 neurons, and the evaluation network uses two hidden layers with 256 neurons. The parameters of the neural network are updated using the Adam optimizer, and the learning rate is set to 0.0003, where γ = 0.99, λ = 0.95, ∈ = 0.2, β = 0.01, and the number of parallel network environments N = 16. After each T learn = 32 steps to update the neural network parameters, number of training rounds The size of the mini-batch For the small traffic matrix dataset, Adpt-SRTE is trained offline for M=500 rounds, with T=100 steps in each round. For the large traffic matrix dataset, Adpt-SRTE is trained offline for M=50 rounds, with T=4320 steps in each round. For both the small and large traffic matrix datasets, this experiment first uses the training set to train Adpt-SRTE offline, and then uses the traffic matrix in the test set to test and evaluate the online traffic engineering performance of Adpt-SRTE.

[0130] In this experiment, Adpt-SRTE is compared with the following traffic engineering methods using reinforcement learning:

[0131] (1) MARL (Multiagent Reinforcement Learning) + GNN (Graph Neural Networks). This method combines multi-agent reinforcement learning and graph neural networks to determine the weights of network links and perform online traffic engineering optimization in traditional IP networks that use IGP protocol routing.

[0132] (2) DATE. This method uses multi-agent reinforcement learning to decide the diversion ratio of network traffic on a preset path and performs online traffic engineering optimization in a network based on explicit path routing.

[0133] Since the offline training of the above two methods is very time-consuming, for small traffic matrix data sets, this experiment uses all 100 traffic matrices in the training set to train these two methods; for large traffic matrix data sets, 100 or 200 traffic matrices are selected from the 4320 traffic matrices in the training set for training. For MARL+GNN, this experiment selects the last 100 or 200 traffic matrices in chronological order from the 4320 traffic matrices in the training set; for DATE, this experiment first uses a 15th-order polynomial function to fit the size of each traffic demand in all traffic matrices in the training set, and obtains the fitted traffic matrix data to eliminate outliers, and then extracts 100 or 200 traffic matrices from all 4320 fitted traffic matrices at uniform intervals. After the training is completed, all 100 or 4320 traffic matrices in the test set are used to evaluate the traffic engineering performance of the method.

[0134] In addition, this experiment also compares the following non-machine learning methods:

[0135] (1) 2-SR-ILP, for each traffic matrix in the test set, solve the 2-SR-ILP problem and obtain the theoretical minimum maximum link utilization of the traffic matrix under the 2-SR routing mode.

[0136] (2) SPF (Shortest Path First): For each traffic matrix in the test set, the traffic demand is routed along the IGP shortest route and the corresponding maximum link utilization is obtained.

[0137] The performance ratio is used to evaluate the traffic engineering performance of each method. For a method, the performance ratio under a certain traffic matrix refers to the ratio of the maximum link utilization obtained by the method to the maximum link utilization obtained by solving the MCF (network service-based technology) problem (i.e. the theoretically optimal maximum link utilization). The closer the performance ratio is to 1, the better the traffic engineering performance of the method.

[0138] For the comparison of traffic engineering performance, this experiment evaluates the traffic engineering performance on larger and smaller traffic matrix datasets.

[0139] First, the traffic engineering performance of each method on a small traffic matrix dataset is evaluated. This experiment trains the method using reinforcement learning using the traffic matrix in the small training set, then tests and records the maximum link utilization obtained by all methods on the traffic matrix in the small test set, and calculates the performance ratio. Figure 6 The cumulative probability distribution of the performance ratios of all methods on the traffic matrix of the small test set is shown in , where Figure 6 (a) is the cumulative probability distribution of the performance ratio obtained by the Abilene method on the traffic matrix of the small test set, Figure 6 (b) is the cumulative probability distribution of the performance ratio obtained by the CERNET method on the traffic matrix of the small test set.

[0140] Depend on Figure 6 It can be seen that on the topologies Abilene and CERNET, the 2-SR-ILP method achieves the same maximum link utilization (i.e., the performance ratio is 1) as the theoretical optimal solution of the MCF problem for more than 80% of the traffic matrices, which indicates that the 2-SR routing mode can provide sufficient traffic engineering capabilities on the topologies Abilene and CERNET.

[0141] It should be noted that 2-SR-ILP calculates the corresponding routing configuration based on the known traffic matrix, while Adpt-SRTE first trains with the traffic matrix in the training set and then outputs the routing configuration for the traffic matrix in the test set when only the link utilization is known. Figure 6 The results in show that the maximum link utilization obtained by Adpt-SRTE is not as good as that of 2-SR-ILP. The experimental results show that the traffic engineering performance of Adpt-SRTE on the small traffic matrix dataset meets the requirements.

[0142] For the three online traffic engineering methods using reinforcement learning, on the topology Abilene, Adpt-SRTE, MARL+GNN, and DATE have similar traffic engineering performance, and their performance ratios are better than the shortest route SPF, among which DATE has poor traffic engineering performance when the performance ratio is large. On the topology CERNET, Adpt-SRTE performs well, but MARL+GNN and DATE have poor traffic engineering performance. Similarly, except for MARL+GNN, which has slightly poor traffic engineering performance when the performance ratio is large, the performance ratios of the three methods are better than the shortest route SPF.

[0143] For the traffic engineering performance evaluation of each method on the large traffic matrix dataset, the experiment was also conducted according to the aforementioned experimental settings. The method using reinforcement learning was trained using the traffic matrix in the large training set. Then, the maximum link utilization obtained by all methods on the traffic matrix in the large test set was tested and recorded, and the performance ratio was calculated. Figure 7 The cumulative probability distribution of the performance ratios of all methods on the traffic matrix of the large test set is shown in Figure 7 (a) is the cumulative probability distribution of the performance ratio obtained by the Abilene method on the traffic matrix of the large test set, Figure 7 (b) is the cumulative probability distribution of the performance ratio obtained by the CERNET method on the traffic matrix of the large test set.

[0144] Depend on Figure 7 It can be seen that the performance ratio of the 2-SR-ILP method on more than 80% of the traffic matrices in the Abilene and CERNET topologies is 1, which further verifies that the 2-SR routing mode can provide sufficient traffic engineering capabilities on these two topologies. On the large traffic matrix data set, Adpt-SRTE also has good traffic engineering performance, and its performance ratio is much better than the shortest route SPF, and the gap in traffic engineering performance with the 2-SR-ILP method is within a reasonable range.

[0145] As for MARL+GNN and DATE, since the offline training time of these two methods is too time-consuming, this experiment only selects 100 or 200 flow matrices from all 4320 flow matrices in the large flow matrix training set for offline training. Figure 7 The legend of the figure indicates the number of traffic matrices used in the training of the corresponding curve. On the large traffic matrix dataset, the traffic engineering performance of both methods is significantly worse than Adpt-SRTE, and the performance of the two methods on different network topologies is inconsistent. On the Abilene topology, the performance of MARL+GNN is unsatisfactory, while the performance of DATE is worse than the shortest route SPF. However, on the CERNET topology, the performance of DATE is unsatisfactory, but the performance of MARL+GNN is much worse than the shortest route SPF. In addition, the number of traffic matrices used in the offline training phase has no obvious effect on the traffic engineering performance of the two methods. Figure 7 As can be seen in the figure, after offline training with 100 and 200 traffic matrices, there is no significant difference in the online traffic engineering performance of the two methods. Compared with MARL+GNN and DATE, Adpt-SRTE also has good traffic engineering performance on large traffic matrix datasets. In the case of a large amount of historical traffic matrix data, Adpt-SRTE can be applied to long-term online traffic engineering optimization.

[0146] For the comparison of time cost, this experiment evaluates the time cost on larger and smaller traffic matrix data sets, as shown in Tables 1 and 2. Table 1 shows the training time of each method on the small traffic matrix training set, and Table 2 shows the training time of each method on the large traffic matrix training set.

[0147] Table 1

[0148]

[0149] Table 2

[0150]

[0151] As shown in Tables 1 and 2, on a small dataset, the offline training time of the Adpt-SRTE method is much shorter than that of the other two methods. The training time of Adpt-SRTE is less than one hour, while MARL+GNN offline training takes several hours and DATE offline training takes more than one day. Nevertheless, Adpt-SRTE outperforms the other two methods in traffic engineering on the small traffic matrix test set. On a large dataset, Adpt-SRTE uses 4320 traffic matrices for offline training, while MARL+GNN and DATE only use 100 or 200 selected traffic matrices. However, the offline training time of Adpt-SRTE is still relatively short. The offline training time of Adpt-SRTE is about 4 hours, while MARL+GNN takes 3 to 4 hours to train if it uses 100 traffic matrices. In addition, the training time of Adpt-SRTE is longer than that of MARL+GNN and DATE. Adpt-SRTE outperforms the other two methods in traffic engineering on the large traffic matrix test set. Increasing the number of traffic matrices used for offline training of MARL+GNN and DATE can improve the traffic engineering performance of these two methods on the test set, but the huge training time overhead that comes with it is unbearable.

[0152] Table 3 gives the average computation time required by each method to make online routing decisions for each traffic matrix in the test set. As can be seen from Table 3, the online routing decision time of all methods is around tens of milliseconds.

[0153] Table 3

[0154]

[0155] Based on the above content, the experimental results of this experiment show that Adpt-SRTE can make full use of a large amount of historical traffic matrix data to perform long-term online traffic engineering optimization after a short period of offline training, and the online decision-making time is short, which meets the requirements of online traffic engineering.

[0156] The present application also provides a traffic routing device in a segmented routing network, such as Figure 8 As shown, the device 800 includes: a policy acquisition module 801, a link calculation module 802, a routing decision module 803 and a traffic routing module 804.

[0157] The strategy acquisition module 801 is used to acquire an initial traffic matrix corresponding to the first time period and an initial routing configuration strategy corresponding to the initial traffic matrix, wherein the initial traffic matrix is ​​a traffic matrix formed by traffic demands in the segment routing network in the first time period;

[0158] The link calculation module 802 is used to calculate the initial link utilization of the target traffic matrix under the initial routing configuration strategy when a difference is detected between the target traffic matrix corresponding to the second time period and the initial traffic matrix, wherein the second time period is a next time period corresponding to the first time period;

[0159] The routing decision module 803 is used to input the initial link utilization into the routing decision model, and determine the target routing configuration strategy from multiple routing configuration strategies based on the initial link utilization through the routing decision model, wherein the target routing configuration strategy is the routing configuration strategy with the minimum maximum link utilization;

[0160] The traffic routing module 804 is used to route the target traffic matrix based on the target routing configuration policy.

[0161] In one example, a routing decision model includes at least an action network and an evaluation network, wherein the action network is used to determine the probability distribution of target state data on the routing strategy, and the target state data is used to characterize the link utilization of the target traffic matrix under the initial routing configuration strategy; the evaluation network is used to determine the expected cumulative reward obtained by routing the target traffic matrix according to a first routing configuration strategy, and the first routing configuration strategy is any one of a plurality of routing configuration strategies; the evaluation network includes at least an input layer, a plurality of hidden layers, and an output layer; the action network includes at least an input layer, a plurality of hidden layers, and a plurality of branch output layers, wherein the plurality of branch output layers are separated from each other, each branch output layer corresponds to a routing configuration strategy, and each branch output layer is used to output the intermediate routing nodes through which the traffic flowing from the first routing node to the second routing node flows under the corresponding routing configuration strategy.

[0162] In one example, a traffic routing device in a segmented routing network includes a model training module for determining network parameters of an action network and network parameters of an evaluation network in the following manner: obtaining initial network parameters of the action network and initial network parameters of the evaluation network; for each training, determining state transition data of each historical traffic matrix in a historical traffic matrix set in multiple parallel network environments, and obtaining multiple state transition data groups corresponding to each historical traffic matrix, wherein the multiple parallel network environments have the same network topology results, and the multiple parallel network environments correspond to different routing configuration strategies, respectively; storing the multiple state transition data groups corresponding to each historical traffic matrix in a target buffer, and obtaining a state transition data set; and updating the initial network parameters of the action network and the initial network parameters of the evaluation network based on the state transition data groups in the state transition data set, and obtaining target network parameters of the action network and target network parameters of the evaluation network.

[0163] In one example, the model training module is specifically used to determine the maximum link utilization of the first historical traffic matrix under the previous routing configuration strategy in a first parallel network environment, and obtain first state transfer data, wherein the first parallel network environment is any one of multiple parallel network environments, and the first historical traffic matrix is ​​any one of the traffic matrices in the historical traffic matrix set; determine the current routing configuration strategy according to the first state transfer data; determine the maximum link utilization of the first historical traffic matrix under the current routing configuration strategy in the first parallel network environment, and obtain second state transfer data; calculate the reward value of the first historical traffic matrix under the current routing configuration strategy; and form a state transfer data group corresponding to the first historical traffic matrix based on the first state transfer data, the current routing configuration strategy, the reward value, and the second state transfer data.

[0164] In one example, the model training module is specifically used to calculate the advantage value corresponding to each state transition data group in the target buffer; based on the advantage value, the network parameters when the function value of the objective function corresponding to the action network is maximum are calculated to obtain the target network parameters of the action network; based on the advantage value, the network parameters when the function value of the objective function corresponding to the evaluation network is minimum are calculated to obtain the target network parameters of the evaluation network.

[0165] In one example, the model training module determines the target network parameters of the action network through the following function:

[0166]

[0167] in, is the objective function of the action network, θ is the target network parameter of the action network, and θ′ is the network parameter before update; is the state transition data set, ρ is any state transition data set in the state transition data set, T is the number of training times of the action network, a i,j is the intermediate routing node that the traffic demand from routing node i to routing node j passes through; π θ (a i,j |s) is the intermediate routing node a under the target network parameter θ of the action network. i,j Routing configuration strategy in state s; π θ′ (a i,j |s) is the intermediate routing node a under the network parameter θ′ before the update. i,j In state a i,j Routing configuration policy under t is the state corresponding to the tth step in each training; a t is the action taken in the tth step in each training; A θ′ (s t ,a t ) is the network parameter θ′ before updating, at s t In state a t The corresponding advantage function; ∈ is a preset constant; β is a preset coefficient; L is the number of branch output layers; The entropy function corresponding to the branch output layer, is the target network parameter θ of the action network in state s t The probability that the traffic from routing node i to routing node j flows through the intermediate routing nodes is .

[0168] In one example, the model training module determines the target network parameters of the evaluation network through the following function:

[0169]

[0170] Where L(φ) is the objective function corresponding to the evaluation network, φ is the target network parameter of the evaluation network; D is the state transition data set, ρ is any state transition data set in the state transition data set, T is the number of training times of the action network; r t′ is the reward value corresponding to the t′th step in each training; γ t′-t is the discount factor of the t′th step relative to the tth step in each training; V φ (s t ) is state s t The corresponding state value function.

[0171] The traffic routing device in the segmented routing network provided in the embodiment of the present application can implement each process implemented in the aforementioned method embodiment, and will not be described again here to avoid repetition.

[0172] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0173] Fig. 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0174] The electronic device may include a processor 901 and a memory 902 storing computer program instructions.

[0175] Specifically, the processor 901 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0176] The memory 902 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 902 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 902 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 902 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 902 is a non-volatile solid-state memory.

[0177] The memory may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical or other physical / tangible memory storage devices. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0178] The processor 901 implements any one of the traffic routing methods in the segment routing network in the above embodiments by reading and executing the computer program instructions stored in the memory 902 .

[0179] In one example, the electronic device may further include a communication interface 903 and a bus 910. Fig. 9 As shown, the processor 901, the memory 902, and the communication interface 903 are connected via a bus 910 and communicate with each other.

[0180] The communication interface 903 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0181] Bus 910 includes hardware, software or both, and the parts of electronic equipment are coupled to each other.For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 910 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0182] In addition, in combination with the traffic routing method in the segmented routing network in the above embodiments, the embodiments of the present application may provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any traffic routing method in the segmented routing network in the above embodiments is implemented.

[0183] In addition, in combination with the traffic routing method in the segmented routing network in the above embodiments, the embodiments of the present application may provide a computer program product for implementation. When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device executes and implements the traffic routing method in the segmented routing network in any of the above embodiments.

[0184] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0185] The functional modules shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. Programs or code segments can be stored in machine-readable media, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable media" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0186] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0187] Aspects of the present disclosure are described above with reference to the flowcharts and / or block diagrams of the traffic routing methods, devices, equipment, and storage media in the segmented routing network according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of boxes in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the functions / actions specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0188] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A traffic routing method in a segment routing network, characterized in that: include: Acquire an initial traffic matrix corresponding to a first time period and an initial routing configuration strategy corresponding to the initial traffic matrix, wherein the initial traffic matrix is ​​a traffic matrix formed by traffic demands in a segment routing network in the first time period; When a difference is detected between the target traffic matrix corresponding to the second time period and the initial traffic matrix, calculating the initial link utilization of the target traffic matrix under the initial routing configuration strategy, wherein the second time period is a next time period corresponding to the first time period; Inputting the initial link utilization into a routing decision model, and determining a target routing configuration strategy from a plurality of routing configuration strategies based on the initial link utilization through the routing decision model, wherein the target routing configuration strategy is a routing configuration strategy with the minimum maximum link utilization; The target traffic matrix is ​​routed based on the target routing configuration policy.

2. The method according to claim 1, characterized in that The routing decision model at least includes an action network and an evaluation network, wherein the action network is used to determine the probability distribution of target state data on the routing strategy, and the target state data is used to characterize the link utilization of the target traffic matrix under the initial routing configuration strategy; the evaluation network is used to determine the expected cumulative reward obtained by routing the target traffic matrix according to the first routing configuration strategy, and the first routing configuration strategy is any one of the multiple routing configuration strategies; The evaluation network at least includes an input layer, a plurality of hidden layers and an output layer; The action network includes at least an input layer, multiple hidden layers and multiple branch output layers, wherein the multiple branch output layers are separated from each other, each branch output layer corresponds to a routing configuration strategy, and each branch output layer is used to output the intermediate routing nodes through which the traffic flowing from the first routing node to the second routing node flows under the corresponding routing configuration strategy.

3. The method according to claim 2, characterized in that The network parameters of the action network and the network parameters of the evaluation network are determined in the following manner: Acquiring initial network parameters of the action network and initial network parameters of the evaluation network; For each training, determine the state transition data of each historical traffic matrix in the historical traffic matrix set in multiple parallel network environments, and obtain multiple state transition data groups corresponding to each historical traffic matrix, wherein the multiple parallel network environments have the same network topology, and the multiple parallel network environments correspond to different routing configuration strategies respectively; The plurality of state transition data groups corresponding to each of the historical traffic matrices are stored in a target buffer to obtain a state transition data set; Based on the state transition data group in the state transition data set, the initial network parameters of the action network and the initial network parameters of the evaluation network are updated respectively to obtain the target network parameters of the action network and the target network parameters of the evaluation network.

4. The method according to claim 3, characterized in that Determine the state transition data of each historical traffic matrix in the historical traffic matrix set in multiple parallel network environments, and obtain multiple state transition data groups corresponding to each historical traffic matrix, including: Determine a maximum link utilization of a first historical traffic matrix under a previous routing configuration strategy in a first parallel network environment, and obtain first state transition data, wherein the first parallel network environment is any one of the multiple parallel network environments, and the first historical traffic matrix is ​​any one of the traffic matrices in the set of historical traffic matrices; Determine a current routing configuration strategy according to the first state transition data; Determine, in the first parallel network environment, a maximum link utilization of the first historical traffic matrix under the current routing configuration strategy, and obtain second state transition data; Calculate the reward value of the first historical traffic matrix under the current routing configuration strategy; A state transition data group corresponding to the first historical traffic matrix is ​​formed based on the first state transition data, the current routing configuration strategy, the reward value, and the second state transition data.

5. The method according to claim 3, characterized in that: Based on the state transition data group in the target buffer, the initial network parameters of the action network and the initial network parameters of the evaluation network are updated respectively to obtain the target network parameters of the action network and the target network parameters of the evaluation network, including: Calculating an advantage value corresponding to each state transition data group in the target buffer; Calculate the network parameters when the function value of the objective function corresponding to the action network is maximum based on the advantage value, and obtain the target network parameters of the action network; Based on the advantage value, the network parameters when the function value of the objective function corresponding to the evaluation network is minimized are calculated to obtain the target network parameters of the evaluation network.

6. The method according to claim 5, characterized in that Calculating the network parameters when the function value of the objective function corresponding to the action network is maximum based on the advantage value, and obtaining the target network parameters of the action network, including: The target network parameters of the action network are determined by the following function: in, is the objective function of the action network, θ is the target network parameter of the action network, and θ′ is the network parameter before updating; is the state transition data set, ρ is any state transition data set in the state transition data set, T is the number of training times of the action network, a i,j is the intermediate routing node that the traffic demand from routing node i to routing node j passes through; π θ (a i,j |s) is the intermediate routing node a under the target network parameter θ of the action network. i,j Routing configuration strategy in state s; π θ′ (a i,j |s) is the network parameter θ′ before the update, the intermediate routing node a i,j In state a i,j Routing configuration policy under t is the state corresponding to the tth step in each training; a t is the action taken in the tth step in each training; A θ′ (s t ,a t ) is the network parameter θ′ before the update, at s t In state a, take action t The corresponding advantage function; ∈ is a preset constant; β is a preset coefficient; L is the number of branch output layers; The entropy function corresponding to the branch output layer, is the target network parameter θ of the action network at state s t The probability that the traffic from routing node i to routing node j flows through the intermediate routing nodes is .

7. The method according to claim 5, characterized in that Calculating the network parameters when the function value of the objective function corresponding to the evaluation network is the minimum based on the advantage value, and obtaining the target network parameters of the evaluation network includes: The target network parameters of the evaluation network are determined by the following function: Wherein, L(φ) is the objective function corresponding to the evaluation network, and φ is the target network parameter of the evaluation network; is the state transition data set, ρ is any state transition data set in the state transition data set, and T is the number of training times of the action network; r t′ is the reward value corresponding to the t′th step in each training; γ t′-t is the discount factor of the t′th step relative to the tth step in each training; V φ (s t ) is state s t The corresponding state value function.

8. A traffic routing device in a segment routing network, characterized in that: include: A strategy acquisition module, used to acquire an initial traffic matrix corresponding to a first time period and an initial routing configuration strategy corresponding to the initial traffic matrix, wherein the initial traffic matrix is ​​a traffic matrix formed by traffic demands in a segment routing network during the first time period; A link calculation module, configured to calculate an initial link utilization rate of the target traffic matrix under the initial routing configuration strategy when a difference is detected between the target traffic matrix corresponding to a second time period and the initial traffic matrix, wherein the second time period is a next time period corresponding to the first time period; A routing decision module, used for inputting the initial link utilization into a routing decision model, and determining a target routing configuration strategy from a plurality of routing configuration strategies based on the initial link utilization through the routing decision model, wherein the target routing configuration strategy is a routing configuration strategy with the minimum maximum link utilization; A traffic routing module is used to route the target traffic matrix based on the target routing configuration strategy.

9. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the traffic routing method in the segmented routing network as described in any one of claims 1-7 is implemented.

10. A computer-readable storage medium, characterized in that: Computer program instructions are stored on a computer-readable storage medium, and when the computer program instructions are executed by a processor, a traffic routing method in a segmented routing network as described in any one of claims 1 to 7 is implemented.

11. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the traffic routing method in a segmented routing network as described in any one of claims 1-7.

Images