Key flow routing optimization method based on deep reinforcement learning

Abstract

The invention provides a key flow routing optimization method based on deep reinforcement learning, and belongs to the field of flow engineering. Under the constraint of routing overhead, a part of the total flow is selected as a key flow and rerouting is carried out, so that the purposes of minimizing the maximum link utilization rate and reducing the time delay are achieved. And splicing the traffic matrix, the topological matrix and the feature matrix as input, and carrying out feature extraction with higher dimension through a graph neural network containing a K-hop peripheral sub-graph. For the discrete action space domain and maximum entropy problem, the optimal selection of the key flow is obtained by using a discrete soft behavior person-reviewer, and finally, the optimal routing scheme is obtained by solving through linear programming constraint. Simulation results show that compared with a traditional algorithm and other reinforcement learning algorithms, the provided algorithm has better time delay performance and ability to adapt to dynamic network topology.

Description

Technical Field

[0001] The present invention belongs to the field of Traffic Engineering (TE), and relates to a key flow routing optimization method based on Deep Reinforcement Learning (DRL), and more specifically, to a key flow routing optimization method based on discretized soft actor-critic (SoftActor Critic, SAC) and graph neural network (Graph Neural Network, GNN). Background Art

[0002] In recent years, the rapid development of emerging technologies such as autonomous driving, drones, and Industry 4.0 in various service applications has made communication networks increasingly complex. As users' demand for high-quality services across the entire Internet continues to escalate, traditional routing protocols have proven to be insufficient to meet these evolving needs. Therefore, new traffic engineering methods are urgently needed to provide enhanced performance, robustness, and reliability to meet these challenges. In addition, machine learning has been widely used in intelligent routing, extending network optimization from traditional analysis protocols to self-driving solutions. The current mainstream routing method based on deep reinforcement learning (DRL) optimizes network performance by rerouting all traffic, but ignores the network interference caused by frequent rerouting of all traffic and the resources it consumes. Therefore, a new method is needed to reduce network resource overhead while ensuring network performance optimization.

[0003] The routing optimization method based on the key flow idea requires a deep reinforcement learning algorithm suitable for discrete action space. The proximal policy optimization has the limitations of low sampling rate and tendency to converge to the local optimum in high-dimensional space; the deep deterministic policy gradient only focuses on the optimal behavior in each state without considering the exploration ability, which leads to convergence difficulties and instability during training. Therefore, the selection of deep reinforcement learning algorithms for the key flow idea is particularly important.

[0004] Given the limitations of DRL that only relies on traditional neural networks and its inadequacies in managing a variety of network topologies, the application of GNN in DRL in various fields has given it the potential value to be further explored. By utilizing the graph features extracted from GNN, integrating GNN with DRL can enhance the generalization ability in transfer learning and further improve the ability to express the overall network topology.

[0005] The present invention proposes a critical flow routing optimization framework based on discretized SAC and k-hop message passing graph neural network. By determining the traffic that has the greatest impact on the overall network performance, i.e., the critical flow, and rerouting these traffic, the network resource overhead is reduced while ensuring network performance optimization. Solving the critical flow selection problem is a problem in the discrete action space domain. The discretized SAC method is introduced to balance its exploration and utilization. Considering the adaptability of routing optimization to dynamic topology, the k-hop message propagation graph neural network is added to the deep reinforcement learning method. Through its internal graph attention mechanism and peripheral subgraph enhancement, the algorithm's latency performance and robustness are further improved. Summary of the invention

[0006] The purpose of the present invention is to provide a key flow routing optimization method based on deep reinforcement learning to solve the above-mentioned existing routing network performance requirement problems, routing optimization overhead problems and robustness problems to dynamic networks.

[0007] In order to achieve the above object, the present invention provides the following scheme:

[0008] The present invention provides a key flow routing optimization method based on deep reinforcement learning, comprising the following steps:

[0009] S1: The topology matrix TM at time t is periodically obtained by the Software Defined Network (SDN) controller t , Traffic MatrixTM l And the link characteristic matrix TM f , and aggregate them into a joint matrix;

[0010] S2: The joint matrix obtained in S1 is used as the input of the graph neural network module, and the graph attention mechanism and peripheral subgraphs are used to enhance the expression of the joint matrix to obtain a high-dimensional information matrix H;

[0011] S3: Use the information matrix H generated in S2 as the input of the deep reinforcement learning agent, and establish the action space and reward function of deep reinforcement learning according to the discrete action space domain and performance indicators;

[0012] S4: Use the discretized soft actor-critic algorithm to obtain the optimal selection of key flows;

[0013] S5: Rerouting is performed according to linear programming based on the selected key flow, while non-key flows are still routed according to the equal-cost multi-path method.

[0014] Furthermore, in S1, the SDN controller periodically obtains the topology matrix TM t , Traffic MatrixTM l And the link characteristic matrix TMf Specifically:

[0015] First, define the topology graph in the communication network as G(V,E), where V represents the set of communication nodes in the graph, E represents the edge set of links connecting these nodes, and each edge e(u,v)∈E has a certain bandwidth capacity, which represents the maximum flow that can pass within a specified time period; define (src,dst,cap) as the representation of the data flow, where src, dst and cap correspond to the source node, destination node and bandwidth capacity of the data flow respectively;

[0016] The SDN controller can obtain the topology matrix TM from the switch in the medium term by setting a timed task internally. t , Traffic MatrixTM l , and the link characteristic matrix TM f It is necessary to select specific data, whose first three features are set as link capacity, link load and link weight, and the remaining features are filled with 0 as hidden features, making its size N×N×N; and the topological matrix TM of size N×N×1 t and Traffic MatrixTM l Merge, and finally become a joint matrix of size N×N×(N+2);

[0017] Furthermore, in S2, the high-dimensional information matrix H is calculated by the graph neural network as follows:

[0018] Multi-hop message passing and multi-hop adjacent subgraph information integration are used to enhance node features. in represents the information of node v aggregating k-hop neighbors at the l-th layer of the graph neural network; the function MES() is used to aggregate information from the node itself and its neighbors; and Among them, EMB() represents a learnable embedding function; represents the set of k-hop neighbors of node v when a specific kernel t is selected in a given graph G; C represents the configuration, which has an effect on the k-hop subgraph The number of peripheral edges and node configurations of all nodes in are encoded, which can be viewed as running another 1-layer k-hop message passing graph neural network and readout function on each peripheral subgraph.

[0019] After completing the message delivery, through and Update node features, where represents the information transmitted by node v in the k-hop nodes of layer l; It refers to the node representation information of node v in the lth layer; COMBINE() is a function used to combine k-hop messages.

[0020] Again, through Introduce the graph attention mechanism and calculate the dynamic attention coefficient α vu , where W is the parameter matrix, a is the parameter vector, and LeakyReLU() is the activation function. and Represent the feature information vectors of the target central node v and the adjacent node u respectively. Specifically, firstly, W is used to transform the input feature and Concatenate, T and || represent transposition and concatenation operations respectively. Then use the LeakyReLU() activation function and its parameter a for nonlinear activation to get the attention score, and finally normalize it to get the dynamic attention coefficient.

[0021] Finally, based on α vu Update the node features, and the update formula is: in Represents the output features of the graph attention layer GAL about node v; represents the adjacent node of node i; σ represents the activation function.

[0022] Furthermore, in S3, the deep reinforcement learning problem in the discrete action space domain is transformed into the key flow identification problem as follows:

[0023] State space: Initially composed of the topological matrix TM t , Traffic MatrixTM l And the link characteristic matrix TM f , the converged joint matrix is ​​input into the k-hop message passing graph neural network module to obtain the state space of the deep reinforcement learning agent, that is, the high-dimensional information matrix H;

[0024] Action space: We select fixed k key flows (i.e., 10% of the total flow) according to different networks. For a network with N nodes, we assume that the agent can perform multiple actions at each time step, thus reducing the complex action space from (N×(N-1)) k Reduced to Therefore, the action space is set to {0, 1, ..., N × (N-1)}, and the agent is allowed to sample K different actions in each time step t (i.e. );

[0025] Reward function: In order to minimize the end-to-end (E2E) delay of the entire network, the reward function is defined as Among them l i,jIndicates link<i,j> ∈ Link load on E; c i,j Indicates link<i,j> ∈E; the reward r is defined as P D , which is set to reflect the network performance after rerouting the key flows to reduce the overall E2E delay, P D A larger value indicates better overall latency performance.

[0026] Furthermore, in S4, the discretized soft actor-critic algorithm is used to obtain the optimal selection of key flows. The specific method is as follows:

[0027] The discretized soft actor-critic algorithm adds an entropy term H(π(·|s)) on top of the maximum entropy reinforcement learning framework as a regularization term for the policy gradient objective: in Represents all possible states s t and a t The expectation under t ,a t ) means in state s t Next, perform action a t Immediate rewards after π represents strategy; * represents the optimal strategy; α is the temperature parameter, which is used to balance the relative importance of the entropy term and the reward r. It represents the "uncertainty" of action selection. The larger the entropy, the more random the policy's action selection is, which helps promote exploration in the early stages of training and avoid falling into local optima.

[0028] For the discrete action space domain, the soft actor-critic algorithm based on maximum entropy reinforcement learning is selected to establish two value function networks and In order to stabilize the learning process, two target value function networks are created and Create a policy network π to generate the agent's behavior strategy and initialize the value function network parameters θ 1 and θ 2 , and the policy network parameters φ, copy the parameters of the value function network to the corresponding target value function network, that is, θ' 1 ←θ 1 and θ' 2 ←θ 2 , initialize the experience replay library Used to store experience data of the interaction between the agent and the environment;

[0029] At the beginning of each episode, the starting state s is received from the environment. 0 , for each time step t, the agent selects action a according to the current policy network πt =π(s t ), the environment executes the action and obtains an immediate reward r t And return the next state s t+1 , the quaternion (s t ,a t ,r t ,s t+1 ) stored in the experience playback library , as the training data set;

[0030] From the Experience Replay Library A mini-batch of size N is randomly sampled from the t ,a t ,r t ,s t+1 ), using the target value function network and Calculate the next state s respectively t+1 The Q value is and To avoid the underestimation bias that is common in discrete soft actor-critic algorithms, we take the average of the two As the target Q value, and through the formula Update the value function, where represents an estimate of the target reviewer network, in this way, the underestimation bias of the lower bound of the two reviewers can be mitigated;

[0031] Using the mean square error loss function J Q (θ), update the value function network Q by gradient descent 1 and Q 2 Parameters to make it close to the target value y, according to the formula Perform gradient update, where i = {1, 2}, λ Q represents the learning rate, Indicates the introduction of a clipping operator to avoid drastic updates of the Q network, where c represents a hyperparameter of the clipping range. The clipping operator prevents the Q network from performing reward updates beyond the clipping range, thereby enhancing its robustness.

[0032] Every several time steps, the policy network π is used to generate a new behavior strategy, and the target value function network is used to calculate the estimated value of the policy gradient and update the policy network parameters. The policy network is calculated by the formula Update, and finally update the target value function network Q' according to the maximum entropy update method 1 and Q' 2 The parameters, that is Among them, τ is the soft update coefficient. As the network is continuously updated, the policy network and the value function network gradually converge to the optimal strategy. When the policy network converges, the agent can effectively select the optimal key flow.

[0033] Furthermore, in S5, the selected key flow is rerouted according to the linear programming, and the non-key flow is still routed according to the equal-cost multi-path method. The specific method is as follows:

[0034] The goal is to minimize the network E2E delay P D , set seven constraints to solve, namely constraint (a) Indicates the link through which the transmission reaches the destination d<i,j> The total flow on the link is equal to the total flow on the link Mi,j>, where i,j:<i,j> ∈E, constraint (b)l i,j ≤c i,j U: indicates a link<i,j> The load of the link<i,j> The maximum capacity of bandwidth, where i,j:<i,j> ∈E, constraint (c) The critical flow demand from node i to node d is equal to the sum of the critical flows from node i to node d minus the sum of the critical flows from other nodes to node i, where Constraint (d) Indicates traffic balancing based on equal-cost multipaths, where k:<i,k> ∈E, constraint (e) Similar to constraint (c), it is aimed at the sum of all flows, where d∈V, constraint (f) It is restricted that all traffic sizes cannot be negative, where d∈V,i,j:<i,j> ∈E, constraint (g) Indicates link<i,k> The above accounts for the key flow split ratio, where j:<i,j> ∈E, constraints (a), (c) and (e) are used to maintain the total number of non-critical flows and critical flows, constraints (b) and (f) ensure that the load of each link remains within a controllable range, constraint (d) indicates the balanced distribution of traffic according to the equal-cost multi-path method, and constraint (g) is used to determine the proportion of critical flows allocated to each path. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A flowchart of a key flow routing optimization method based on deep reinforcement learning provided by an embodiment of the present invention;

[0036] Figure 2 A framework diagram of a key flow routing optimization method based on deep reinforcement learning provided by an embodiment of the present invention;

[0037] Figure 3 A simulation result diagram of delay performance under the key flow routing optimization method based on deep reinforcement learning provided by an embodiment of the present invention;

[0038] Figure 4 A diagram of simulation results of dynamic network robustness under the key flow routing optimization method based on deep reinforcement learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] The core of the present invention is to provide a key flow routing optimization method based on deep reinforcement learning, which can optimize network delay and improve the robustness to dynamic networks by selecting and rerouting key flows.

[0041] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0042] Figure 1 A flowchart of a key flow routing optimization method based on deep reinforcement learning provided by an embodiment of the present invention is as follows: Figure 1 As shown, the following steps are included:

[0043] S1: The SDN controller periodically obtains the topology matrix TM at time t t , Traffic MatrixTM l And the link characteristic matrix TM f , and aggregate them into a joint matrix;

[0044] In the SDN controller, the topology matrix TM t and Traffic MatrixTM l can be collected directly from the switch periodically, and the link feature matrix TM f A special design is made to set the first three features as link capacity, link load and link weight, and fill the remaining features as hidden features with 0, so that its size is N×N×N, and the topological matrix TM of size N×N×1 t and Traffic MatrixTM l Merge, and finally become a joint matrix of size N×N×(N+2).

[0045] S2: The joint matrix obtained in S1 is used as the input of the graph neural network module, and the graph attention mechanism and peripheral subgraphs are used to enhance the expression of the joint matrix to obtain a high-dimensional information matrix H;

[0046] Multi-hop message passing and multi-hop adjacent subgraph information integration are used to enhance node features. in represents the message of node v aggregating k-hop neighbors at the l-th layer of the graph neural network. The function MES() is used to aggregate information from the node itself and its neighbors, while Among them, EMB() represents a learnable embedding function, represents the set of k-hop neighbors of node v when a specific kernel t is selected in a given graph G, and C represents the configuration, which has an impact on the k-hop subgraph The number of peripheral edges and node configurations of all nodes in are encoded, which can be viewed as running another 1-layer k-hop message passing graph neural network and readout function on each peripheral subgraph;

[0047] After completing the message delivery, through and Update node features, where represents the information transmitted by node v in the k-hop node of layer l, and Then it refers to the node representation information of node v in layer l, COMBINE() is the function used to combine k-hop messages;

[0048] Again, through Introduce the graph attention mechanism and calculate the dynamic attention coefficient α vu , where W is the parameter matrix, a is the parameter vector, and LeakyReLU() is the activation function. and Represent the feature information vectors of the target central node v and the adjacent node u respectively. Specifically, firstly, W is used to transform the input feature and Concatenate, T and || represent transposition and cascade operations respectively, then use the LeakyReLU() activation function and its parameter a for nonlinear activation to get the attention score, and finally normalize it to get the dynamic attention coefficient. Finally, based on α vu Update the node features, and the update formula is: in represents the output features of the graph attention layer GAL about node v, represents the adjacent node of node i, and σ represents the activation function.

[0049] S3: Use the information matrix H generated in S2 as the input of the deep reinforcement learning agent, and establish the action space and reward function of deep reinforcement learning according to the discrete action space domain and performance indicators;

[0050] State space: Initially composed of the topological matrix TM t , Traffic MatrixTM l And the link characteristic matrix TM f , the converged joint matrix is ​​input into the k-hop message passing graph neural network module to obtain the state space of the deep reinforcement learning agent, that is, the high-dimensional information matrix H;

[0051] Action space: We select fixed k key flows (i.e., 10% of the total flow) according to different networks. For a network with N nodes, we set the agent to be able to perform multiple actions at each time step, thus reducing the complex action space from (N×(N-1)) k Reduced to Therefore, the action space is set to {0, 1, ..., N × (N-1)}, and the agent is allowed to sample K different actions in each time step t (i.e. );

[0052] Reward function: In order to minimize the E2E delay of the entire network, the reward function is defined as Among them l i,j Indicates link<i,j> ∈E link load, c i,j Indicates link<i,j> ∈E, the link capacity, the reward r is defined as P D , which is set to reflect the network performance after rerouting the key flows to reduce the overall E2E delay, P D A larger value indicates better overall latency performance.

[0053] S4: Use the discretized soft actor-critic algorithm to obtain the optimal selection of key flows;

[0054] For the discrete action space domain, the soft actor-critic algorithm based on maximum entropy reinforcement learning is selected to establish two value function networks and In order to stabilize the learning process, two target value function networks are created and Create a policy network π to generate the agent's behavior strategy and initialize the value function network parameters θ 1 and θ 2 , and the policy network parameters φ, copy the parameters of the value function network to the corresponding target value function network, that is, θ' 1 ←θ 1 and θ' 2 ←θ 2, initialize the experience replay library Used to store experience data of the interaction between the agent and the environment;

[0055] At the beginning of each episode, the starting state s is received from the environment. 0 , for each time step t, the agent selects action a according to the current policy network π t =π(s t ), the environment executes the action and obtains an immediate reward r t And return the next state s t+1 , the quaternion (s t ,a t ,r t ,s t+1 ) stored in the experience playback library , as the training data set;

[0056] From the Experience Replay Library A mini-batch of size N is randomly sampled from the t ,a t ,r t ,s t+1 ), using the target value function network and Calculate the next state s respectively t+1 The Q value is and To avoid the underestimation bias problem that is common in discrete soft actor-critic algorithms, we take the average value avg(Q' 2 ,Q' 1 ) as the target Q value, and through the formula Update the value function;

[0057] Using the mean square error loss function J Q (θ), update the value function network Q by gradient descent 1 and Q 2 Parameters to make it close to the target value y, according to the formula Perform gradient update, where i = {1, 2}, λ Q represents the learning rate, It means that the pruning operator is introduced to avoid drastic updates of the Q network;

[0058] Every several time steps, the policy network π is used to generate a new behavior strategy, and the target value function network is used to calculate the estimated value of the policy gradient and update the policy network parameters. The policy network is calculated by the formula Update, and finally update the target value function network Q' according to the maximum entropy update method 1 and Q' 2parameters, namely Among them, τ is the soft update coefficient. As the network is continuously updated, the policy network and the value function network gradually converge to the optimal policy. After the policy network converges, the agent can effectively select the optimal critical flow.

[0059] S5: Re-route according to the selected critical flow using linear programming, while non-critical flows are still routed in the equivalent multi-path manner.

[0060] Define the objective as minimizing the network E2E delay P D , and set seven constraint conditions for solving, namely constraint condition (a) It means that the total traffic on the link <i,j> through which the traffic is transmitted to the destination d is equal to the total traffic on the link <i,j>, where i,j: <i,j> ∈ E. Constraint condition (b) l i,j ≤ c i,j ·U; It means that the load of the link <i,j> should be less than or equal to the maximum capacity of the bandwidth of the link <i,j>, where i,j: <i,j> ∈ E. Constraint condition (c) It means that the critical traffic demand from node i to node d is equal to the sum of the critical traffic passing through the links from node i to node d minus the sum of the critical traffic converging from other nodes to node i, where Constraint condition (d) It means that traffic is evenly distributed according to the equivalent multi-path, where k: <i,k> ∈ E. Constraint condition (e) Similar to constraint condition (c), it is for the total traffic sum relationship, where d ∈ V. Constraint condition (f) It restricts that the size of all traffic cannot be negative, where d ∈ V, i,j: <i,j< ∈ E. Constraint condition (g) It means the proportion of the critical flow occupied on the link <i,k<, where j: <i,j< ∈ E. Constraint conditions (a), (c) and (e) are used to maintain the total number of non-critical and critical flows. Constraint conditions (b) and (f) ensure that the load of each link remains within a controllable range. Constraint condition (d) indicates traffic balance distribution according to the equivalent multi-path method. Constraint condition (g) is used to determine the proportion of the critical flow allocated to each path.

[0061] Specifically, as Figure 3 describes the simulation results of the traffic routing delay performance obtained by different algorithms under a large-scale Tiscali network. From this cumulative distribution graph, it can be observed that the critical flow routing optimization method KCS based on deep reinforcement learning can obtain the best delay performance, with significant improvements compared to CFR-RL and Top-K Critical.

[0062] Specifically, Figure 4 The simulation results of the delay performance of different algorithms in the Ebone network when random sudden failures of network nodes are encountered are described. It can be observed from the figure that compared with other algorithms, the key flow routing optimization method KCS algorithm based on deep reinforcement learning has the best performance and is least affected in dynamic networks, and has high robustness.

[0063] The above is a detailed introduction to the key flow routing optimization method based on deep reinforcement learning provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other.

[0064] The contents not described in detail in the specification of the present invention belong to the prior art known to the professional and technical personnel in this field.

Claims

1. A key flow routing optimization method based on deep reinforcement learning, characterized in that: The following steps are involved: S1: The topology matrix TM at time t is periodically obtained by the Software Defined Network (SDN) controller t , Traffic MatrixTM l And the link characteristic matrix TM f , and aggregate them into a joint matrix; S2: The joint matrix obtained in S1 is used as the input of the graph neural network module, and the graph attention mechanism and peripheral subgraphs are used to enhance the expression of the joint matrix to obtain a high-dimensional information matrix H; S3: Use the information matrix H generated in S2 as the input of the deep reinforcement learning agent, and establish the action space and reward function of deep reinforcement learning according to the discrete action space domain and performance indicators; S4: Use the discretized soft actor-critic algorithm to obtain the optimal selection of key flows; S5: Rerouting is performed according to linear programming based on the selected key flow, while non-key flows are still routed according to the equal-cost multi-path method.

2. The key flow routing optimization method based on deep reinforcement learning according to claim 1 is characterized in that: In step S1: Link Characteristic MatrixTM f The first three features of are set as link capacity, link load and link weight, and the remaining features are filled with 0 as hidden features to make its size N×N×N, and the topological matrix TM of size N×N×1 is t and Traffic MatrixTM l Merge, and finally become a joint matrix of size N×N×(N+2).

3. The key flow routing optimization method based on deep reinforcement learning according to claim 1 is characterized in that: In S2, the high-dimensional information matrix H is generated by the graph neural network module as follows: Multi-hop message passing and multi-hop adjacent subgraph information integration are used to enhance node features. in represents the message of node v aggregating k-hop neighbors at the l-th layer of the graph neural network. The function MES() is used to aggregate information from the node itself and its neighbors, while Among them, EMB() represents a learnable embedding function, represents the set of k-hop neighbors of node v when a specific kernel t is selected in a given graph G, and C represents the configuration, which has an impact on the k-hop subgraph The number of peripheral edges and node configurations of all nodes in are encoded, which can be viewed as running another 1-layer k-hop message passing graph neural network and readout function on each peripheral subgraph; After completing the message delivery, through and Update node features, where represents the information transmitted by node v in the k-hop node of layer l, and Then it refers to the node representation information of node v in layer l, COMBINE() is the function used to combine k-hop messages; Again, through Introduce the graph attention mechanism and calculate the dynamic attention coefficient α vu , where W is the parameter matrix, a is the parameter vector, and LeakyReLU() is the activation function. and Represent the feature information vectors of the target central node v and the adjacent node u respectively. Specifically, firstly, W is used to transform the input feature and Concatenate, T and || represent transposition and cascade operations respectively, then use the LeakyReLU() activation function and its parameter a for nonlinear activation to get the attention score, and finally normalize it to get the dynamic attention coefficient. Finally, based on α vu Update the node features, and the update formula is: in represents the output features of the graph attention layer GAL about node v, represents the adjacent node of node i, and σ represents the activation function.

4. The key flow routing optimization method based on deep reinforcement learning according to claim 1 is characterized in that: In S3, the deep reinforcement learning problem in the discrete action space domain is transformed into a key flow identification problem: State space: Initially composed of the topological matrix TM t , Traffic MatrixTM l And the link characteristic matrix TM f , the converged joint matrix is ​​input into the k-hop message passing graph neural network module to obtain the state space of the deep reinforcement learning agent, that is, the high-dimensional information matrix H; Action space: We select fixed k key flows (i.e., 10% of the total flow) according to different networks. For a network with N nodes, we set the agent to be able to perform multiple actions at each time step, thereby reducing the complex action space from (N×(N-1)) k Reduced to Therefore, the action space is set to {0, 1, ..., N × (N-1)}, and the agent is allowed to sample K different actions in each time step t (i.e. ); Reward function: In order to minimize the end-to-end (E2E) delay of the entire network, the reward function is defined as Among them l i,j Indicates link<i,j> ∈E link load, c i,j Indicates link<i,j> ∈E, the link capacity, the reward r is defined as P D , which is set to reflect the network performance after rerouting the key flows to reduce the overall E2E delay, P D A larger value indicates better overall latency performance.

5. The key flow routing optimization method based on deep reinforcement learning according to claim 1 is characterized in that: In S4, the discretized soft actor-critic algorithm is used to obtain the optimal selection of key flows. The specific method is as follows: For the discrete action space domain, the soft actor-critic algorithm based on maximum entropy reinforcement learning is selected to establish two value function networks and In order to stabilize the learning process, two target value function networks are created and Create a policy network π to generate the agent's behavior strategy, initialize the value function network parameters θ1 and θ2, and the policy network parameters φ, copy the value function network parameters to the corresponding target value function network, i.e. θ'1←θ1 and θ'2←θ2, and initialize the experience replay library Used to store experience data of the interaction between the agent and the environment; At the beginning of each episode, the agent receives the starting state s0 from the environment. For each time step t, the agent selects an action a according to the current policy network π. t =π(s t ), the environment executes the action and obtains an immediate reward r t And return the next state s t+1 , the quaternion (s t ,a t ,r t ,s t+1 ) stored in the experience playback library , as the training data set; From the Experience Replay Library A mini-batch of size N is randomly sampled from the t ,a t ,r t ,s t+1 ), using the target value function network and Calculate the next state s respectively t+1 The Q value is and In order to avoid the underestimation bias problem that is common in discrete soft actor-critic algorithms, the average value avg(Q'2,Q'1) is taken as the target Q value, and the formula is used Update the value function; Using the mean square error loss function J Q (θ), update the parameters of the value function network Q1 and Q2 by gradient descent method to make it close to the target value y, according to the formula Perform gradient update, where i = {1, 2}, λ Q represents the learning rate, It means that the pruning operator is introduced to avoid drastic updates of the Q network; Every several time steps, the policy network π is used to generate a new behavior strategy, and the target value function network is used to calculate the estimated value of the policy gradient and update the policy network parameters. The policy network is calculated by the formula Update, and finally update the parameters of the target value function network Q'1 and Q'2 according to the maximum entropy update method, that is, Among them, τ is the soft update coefficient. As the network is continuously updated, the policy network and the value function network gradually converge to the optimal strategy. When the policy network converges, the agent can effectively select the optimal key flow.

6. The key flow intelligent routing method of deep reinforcement learning according to claim 1 is characterized by: In S5, linear programming is used to reroute key flows. The specific method is as follows: The goal is to minimize the network E2E delay P D , set seven constraints to solve, namely constraint (a) Indicates the link through which the transmission reaches the destination d<i,j> The total traffic on the link is equal to<i,j> The total flow on , where i,j:<i,j> ∈E, constraint (b)l i,j ≤c i,j U: indicates a link<i,j> The load of the link<i,j> The maximum capacity of bandwidth, where i,j:<i,j> ∈E, constraint (c) The critical flow demand from node i to node d is equal to the sum of the critical flows from node i to node d minus the sum of the critical flows from other nodes to node i, where Constraint (d) Indicates traffic balancing based on equal-cost multipaths, where k:<i,k> ∈E, constraint (e) Similar to constraint (c), it is aimed at the sum of all flows, where d∈V, constraint (f) It is restricted that all traffic sizes cannot be negative, where d∈V,i,j:<i,j> ∈E, constraint (g) Indicates link<i,k> The above accounts for the key flow split ratio, where j:<i,j> ∈E, constraints (a), (c) and (e) are used to maintain the total number of non-critical flows and critical flows, constraints (b) and (f) ensure that the load of each link remains within a controllable range, constraint (d) indicates the balanced distribution of traffic according to the equal-cost multi-path method, and constraint (g) is used to determine the proportion of critical flows allocated to each path.

Images