A routing optimization method based on graph neural network and deep reinforcement learning

Abstract

The application discloses a routing optimization method based on a graph neural network and deep reinforcement learning, which is improved from two aspects of network topology perception and intelligent decision-making by combining GNN and DRL technologies, starting from the actual needs of modern communication networks. Under the condition of ensuring network stability and user experience, the algorithm has been significantly improved in network resource allocation, routing path calculation and network congestion control, and the communication network is promoted to develop in a more intelligent and greener direction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer network communication, and particularly relates to a routing optimization method based on a graph neural network (GNN) and deep reinforcement learning (DRL). BACKGROUND

[0002] In the information age, communication networks as the infrastructure of global information transmission, its importance is increasingly prominent. The rapid development of Internet, Internet of Things, mobile communication and cloud computing, etc. makes the scale of communication network expanding, and the network structure becoming more complex. Modern communication networks need more efficient, flexible and intelligent routing strategies to meet the growing network demand. However, the existing routing technology has deficiencies in fast and reliable transmission when facing the diversification of network structure and the change of traffic demand, and further research is needed to adapt to the needs of modern communication networks.

[0003] Among the existing communication network routing strategies:

[0004] Technical solution 1 (publication number CN115333947A) proposes a guided routing optimization method. This method obtains system requirements, divides network domains, sets nodes of network domains, generates an incomplete graph, calculates the weight between each node, and takes the shortest path as the optimized routing corresponding to the network domain. This method classifies nodes and generates an incomplete graph, which can make routing optimization more controllable.

[0005] Technical solution 2 (publication number CN114423061A) proposes a wireless routing optimization method based on attention mechanism and deep reinforcement learning. This method includes obtaining the latest decision model parameters from the server when each node enters the network, listening to neighbor node information and building a candidate parent node set, and selecting the energy maximum parent node information modeling as a graph vector as input, using CNN-based attention mechanism to extract graph vector features, and using deep reinforcement learning to select the optimal parent node as the relay node for data transmission.

[0006] Technical solution 1 optimizes routing by node classification and incomplete graph generation when designing routing solutions, but does not fully consider the dynamic allocation of network resources and the challenges brought by the dynamic changes of network devices. Technical solution 2 uses a routing optimization method based on attention mechanism and deep reinforcement learning, which has high scalability and can be applied to scenarios with dynamic changes in network nodes, but has not discussed in detail the volatility of network traffic and how to achieve fast and efficient routing optimization for large-scale networks. SUMMARY

[0007] The application proposes a routing optimization method based on a graph neural network and deep reinforcement learning to solve the problems of dynamic resource allocation, dynamic device change and traffic fluctuation in modern communication networks, aiming to find a high-reliability and high-efficiency routing path between the source node and the destination node of the communication network, so as to complete more service demands under the premise of limited network resources and improve the transmission efficiency and service quality of the network.

[0008] In order to achieve the above purpose, the application provides the following technical scheme:

[0009] A routing optimization method based on a graph neural network and deep reinforcement learning introduces a main network, an experience replay buffer and a target network in a DQN model, wherein the main network interacts with the environment to constantly generate training experience and receives the current network state s as input and outputs the estimated Q value of each possible action a; the experience replay buffer is used to update and store training experience in real time, and part of the samples are randomly extracted for training; the target network is used to periodically copy parameters from the main network, update, calculate the target Q value and enhance the convergence speed of the gradient training algorithm; the routing optimization method comprises the following steps:

[0010] S1, input the network state information of each link and all its neighbor links and the service traffic demand, integrate the network state and the service traffic demand by using the graph neural network, thereby generating a message integrated according to the node and presented in the structure of a graph;

[0011] S2, run an iterative message passing algorithm between the hidden states of the link nodes in the graph, and after multiple iterations, use an aggregation function to aggregate the updated node representation to obtain a global hidden state;

[0012] S3, based on the global hidden state of step S2, use the graph neural network to calculate the action estimated Q value Q(s,a) under the input state;

[0013] S4, select the optimal action according to the Q value of step S3, and then execute the routing decision action in the environment; if the action is successfully executed, i.e. the remaining bandwidth in each link of the selected path is sufficient to meet the traffic demand of the service, a normalized reward value is returned, and the network state s' is updated; if the action cannot be completed, the training is ended;

[0014] S5, after multiple training, the model parameters are constantly updated according to the network state s, the action a and the reward value r, and the training experience (s,a,r,s') generated by the main network is stored in the experience replay buffer;

[0015] S6, every M iterations, a small batch of samples are randomly extracted from the buffer and copied to the target network for calculating the target Q value.

[0016] S7, constantly updating the network parameters of the target network, and sending the updated model parameters to each node in the main network, when the main network receives the updated parameters, each node updates its routing planning model parameters; repeat step S7 until convergence.

[0017] Further, in step S1, the network state information includes link characteristics and network topology.

[0018] Further, in step S2, in each iteration step of the message passing algorithm, the node receives the current state from the adjacent node, and uses the update function to update its own hidden state.

[0019] Further, in step S2, the message passing algorithm network contains three layers, namely Message layer, Update layer and Readout layer, in Message layer and Update layer, a total of T times of information passing process needs to be performed, the specific process is as follows:

[0020] For a specific node v, first perform formula (1) to make node v receive messages from the state of node v itself and the state of all surrounding nodes, and aggregate messages by element-wise summation;

[0021]

[0022] In formula (1), m v t+1 is the message received by node v at t+1 time step, N(v) represents the neighbor nodes of v, h v t is the feature vector of node v at t time step, m(·) is the message function, w is the target node, and W is an adjacency matrix containing all neighbor nodes of the target node;

[0023] Then send the aggregated message to a neural network implemented by a gated recurrent unit, and continuously update the hidden state of the link by recursive iteration, calculate the new hidden state of the node as shown in formula (2):

[0024]

[0025] U t is the update function, which takes the original node state h v t and information m v t+1 as input, and gets the new node state h v t+1 ;

[0026] After T iterations, the final hidden state of each link is aggregated again to enter the Readout layer, and the node state based on the global hidden state graph is calculated using the readout function R, as shown in equation (3):

[0027]

[0028] where G is the network topology.

[0029] Further, in step S4, the calculation of the reward value r is shown in equation (4):

[0030] r = bw / max(bw) (4)

[0031] where bw is the traffic demand.

[0032] Further, in step S3, the estimated Q value returned after following the strategy π to perform action a at state s is calculated as shown in equation (5):

[0033]

[0034] where the strategy function π(a|s): S × A→ [0,1] is the conditional probability distribution when selecting action a at state s, and the goal of the strategy at time τ is to maximize the accumulated reward r in the historical experience τ , γ ∈ (0,1) is the discount factor that weighs historical and current reward data, k is a certain time interval, and E π is the expected value when using the π strategy.

[0035] Further, in step S6, the target Q value is the sum of the reward value r at the current time step and the value after decay of the maximum estimated Q value at the next state, and the calculation method is shown in equation (6):

[0036] Q target (s,a,w) = r + γmax a' Q(s',a',w) (6)

[0037] where w represents the neural network parameters, γ ∈ (0,1) is the discount factor that weighs historical and current reward data, s' is the next state, a' is the action selected at the next state, and w is the target node.

[0038] Further, the loss value is calculated using the mean square error between the estimated Q value and the target Q value, which is used to calibrate the neural network parameters, as shown in equation (7):

[0039] L(w) = E[(Q target (s,a,w) - Q(s,a,w)) 2 ] (7)

[0040] wherein E is the expected value.

[0041] Further, the model is continuously trained using the gradient descent method, so that the model loss value is continuously reduced, and the gradient calculation method is shown in formula (8):

[0042]

[0043] wherein a k is the selected action at time interval k.

[0044] Further, the network parameters of the target network are continuously updated using formula (9):

[0045]

[0046] wherein alpha is the learning rate, and v k is calculated from the discounted return value, as a weighting factor for the gradient update value.

[0047] Compared with the prior art, the present application has the following beneficial effects:

[0048] The routing optimization method based on the graph neural network and the deep reinforcement learning provided by the present application focuses on the communication network routing optimization problem, and the core is to apply the graph neural network (GNN) and the deep reinforcement learning (DRL) technology to cope with the challenges brought by the expansion of network scale and the complication of network structure. Under the premise of ensuring the information transmission efficiency and the network stability, the present application aims to realize the dynamic routing optimization of the communication network.

[0049] Firstly, the present application designs a network topology dynamic perception algorithm based on the graph neural network. The algorithm takes the dynamic change of the communication network as the demand constraint, learns the features of the nodes and edges in the network topology by using the GNN, and realizes the real-time perception of the network state. Through the dynamic updating capability of the GNN, the algorithm can capture the continuous change of the network topology structure, and provide accurate network representation for the routing decision.

[0050] Secondly, the present application proposes an intelligent routing decision method driven by the deep reinforcement learning. The method uses the DRL technology to learn the mapping relationship between the network state and the action, realizes the dynamic allocation of network resources and the intelligent adjustment of routing path. The autonomous learning and decision-making capability of the DRL enables the algorithm to adapt to the volatility of network traffic and the dynamic change of network equipment, and optimizes the use of network resources.

[0051] Further, the application combines GNN with DRL to form a brand-new communication network routing optimization algorithm. The algorithm makes full use of the feature information extracted by GNN to guide the DRL model to make more accurate routing decisions. Through this combination, the algorithm can achieve fast and efficient routing optimization for large-scale networks, improving the transmission efficiency and service quality of the network.

[0052] Finally, the application improves routing optimization from the actual needs of modern communication networks, combining GNN and DRL technology from network topology perception and intelligent decision-making. Under the condition of ensuring network stability and user experience, the algorithm has been significantly improved in network resource allocation, routing path calculation, and network congestion control, promoting the development of communication networks towards more intelligent and greener directions. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0054] Figure 1 The model architecture of the routing optimization method based on graph neural network and deep reinforcement learning provided by the embodiments of the present application.

[0055] Figure 2 The node state update process taking node 5 as an example provided by the embodiments of the present application.

[0056] Figure 3 The process of message passing using the MPNN structure used by the GNN-DRL model provided by the embodiments of the present application.

[0057] Figure 4 The implementation process of the DQN algorithm in the GNN-DRL model provided by the embodiments of the present application.

[0058] Figure 5 The four typical topologies in the SNDLib dataset used by the GNN-DRL model provided by the embodiments of the present application. Figure 5 In the figure, a is Geant2, b is Small Top, c is Gbn, and d is Nsfnet.

[0059] Figure 6 The average loss value change trend in the training process of the optimal routing planning model of the four topologies provided by the embodiments of the present application. Figure 6 In the figure, a is Geant2, b is Small Top, c is Gbn, and d is Nsfnet.

[0060] Figure 7 The average score of each model in the training process of the four topologies provided by the embodiment of the application. Figure 7 In the table, a is Geant2, b is Small Top, c is Gbn, and d is Nsfnet.

[0061] Figure 8 The change trend of the Epsilon value of the four topologies provided by the embodiment of the application. Figure 8 In the table, a is Geant2, b is Small Top, c is Gbn, and d is Nsfnet.

[0062] Figure 9 The routing planning result of the GNN-DRL model provided by the embodiment of the application on the Geant2 topology structure.

[0063] Figure 10 The performance comparison result of the three methods provided by the embodiment of the application on the four different topologies. Figure 10 In the table, a is Geant2, b is Small Top, c is Gbn, and d is Nsfnet. DETAILED DESCRIPTION

[0064] The application is based on deep reinforcement learning algorithm (DRL) and graph neural network (GNN), and proposes a new network routing optimization method for modern communication network routing optimization problems. The method fully considers the complex relationship between dynamic network topology, routing configuration and traffic information, so as to realize efficient allocation of network resources.

[0065] Through simulation experiments, the effectiveness and applicability of the proposed model in different network scenarios are verified, and under the condition of limited network resources, the advantages of resource allocation can be played to complete more business demands.

[0066] In order to better understand the technical solution, the method of the application will be described in detail below with reference to the accompanying drawings.

[0067] The application proposes a routing optimization method based on graph neural network and deep reinforcement learning (hereinafter referred to as GNN-DRL model), and the model architecture is as shown in Figure 1 The model takes Deep Q-Network (DQN) algorithm as the main implementation framework of DRL, and vectorizes the attributes of nodes and edges in the network topology through GNN. GNN uses the iterative message passing algorithm to update the vectorized representation of nodes and edges in real time by using the dependency relationship of the network topology, so that the algorithm can effectively adapt to different network topologies and structural changes, provide optimized routing schemes for various network traffic demands, and maximize the utilization of routing resources.

[0068] As Figure 1 shown, the method of the present application mainly consists of the following two operation processes:

[0069] First, in the execution process of the present application, the network state information (s) such as link characteristics, network topology, and traffic demand are received as the input of the model. Then, the GNN model can integrate these information and present them in the structure of a graph. By running an iterative message passing algorithm between the hidden states of the links, the global hidden state can be aggregated after multiple iterations. Finally, the GNN can calculate the Q value Q(s,a) corresponding to a specific set of routing actions that need to be evaluated, i.e. the model predicts the next operation.

[0070] Second, the optimal action (a) is selected according to the Q value, and then the routing decision action is tried to be executed in the environment. If the action is successfully executed, i.e. the remaining bandwidth of each link in the selected path is sufficient to meet the traffic demand of this service, a reward value r calculated by reasonable normalization is returned, and the network state (s') is updated. If the action cannot be completed, the training is ended, so no reward is given for this operation. In the process of multiple training, the model parameters are constantly updated according to the state, action and reward value, so as to finally learn the optimal model with the highest score. This model can maximize the completion of more business demand under the condition of limited network resources.

[0071] The goal of the present application is to select the best routing configuration and optimize network resource allocation by taking actions that can maximize the reward value according to the current dynamically changing network state information. In order to enhance the effect of model training faster and more stably, the present application introduces a main network, an experience replay buffer and a target network in the DQN framework. The main network constantly generates training experience (s,a,r,s') by interacting with the environment. It can receive the current network state as input and output the estimated Q value of each possible action. The experience replay buffer not only updates and stores the training samples in real time, but also randomly extracts some samples for training, reduces the correlation between samples, and improves the learning efficiency of the model. The target network has the same architecture as the main network, but it is relatively stable because it only copies and updates the parameters from the main network periodically, which can provide a relatively stable target Q value and further enhance the convergence speed of the gradient training algorithm. Through this method combining GNN and DRL, the method of the present application can achieve efficient routing optimization in a complex network environment, adapt to the dynamic changes of the network, and improve the network transmission efficiency and service quality.

[0072] The present application uses a message passing network (MPNN) commonly used in GNN to capture the relationship and local structure between nodes in the topology graph. In a specific embodiment, the MPNN needs to perform T = 8 message iterations in total to be able to integrate enough topology graph link information for path planning. In each iteration step, the node receives the current state from the adjacent nodes and updates its own hidden state using the update function. In this way, each node will contain state information from surrounding nodes. Then, the aggregation function aggregates the updated node representation to produce a new global hidden state representation.

[0073] Specifically, the MPNN used in the present application includes three layers, namely the "Message"

[0074] layer, the "Update" layer and the "Readout" layer.

[0075] In the "Message" layer and the "Update" layer, a total of T message passing processes need to be performed. For a particular node v, formula (1) is first executed so that node v receives messages from the state of node v itself and the state of all surrounding nodes, and the messages are integrated by summation.

[0076]

[0077] In formula (1), m v t+1 is the message received by node v at time step t+1. N(v) represents the neighbor nodes of v, h v t is the feature vector of node v at time step t. m(·) is the message function.

[0078] Then the new hidden state of the node is calculated using formula (2).

[0079]

[0080] U t is the update function, which takes the original node state h v t and the information m v t+1 as input to obtain the new node state h v t+1 . Figure 2 The node state update process is shown taking node 5 as an example.

[0081] After T iterations, the node collects enough state information in a large range, and enters the "Readout" layer. The readout function R is used to calculate the node state based on the whole graph, as shown in formula (3).

[0082]

[0083] where h v T is the final hidden state of node v after T iterations of the graph neural network, and G is the network topology.

[0084] Figure 3 The flow of message passing using the MPNN structure used in the present application is shown. First, the state of each link and all its neighbor links is input into the model. Then, these hidden states are combined together using a fully connected neural network, generating a message that integrates information by node. Then, each node and its neighbor node messages are aggregated (Aggr) using an element-wise summation. These new messages are sent to a neural network implemented by a gated recurrent unit (GRU), which constantly updates the hidden state of the link using recursive iterations. In this process, the range of collected link information is gradually expanded around each node as the center. After T iterations, the final hidden state of each link is obtained by aggregation again. Finally, the readout is performed, and the action prediction Q value under the input state is calculated again using a fully connected neural network.

[0085] In the GNN-DRL model designed in the present application, the DQN algorithm commonly used in DRL is used to realize routing optimization. This enables the model to interact with the real-time state of the environment to update the neural network parameters. And according to the Q value calculated by the GNN, the action is selected and executed, so as to maximize the reward value and realize the planning of the optimal route with the maximum overall network resource utilization.

[0086] The implementation flow of the DQN algorithm in the GNN-DRL model is shown in Figure 4 The routing selection process of DQN includes three parts: environment, model, and reward. First, the topology environment is initialized, that is, the link capacity is set to the maximum value and there is no occupied traffic at present, and the link betweenness is calculated as a link feature and saved in the network. Then, a group of source nodes (src) and destination nodes (dst) are randomly extracted from the topology as the service of this path planning, and a tuple {src, dst, bw} is formed with the randomly injected traffic demand (bw). At this time, no service demand is completed, so the initial reward value is 0.

[0087] The application realizes real-time decision-making combined with the current state (s) of the node through interaction with the network simulation environment. During execution, first, the optional paths from the source node to the destination node are sorted by hop count from few to many according to the link relationship of the topology graph. In order to occupy less computing resources under the condition of finding the best resource allocation route, I only keep the k=4 shortest paths as the candidate paths for route planning. Then, the Q value of each path is fitted using a deep neural network, which represents the probability of the model selecting this path and helps to evaluate the pros and cons of selecting the path action. The set of these Q values is called the Q value of the corresponding action under the current state. After using the epsilon-greedy strategy to select the route action to be executed according to the Q value, the corresponding path in the environment can be found according to the action. The model tries to inject traffic demand in the path, and checks whether each link along the path can complete the service, that is, whether the remaining capacity of the link is greater than the traffic demand. If the path is feasible, the reward value is increased, and the environment state s' is updated. Then continue to extract other service tuples {src', dst', bw'} for route planning, until the selected path cannot meet the demand and stop this round of operation.

[0088] The reward function represents different immediate rewards brought by different routing choices. The goal of the model is to maximize the amount of completed demand and improve network resource utilization in the long run. The reward function r is determined as shown in equation (4), where bw is the traffic demand and maxbw is the set maximum traffic demand. This normalization method improves the problem of gradient explosion and improves learning ability:

[0089] r=bw / maxbw (4)

[0090] The policy function π(a|s): SxA→[0,1] is the conditional probability distribution of the model selecting action a under state s. The goal of the policy at time τ is to maximize the accumulated reward r τ in the history experience. γ∈(0,1) is the discount factor that weighs the historical and current reward data of the model. According to the Bellman equation, the GNN-DRL model returns the estimated Q value after the action a is performed under the state s following the policy π, and the calculation method is shown in equation (5):

[0091]

[0092] DQN solves the instability problem of function approximation by introducing experience replay and target network. These training experiences (s, a, r, s') generated by the main network are first stored in the experience replay buffer. Every M iterations, a small batch of samples is randomly drawn from the buffer and copied to the target network to calculate the target Q value. This strategy not only allows the target network to effectively sample from rich experiences, reducing the correlation between samples, but also eliminates the need for real-time interaction with the environment, enhancing the stability of calculating the target Q value and thus the learning ability of the model. The target Q value is the evaluation value of the network state after implementing the routing strategy, which is the sum of the reward value r at the current time step and the decayed maximum estimated Q value at the next state, as shown in equation (6):

[0093] Q target (s,a,w)=r+γmax a' Q(s',a',w) (6)

[0094] where w represents the neural network parameters. Then, the mean square error between the estimated Q value and the target Q value is used to calculate the loss value, which is used to calibrate the neural network parameters, as shown in equation (7).

[0095] L(w)=E[(Q target (s,a,w)-Q(s,a,w)) 2 ] (7)

[0096] Then, the gradient descent method is used to continuously train the model, so that the model loss value continuously decreases. The gradient calculation method is shown in equation (8):

[0097]

[0098] At the same time, equation (9) is used to continuously update the network parameters, so that the fitting effect of the Q value function becomes better and better.

[0099]

[0100] where α is the learning rate, which controls the speed of parameter update. vk is calculated from the discounted return value and serves as a weighting factor for the gradient update value. After updating the parameters of the target network at each stage, the updated model parameters are sent to each node in the main network. When the main network receives the updated parameters, each node updates its routing planning model parameters. Repeat this process until convergence, i.e., reach the predefined standard.

[0101] The computer hardware used in the embodiments of the application is Intel Xeon Gold 6226R CPU (2.90GHz), and the operating system version used is Ubuntu 18.04.6. At the same time, the embodiments realize the GNN-DRL model algorithm based on TensorFlow and OpenAIGym framework, and build a learning environment through Keras2.7.0+Tensorflow2.7.0 in pthon3.7.

[0102] The application uses four different topologies in SDNLib based on the GNN-DRL model for experiments, which are Geant2, Small Top, Gbn and Nsfnet. Figure 5 The structure of each topology and the connection relationship between nodes are clearly shown. Training on these different scale network topologies helps to evaluate the performance of the GNN-DRL model under various complexities and scales.

[0103] When resetting the environment, all link features will be initialized, including the following five, edgeId, betweenness, numsp, capacity, and bw_allocated. Before each resource allocation, it needs to be recalculated. Among them, edgeId represents the encoding of each edge, betweenness represents the link betweenness, that is, the proportion of the number of shortest paths passing through the link in all shortest paths, which is used to measure the importance of the link in the topology. numsp records the number of shortest paths that need to use the link. Capacity represents the total capacity of the link, and in the experiment, the maximum capacity of all links is set to 200Mbps. bw_allocated represents the bandwidth of the link that has been occupied by other services.

[0104] Three types of demand are defined in the form of custom arrays, which are 8, 32, and 64, i.e., listofDemand = [8, 32, 64]. Among them, demand can be regarded as the comprehensive demand for resources (such as bandwidth). listofDemand is a set of arrays containing all custom business demands, and max_demand is the maximum demand among the custom business demands. Each number in the array represents a different demand level, and the higher the value, the greater the demand for resources. If the state of the path selected by the current GNN-DRL model can meet the business demand, the action can obtain a reward value. In order to avoid the accumulation of reward value being too large, the normalization method is adopted, i.e., the reward value is calculated by the completed business demand in this time / the maximum demand set in the experiment. For example, the business demand is demand = 32, and max_demand = 64. If the residual capacity of each link of the path selected by the model is greater than 32, it is considered that the business can be completed. The action corresponding to the path obtains a reward value of 32 / 64 = 0.5 for this business.

[0105] In order to make the GNN-DRL model have strong learning and decision-making ability, appropriate model parameters are set, including the gradient optimization algorithm parameters of the graph neural network and the hyperparameter values of the deep reinforcement learning. In this model, a graph neural network is defined to perceive the network state and predict the Q value of the action. The hidden state of the link is set to contain link_state_dim = 20 dimensions, i.e., each link contains at most 20 feature vectors. The more complex the topology graph is, the more capacity is required to meet the requirement of storing enough encoding information. In each forward propagation process of the message, a batch of samples is used to perform T = 8 rounds of message passing. The sample capacity of each batch is set to batch_size = 32. The maximum output value can be readout_units = 35 units.

[0106] In the GNN-DRL model, the action can be selected according to the Q value output by the GNN. The epsilon-greedy strategy is used to help train the model that can select the global optimal solution. The initial value of epsilon is set to 1.0, epsilon_start_decay = 70 iterations, and the value of epsilon is reduced to epsilon_decay = 0.995 every iteration until it reaches the minimum value epsilon_min = 0.01. In order to stabilize the parameter learning process, a target network is introduced to calculate the target Q value. Therefore, an experience replay buffer with a capacity of MAX_QUEUE_SIZE = 4000 is set to temporarily store samples, and once it is full, the old experience will be deleted. Every time, MULTI_FACTOR_BATCH = 6 batches of samples (each batch has batch_size = 32 samples) are randomly extracted from the experience replay buffer and put into the target network. By using the gradient descent method, the loss value can be used to strengthen the model fitting. Therefore, every store_loss = 3 batches, the current loss value and gradient value are calculated and stored. Every copy_weights_interval = 50 iterations, the target network parameters are updated once. Among them, the learning rate is set to learning_rate = 0.0001 to control the parameter update speed of the model, that is, the parameter update amount is gradient value * 0.0001. At the same time, samples have a dropout_rate = 0.01 probability of being discarded to prevent the model from overfitting.

[0107] During the process of ITERATIONS = 10000 model iterations of the GNN-DRL model, the first iteration inputs FIRST_WORK_TRAIN_EPISODE = 60 training sets, and each subsequent iteration inputs TRAINING_EPISODES = 20 training sets. Because there is no training experience in the model at the first iteration, more samples are needed to help update the model parameters. During the training process, the environment state needs to be reset using reset() and a new set of traffic demand, source node and target node needs to be extracted at the beginning of each training iteration. In the current state, the model first selects k = 4 shortest paths as candidate paths. Then the GNN-DRL model interacts with the environment by calling agent.act() to select the optimal action. According to the selected action, the make_step() method is used to attempt to execute in the environment, that is, to check whether the residual capacity of all links of the path is greater than the traffic demand of this time. If it can be successfully executed, the network state and reward value are updated. Then, the sample information (including state, action, reward, next state, etc.) in the state transition process is saved to the experience replay buffer of the model using agent.add_sample(). Then the business demand, source node and target node information is extracted again to continue the training of this round, until the residual capacity of the selected path link cannot meet the traffic demand, then stop this iteration. After each training iteration, the experience replay agent.reply() of the model is called, and part of the sample information in the experience replay buffer is copied to the target network for stable updating of the network parameters, so as to improve the decision-making performance of the model in the environment. Through multiple iterations, the model can gradually learn and optimize its strategy to achieve better performance.

[0108] Every evaluation_interval = 20 iterations, input EVALUATION_EPISODES = 40 data sets to the current model for model evaluation. At the beginning of each evaluation round, the environment state is reset and the relevant parameters are initialized. Then select the action according to the current network state, execute the action and observe the feedback of the environment. Every time a business requirement is completed, the reward value rewardAddTest is accumulated. Then a set of business is extracted again and the above steps are repeated for route planning. If a link resource is exhausted, resulting in the inability to complete this requirement, the next round of evaluation round is started. After all the evaluation rounds are completed, the average reward value (Average Score) of the current model is calculated, which can be regarded as the performance of the model during the evaluation test. If the average reward value of the current model is greater than the historical maximum reward value, the maximum reward value is updated and the current optimal model ID is recorded. At the same time, the trained model parameters are saved, so that the model can be loaded and used when needed. Through multiple iterations, the learning process score of each route planning model and the optimal model serial number with the highest score can be obtained. Finally, the maximum reward value and the corresponding model ID are written into the log file, and the model counter is updated. When using the model, only the serial number is needed to match the detailed parameters and settings of the model in the record file.

[0109] In the model training process, three parameters of loss value, Score, and epsilon value are used to evaluate the excellence of the GNN-DRL model trained by the model:

[0110] During the experiment, the loss is used to measure the difference between the model's estimated Q value and the target Q value. It is used to evaluate the accuracy of the model's prediction and serves as a feedback signal to assist in updating the model parameters. By minimizing the loss function, the model can more accurately fit the training data, thereby improving the model's prediction performance and generalization ability. In the experiment, the Mean Squared Error (MSE) is chosen as the loss function, which is the average of the squared differences between the model's predicted value and the target value. At the same time, L2 regularization is used to reduce the model complexity, which is calculated by multiplying the square root of the sum of the squares of all weights by the regularization parameter L2 = 0.1. The final stored loss value is obtained by adding the loss value and the regularization value. Among them, the estimated Q value is the action Q value calculated by passing the current state into the main network and performing forward propagation. The target Q value is based on the next state of the network. According to the Bellman equation, the target Q value = current reward + discount factor (gamma) * next state maximum Q value. In this experiment, gamma = 0.95. Then, based on the loss function, the gradient is calculated and gradient clipping is performed to avoid the problem of gradient explosion. Finally, the gradient descent method is used to update the model parameters according to the clipped gradient, so that the loss gradually decreases. During the training process of the optimal routing planning model of the four topologies, the loss value changes as shown in Figure 6 The dark blue line represents the average loss value of each batch. It can be observed that the average loss value gradually decreases to 10-1 and finally converges, which indicates that the Q value estimated by the final model can well satisfy the target, i.e. the current model has good network state prediction and policy selection ability.

[0111] In ITERATIONS = 10000 training iterations, a total of 500 model evaluations were performed, with EVALUATION_EPISODES = 40 data sets input each time. In each evaluation round, the average reward value (Average Score) of the current model in the 40 rounds was calculated according to the completed demand. If the average reward value of the current model is greater than the historical maximum reward value, the maximum reward value is updated, and the current model is marked as the new best model. Figure 7 The average score of each model of the four topologies during the training process is shown. It can be observed that the score gradually increases from 0 and finally converges. This indicates that as the model training degree deepens, through multiple iterations of optimal model selection, the effect of model selection path becomes better and better, i.e. more business demands can be completed under the condition of limited network resources.

[0112] During the training process, if the model only adopts the best strategy that has been learned, it may lead to the strategy falling into a local optimal solution and missing a better strategy. By introducing the epsilon-greedy strategy, the model has a certain probability of selecting a random action, thereby avoiding excessive reliance on learned knowledge and jumping out of a local optimal solution to find a better strategy. When epsilon is small, the model is more inclined to use existing knowledge, but still has a certain probability of exploration; when epsilon is large, the model is more inclined to random exploration. With the training, gradually reduce the value of epsilon, which can make the DQN model more dependent on exploration in the early stage of learning, and gradually transition to using existing experience to make decisions, so as to more stably select the optimal action. The trend of the change of the Epsilon value of the four topologies is shown in Figure 8 It can be observed that it gradually decreases from 1 and finally tends to 0.1. This indicates that the model relies more on random exploration in the initial stage of training. However, as the training experience increases, the model becomes more mature, and its parameters become more stable. At this time, the routing decision can rely more on the model's selection, thereby performing path planning based on the optimal solution of the whole set and maximizing the utilization of network resources.

[0113] Through the above training and routing planning process, I obtained the optimal routing configuration of the GNN-DRL model, thereby achieving the goal of optimal allocation of network resources. I presented the results of the GNN-DRL model's path selection in the form of a network topology graph. In the graph, the unused nodes and links are displayed in gray, while the source node, target node, and GNN-DRL model planned path are marked in yellow. Taking the GEANT2 topology structure in the SNDLib dataset as an example, the routing planning results of the GNN-DRL model are shown in Figure 9 , which clearly shows its optimization of network path planning.

[0114] The proposed GNN-DRL model is compared with other traditional models in terms of performance on a series of topologies and data. Figure 5 The specific topology structure of each dataset is shown. In the experiments, the source node and destination node are randomly selected in the topology, and the capacity of each link in the topology is set according to the real data in the dataset, aiming to simulate a more realistic network environment. The task of the GNN-DRL model is to reasonably allocate the demand between the source node and the destination node to ensure that the communication demand is met. According to the definition of the reward function, the more demand the model achieves (i.e., the larger the normalized sum), the higher the reward the model's action will obtain.

[0115] The GNN-based DRL method (referred to as the GNN-DRL model) adopts a deep neural network to estimate the Q value of each state-action pair and selects an action by maximizing the Q value. In addition, in order to improve the stability and effect of learning, the method also introduces technologies such as experience replay and target network. Experience replay enables the agent to learn from historical experience, while the target network is used to reduce Q value fluctuations during training, thereby improving the stability of training. In order to verify that the method has good performance when running on different topologies, I compared the Score of its running with two baseline solutions.

[0116] The SAP (Shortest Available Path) model is a path-based demand allocation model. First, the model obtains all possible paths from the source node to the destination node, and checks whether each path can meet the current communication demand in turn. Once a path is found that can meet the demand, the model will immediately allocate resources and return the selected path for routing.

[0117] The RAND (Random) model is a strategy model based on random selection of paths, which does not consider environmental states or other information, and is a completely random strategy. The RAND model traverses all possible paths and performs capacity checks on each path to determine whether the path is available. A path is randomly selected from all paths that can meet the demand for routing.

[0118] The present application inputs 50 data sets into the GNN-DRL trained optimal model and calculates the cumulative reward of each data set to evaluate the performance of the model. According to the definition of the reward function, the cumulative reward is normalized to Score to represent the overall performance level of the model in allocating network resources. The more total demand is achieved, the higher the score is obtained. At the same time, these data sets are input into the SAP and RAND algorithms to calculate the Score for comparison.

[0119] Figure 10The comparison of the performance of the three model methods under different topologies is shown, where the dashed line represents the average score of various methods. In Geant2, the average score of the GNN-DRL model (red line) is 13.9, while the average scores of the SAP and RAND models are only 13.2 and 11.7, respectively. In Small Top, the average score of the GNN-DRL model is 18.1, while the average scores of the SAP and RAND models are only 17.6 and 13.9, respectively. In Gbn and Nsfnet, the average scores of the GNN-DRL model are 12.6 and 14.8, respectively, while the average scores of the RAND model are only 11.1 and 11.5, respectively. As can be clearly seen from the figure, compared with the baseline SAP model and the RAND model, the GNN-DRL model shows better performance under various topologies. This shows that the GNN-DRL model has better performance in processing network resources, regardless of the topology.

[0120] In summary, the present application has the ability to perceive and respond to network status in real time and find the optimal transmission path for data flow, which can promote the development of communication networks in the direction of intelligence and green.

[0121] The above examples are only used to illustrate the technical solutions of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features, but these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A route optimization method based on graph neural networks and deep reinforcement learning, characterized in that, The DQN model introduces a main network, an experience replay buffer, and a target network. The main network continuously generates training experience by interacting with the environment, receives the current network state s as input, and outputs the predicted Q-value for each possible action a. The experience replay buffer is used to update and store training experience in real time, while randomly sampling a portion of samples for training. The target network is used to periodically copy and update parameters from the main network, calculate the target Q-value, and enhance the convergence speed of the algorithm through gradient training. The route optimization method includes the following steps: S1. Input the network status information and service traffic requirements of each link and all its neighboring links. Use a graph neural network to integrate the network status and service traffic requirements to generate messages that are integrated according to nodes and presented in a graph structure. S2. Run an iterative message passing algorithm between the hidden states of the link nodes in the graph. After multiple iterations, use an aggregation function to aggregate the updated node representations to obtain the global hidden state. S3. Based on the global hidden state in step S2, use a graph neural network to calculate the action prediction Q value Q(s,a) under the input state; S4. Select the optimal action based on the Q value of step S3, and then execute this routing decision action in the environment. If the action is successfully executed, that is, the remaining bandwidth in each link of the selected path is sufficient to meet the traffic demand of this service, then return a normalized reward value and update the network state s'. If the action cannot be completed, then end this training. S5. After multiple training sessions, the model parameters are continuously updated based on the network state s, action a, and reward value r, and the training experience (s,a,r,s') generated by the main network is stored in the experience replay buffer. S6. Every M iterations, a small batch of samples will be randomly drawn from the buffer and copied into the target network for calculating the target Q value. S7. Continuously update the network parameters of the target network and send the updated model parameters to each node in the main network. When the main network receives the updated parameters, each node will update its routing planning model parameters. Repeat step S7 until convergence.

2. The routing optimization method based on graph neural networks and deep reinforcement learning according to claim 1, characterized in that, In step S1, the network status information includes link characteristics and network topology.

3. The routing optimization method based on graph neural networks and deep reinforcement learning according to claim 1, characterized in that, In step S2, in each iteration of the message passing algorithm, a node receives the current state from its neighboring nodes and uses an update function to update its own hidden state.

4. The routing optimization method based on graph neural networks and deep reinforcement learning according to claim 1, characterized in that, In step S2, the message passing algorithm network consists of three layers: the Message layer, the Update layer, and the Readout layer. In the Message and Update layers, a total of T message passing processes are required, as detailed below: For a specific node v, first execute formula (1) so that node v receives messages from the state of node v itself and the states of all surrounding nodes, and aggregate the messages by summing them element by element. In formula (1), N(v) represents the message received by node v at time step t+1, where N(v) represents the neighboring nodes of v. is the feature vector of node v at time step t, m(·) is the message function, w is the target node, and W is an adjacency matrix containing all the neighboring nodes of the target node. Then, the aggregated message is sent to a neural network implemented by a gated recursive unit, and the hidden state of the link is continuously updated in a recursive iterative manner. The new hidden state of the node is calculated as shown in formula (2): U t It is an update function that updates the original node state h. v t and information As input, we obtain the new node state. After T iterations, the final hidden state of each link is obtained by aggregation again and enters the Readout layer. The reading function R is used to calculate the node state based on the global hidden state graph, as shown in Equation (3): in, Let G be the final hidden state of node v after all T iterations of the graph neural network, and let G be the network topology.

5. The routing optimization method based on graph neural networks and deep reinforcement learning according to claim 1, characterized in that, In step S4, the reward value r is calculated as shown in formula (4): r = bw / max(bw) (4) Where bw represents business traffic requirements.

6. The routing optimization method based on graph neural networks and deep reinforcement learning according to claim 1, characterized in that, In step S3, the estimated Q value returned after performing action a according to policy π in state s is calculated as shown in equation (5): Wherein, the policy function π(a|s): S×A→[0,1] is the conditional probability distribution when choosing action a in state s, and the objective of the policy at time τ is to maximize the reward r accumulated from historical experience. τ γ∈(0,1) is the discount factor that balances historical and current reward data, k is a certain time interval, and E π To obtain the expected value when using the π strategy.

7. The routing optimization method based on graph neural networks and deep reinforcement learning according to claim 1, characterized in that, In step S6, the target Q value is the sum of the reward value r at the current time step and the maximum estimated Q value in the next state after decay, and is calculated as shown in equation (6): Q target (s,a,w)=r+γmax a' Q(s',a',w) (6) Where w represents the neural network parameters, γ∈(0,1) is the discount coefficient for weighing historical and current reward data, s' is the next state, a' is the action selected in the next state, and w is the target node.

8. The routing optimization method based on graph neural networks and deep reinforcement learning according to claim 7, characterized in that, The loss value is calculated using the mean squared error between the estimated Q value and the target Q value, and is used to calibrate the neural network parameters, as shown in Equation (7): L(w)=E[(Q target (s,a,w)-Q(s,a,w)) 2 ] (7) Where E represents the expected value.

9. The routing optimization method based on graph neural networks and deep reinforcement learning according to claim 7, characterized in that, The model is continuously trained using the gradient decrease method, which continuously reduces the model loss value. The gradient calculation method is shown in Equation (8): Among them, a k It is the action selected at time interval k.

10. The routing optimization method based on graph neural networks and deep reinforcement learning according to claim 7, characterized in that, Use equation (9) to continuously update the network parameters of the target network: Where α is the learning rate, v k It is calculated from the discount return value and used as a weighting factor for the gradient update value.

Images