On-line routing method and system based on deep reinforcement learning

Abstract

The invention discloses an on-line routing method and system based on deep reinforcement learning. The method specifically comprises the following steps: 1, calculating K paths from a source node to a destination node; 2, predicting a prediction performance index of the optimal path p obtained in the (n-1) th period; 3, calculating the actual performance index of the optimal path y obtained in the (n-2) th period; 4, calculating the relative difference between the predicted performance index in the step 2 and the actual performance index in the step 3, and updating the weights of the end-to-end time delay, the end-to-end residual bandwidth and the end-to-end packet loss probability according to the relative difference; and 5, updating the reward function of the nth period based on the weight obtained in the step 4, calculating the optimal path of the nth period by adopting an SAC algorithm based on the reward function of the nth period, and then turning to the step 2. According to the method, the change of a network environment can be more quickly adapted, and a stable and optimal routing scheme is obtained.

Description

Technical Field

[0001] The present invention belongs to the field of traffic control, and in particular relates to an online routing method and system based on deep reinforcement learning. Background Art

[0002] In recent years, there are heterogeneous mixed traffic such as voice, text, images, and videos in communication networks. Different services have different requirements for service quality, which makes it difficult for heterogeneous services to match the optimal path. On the other hand, the strong burstiness of network services may cause a sudden increase in the load of the selected link, which cannot meet the traffic demand in time, thus affecting the stability of the network and the data transmission rate. In the communication environment, adverse factors such as electromagnetic interference and complex terrain may cause link failures and even cause communication interruptions. However, the optimization goals of most current routing methods based on Deep Reinforcement Learning (DRL) are static, while the network environment is dynamic. Traffic fluctuations or readjustment of network equipment often lead to periodic fluctuations in network performance, such as decreased throughput or latency peaks. Existing technologies cannot effectively reflect the differences in the importance of different metrics in a dynamic network environment, resulting in a lack of sufficient flexibility in routing strategies. Summary of the invention

[0003] In order to solve the above problems in the prior art, the present invention provides an online routing method and system based on deep reinforcement learning.

[0004] Technical solution: The present invention provides an online routing method based on deep reinforcement learning, which specifically includes the following steps: Step 1: Calculate K paths from the source node to the destination node; Step 2: predicting the performance indicators of the optimal path p obtained in the n-1th cycle, wherein the performance indicators include end-to-end delay, end-to-end throughput, and end-to-end packet loss rate; n is a positive integer greater than 2; Step 3: Calculate the actual performance index of the optimal path y obtained in the n-2th cycle; Step 4: Calculate the relative difference between the predicted performance indicator in step 2 and the actual performance indicator in step 3, and update the weight of the end-to-end delay based on the relative difference , the weight of the end-to-end remaining bandwidth and the weight of the end-to-end packet loss rate ; Step 5: Based on the weights updated in step 4, update the reward function of the nth cycle. Based on the reward function of the nth cycle, use the SAC algorithm to calculate the optimal path of the nth cycle, generate the flow table, send it to the corresponding switch device, install the path and forward the traffic, and then go to step 2.

[0005] Furthermore, in step 1, a K shortest path algorithm is used to calculate K paths between the source node and the destination node.

[0006] Furthermore, in step 2, the message passing neural network is used to predict the prediction performance index of the optimal path obtained in the n-1th cycle. The training of the message passing neural network is specifically as follows: Step 2.1: Construct a topology diagram of the communication network; Step 2.2: Simulate the constructed communication network topology using different traffic conditions to generate a data set; Step 2.3: Preprocess the data of link-level features and path-level features in the data set, and initialize the hidden state of the link and the hidden state of the path; Step 2.4: Perform T cycles of iterations on the message passing neural network to update the hidden state of the link and the hidden state of the path, specifically: Path message passing process: During the t+1th iteration, a recurrent neural network is used to encode the hidden state of the link at the tth iteration , get the link state sequence associated with the path ,Will and Input to the attention network , get the dynamically weighted link state aggregation information , the hidden state of the path at the tth iteration and Input to the neural network μ and get the hidden state of the path at the t+1th iteration output by the neural network μ ; Message passing process of the link: During the t+1th iteration, the state information of all paths in the link during the t+1th iteration is aggregated by summing to obtain the transition state information of the path. , and then the hidden state of the link and Input into the recurrent neural network to obtain the hidden state of the link at the t+1th iteration ; Step 2.5: Finally, the hidden state of the path obtained in the Tth cycle , input into the readout function to obtain the prediction performance index of the optimal path.

[0007] Furthermore, the preprocessing in step 2.3 is: normalizing the data represented by numerical values ​​and performing hot encoding on the data represented by categories.

[0008] Furthermore, step 3 calculates the actual end-to-end delay of the optimal path y according to the following formula: , actual end-to-end throughput and the actual end-to-end packet loss rate : ; ; ; Among them, i represents the source node, j represents the destination node, and e ij represents the link in the optimal path y, d(e ij ) indicates link e ij The delay of min represents the minimum function, thr (e ij ) is link e ij The throughput, Link e ij The packet loss rate.

[0009] Furthermore, the step 4 is specifically as follows: Step 4.1: Calculate the relative difference between the predicted performance metric and the actual performance metric: ; ; ; Where rel_delta_delay is the relative difference between end-to-end delays, rel_delta_thr is the relative difference between end-to-end throughputs, rel_delta_loss is the relative difference between end-to-end packet loss rates, p_delay is the predicted end-to-end delay of the optimal path p, p_thr is the predicted end-to-end throughput of the optimal path p, and p_loss is the predicted end-to-end packet loss rate of the optimal path p; is the actual end-to-end delay of the optimal path y, is the actual end-to-end throughput of the optimal path y, is the actual end-to-end packet loss rate of the optimal path y, is a constant; Step 4.2: Calculate the weights of end-to-end delay, end-to-end remaining bandwidth, and end-to-end packet loss rate: ; ; ; ; ; ; Where k1 represents the scaling factor, , as well as All are intermediate amounts.

[0010] Furthermore, the expression of the reward function r in the SAC algorithm is: ; in, Indicates Path, ; Indicates The normalized value of the end-to-end delay of the path, Indicates The normalized value of the end-to-end residual bandwidth of the path, Indicates The normalized value of the end-to-end packet loss rate of each path.

[0011] An online routing system based on deep reinforcement learning, comprising a network perception module, a network monitoring module, a data processing module, a prediction module, an intelligent optimization module and a path installation module; The network monitoring module periodically sends status request information to the forwarding device, asynchronously receives port status information of the forwarding device, and transmits it to the data processing module and the prediction module, the port status information includes the enabled state of the port, traffic load, packet loss rate and delay; The network perception module collects network topology, global routing scheme, path-level information and link-level information; and transmits them to the data processing module and the prediction module; The data processing module calculates the actual values ​​of K paths and performance indicators; The prediction module predicts performance indicators based on network topology, global routing scheme, flow-level information and path-level information; The intelligent optimization module calculates the weight of the end-to-end delay, the weight of the end-to-end remaining bandwidth and the weight of the end-to-end packet loss rate in the reward function, and selects the optimal path from K paths as the global routing solution; The path installation module obtains the optimal path calculated by the intelligent optimization module, generates a flow table, and sends it to the switch device to perform path installation and traffic forwarding.

[0012] An electronic device / system for an online routing method comprises a processor and a memory, wherein the memory stores execution instructions of the processor, and the processor is configured to execute the execution instructions to implement the online routing method.

[0013] A computer-readable storage medium is used to store a program, and the program is executed to implement the above-mentioned online routing method.

[0014] Beneficial effects: The present invention proposes a new online routing method (abbreviated as GSAC-P), which combines a performance prediction model based on a message passing neural network (MPNN) and a routing generation model based on a soft actor-critic (SAC) algorithm, and introduces a reward function that can dynamically adjust the weight of indicators. This method effectively solves the problem that some existing online routing algorithms cannot respond to network changes in a timely manner when dealing with strong burst services and strong adversarial network environments. Through this innovative design, in a complex network, when encountering a surge in burst services or link failures, the present invention can not only improve the transmission performance of services with different service quality, but also adapt to changes in network status more quickly and accurately, significantly improve network throughput, and effectively reduce end-to-end latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of the present invention.

[0016] Figure 2 It is a schematic diagram of the architecture of the present invention.

[0017] Figure 3 It is a graph neural network model diagram based on MPNN.

[0018] Figure 4 It is a communication network topology diagram.

[0019] Figure 5 It is a communication network topology diagram after simulating partial link failure.

[0020] Figure 6 It is the average end-to-end delay curve under traffic changes.

[0021] Figure 7 It is a curve chart of average network throughput ratio under traffic changes.

[0022] Figure 8 It is the box plot of the average end-to-end delay under traffic changes.

[0023] Fig. 9 It is a histogram of average network throughput under traffic changes.

[0024] Fig.10 It is a curve of the average end-to-end delay before and after some link failures.

[0025] Fig.11 It is a curve chart of the average network throughput ratio before and after partial link failure.

[0026] Fig.12 It is a box plot of the average end-to-end delay before and after partial link failure.

[0027] Fig.13 It is a histogram of the average network throughput before and after partial link failure. DETAILED DESCRIPTION

[0028] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0029] The specific process of the method of the present invention is as follows Figure 1 As shown: Step 1: Define the topology of the communication network as , G represents the topology graph of the entire communication network, which includes the set V of communication entities, the edge set E1 of communication links between entities, the transmission flow set F, the link set L and the path set P1.

[0030] Take the communication entity as a node, and set , where k2 represents the total number of nodes, represents the k2th node, the link set , Indicates the total number of links. Indicates Links; transport stream set , Indicates the total number of transmission streams, Indicates Transport streams, path set , Indicates the total number of paths. Indicates A path is a transmission path that a flow follows from source to destination, so a path is defined as a sequence of flows and traversed links. ,in , Respectively represent the paths The qth index of the flow and link, J represents the total number of indexes.

[0031] Step 2: Use the simulation platform to build the communication network topology diagram required for prediction, simulate 50 traffic conditions, and generate a data set.

[0032] The present invention simulates 50 kinds of random traffic, and these random traffics have different traffic characteristics, including the bandwidth size of the traffic, the arrival time distribution of the data packets, the size of the data packets and the source-destination nodes.

[0033] Step 3: preprocess the data about link-level features and path-level features in the dataset, and initialize the hidden state of the link and the hidden state of the path through the feature embedding H function.

[0034] First, the data in the dataset is classified according to link-level features and path-level features. Link-level features include bandwidth, link utilization, and load, etc. Path-level features include traffic bandwidth and data packet size, etc.

[0035] For data expressed as numerical values, the actual value is subtracted from its mean value and then divided by the standard deviation for normalization.

[0036] Data represented by categories can be encoded in hot encoding to facilitate subsequent matrix operations.

[0037] Finally, the link and path features are passed through an input layer to initialize the hidden state.

[0038] Step 4: The message passing mechanism of the message passing neural network is used to propagate and update information between states. The structure of the network is shown in the figure below: Figure 2 shown.

[0039] First, the link set L, the path set P, the initial link hiding state and the initial path hiding state are input.

[0040] Path message passing process: At the current t+1th iteration, a recurrent neural network (RNN) is used to encode the hidden state of the link at the tth iteration , get the link state sequence associated with the path ,Will and Input to the attention network , get the dynamically weighted link state aggregation information , the hidden state of the path of the tth iteration and Input to the neural network μ and get the hidden state of the path at the t+1th iteration output by the neural network μ ; Link message passing process: During the t+1th iteration, for any link, the state information of all paths passing through the link during the t+1th iteration is aggregated by summing, thereby obtaining the transition state information of the path. , and then hide the link status and Input into the RNN network to obtain the hidden state of the link at the t+1th iteration .

[0041] Step 5: Repeat the three-stage message passing process in step 4 T times to achieve message passing and updating in more distant nodes.

[0042] In graph neural networks, nodes update their features by interacting with neighboring nodes. In each iteration, nodes obtain information from their neighboring nodes and adjust their own features based on this information. The depth of information transmission is determined by the number of iterations. Increasing the number of iterations allows information to propagate further, allowing each node to integrate feature data from more nodes.

[0043] Step 6: The readout function outputs the path-level hidden state after the Tth cycle To predict some path-level information, namely performance indicators, including end-to-end delay, end-to-end throughput and end-to-end packet loss rate; and use triples to store the performance indicator prediction information on the path from source node i to destination node j.

[0044] Step 7, using smooth L1 loss as the loss function, train the overall model, iterate to obtain the convergence value, and store the trained model. The above steps 1 to 7 are all the training process of the message passing neural network.

[0045] Step 8: The system initializes the network perception module, network monitoring module, data processing module, prediction module, intelligent optimization module and path installation module; Figure 3 It is a schematic diagram of the architecture of the present invention, including a three-layer structure of a data layer, a control layer and an application layer. The control layer also includes a network perception module, a network monitoring module, a data processing module, a prediction module, an intelligent optimization module and a path installation module.

[0046] The data layer consists of programmable switches and communication links. It accepts the policies of the control layer through the southbound interface and processes data packets. It is mainly used to model complex networks.

[0047] The control layer is used to connect the data layer and the application layer, maintain the global view, pass the control policy downward to the data layer, provide network resource information to the application layer upward, and periodically send query messages to the data layer.

[0048] The application layer communicates with the control layer through the northbound interface, uses the network status information provided by the data layer, and supports various application services.

[0049] Step 9: The data processing module executes K Shortest Paths Algorithm (KSP) according to the topological structure and source-destination node information to calculate K feasible paths.

[0050] Step 10: The network perception module and the network monitoring module periodically collect raw data of the network to detect changes in network status, including network topology, global routing scheme, port status information, path-level information, link-level information, etc., and store these data.

[0051] The network perception module periodically sends feature request information to the forwarding devices (such as switches) at the data layer to obtain the latest status of the network topology, including the connection relationship of each node, link status, etc. In addition, the network perception module also collects global routing plan information to track the path selection and traffic distribution of each flow.

[0052] The network monitoring module periodically sends status request information to the forwarding device of the data layer and asynchronously receives the status statistics of the device. It mainly focuses on the port status information of the device, such as the port's enabled status, traffic load, packet loss rate, latency, etc. Through these real-time monitoring data, the network monitoring module can fully reflect the health status of the network.

[0053] Step 11: The data processing module calculates the actual performance index of the optimal path in the n-2th cycle based on the network topology and port status information. The performance index includes the end-to-end delay , end-to-end throughput and end-to-end packet loss rate (Due to the influence of delay, this embodiment calculates the performance index of the optimal path of the first cycle from the third cycle): ; ; ; Where y represents the optimal path obtained in the n-2th cycle, i represents the source node, j represents the destination node, and e represents the ij represents the link in the optimal path y, d(e ij ) indicates link e ij delay; min represents the minimum function, thr (e ij ) is link e ij The throughput, Link e ij The packet loss rate.

[0054] Step 12: The network topology, global routing scheme, link-level information, and path-level information are input into the prediction module (i.e., the message passing neural network). The prediction module outputs the predicted end-to-end delay, end-to-end throughput, and end-to-end packet loss rate of the optimal path obtained in the n-1th cycle.

[0055] The intelligent optimization module updates the weights of the end-to-end delay, the end-to-end remaining bandwidth and the end-to-end packet loss rate in the reward function according to the predicted performance indicators and the actual performance indicators calculated in step 11.

[0056] By calculating the relative difference between the predicted value and the actual value, these differences are weighted using the Softmax function to obtain the weights of the end-to-end delay, end-to-end remaining bandwidth, and end-to-end packet loss rate, thereby updating the reward function. First, calculate their relative differences according to the following formula: ; ; ; Where rel_delta_delay is the relative difference between end-to-end delays, rel_delta_thr is the relative difference between end-to-end throughputs, rel_delta_loss is the relative difference between end-to-end packet loss rates, p_delay is the predicted end-to-end delay of the optimal path p in the n-1th cycle, p_thr is the predicted end-to-end throughput of the optimal path p, and p_loss is the predicted end-to-end packet loss rate of the optimal path p. is a constant.

[0057] Calculate the weights of end-to-end delay, remaining bandwidth, and packet loss rate: ; ; ; ; ; ; in, is the weight of the end-to-end delay, is the weight of the end-to-end remaining bandwidth, is the weight of the end-to-end packet loss rate, k1 represents the scaling factor, , as well as All are intermediate amounts.

[0058] Step 13: Build a deep reinforcement learning agent based on the SAC algorithm. The intelligent optimization module inputs the information obtained by the data processing module into the agent, and the agent obtains the current network status. , select the optimal path action for it , and then get the network status at the next moment , and get the current reward according to the latest reward function , and in the form of a quad Stored in the experience replay pool for model training.

[0059] The SAC algorithm is a deep reinforcement learning algorithm based on the maximum entropy idea. Different from traditional deep reinforcement learning algorithms, it uses a random strategy instead of a deterministic strategy. At the same time, the SAC algorithm shows significant advantages in offline strategy learning and exploration capabilities, and can better balance exploration and utilization. Its core idea is to maximize the long-term cumulative rewards while maximizing the entropy of the strategy, encouraging the strategy to maintain sufficient randomness. Specifically, it is achieved by optimizing the following objective function: ; in, represents the optimized strategy, π represents the strategy network, E represents the expected function, is the discount factor, is the entropy regularization coefficient, Indicates that the input status is , the action is The reward value at time is the entropy value. The introduction of the concept of entropy helps to keep the strategy sufficiently random, promote the agent to explore the state space more comprehensively, and avoid the strategy from falling into the local optimal solution. At the same time, the agent can explore multiple possible solutions, enhance its ability to adapt to environmental changes, and improve the robustness and anti-interference ability of the strategy. max represents the maximum function.

[0060] The SAC algorithm is an off-policy algorithm that uses an experience replay mechanism to eliminate data correlation, reuse historical data, and improve sample utilization. It uses a dual Q value network structure to avoid Q value overfitting and ensure training stability. Let Q (.) represent the source Q value network, represents the parameters of the policy network. The source Q-value network is learned by minimizing the flexible Bellman residual:

[0061] in, is the loss function of the source Q-value network, is the intermediate function, , Indicates that the parameter is The strategic network is the target Q value network, whose parameters The parameters of the source Q network are Soft update by Polyak averaging method:

[0062] in, is the soft update parameter, , the parameters of the target network are updated smoothly, which improves the stability of the learning process.

[0063] The policy function is obtained by minimizing the KL divergence of the following formula: ; in, is the objective function of the policy network, s~D are the states in the sampling distribution, For policy-based networks Actions sampled from a probability distribution.

[0064] Entropy regularization coefficient Updated via automatic entropy regulation method: ; in, is the hyperparameter of the target entropy.

[0065] State space. The state space consists of all states that the agent can observe. Each state reflects the current state of all traffic requests in the network (including information about the source and destination nodes), as well as the states of feasible paths associated with these requests. The state information is organized into a state matrix , and through the state N represents the source node and destination node of all traffic requests in the network at the current moment. PM consists of K feasible path status information (delay, remaining bandwidth and packet loss rate), and the specific form is as follows: ; in, Indicates The end-to-end delay of the path, Indicates The end-to-end residual bandwidth of the path, Indicates The end-to-end packet loss rate of the path.

[0066] Due to the large differences in the values ​​of the elements in the state matrix, the agent fluctuates during training and the model is difficult to converge. To solve this problem, the Min-Max normalization method is used to standardize the elements in the PM. The value of each element can be adjusted to within the range.

[0067] Action space. The action space refers to the set of actions that the agent can choose to perform in each state. Each action corresponds to a set of feasible paths from the source node to the destination node in a given state. The agent selects an optimal path from this set to achieve the best routing of network traffic.

[0068] Reward function. The reward function is used to measure the feedback obtained by the agent after taking an action, thereby helping it optimize its decision-making strategy. In reinforcement learning, the goal of the agent is to continuously maximize the reward value obtained. Therefore, the calculation method of the reward value r is as follows: ; in, Indicates Path, ; represents the normalized value of the end-to-end delay of the kth path, represents the normalized value of the end-to-end residual bandwidth of the kth path, Indicates The normalized value of the end-to-end packet loss rate of each path. The weights of the end-to-end delay in the initial cycle and the second cycle are both 0.3, the weights of the end-to-end residual bandwidth in the initial cycle and the second cycle are both 0.4, and the weights of the end-to-end packet loss rate in the initial cycle and the second cycle are both 0.4. Starting from the third cycle, the weights of each cycle are dynamically updated according to step 12.

[0069] Step 14: When the experience replay pool reaches the set threshold, the agent performs model training and parameter update, and the training is iterated until the model converges. Then the intelligent optimization module calculates the optimal path of the global network node pair and stores it. First, sample N tuples from the experience replay pool. , the target Q-value network is used to calculate the target value, the source Q-value network is updated by minimizing the Bellman residual, and the policy network is optimized by minimizing the KL divergence. The reparameterization technique is used to ensure that the gradient can be back-propagated, so that the network parameters can be directly optimized using the gradient descent method. For the stability of training, the parameters of the two target Q-value networks are soft-updated by Polyak averaging.

[0070] After updating the neural network parameters at each step, the entropy regularization coefficient is updated by the automatic entropy adjustment method. , to balance the exploration (maximizing entropy) and exploitation (maximizing Q-value) of the strategy.

[0071] Repeat the above training process and continuously optimize the parameters of the neural network until the algorithm converges and the learned strategy reaches the expected performance level.

[0072] The values ​​of the parameters in the SCA algorithm of this embodiment are shown in Table 1 below: Table 1 parameter value Optimizer Adam Entropy regularization coefficient 0.008 Discount Factor 0.95 Soft Update Parameters 0.01 Experience pool size 15000 Number of samples 32 Target Entropy -2 Gradient clipping parameters 1 Temperature parameter in softmax 2 Total number of training rounds 120 Number of feasible paths 10 Step 15: The path installation module obtains the optimal path result calculated by the intelligent optimization module, generates a flow table, and sends it to the corresponding switch device in the data layer for path installation and traffic forwarding. In the next network cycle, the prediction model predicts the performance index of the optimal path calculated by the intelligent optimization module in the current cycle.

[0073] According to the optimal path generated by the intelligent optimization module, the system first locates the corresponding host node, then combines the optimal path and host node information to build the best traffic transmission path and update the corresponding flow table entries. Finally, these updated flow table entries will be sent down to the data layer to ensure that network traffic can be efficiently transmitted along the optimal path.

[0074] Figure 4 It is the communication network topology of this embodiment, which consists of a backbone network and an access network. Its topology includes 44 switches and 61 links, where numbers 1-47 represent nodes, each node represents a switch that supports the OpenFlow protocol, and each switch is connected to a host, which is responsible for the output and reception of traffic. Nodes 18, 19, 20 and nodes 32, 33, 34 simulate sensor nodes, while nodes 42, 43, 44 simulate command and control nodes. Nodes 11, 12, 13, nodes 25, 26, 27 and nodes 39, 40, 41 represent synthetic nodes. The traffic transmission of the network follows the path planning from "sensor" to "command and control" and then to "synthesis". Therefore, this embodiment selects 6 sensor nodes as source nodes and 9 synthetic nodes as destination nodes. The link connection of the command and control network uses different types of heterogeneous links such as microwave, optical fiber, regional broadband, VHF, and UHF. In order to simulate these heterogeneous links, different transmission bandwidths were set in the Mininet simulation platform. The operating system used in the experiment of this embodiment was Ubuntu 18.04 with 8GB memory and a 2-core processor. The network simulation was implemented through the Mininet 2.3.0 platform, and the Ryu 4.34 controller was used to control the entire network. The communication network topology diagram after the partial link failure simulated in this embodiment is as follows: Figure 5 shown.

[0075] Figure 6 , Figure 7 , Figure 8 , Fig. 9 The performance comparison diagram of various algorithms in the process of traffic surge from 100 kbps to 500 kbps and then to 1000 kbps is shown in Figure 2. In order to verify the advantages of the algorithm of the present invention, the comparison algorithm used in this embodiment is as follows: SPR algorithm: shortest path routing algorithm, which obtains the weight information of each link in the network through the SDN measurement mechanism and calculates the path with the shortest link weight.

[0076] DRL-ST algorithm: Dueling DQN is used to build an end-to-end transmission path decision model, and the sampling mechanism is optimized using a tree storage structure. For a fair comparison, DRL-ST maintains the same training settings (such as state, action, and initial reward) as the algorithm of the present invention.

[0077] This embodiment uses network throughput and end-to-end delay to evaluate the impact of different routing algorithms on network performance.

[0078] Figure 6 The end-to-end delay of each algorithm under traffic changes is shown. When the DRL-ST algorithm is used, since this algorithm calculates the optimal path based on the periodically detected network state, when the traffic surges, the agent cannot respond to the changes in the network environment in time, resulting in a sharp increase in delay. But with the arrival of the next cycle, the delay will gradually decrease. However, since the algorithm uses a fixed reward function, the delay cannot be reduced to the optimal level after entering the stable state. When the environment changes, the SPR algorithm always chooses the shortest path. During the first change, the performance of SPR was almost unaffected due to sufficient bandwidth, but when the second change occurred, the bandwidth of the link could not meet the high load demand, resulting in a sharp increase in delay. In contrast, the GSAC-P algorithm proposed in the present invention introduces a prediction model, predicts the performance of the next cycle based on the network topology, traffic and routing scheme of the current cycle, and adjusts the reward function in time, which can quickly adapt to changes in the network environment and maintain a low delay after the system is stable. Therefore, GSAC-P can obtain a stable and optimal routing solution more quickly.

[0079] Figure 7 The network throughput ratio of each algorithm under traffic changes is shown. The throughput ratios of the two deep reinforcement learning (DRL)-based routing algorithms (DRL-ST and GSAC-P) both decrease after traffic changes, but the GSAC-P algorithm of the present invention still has a small change and a higher throughput after the routing is stabilized, indicating that the method of the present invention is more stable.

[0080] Figure 8 The box plot shows the end-to-end delay under traffic changes. It can be seen that the DRL-ST algorithm has a large number of outliers after the traffic changes, indicating that the delay fluctuates greatly, which means that the algorithm cannot respond to environmental changes in a timely manner.

[0081] Fig. 9 The average network throughput under traffic changes is shown in a bar graph, and it can be seen that under the three network states, the throughput corresponding to the present invention is the maximum.

[0082] Fig.10 The end-to-end delay of each algorithm before and after partial link failure at a traffic transmission rate of 1000 kbps is shown. Fig.10 It can be seen that the present invention can adapt to changes in the network environment more quickly and maintain a lower delay after the system is stable.

[0083] Fig.11 The network throughput ratio of each algorithm before and after partial link failure at a traffic sending rate of 1000 kbps is shown. Except for the SPR algorithm, the throughput ratio of the two deep reinforcement learning-based routing algorithms (DRL-ST and GSAC-P) decreases after the traffic changes, but the present invention still has a small change and a higher throughput after the routing is stable, indicating that the present invention is more stable.

[0084] Fig.12 The box plot shows the end-to-end delay before and after some link failures at a traffic sending rate of 1000 kbps. It can be seen that the DRL-ST algorithm has a large number of outliers after the traffic changes, indicating that the delay fluctuates greatly and the algorithm cannot respond to environmental changes in a timely manner.

[0085] Fig.13 The average network throughput before and after the partial link failure at a traffic transmission rate of 1000 kbps is shown in a bar graph. It can be seen that under the two network conditions, the throughput corresponding to the present invention is the largest.

[0086] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present invention will not further describe various possible combinations.

Claims

1. An online routing method based on deep reinforcement learning, characterized in that: The specific steps include: Step 1: Calculate K paths from the source node to the destination node; Step 2: predicting the performance indicators of the optimal path p obtained in the n-1th cycle, wherein the performance indicators include end-to-end delay, end-to-end throughput, and end-to-end packet loss rate; n is a positive integer greater than 2; Step 3: Calculate the actual performance index of the optimal path y obtained in the n-2th cycle; Step 4: Calculate the relative difference between the predicted performance indicator in step 2 and the actual performance indicator in step 3, and update the weight of the end-to-end delay based on the relative difference , the weight of the end-to-end remaining bandwidth and the weight of the end-to-end packet loss rate ; Step 5: Based on the weights updated in step 4, update the reward function of the nth cycle. Based on the reward function of the nth cycle, use the SAC algorithm to calculate the optimal path of the nth cycle, generate the flow table, send it to the corresponding switch device, install the path and forward the traffic, and then go to step 2.

2. The online routing method based on deep reinforcement learning according to claim 1, characterized in that: In step 1, the K shortest path algorithm is used to calculate K paths between the source node and the destination node.

3. The online routing method based on deep reinforcement learning according to claim 1, characterized in that: In step 2, the message passing neural network is used to predict the prediction performance index of the optimal path obtained in the n-1th cycle. The training of the message passing neural network is specifically as follows: Step 2.1: Construct a topology diagram of the communication network; Step 2.2: Simulate the constructed communication network topology using different traffic conditions to generate a data set; Step 2.3: Preprocess the data of link-level features and path-level features in the data set, and initialize the hidden state of the link and the hidden state of the path; Step 2.4: Perform T cycles of iterations on the message passing neural network to update the hidden state of the link and the hidden state of the path, specifically: Path message passing process: During the t+1th iteration, a recurrent neural network is used to encode the hidden state of the link at the tth iteration , get the link state sequence associated with the path ,Will and Input to the attention network , get the dynamically weighted link state aggregation information , the hidden state of the path at the tth iteration and Input to the neural network μ and get the hidden state of the path at the t+1th iteration output by the neural network μ ; Message passing process of the link: During the t+1th iteration, the state information of all paths in the link during the t+1th iteration is aggregated by summing to obtain the transition state information of the path. , and then the hidden state of the link and Input into the recurrent neural network to obtain the hidden state of the link at the t+1th iteration ; Step 2.5: Finally, the hidden state of the path obtained in the Tth cycle , input into the readout function to obtain the prediction performance index of the optimal path.

4. The online routing method based on deep reinforcement learning according to claim 3, characterized in that: The preprocessing in step 2.3 is: normalize the data represented by numerical values ​​and perform hot encoding on the data represented by categories.

5. The online routing method based on deep reinforcement learning according to claim 1, characterized in that: Step 3 calculates the actual end-to-end delay of the optimal path y according to the following formula , actual end-to-end throughput and the actual end-to-end packet loss rate : ； ； ； Among them, i represents the source node, j represents the destination node, and e ij represents the link in the optimal path y, d(e ij ) indicates link e ij The delay of min represents the minimum function, thr (e ij ) is link e ij The throughput, Link e ij The packet loss rate.

6. The online routing method based on deep reinforcement learning according to claim 1, characterized in that: The step 4 is specifically as follows: Step 4.1: Calculate the relative difference between the predicted performance metric and the actual performance metric: ； ； ； Where rel_delta_delay is the relative difference between end-to-end delays, rel_delta_thr is the relative difference between end-to-end throughputs, rel_delta_loss is the relative difference between end-to-end packet loss rates, p_delay is the predicted end-to-end delay of the optimal path p, p_thr is the predicted end-to-end throughput of the optimal path p, and p_loss is the predicted end-to-end packet loss rate of the optimal path p; is the actual end-to-end delay of the optimal path y, is the actual end-to-end throughput of the optimal path y, is the actual end-to-end packet loss rate of the optimal path y, is a constant; Step 4.2: Calculate the weights of end-to-end delay, end-to-end remaining bandwidth, and end-to-end packet loss rate: ； ； ； ； ； ； Where k1 represents the scaling factor, , as well as All are intermediate amounts.

7. The online routing method based on deep reinforcement learning according to claim 1, characterized in that: The expression of the reward function r in the SAC algorithm is as follows: ； in, Indicates Path, ; Indicates The normalized value of the end-to-end delay of the path, Indicates The normalized value of the end-to-end residual bandwidth of the path, Indicates The normalized value of the end-to-end packet loss rate of each path.

8. A system for an online routing method based on deep reinforcement learning as claimed in claim 1, characterized in that: It includes network perception module, network monitoring module, data processing module, prediction module, intelligent optimization module and path installation module; The network monitoring module periodically sends status request information to the forwarding device, asynchronously receives port status information of the forwarding device, and transmits it to the data processing module and the prediction module, the port status information includes the enabled state of the port, traffic load, packet loss rate and delay; The network perception module collects network topology, global routing scheme, path-level information and link-level information; and transmits them to the data processing module and the prediction module; The data processing module calculates the actual values ​​of K paths and performance indicators; The prediction module predicts performance indicators based on network topology, global routing scheme, flow-level information and path-level information; The intelligent optimization module calculates the weight of the end-to-end delay, the weight of the end-to-end remaining bandwidth and the weight of the end-to-end packet loss rate in the reward function, and selects the optimal path from K paths as the global routing solution; The path installation module obtains the optimal path calculated by the intelligent optimization module, generates a flow table, and sends it to the switch device to perform path installation and traffic forwarding.

9. An electronic device / system for an online routing method, characterized in that: The invention comprises a processor and a memory, wherein the memory stores execution instructions of the processor, and the processor is configured to execute the execution instructions to implement the online routing method according to any one of claims 1 to 7.

10. A computer-readable storage medium for storing a program, characterized in that: Execute the program to implement the online routing method described in any one of claims 1-7.

Images