Internet routing optimization method and device based on graph convolutional neural network

Abstract

The embodiment of the application provides an Internet routing optimization method and device based on a graph convolutional neural network, the method comprising: inputting current network state network graph data into a pre-trained routing optimization model to obtain a network link state weight matrix; determining a shortest routing path from a current network device node to a target network device node according to the network link state weight matrix; and transmitting a data packet from the current network device node to a first network device node in the shortest routing path, the first network device node being a next-hop network device node of the current network device node. In the embodiment of the application, the routing optimization model learns the rules between network device nodes in the network topology according to graph data, and a reinforcement learning algorithm is used to achieve a low network delay, so that an adaptive routing strategy is realized according to the network link state, the dynamic nature and real-time nature of network services can be adapted, and the communication efficiency of the entire network is improved.

Description

Technical Field

[0001] This application relates to the field of network routing optimization, and in particular to an Internet routing optimization method and apparatus based on graph convolutional neural networks. Background Technology

[0002] With the emergence of various new network applications such as online conferencing, short videos, live streaming, and autonomous driving, network traffic has increased exponentially, frequently resulting in network outages, disconnections, and buffering. Simultaneously, many similar network applications place higher demands on network real-time performance, end-to-end low latency, and low jitter. Network routing optimization is a crucial issue in achieving goals such as real-time performance and low latency.

[0003] Traditional routing optimization protocols include Routing Information Protocol (RIP), Open Shortest Path First (OSF), and Border Gateway Protocol (BGP). RIP uses a distance-vector algorithm, making it suitable only for small networks; it can easily cause network congestion in larger networks. OSF uses shortest path algorithms such as Dijkstra's algorithm to determine routing strategies; under high network load, frequent router information exchange exacerbates congestion. BGP's primary function is to find a suitable path, not necessarily the shortest path, making it difficult to achieve fast packet delivery in complex network environments.

[0004] Therefore, traditional routing optimization methods are difficult to adapt to the ever-changing network environment. There is an urgent need for a new routing strategy that can meet the constraints of low latency and low jitter in complex network environments and better adapt to the needs of dynamic and complex network environments. Summary of the Invention

[0005] In view of the above problems, this application provides an Internet routing optimization method and apparatus based on graph convolutional neural networks to overcome or at least partially solve the above problems.

[0006] A first aspect of this application discloses an Internet routing optimization method based on a graph convolutional neural network, the method comprising:

[0007] Input the current network state graph data into the pre-trained routing optimization model to obtain the network link state weight matrix;

[0008] The shortest route from the current network device node to the target network device node is determined based on the network link state weight matrix.

[0009] The data packet is transmitted from the current network device node to the first network device node in the shortest routing path, where the first network device node is the next-hop network device node of the current network device node;

[0010] The routing optimization model is based on training data generated from graph data and trained using reinforcement learning. The device nodes in the network correspond to the nodes in the graph data, and the topological connections between network device nodes correspond to the edges between nodes in the graph data.

[0011] Optionally, the training data is generated in the following way:

[0012] Initialize network characteristic information parameters and routing optimization model parameters based on the network state at time t. The network characteristic information parameters include: network topology, available bandwidth, and routing configuration information.

[0013] Obtain the network link status data under the network state at time t, and process the link status data into a network link status matrix at time t;

[0014] The network topology and routing configuration information are input into the routing optimization model to obtain the network latency at time t.

[0015] The network link state matrix at time t is input into the routing optimization model to obtain the network link state weight matrix at time t, and the network state in the routing optimization model is initialized according to the network latency at time t and the available bandwidth.

[0016] Based on the network link state weight matrix at time t and the shortest path algorithm, a shortest path is determined for packet forwarding, and the reward value of the network state at time t is calculated.

[0017] The network link state matrix at time t, the network link state weight matrix at time t, the reward value of the network state at time t, and the network link state matrix at time t+1 are used as one training data.

[0018] Optionally, the routing optimization model models the network data packet transmission process as a Markov decision process and is constructed based on a dual-delay deep deterministic policy gradient algorithm, and is trained in the following manner:

[0019] After each training data point is generated, it is placed into the experience replay pool of the route optimization model. The experience replay pool is used to store multiple training data points of the route optimization model.

[0020] A training data point is randomly sampled from the experience replay pool to update the parameters of the reinforcement learning model constructed based on the dual-delay deep deterministic policy gradient algorithm;

[0021] By performing multiple iterations of training following the steps described above, and finally obtaining the trained routing optimization model when the training termination condition is met, a well-trained model can be obtained.

[0022] Optionally, modeling the transmission process of network data packets as a Markov decision process includes:

[0023] The network link state matrix is ​​used as the state of the Markov decision process;

[0024] The link state weight matrix output by the routing optimization model is used as the selection probability of the next node in the Markov decision process.

[0025] The reward value of the Markov decision process is calculated based on the available bandwidth of the network link and the packet forwarding delay time.

[0026] Optionally, the reinforcement learning model constructed based on the dual-delay deep deterministic policy gradient algorithm includes: an Actor network, a Critic1 network, and a Critic2 network; randomly sampling a training data point from the experience replay pool to update the parameters of the reinforcement learning model constructed based on the dual-delay deep deterministic policy gradient algorithm includes:

[0027] The network link state matrix is ​​input into the Actor network to obtain the predicted network link state weight matrix;

[0028] The network link state matrix and the predicted network link state weight matrix are input into the Critic1 network and the Critic2 network, respectively, to obtain the Q value of the action corresponding to the network state;

[0029] The parameters of the Actor network, the Critic1 network, and the Critic2 network are updated based on the Q-value of the action corresponding to the network state.

[0030] Optionally, the Q-value of the action corresponding to the network state is represented as:

[0031]

[0032] Where R represents the reward value of the network state, and S m+1 This represents the next network state, D is the network delay in the current network state, i = 1 and 2 represent Critic1 and Critic2 networks, and a TD3 This represents the action matrix in the current network state, where γ∈(0,1) is the discount factor.

[0033] A second aspect of this application discloses an Internet routing optimization device based on a graph convolutional neural network, the device comprising:

[0034] The state weight module is used to input the current network state graph data into a pre-trained routing optimization model to obtain the network link state weight matrix.

[0035] The path determination module is used to determine the shortest route path from the current network device node to the target network device node based on the network link state weight matrix.

[0036] The data transmission module is used to transmit data packets from the current network device node to the first network device node in the shortest routing path, wherein the first network device node is the next-hop network device node of the current network device node;

[0037] The routing optimization model is based on training data generated from graph data and trained using reinforcement learning. The device nodes in the network correspond to the nodes in the graph data, and the topological connections between network device nodes correspond to the edges between nodes in the graph data.

[0038] A third aspect of this application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the Internet routing optimization method based on graph convolutional neural networks as described in the first aspect of this application.

[0039] A fourth aspect of this application discloses a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the Internet routing optimization method based on a graph convolutional neural network as described in the first aspect of this application.

[0040] A fifth aspect of this application discloses a computer program product including computer-readable code that, when executed, implements the Internet routing optimization method based on graph convolutional neural networks as described in the first aspect of this application.

[0041] The embodiments of this application have the following advantages:

[0042] In this embodiment, the routing optimization model obtains a network link state weight matrix based on the current network graph data, then determines the shortest route from the current network device node to the target network device node based on the network link state weight matrix, and transmits the data packet from the current network device node to the first network device node in the shortest route. Since the routing optimization model learns the rules between network device nodes in the network topology based on the graph data, and the reinforcement learning algorithm can achieve low network latency, it implements an adaptive routing strategy based on the network link state. This strategy can adapt to the dynamic and real-time requirements of network services, improve data transmission efficiency, select the optimal route in dynamic and complex environments, improve the overall network communication efficiency while ensuring global network load balancing, and has strong generalization ability for unknown network topologies. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart illustrating the steps of an Internet routing optimization method based on a graph convolutional neural network, as provided in an embodiment of this application.

[0045] Figure 2 This is a schematic diagram of the training process of a routing optimization model provided in an embodiment of this application;

[0046] Figure 3 This is a schematic diagram of the structure of an Internet routing optimization device based on a graph convolutional neural network provided in an embodiment of this application. Detailed Implementation

[0047] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0048] Reference Figure 1 As shown, Figure 1 This document illustrates a flowchart of the steps involved in an Internet routing optimization method based on a graph convolutional neural network, as provided in an embodiment of this application. Figure 1As shown, the Internet routing optimization method based on graph convolutional neural networks provided in this application embodiment may specifically include steps S110 and S130:

[0049] Step S110: Input the current network state network graph data into the pre-trained routing optimization model to obtain the network link state weight matrix.

[0050] The routing optimization model is based on training data generated from graph data and trained using reinforcement learning. The device nodes in the network correspond to the nodes in the graph data, and the topological connections between network device nodes correspond to the edges between nodes in the graph data.

[0051] In this embodiment, the current network state graph data is matrix-form data determined based on monitored current network state data. This data includes packet transmission delays between current network device nodes (i.e., routers or switches in the network) and bandwidth information (i.e., traffic volume) between network device nodes. The trained routing optimization model learns the rules between network device nodes in the network topology based on the graph data. This model effectively perceives the dynamic information of a specific network device node and its surroundings, and then uses this model to calculate network link state weights.

[0052] In practice, the current network link state data is determined based on the packet transmission delay and bandwidth information between network device nodes (i.e., network links). This network link state data is then processed into a matrix to obtain the network link state matrix, which is the current network state network graph data. After receiving the current network state network graph data, the routing optimization model outputs a network link state weight matrix based on the rules between network device nodes in the pre-learned network topology. Each element in the network link state weight matrix represents the weight value of each network link. If a network link has a large packet transmission delay and limited available bandwidth, its weight value is relatively small; conversely, if a network link has a small packet transmission delay and limited available bandwidth, its weight value is relatively large. Therefore, in subsequent steps, the shortest route path can be directly determined based on the network link state weight matrix.

[0053] Step S120: Determine the shortest route from the current network device node to the target network device node based on the network link state weight matrix.

[0054] In this embodiment, the current network device node refers to the network device node where the current data packet resides, the target network device node refers to the network device node receiving the data packet, and the shortest route path refers to the path with the shortest transmission delay from the current network device node to the target network device node. Specifically, the shortest route path from the current network device node to the target network device node is calculated based on the shortest path algorithm and the network link state weight matrix. Since the network state weight matrix is ​​determined based on the current network state network graph data, it accurately reflects the current network state information. Therefore, the shortest route path obtained based on the network link state weight matrix is ​​adaptively selected according to the network link state, improving the routing algorithm's adaptability to dynamic network topologies and better meeting the requirements of network dynamism and real-time performance.

[0055] Step S130: Transmit the data packet from the current network device node to the first network device node in the shortest routing path, where the first network device node is the next-hop network device node of the current network device node.

[0056] In this embodiment, considering the dynamic changes in network status, after obtaining the shortest route from the current network device node to the target network device node, it is only necessary to transmit the data packet to the first network device node in the shortest route. This avoids the problem of the shortest route changing due to dynamic changes in network status during data packet transmission. Furthermore, after sending the data packet to the first network device node in the shortest path, this first network device node is used as the current network device node, and the process returns to step S110 for a new round of shortest route planning until the data packet is transmitted to the target network device node. Since the shortest route is adaptively planned based on the current network link status, and transmission is performed at a single network device node at each moment, changes in the shortest route due to changes in network status are effectively avoided. Therefore, this method can adapt to the dynamic and real-time requirements of network services, select the optimal route in dynamic and complex environments, and improve the overall network communication efficiency.

[0057] For example, if the shortest route path obtained in step S120 is A→B→C→D→E (A is the current network device node, and E is the target network device node), then in step S130, it is only necessary to transmit the data packet from network device node A to network device node B, take network device node B as the current network device node, and return to step S110 to perform a new shortest route path planning, obtain a new shortest route path B→F→D→E, and transmit the data packet from network device node B to network device node F. Repeat the above method until the data packet is transmitted to the target network device node E.

[0058] In this embodiment, the routing optimization model is trained using training data generated from graph data and trained based on reinforcement learning. The routing optimization model learns the rules between network device nodes in the network topology based on the graph data, and at the same time uses reinforcement learning algorithms to achieve low network latency, thereby implementing an adaptive routing strategy based on the network link status.

[0059] In an optional embodiment, the training data is generated via steps A1 to A5:

[0060] Step A1: Initialize network characteristic information parameters and routing optimization model parameters according to the network state at time t. The network characteristic information parameters include: network topology, available bandwidth, and routing configuration information.

[0061] In this context, network topology refers to the connectivity between network nodes, available bandwidth refers to the available bandwidth between two network nodes (network links), and routing configuration information refers to the configuration information of network nodes. Network topology, available bandwidth, and routing configuration information reflect the network state. The network characteristic information parameters are initialized using the network state at time t, resulting in parameters that reflect the network state at time t. These parameters are then used to initialize the routing optimization model parameters, enabling the model to learn subsequently based on the network state at time t.

[0062] Step A2: Obtain the network link state data under the network state at time t, and process the link state data into a network link state matrix at time t.

[0063] Specifically, based on the network topology and routing configuration information in step A1, the controller in Software-Defined Networks (SDN) monitors and acquires the network link state data at time t, and then saves the network link state data at time t in the form of a matrix to obtain the network link state matrix at time t.

[0064] Step A3: Input the network topology and the routing configuration information into the routing optimization model to obtain the network delay at time t.

[0065] Here, the network delay at time t refers to the transmission delay time of data packets in each network link under the network state at time t. The network delay at time t is a matrix of data, where each element represents the data packet transmission delay time of a network link. Specifically, the network topology and routing configuration information are calculated using the routing optimization model initialized with parameters in step A1 to obtain the network delay at time t, which is then used to initialize the network state in the routing optimization model in subsequent steps.

[0066] Step A4: Input the network link state matrix at time t into the routing optimization model to obtain the network link state weight matrix at time t, and initialize the network state in the routing optimization model according to the network latency at time t and the available bandwidth.

[0067] Specifically, the network link-state matrix at time t in step A2 is input into the route optimization model initialized with parameters in step A1, and the network link-state weight matrix at time t is calculated using this route optimization model. Initializing the network state in the route optimization model specifically means that the network delay matrix and available bandwidth matrix at time t are horizontally concatenated to serve as the initial network state in the route optimization model.

[0068] Step A5: Based on the network link state weight matrix at time t and the shortest path algorithm, determine a shortest path for packet forwarding and calculate the reward value of the network state at time t.

[0069] Packet forwarding refers to transmitting packets from the current network device node to the target network device node based on the shortest routing path. Since the shortest path consists of multiple network links, the network state reward value is calculated based on the transmission delay time of the packet in each network link and the bandwidth of each network link.

[0070] Step A6: Use the network link state matrix at time t, the network link state weight matrix at time t, the reward value of the network state at time t, and the network link state matrix at time t+1 as a training data set.

[0071] Based on the method in step A1, obtain the network link state matrix M at time t+1. t+1 Then the network link state matrix M at time t will be... t The network link state weight matrix W at time t t The reward value R of the network state at time t t Network link state matrix M at time t+1 t+1 Composed of multiple groups (M) t W t ,R t Mt+1 This is used as a training data point to train the routing optimization model.

[0072] In an optional embodiment, the routing optimization model models the network packet transmission process as a Markov decision process and is constructed based on a dual-delay deep deterministic policy gradient algorithm, and is trained in the following manner:

[0073] After each training data point is generated, it is placed into the experience replay pool of the route optimization model. The experience replay pool is used to store multiple training data points of the route optimization model.

[0074] A training data point is randomly sampled from the experience replay pool to update the parameters of the reinforcement learning model constructed based on the dual-delay deep deterministic policy gradient algorithm;

[0075] By performing multiple iterations of training following the steps described above, and finally obtaining the trained routing optimization model when the training termination condition is met, a well-trained model can be obtained.

[0076] In this embodiment, considering the temporal nature of the generated training data, to ensure that the data used for training the routing optimization model is independent, thereby giving the final routing optimization model better robustness and stronger generalization ability, each training data point is added to the experience replay pool of the routing optimization model after it is generated. Then, during each training iteration, a training data point is randomly sampled from the experience replay pool to update the parameters of the reinforcement learning model (i.e., the routing optimization model) constructed based on the dual-delay deep deterministic policy gradient algorithm. After one parameter update, new training data is generated based on the above training data generation method, and the new training data is randomly added to the experience replay pool, with a new training data point randomly sampled for another parameter update. Finally, the trained routing optimization model is obtained when the training termination condition is met, which can be reaching a preset number of iterations.

[0077] In practical applications, in order to ensure the robustness of the routing optimization model, multiple training data are generated in advance based on the training data generation method described in steps A1 to A6 above, and put into the experience replay pool of the routing optimization model. When the number of training data in the experience replay pool reaches the preset number, training data is randomly sampled from the experience replay pool to update the parameters of the routing optimization model. In subsequent training processes, a new training data is generated each time based on the network link data at this time.

[0078] In this embodiment, since the routing optimization model is constructed using Markov decision processes and reinforcement learning, it can better perceive the dynamics and real-time nature of the network environment, thereby improving the dynamics, real-time nature, and reliability of network routing. At the same time, since the training data used for each parameter update of the routing optimization model is randomly sampled, the independence of the training data used for each parameter update is guaranteed. As a result, the trained optimization model parameters have good robustness and strong generalization ability for unknown network topologies.

[0079] In an optional embodiment, the process of modeling the transmission of network data packets as a Markov decision process includes steps B1 to B3:

[0080] Step B1: Use the network link state matrix as the state of the Markov decision process.

[0081] Specifically, the network link state matrix M is obtained by horizontally connecting the network link bandwidth matrix B and the network link data transmission delay matrix D. The state S of the Markov decision process is represented as:

[0082] S = M = B||D,

[0083] Among them, b ij and d ij They represent network link l respectively ij The network link bandwidth and packet transmission delay are given by 1 ≤ i, j ≤ n, where n is the total number of network device nodes, and || represents a horizontal connection.

[0084] Step B2: Use the link state weight matrix output by the routing optimization model as the selection probability of the next node in the Markov decision process.

[0085] Specifically, the selection probability A of the next node in a Markov decision process can be expressed as:

[0086]

[0087] Where W represents the network link state weight matrix output by the routing optimization model, W ij It is a network link ij The weight of the link is such that the larger the weight, the greater the probability that the network link will be selected, that is, the greater the probability that the node corresponding to the network link will be selected.

[0088] Step B3: Calculate the reward value of the Markov decision process based on the available bandwidth of the network link and the packet forwarding delay time.

[0089] Specifically, the reward value R of the Markov decision process is represented as:

[0090]

[0091] Where α∈(0,1) is the influence factor determined by the routing strategy, b ij and d ij These are network links l ij The link bandwidth and data transmission latency are given by 1 ≤ i, j ≤ n, where n is the total number of network device nodes.

[0092] In an optional embodiment, the reinforcement learning model constructed based on the dual-delay deep deterministic policy gradient algorithm includes: an Actor network, a Critic1 network, and a Critic2 network; randomly sampling a training data point from the experience replay pool to update the parameters of the reinforcement learning model constructed based on the dual-delay deep deterministic policy gradient algorithm includes:

[0093] The network link state matrix is ​​input into the Actor network to obtain the predicted network link state weight matrix;

[0094] The network link state matrix and the predicted network link state weight matrix are input into the Critic1 network and the Critic2 network, respectively, to obtain the Q value of the action corresponding to the network state;

[0095] The parameters of the Actor network, the Critic1 network, and the Critic2 network are updated based on the Q-value of the action corresponding to the network state.

[0096] In this embodiment, the Actor network, Critic1 network, and Critic2 network each consist of an online network and a target network, and the online network and the target network have the same neural network structure. Specifically, the Actor network is used to calculate the network link-state weight matrix. The Actor network consists of two layers of gated recurrent unit (GRU) networks and five layers of fully connected (FC) networks. The input of the Actor network is the network link-state matrix, and the output of the Actor network is the network link-state matrix composed of the weights of each link in the network.

[0097] The Critic1 and Critic2 networks are used to calculate the Q-values ​​of actions corresponding to network states. The Critic1 and Critic2 networks have the same structure, both consisting of two layers of Gated Recurrent Unit (GRU) networks and five layers of fully connected layers. The inputs of the Critic1 and Critic2 networks consist of two parts: the network link-state matrix, which is the same as the input of the Actor network, and the predicted network link-state weight moments output by the Actor network. The outputs of the Critic1 and Critic2 modules are the Q-values ​​of actions corresponding to the current state.

[0098] Specifically, the Q-value corresponding to the action of the network state is represented as:

[0099]

[0100] Where R represents the reward value of the network state, and S m+1 This represents the next network state, D is the network delay in the current network state, i = 1 and 2 represent Critic1 and Critic2 networks, and a TD3 This represents the action matrix in the current network state, where γ∈(0,1) is the discount factor.

[0101] In practical applications, to obtain a more accurate routing optimization model, a delayed update strategy is adopted when updating the parameters of the Actor network, Critic1 network, and Critic2 network in the routing optimization model. That is, the Actor network parameters are updated only once after the parameters of the Critic1 network and Critic2 network have been updated multiple times.

[0102] Figure 2The diagram illustrates the training process of the routing optimization model provided in this application embodiment. First, network feature information parameters and routing optimization model parameters are initialized based on the network state at time t. Then, network link state data under the network state at time t is monitored and processed into a network link state matrix at time t for storage. Next, a reinforcement learning model is constructed using the dual-delay deep deterministic policy gradient algorithm (TD3), and the network state in the routing optimization model is initialized based on the network latency and available bandwidth at time t (i.e., the network state of TD3 is initialized). Simultaneously, the network link state matrix at time t is input into the routing optimization model at this time to obtain... The network link state weight matrix at time t is obtained; based on the network link state weight matrix at time t and the shortest path algorithm, a shortest path is determined for packet forwarding, and the reward value R of the network state at time t is calculated; then, the network link state matrix at time t, the network link state weight matrix at time t, the reward value of the network state at time t, and the network link state matrix at time t+1 are used as training data and put into the experience replay pool of the routing optimization model. A training data is randomly sampled from the experience replay pool for training, and the Q value of the action corresponding to the network state is calculated. The parameters of the routing optimization model are updated using the Q value of the action corresponding to the network state.

[0103] In this embodiment, the routing optimization model obtains a network link state weight matrix based on the current network graph data, then determines the shortest route from the current network device node to the target network device node based on the network link state weight matrix, and transmits the data packet from the current network device node to the first network device node in the shortest route. Since the routing optimization model learns the rules between network device nodes in the network topology based on the graph data, and the reinforcement learning algorithm can achieve low network latency, it implements an adaptive routing strategy based on the network link state. This strategy can adapt to the dynamic and real-time requirements of network services, improve data transmission efficiency, select the optimal route in dynamic and complex environments, improve the overall network communication efficiency while ensuring global network load balancing, and has strong generalization ability for unknown network topologies.

[0104] Reference Figure 3 The diagram illustrates a structural schematic of an Internet routing optimization device based on a graph convolutional neural network, according to an embodiment of this application. Figure 3 As shown, the device includes:

[0105] The state weight module 31 is used to input the current network state network graph data into the pre-trained routing optimization model to obtain the network link state weight matrix;

[0106] Path determination module 32 is used to determine the shortest route from the current network device node to the target network device node based on the network link state weight matrix;

[0107] Data transmission module 33 is used to transmit data packets from the current network device node to the first network device node in the shortest routing path, wherein the first network device node is the next-hop network device node of the current network device node;

[0108] The routing optimization model is based on training data generated from graph data and trained using reinforcement learning. The device nodes in the network correspond to the nodes in the graph data, and the topological connections between network device nodes correspond to the edges between nodes in the graph data.

[0109] In an optional embodiment, the apparatus further includes a training data module for generating training data, the training data module comprising:

[0110] The first training data submodule is used to initialize network feature information parameters and routing optimization model parameters according to the network state at time t. The network feature information parameters include: network topology, available bandwidth, and routing configuration information.

[0111] The second training data submodule is used to obtain network link state data under the network state at time t, and process the link state data into a network link state matrix at time t.

[0112] The third training data submodule is used to input the network topology and the routing configuration information into the routing optimization model to obtain the network latency at time t.

[0113] The fourth training data submodule is used to input the network link state matrix at time t into the routing optimization model to obtain the network link state weight matrix at time t, and to initialize the network state in the routing optimization model according to the network latency at time t and the available bandwidth.

[0114] The fifth training data submodule is used to determine a shortest path for packet forwarding based on the network link state weight matrix at time t and the shortest path algorithm, and to calculate the reward value of the network state at time t.

[0115] The sixth training data submodule is used to take the network link state matrix at time t, the network link state weight matrix at time t, the reward value of the network state at time t, and the network link state matrix at time t+1 as a training data.

[0116] In an optional embodiment, the routing optimization model models the network data packet transmission process as a Markov decision process and constructs it based on a dual-delay deep deterministic policy gradient algorithm; the device further includes a training module for training the routing optimization model, the training module comprising:

[0117] The first training submodule is used to put each piece of training data into the experience replay pool of the route optimization model after each piece of training data is generated. The experience replay pool is used to store multiple pieces of training data of the route optimization model.

[0118] The second training submodule is used to randomly sample a training data point from the experience replay pool and update the parameters of the reinforcement learning model constructed based on the dual-delay deep deterministic policy gradient algorithm.

[0119] The third training submodule is used to perform multiple iterations of training according to the above steps, and obtain the trained routing optimization model when the training termination condition is met.

[0120] In an optional embodiment, modeling the transmission process of network data packets as a Markov decision process includes:

[0121] The network link state matrix is ​​used as the state of the Markov decision process;

[0122] The link state weight matrix output by the routing optimization model is used as the selection probability of the next node in the Markov decision process.

[0123] The reward value of the Markov decision process is calculated based on the available bandwidth of the network link and the packet forwarding delay time.

[0124] In an optional embodiment, the reinforcement learning model constructed based on the dual-delay deep deterministic policy gradient algorithm includes: an Actor network, a Critic1 network, and a Critic2 network; the second training submodule includes:

[0125] The first training unit is used to input the network link state matrix into the Actor network to obtain the predicted network link state weight matrix.

[0126] The second training unit is used to input the network link state matrix and the predicted network link state weight matrix into the Critic1 network and the Critic2 network respectively, and obtain the Q value of the action corresponding to the network state.

[0127] The third training unit is used to update the parameters of the Actor network, the Critic1 network, and the Critic2 network based on the Q-value of the action corresponding to the network state.

[0128] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes an Internet routing optimization method based on a graph convolutional neural network as described in this application.

[0129] This application also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the Internet routing optimization method based on graph convolutional neural networks described in this application.

[0130] This application also provides a computer program product, including computer-readable code, which, when executed, implements the Internet routing optimization method based on graph convolutional neural networks described in this application.

[0131] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0132] This application describes embodiments of methods and apparatus according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0133] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0134] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0135] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0136] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0137] The above provides a detailed description of the Internet routing optimization method and apparatus based on graph convolutional neural networks provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An Internet routing optimization method based on a graph convolutional neural network, characterized in that, The method comprises: inputting current network state network graph data into a pre-trained route optimization model to obtain a network link state weight matrix; determining a shortest route path from a current network device node to a target network device node according to the network link state weight matrix; transmitting a data packet from the current network device node to a first network device node in the shortest route path, the first network device node being a next-hop network device node of the current network device node. The route optimization model is trained based on training data generated based on graph data and based on reinforcement learning, wherein the device nodes in the network correspond to the nodes in the graph data, and the topology connections between the network device nodes correspond to the edges between the nodes in the graph data.

2. The method of claim 1, wherein, The training data is generated in the following manner: initializing network feature information parameters and route optimization model parameters according to a network state at time t, wherein the network feature information parameters include a network topology structure, available bandwidth, and route configuration information; obtaining network link state data under the network state at time t and processing the link state data into a network link state matrix at time t; inputting the network topology structure and the route configuration information into the route optimization model to obtain a network time delay at time t; inputting the network link state matrix at time t into the route optimization model to obtain a network link state weight matrix at time t, and initializing the network state in the route optimization model according to the network time delay at time t and the available bandwidth; determining a shortest path for data packet forwarding according to the network link state weight matrix at time t and a shortest path algorithm, and calculating a reward value of the network state at time t; taking the network link state matrix at time t, the network link state weight matrix at time t, the reward value of the network state at time t, and a network link state matrix at time t+1 as a piece of training data.

3. The method of claim 2, wherein, The route optimization model models the transmission process of network data packets as a Markov decision process and is constructed based on a double-delay deep deterministic policy gradient algorithm, and is trained in the following manner: after generating a piece of training data, the piece of training data is put into an experience replay pool of the route optimization model, and the experience replay pool is used to store multiple pieces of training data of the route optimization model; a piece of training data is randomly sampled from the experience replay pool, and a reinforcement learning model constructed based on a double-delay deep deterministic policy gradient algorithm is updated in parameters; the above steps are iteratively trained multiple times, and a trained route optimization model is obtained when a training end condition is met.

4. The method of claim 3, wherein, The modeling of the transmission process of network data packets as a Markov decision process comprises: taking a network link state matrix as a state of the Markov decision process; taking a link state weight matrix output by the route optimization model as a selection probability of a next node selected by the Markov decision process in action selection; calculating a reward value of the Markov decision process according to the available bandwidth of a network link and a data packet forwarding delay time.

5. The method of claim 3, wherein, The reinforcement learning model constructed based on the double-delay deep deterministic policy gradient algorithm comprises an Actor network, a Critic1 network and a Critic2 network; a piece of training data is randomly sampled from the experience replay pool, and the reinforcement learning model constructed based on the double-delay deep deterministic policy gradient algorithm is updated in parameters, comprising: inputting the network link state matrix into the Actor network to obtain a predicted network link state weight matrix; inputting the network link state matrix and the predicted network link state weight matrix into the Critic1 network and the Critic2 network respectively to obtain a Q value of an action corresponding to the network state; updating the Actor network, the Critic1 network and the Critic2 network in parameters based on the Q value of the action corresponding to the network state.

6. The method of claim 5, wherein, The Q value of the action corresponding to the network state is expressed as: where R represents the reward value of the network state, S m+1 represents the next network state, D is the network delay at the current network state, i = 1, 2 represents Critic1 network and Critic2 network, a TD3 represents the action matrix at the current network state, and γ ∈ (0, 1) is a discount factor.

7. An Internet route optimization apparatus based on a graph convolutional neural network, characterized by, The apparatus comprises: a state weight module configured to input current network state network graph data into a pre-trained route optimization model to obtain a network link state weight matrix; a path determination module configured to determine a shortest route path from a current network device node to a target network device node according to the network link state weight matrix; a data transmission module configured to transmit a data packet from the current network device node to a first network device node in the shortest route path, the first network device node being a next-hop network device node of the current network device node. The route optimization model is obtained based on training data generated based on graph data and based on reinforcement learning.

8. An electronic device, comprising: The computer program / instructions, when executed by the processor, implement the internet route optimization method based on the graph convolutional neural network according to any one of claims 1-6.

9. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, The computer program / instructions, when executed by the processor, implement the internet route optimization method based on the graph convolutional neural network according to any one of claims 1-6.

10. A computer program product comprising computer readable code, characterized in that, The computer program / instructions, when executed by the processor, implement the internet route optimization method based on the graph convolutional neural network according to any one of claims 1-6.

Images