Intelligent routing decision method and system based on graph neural network and proximal policy optimization

Abstract

The application discloses an intelligent routing decision method based on a graph neural network and a proximal policy optimization, and mainly solves the problems of complex routing decision process, insufficient local port state perception and difficulty in shielding illegal ports, and the method comprises the following steps: obtaining a communication network to be optimized and defining network global state information; calculating current node candidate port state information according to network topology, node distance and real-time port state; constructing a graph neural network strategy model and a value model, combining the candidate port state information and the network global state information into port-level state input, and training the strategy model by using the state value output by the value model and the network environment feedback reward; constructing a candidate port action mask during routing, shielding unusable ports, and selecting a next-hop forwarding port from a legal candidate port set by the trained strategy model, so as to realize routing optimization decision. The application improves the real-time performance, accuracy and executability of routing decision, and can be used for selecting a next hop of a communication network according to real-time network state.

Description

Technical Field

[0001] This invention belongs to the field of computer network and reinforcement learning technology, specifically relating to an intelligent routing decision method and system, which can be applied to packet switching networks, software-defined networks, data center networks, backbone networks, and other communication networks that need to select the next hop based on real-time network status. Background Technology

[0002] Network routing is the process of selecting forwarding paths for data packets while satisfying network connectivity and service forwarding constraints. In real-world communication networks, service traffic changes over time, and port queue lengths, link loads, and node congestion states also change dynamically. Traditional static routing or fixed-weight routing typically selects paths based on pre-defined link costs, making it difficult to reflect port congestion, sudden load changes, and local link state changes in a timely manner. This can easily lead to decreased delivery rates, increased end-to-end latency, and reduced effective throughput. To improve the adaptability of routing methods to dynamic network conditions, existing technologies have attempted to apply graph neural networks and reinforcement learning to intelligent routing.

[0003] Patent document CN113194034A discloses an "intelligent routing method for dynamic network environments." This method uses graph neural networks to extract network topology features and combines this with PPO reinforcement learning for modeling. It outputs the weights of network links through policy functions and calculates the minimum weight path that satisfies the constraints based on the link weights, thereby achieving network optimization. However, because its reinforcement learning action space is constructed based on link weights, this method cannot directly correspond to the actual data forwarding interfaces executed by network devices.

[0004] Patent document CN114051272A discloses a "Dynamic Topology Network Intelligent Routing Method," which uses a graph neural network to approximate the policy function and value function in the PPO algorithm. The policy function outputs the weights of all links in the network, and the network environment then calculates the minimum weight path satisfying the constraints based on these link weights. While this method can extract network topology features using graph neural networks and improve routing strategies through reinforcement learning, its policy model outputs the total link weights, requiring a further calculation of the complete path using a constrained shortest path algorithm. This results in a long decision chain and a relatively complex calculation process. Furthermore, since the model does not directly output the next-hop port of the current node, there is a conversion process between the policy output and the actual packet forwarding action. When network load changes rapidly or local ports become congested, the conversion result from link weights to the complete path may not reflect the queue status and busy / idle status of each candidate port of the current node in a timely manner, thus affecting the real-time performance and precision of routing decisions. Additionally, in hop-by-hop transitions… In this scenario, each time a data packet arrives at a current node, it is required to select a valid port from the candidate ports of that node as the next hop exit. However, the actual number of candidate ports may vary between different network nodes, and some ports may not be selected due to virtual padding, link failure, or unreachability constraints. A filtering and blocking mechanism is needed to ensure the selection of a valid port. This method, due to the lack of a direct decision-making mechanism for candidate ports and a mechanism for blocking invalid ports, may select an inoperable port, or, under local congestion, still select a port with a long queue and high utilization rate. This results in a decrease in data packet delivery rate, an increase in end-to-end latency, and a decrease in effective throughput, leading to a reduction in the granularity of the decision. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of the prior art by proposing an intelligent routing decision-making method and system based on graph neural networks and near-end policy optimization, so as to shorten the routing decision-making chain, simplify the routing decision-making process, and improve the real-time performance and precision of routing decisions.

[0006] The technical approach to achieving the objective of this invention is as follows: by directly mapping the relationship between network state and next-hop port action through the PPO policy network, the routing decision chain is shortened; by using the legal candidate ports of the current node as the reinforcement learning action space, the decision-making process is simplified and the precision of routing decision is improved by eliminating the need for additional full link search calculations; by introducing an action masking mechanism and constraining the action space before policy decision, invalid repeated exploration is avoided, thereby improving the real-time performance of routing decision in dynamic network environments.

[0007] Based on the above ideas, the technical solution of the present invention includes:

[0008] 1. A smart routing decision-making method based on graph neural networks and near-end policy optimization, characterized in that it includes:

[0009] (1) Obtain the communication network for which the route to be optimized, and define the global state information of the network, including: current service load.

[0010] Load mode and remaining forwardable hops (TTL);

[0011] (2) Calculate based on the topology information of the communication network, the distance information between nodes, and the real-time port status information.

[0012] Current node candidate port status information, including queue percentage Port utilization Normalized distance and normalized distance increment ;

[0013] (3) Construct graph neural network policies including port feature encoder, port feature aggregator, and policy outputter respectively.

[0014] The model includes a simplified model and a value model that includes a state feature extractor and a value estimator;

[0015] (4) Combine the state information of each candidate port of the current node with the known global state information of the network to form the port-level state input of the current node, and input them into the graph neural network strategy model and the value model respectively. Train the graph neural network strategy model according to the state value output by the value model and the environmental feedback reward information generated by the routing action in the network environment to obtain the trained graph neural network strategy model.

[0016] (5) Obtain the communication network status information of the route to be optimized in real time to construct the candidate port action mask, block unavailable candidate ports, and obtain a set of legal candidate ports;

[0017] (6) Deploy the trained policy model in the network environment, select the next-hop forwarding port from the set of legal candidate ports, form the routing forwarding path of the current data packet, and realize the routing optimization decision.

[0018] Furthermore, in step (4), the port-level state of the current node is input into the graph neural network policy model and the value model respectively, and the graph neural network policy model is trained, which includes:

[0019] (4a) Initialize the graph neural network policy model and value model, and set the training hyperparameters to obtain the initial policy model and initial value model;

[0020] (4b) Interact with the current strategy model and the network simulation environment to collect a complete forwarding trajectory, which includes the port-level state input corresponding to each forwarding, the selected target port, action probability, action mask, forwarding reward and trajectory termination result;

[0021] (4c) Calculate the cumulative reward for each forward based on the reward for each forward in the complete forwarding trajectory;

[0022] (4d) Input the port-level state corresponding to each forwarding into the value model to obtain the state value, and calculate the advantage function based on the cumulative reward and state value;

[0023] (4e) Under the same action mask constraint, calculate the action probability of the current policy model and the policy model to be updated selecting the same target port, and calculate the probability ratio of the new and old policies;

[0024] (4f) Calculate the PPO pruning target based on the dominance function and the probability ratio of the new and old strategies;

[0025] (4g) Calculate the value loss based on the error between the state value and the cumulative return;

[0026] (4h) Calculate the policy entropy based on the action probability distribution of the legitimate candidate ports;

[0027] (4i) Calculate the joint training loss based on the PPO pruning target, value loss and policy entropy, and update the parameters of the graph neural network policy model and value model based on the joint training loss;

[0028] (4j) Repeat (4b) to (4i) until the preset training conditions are met, and the trained graph neural network policy model is obtained.

[0029] Furthermore, the process of obtaining the communication network status information of the route to be optimized in real time to construct the candidate port action mask in step (5) includes:

[0030] (5a) Determine the set of executable candidate ports based on the available status information of each candidate port of the current node;

[0031] (5b) Generate an action mask based on the set of executable candidate ports, mark candidate ports that meet the forwarding constraints as valid actions, and mark candidate ports that do not meet the forwarding constraints as invalid actions, so as to block unavailable candidate ports and limit the action selection range of the policy model.

[0032] 2. An intelligent routing decision system based on graph neural networks and near-end policy optimization, characterized in that it comprises:

[0033] The network status acquisition module is used to acquire the communication network status information of the route to be optimized, including the port status information of each candidate port of the current node, network topology information, destination node information, and global network status information.

[0034] A port-level status input construction module is used to construct the port-level status input corresponding to the current node based on the communication network status information.

[0035] The action mask generation module is used to generate action masks for candidate ports based on whether the candidate port actually exists, whether the corresponding link is valid, and whether the forwarding constraints from the current node to the destination node are met, and to block unavailable candidate ports.

[0036] The graph neural network policy model module is used to output the action probability corresponding to each legal candidate port of the current node based on the port-level state input and the candidate port action mask;

[0037] The graph neural network value model module is used to output the state value corresponding to the current state based on the port-level state input;

[0038] The near-end strategy optimization training module is used to train based on the complete forwarding trajectory and cumulative reward. State value Advantage function Probability ratio between new and old strategies PPO trimming target Value loss and policy entropy The parameters of the graph neural network strategy model module and value model module are trained and updated.

[0039] The routing decision execution module is used to select a next-hop forwarding port from the legitimate candidate ports under the action mask constraint using the trained graph neural network policy model module, and forward the data packet through the next-hop forwarding port.

[0040] Compared with the prior art, the present invention has the following advantages:

[0041] Firstly, this invention uses the candidate ports of the current node as the routing decision object. The graph neural network strategy model directly outputs the action probability corresponding to each candidate port and selects the next-hop forwarding port from the set of legal candidate ports. Therefore, the model output can directly correspond to the next-hop port selection actually performed by the current node, avoiding the intermediate process in the prior art of first outputting the entire network link weight, then calculating the complete path, and then converting it into the forwarding action of the current node. This shortens the routing decision chain, simplifies the calculation process, and improves the real-time performance of routing decisions.

[0042] Secondly, because the present invention uses the queue ratio, port utilization, normalized distance, normalized distance increment, and global network state information of each candidate port of the current node together for routing decision when constructing port-level state input, the graph neural network strategy model can simultaneously refer to port congestion status, port busy / idle status, destination node direction information, and service load changes when selecting the next-hop forwarding port. This solves the problem that existing technologies cannot reflect the local port status of the current node in a timely manner, thereby improving the precision of routing decision and the adaptability to dynamic network states.

[0043] Third, by introducing an action masking mechanism, this invention masks unavailable candidate ports based on whether the candidate port actually exists, whether the corresponding link is valid, and whether the forwarding constraints are met. This allows the graph neural network strategy model to select ports only from the set of legal candidate ports, thus avoiding the selection of virtual ports, failed link ports, or ports that do not meet reachability constraints. This solves the problem that existing technologies may output unexecutable routing actions due to the lack of an illegal port masking mechanism, thereby improving the executability of routing actions.

[0044] Fourth, this invention employs a near-end policy optimization algorithm to train the graph neural network policy model and value model, calculates the cumulative reward, state value, and advantage function based on the complete forwarding trajectory, and limits the magnitude of a single policy update by pruning targets through PPO. Therefore, it can maintain the stability of the training process and reduce the routing performance fluctuations caused by excessively rapid policy updates while enhancing the port selection tendency of high-advantage targets and suppressing the port selection tendency of low-advantage targets.

[0045] Fifth, because the present invention adopts a hop-by-hop port selection method, it reacquires network status information when the data packet arrives at each current node and selects the next-hop forwarding port based on the real-time port-level status input and action mask, instead of generating a fixed complete path all at once. Therefore, when there are sudden changes in network load, local port congestion, or changes in port busy / idle status, it can adjust subsequent forwarding decisions in a timely manner, reduce end-to-end latency, and improve data packet delivery rate and effective throughput. Attached Figure Description

[0046] Figure 1 This is the overall flowchart of the intelligent routing decision-making method based on graph neural networks and near-end policy optimization of the present invention;

[0047] Figure 2 This is a sub-flowchart of the training of the graph neural network strategy model and value model in the method of this invention;

[0048] Figure 3 This is a block diagram of the intelligent routing decision system based on graph neural networks and near-end policy optimization of the present invention;

[0049] Figure 4This is a comparison chart of packet delivery rates when using the GNN model of this invention and the existing PPO_ONLY model for routing and forwarding under different service load scenarios;

[0050] Figure 5 This is a comparison chart of the 95th percentile latency of the GNN model using the method of this invention and the existing PPO_ONLY model when performing routing and forwarding under different service load scenarios. Detailed Implementation

[0051] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention and not all embodiments. Based on the embodiments of the present invention, other embodiments obtained by those skilled in the art without creative effort should all fall within the protection scope of the present invention.

[0052] Example 1: Intelligent Routing Decision Method Based on Graph Neural Networks and Proximity Policy Optimization

[0053] This embodiment provides an intelligent routing decision-making method that takes the candidate ports of the current node as the decision object, integrates the local state of the port, the overall state of the candidate port set, and the global state of the network, and directly outputs the next-hop port selection result under the constraint of action mask.

[0054] Reference Figure 1 The implementation steps of this embodiment include the following:

[0055] Step 1: Obtain the communication network of the route to be optimized and define the global network status information.

[0056] The communication network for which the route to be optimized is obtained, which includes multiple network nodes and multiple links connecting the network nodes. Here, a network node represents a routing and forwarding device in the communication network, and a port represents the link exit point between the current network node and its neighboring nodes.

[0057] Based on the preset service traffic model, a data packet to be forwarded is generated between the source node and the destination node, and the global network status information is defined as including: the current service load mode and the remaining forwardable hops (TTL) of the data packet.

[0058] Step 2: Calculate the status information of the candidate ports of the current node.

[0059] When a data packet arrives at the current node but has not yet reached the destination node, the candidate port set of the current node is first determined based on the communication network topology information.

[0060] Then, for each candidate port of the current node, based on network topology information, inter-node distance information, and real-time port status information, calculate the status information of the candidate ports, including queue percentage. Port utilization Normalized distance and normalized distance increment The formula is as follows:

[0061] ;

[0062] ;

[0063] ;

[0064] ;

[0065] in, This indicates the number of packets to be forwarded in the buffer queue corresponding to the i-th candidate port.

[0066] This represents the cache capacity of the i-th candidate port.

[0067] Indicates the first The adjacent nodes connected to each candidate port.

[0068] Indicates the destination node.

[0069] Indicates adjacent nodes To the destination node The topological shortest hop distance,

[0070] This represents the maximum shortest hop distance or a preset distance normalization constant in the network topology.

[0071] Indicates the first The current node where the data packet is located during the next forwarding.

[0072] Indicates the current node To the destination node The shortest hop distance in the topology.

[0073] Utilization The value of is when the th The value is 1 when the candidate port is sending a data packet or its sending resources are occupied; when the candidate port is sending a data packet, the value is 1. When a candidate port has not sent any data packets and is in an idle state, the value is 0.

[0074] Step 3: Construct a graph neural network strategy model and a value model.

[0075] A port feature encoder, a port feature aggregator, and a policy outputter are selected, cascaded, and the output of the port feature encoder is connected to the input of the policy outputter to form a graph neural network policy model, which is used to input and output the action probability corresponding to the candidate port based on the port-level state of the current node.

[0076] The port feature encoder is used to process the state information of each candidate port of the current node using shared parameters, and convert the state information of each candidate port into a port embedding representation. :

[0077] ,

[0078] in, Indicates the first Port characteristics of candidate ports, This represents the mapping function corresponding to the port feature encoder. This represents the encoding parameters shared by all candidate ports;

[0079] The port feature aggregator is used to aggregate the port embedding representations of each candidate port of the current node to obtain a context representation that characterizes the overall state of the candidate port set of the current node. :

[0080] ,

[0081] in, This indicates the maximum number of candidate ports preset.

[0082] The policy outputter is used to embed the port of each candidate port into its own representation. Context representation of candidate port set and global network status information By merging, we obtain the first... Action score of candidate ports :

[0083] ,

[0084] in, Indicates the first The fused feature vectors corresponding to each candidate port This represents the mapping function corresponding to the policy outputter; This represents the trainable parameters of the policy output.

[0085] A value model is constructed by cascading a state feature extractor and a value estimator, which is used to input and output state value based on the port-level state of the current node.

[0086] The state feature extractor is used to receive the context representation of the candidate port set. and network global status information and the context representation With network global status information By fusing the data, we obtain the value feature vector: And transmit its value to the value estimator;

[0087] The value estimator is used to estimate the value feature vector. Calculate the current port-level state input Corresponding state value: ,in, This represents the mapping function corresponding to the value estimation module. This represents the trainable parameters of the value model.

[0088] Step 4: Train the graph neural network policy model and value model.

[0089] The state information of each candidate port of the current node is combined with the known global state information of the network to form the port-level state input of the current node. This input is then fed into the graph neural network policy model and the value model. The graph neural network policy model is trained based on the state value output by the value model and the environmental feedback reward information generated by the routing action in the network environment.

[0090] Reference Figure 2 This step includes the following:

[0091] 4.1) Randomly initialize the weight parameters of the graph neural network policy model and value model, initialize the bias parameters to zero, and set the training hyperparameters, including the learning rate of the policy model. Learning rate of the value model Reward Discount Factor PPO limiting factor Strategy entropy weights and maximum number of iterations Thus, the initial strategy model and the initial value model are obtained;

[0092] 4.2) Interact with the network simulation environment using the current strategy model to collect the complete forwarding trajectory, which includes: port-level state inputs corresponding to each forwarding. Selected target port The current graph neural network strategy model selects the target port. action probability Or logarithmic action probability, action mask Forwarding Rewards And the trajectory termination result, including the forwarding reward. , This indicates the percentage of the port queue for the target port. This represents the port utilization rate of the target port. This represents the distance increment to the destination node caused by forwarding through the target port. These represent the weights corresponding to the port queue percentage, port utilization, and distance increment, respectively.

[0093] 4.3) Based on the rewards for each forward in the complete forwarding trajectory, calculate the cumulative reward for each forward in reverse chronological order. :

[0094] ,

[0095] 4.4) The first Port-level status input for forwarding Inputting into the value model yields the state value. And calculate the advantage function based on the cumulative returns and state value:

[0096] ,

[0097] in, Indicates the first The advantage corresponding to each forwarding is used to characterize the actual forwarding effect of the selected target port relative to the value model prediction result;

[0098] 4.5) Under the same action mask constraint corresponding to the t-th forward, obtain the target port selection of the current model's graph neural network policy model and the graph neural network policy model to be updated. The probability of the action is calculated, and the probability ratio between the old and new strategies is calculated. :

[0099] ,

[0100] in, This represents the graph neural network policy model to be updated. This represents the current graph neural network policy model;

[0101] 4.6) Based on the aforementioned advantage function Probability ratio between old and new strategies Construct PPO trimming goals :

[0102] ,

[0103] in, Represents the amplitude limiting function;

[0104] 4.7) Based on state value With cumulative returns The error between them creates value loss And calculate the policy entropy based on the action probability distribution of the legitimate candidate ports. :

[0105] ,

[0106] ,

[0107] in, Indicates the first The probability of actions corresponding to each valid candidate port. This indicates the number of candidate ports.

[0108] 4.8) Optimize the pruning target based on PPO Value loss and policy entropy Constructing joint training loss :

[0109] ;

[0110] 4.9) Calculate the joint training loss Graph Neural Network Policy Model Parameters gradient and parameters of the value model gradient The adjustment direction and magnitude of each parameter are determined based on the gradient, and the parameters are updated according to the preset learning rate to reduce the joint training loss.

[0111] ,

[0112] ,

[0113] 4.10) Repeat steps 4.2) to 4.9) until the maximum number of iterations is reached. This yields the trained graph neural network policy model.

[0114] Step 5: Construct candidate port action masks.

[0115] 5.1) Obtain real-time communication network status information for the route to be optimized;

[0116] 5.2) Based on the communication network status information, determine the availability of each candidate port in the current node's candidate port set one by one to obtain the action mask of the candidate port:

[0117] If the candidate port is a real port, its corresponding link is in a valid connection state, and the forwarding through the candidate port can satisfy the forwarding constraints from the current node to the destination node, then the candidate port is marked as a valid action.

[0118] If the candidate port is a virtual filler port, its corresponding link is invalid, or it does not meet the forwarding constraints to the destination node, then the candidate port will be blocked and marked as an invalid action.

[0119] By masking the candidate ports corresponding to invalid actions using the action mask, the graph neural network policy model can normalize the action probability only for the legal candidate ports corresponding to valid actions, and select the next-hop forwarding port from the set of legal candidate ports.

[0120] Step 6, online routing decision.

[0121] The trained graph neural network policy model is deployed in the network environment. When a data packet arrives at the current node, the next-hop forwarding port is selected from the set of legal candidate ports based on the real-time network status and action mask. The routing path for the current data packet is formed through hop-by-hop forwarding, thus realizing routing optimization decision-making. Its implementation includes:

[0122] 6.1) Calculate the status information of the candidate ports of the current node using the topology information of the communication network, the distance information between nodes, and the real-time status information of the ports;

[0123] 6.2) Input the state information and the global state information of the network into the trained graph neural network policy model to obtain the action selection probability corresponding to each candidate port;

[0124] 6.3) Based on the action selection probability, select the next-hop forwarding port from the set of legal candidate ports after filtering by the action mask, form the routing forwarding path of the current data packet, and realize the routing decision.

[0125] It should be noted that the step numbers in the specification and claims of this invention are only for the purpose of clearly describing the embodiments of this invention and facilitating understanding, and their order is not limited.

[0126] Example 2: Intelligent Routing Decision System Based on Graph Neural Networks and Proximal Policy Optimization

[0127] Reference Figure 3This embodiment includes a network state acquisition module 1, a port-level state input construction module 2, an action mask generation module 3, a graph neural network policy model module 4, a value model module 5, a near-end policy optimization training module 6, and a routing decision execution module 7. The graph neural network policy model module 4 includes a port feature encoding submodule 41, a port feature aggregation submodule 42, and a policy output submodule 43; the value model module 5 includes a state feature extraction submodule 51 and a value estimation submodule 52; and the near-end policy optimization training module 6 includes a trajectory acquisition submodule 61, a reward and advantage calculation submodule 62, a policy probability ratio calculation submodule 63, a joint loss construction submodule 64, and a parameter update submodule 65.

[0128] The working principle of the entire system is as follows:

[0129] The network status acquisition module 1 is used to acquire the communication network status information of the route to be optimized, including the port status information of each candidate port of the current node, network topology information, destination node information and global network status information, and input the information to the port-level status input construction module 2 and the action mask generation module 3 respectively.

[0130] The port-level state input construction module 2 is used to construct the port-level state input corresponding to the current node based on the communication network state information, so that the local state of each candidate port of the current node and the global state of the network can be uniformly input into the model, and the obtained port-level state input is respectively input into the graph neural network strategy model module 4 and the value model module 5.

[0131] The action mask generation module 3 is used to generate a candidate port action mask based on whether the candidate port actually exists, whether the corresponding link is valid, and whether the forwarding constraint from the current node to the destination node is met. It also masks unusable candidate ports and sends the action mask to the graph neural network strategy model module 4 and the routing decision execution module 7.

[0132] The graph neural network policy model module 4 is used to output the action probability corresponding to each legal candidate port of the current node based on the port-level state input and action mask. Specifically, the port feature encoding submodule 41 encodes the port features of each candidate port of the current node to obtain the port embedding representation corresponding to each candidate port, which is then input to the port feature aggregation submodule 42 and the policy output submodule 43, respectively. The port feature aggregation submodule 42 aggregates the port embedding representations of each candidate port to obtain a context representation representing the overall state of the candidate port set of the current node, which is then input to the policy output submodule 43. The policy output submodule 43 fuses the port embedding representations of each candidate port, the context representation of the candidate port set, and the global network state information, and outputs the action probability corresponding to each legal candidate port under the action mask constraint, which is then input to the near-end policy optimization training module 6 and the routing decision execution module 7, respectively.

[0133] The value model module 5 is used to output the state value corresponding to the current state based on the port-level state input, providing state evaluation for the near-end policy optimization training process. The state feature extraction submodule 51 is used to extract value features based on the context representation of the candidate port set and the global state information of the network, and input them into the value estimation submodule 52. The value estimation submodule 52 is used to output the state value corresponding to the current port-level state input to the near-end policy optimization training module 6 based on the value features.

[0134] The near-end policy optimization training module 6 is used to feed back its output parameter update results to the graph neural network policy model module 4 and the value model module 5, respectively, based on the complete forwarding trajectory, cumulative reward, state value, advantage function, probability ratio of new and old policies, PPO pruning target, value loss, and policy entropy, to train and update the parameters of the graph neural network policy model module 4 and the value model module 5. The trajectory acquisition submodule 61 is used to interact with the network simulation environment using the current policy model to acquire the complete forwarding trajectory and input it to the reward and advantage calculation submodule 62. This reward and advantage calculation submodule 62 is used to calculate the cumulative reward based on the forwarding reward in the complete forwarding trajectory and to calculate the advantage function based on the cumulative reward and state value. The potential function is input to the policy probability ratio calculation submodule 63. This submodule 63 calculates the new and old policy probability ratios based on the target port action probabilities output by the current graph neural network policy model and the graph neural network policy model to be updated, under the same action mask constraint. This ratio is then input to the joint loss construction submodule 64. This submodule 64 constructs a joint training loss based on the advantage function, the new and old policy probability ratios, the PPO pruning target, the value loss, and the policy entropy, and inputs it to the parameter update submodule 65. This parameter update submodule 65 calculates the gradients of the parameters of the graph neural network policy model module 4 and the value model module 5 based on the joint training loss, and updates the parameters according to a preset learning rate.

[0135] The routing decision execution module 7 is used to receive the action probability output by the trained graph neural network policy model module 4 and the action mask output by the action mask generation module 3, select the next-hop forwarding port from the set of legal candidate ports, and forward the data packet through the selected next-hop forwarding port.

[0136] It should be noted that the above functional modules can be implemented, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, they can be implemented, in whole or in part, as program instruction products. A program instruction product includes one or a set of program instructions. When the program instructions are loaded and executed on a computer, the described process or function is generated, in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The program instructions can be stored in a computer-readable and writable storage medium, or transferred from one computer's readable and writable storage medium to another.

[0137] In this embodiment, the direct coupling or communication connection between the modules can be achieved through indirect coupling or communication connection via interfaces, devices, or modules. The functional modules and sub-modules in this embodiment can dynamically reside within a single processing unit, or each module can exist physically independently, or two or more modules can dynamically reside within a single processing unit. When these dynamic components are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable and writable storage medium. This storage medium can be a memory, disk, or optical disc, etc.

[0138] The effectiveness of this invention can be further illustrated by the following simulation results.

[0139] I. Simulation Conditions

[0140] The communication network topology is configured with 14 network nodes and 22 bidirectional links, and based on preset services...

[0141] The traffic model sets up three service load scenarios: light load, medium load, and heavy load. Data packets to be forwarded are generated between the source node and the destination node. For each service load scenario, a static service mode and a burst service mode are set up respectively.

[0142] II. Simulation Content

[0143] Simulation 1: Under the above simulation conditions, the method of this invention and the existing intelligent system that only uses the PPO strategy model are compared respectively.

[0144] The routing method compares packet delivery rates during routing and forwarding, and the results are as follows: Figure 4 As shown.

[0145] from Figure 4 As can be seen, under light, medium, and heavy load scenarios, the method of this invention performs well in static service models.

[0146] The packet delivery rate under both the burst and burst service modes is close to or reaches 1.00; while the intelligent routing method using only the PPO strategy model has a packet delivery rate of approximately 0.93 to 0.94 in the corresponding scenarios. Compared with existing technologies, this invention can maintain a higher packet delivery rate under different service loads and service change modes, indicating that this invention can effectively improve the ability of packets to successfully reach the destination node through port-level state input, graph neural network strategy model, and action mask constraints.

[0147] Simulation 2: Under the above simulation conditions, the present invention and the existing intelligent routing system that only adopts the PPO strategy model are respectively tested.

[0148] The method compares the 95th percentile latency during routing, and the results are as follows: Figure 5 As shown.

[0149] from Figure 5 As can be seen, under light, medium, and heavy load scenarios, the 95th percentile latency of the method of this invention is [data missing].

[0150] The latency is approximately 71.63ms to 80.28ms; while the 95th percentile latency of the intelligent routing method using only the PPO strategy model is 238.65ms to 263.19ms. Compared with existing technologies, the method of this invention significantly reduces high percentile latency under various load scenarios. This indicates that the present invention can make port-level routing decisions based on the queue ratio of candidate ports of the current node, port utilization, normalized distance, and normalized distance increment, reducing the waiting time of data packets on congested ports and improving the real-time performance of routing decisions.

[0151] The simulation results above show that, compared with the intelligent routing method that only uses the PPO strategy model, the present invention can achieve [the desired results].

[0152] It maintains a higher data packet delivery rate under different service load scenarios and significantly reduces 95th percentile latency, thereby improving routing stability and forwarding efficiency in dynamic communication networks.

Claims

1. A smart routing decision-making method based on graph neural networks and near-end policy optimization, characterized in that, include: (1) Obtain the communication network for which the route to be optimized, and define the global state information of the network, including: the current service load mode and the remaining forwardable hops (TTL) of the data packets; (2) Based on the topology information of the communication network, the distance information between nodes, and the real-time status information of the ports, calculate the status information of the candidate ports of the current node, including the queue percentage. Port utilization Normalized distance and normalized distance increment ; (3) Construct a graph neural network policy model including a port feature encoder, a port feature aggregator, and a policy outputter, and a value model including a state feature extractor and a value estimator, respectively; (4) Combine the state information of each candidate port of the current node with the known global state information of the network to form the port-level state input of the current node, and input them into the graph neural network strategy model and the value model respectively. Then, train the graph neural network strategy model according to the state value output by the value model and the environmental feedback reward information generated by the routing action in the network environment to obtain the trained graph neural network strategy model. (5) Obtain the communication network status information of the route to be optimized in real time to construct the candidate port action mask, block unavailable candidate ports, and obtain a set of legal candidate ports; (6) Deploy the trained strategy model in the network environment, select the next-hop forwarding port from the set of legal candidate ports, form the routing forwarding path of the current data packet, and realize the routing optimization decision.

2. The method according to claim 1, characterized in that, In step (2), the status information of the candidate ports of the current node is calculated based on the network topology information, the distance information between nodes, and the real-time status information of the ports. The formula is as follows: Queue percentage: , Normalized distance: , Normalized distance increment: , Port utilization: , in, This indicates the number of packets to be forwarded in the buffer queue corresponding to the i-th candidate port. This represents the cache capacity of the i-th candidate port. Indicates the first The adjacent nodes connected to each candidate port. Indicates the destination node. Indicates adjacent nodes To the destination node The topological shortest hop distance, This represents the maximum shortest hop distance or a preset distance normalization constant in the network topology. Indicates the first The current node where the data packet is located during the next forwarding. Indicates the current node To the destination node The topological shortest hop distance, Utilization The value of is when the th The value is 1 when the candidate port is sending a data packet or its sending resources are occupied; when the candidate port is sending a data packet, the value is 1. When a candidate port has not sent any data packets and is in an idle state, the value is 0.

3. The method according to claim 1, characterized in that, The functions and connections of the port feature encoder, port feature aggregator, and policy outputter in (3) are as follows: The port feature encoder is used to process the state information of each candidate port of the current node using shared parameters, and convert the state information of each candidate port into a port embedding representation. : ， in, Indicates the first Port characteristics of candidate ports, This represents the mapping function corresponding to the port feature encoder. This represents the encoding parameters shared by all candidate ports; The port feature aggregator is used to aggregate the port embedding representations of each candidate port of the current node to obtain a context representation that characterizes the overall state of the candidate port set of the current node. : ， in, This indicates the maximum number of candidate ports preset. The policy outputter is used to embed the port of each candidate port into its own representation. Context representation of candidate port set and global network status information By merging, we obtain the first... Action score of candidate ports : ， in, Indicates the first The fused feature vectors corresponding to each candidate port This represents the mapping function corresponding to the policy outputter; Represents the trainable parameters of the policy output; The port feature encoder, port feature aggregator, and policy outputter mentioned above are cascaded to form a graph neural network policy model.

4. The method according to claim 1, characterized in that, The state feature extractor and value estimator in (3) have the following functions and transmission relationships: The state feature extractor is used to receive the context representation of the candidate port set. and network global status information and the context representation With network global status information By fusing the data, we obtain the value feature vector: And transmit its value to the value estimator; The value estimator is used to estimate the value feature vector. Calculate the current port-level state input Corresponding state value: ,in, This represents the mapping function corresponding to the value estimation module. This represents the trainable parameters of the value model.

5. The method according to claim 1, characterized in that, In step (4), the port-level state of the current node is input into the graph neural network policy model and the value model respectively, and the graph neural network policy model is trained, which includes: (4a) Initialize the graph neural network policy model and value model, and set the training hyperparameters to obtain the initial policy model and initial value model; (4b) Interact with the current strategy model and the network simulation environment to collect a complete forwarding trajectory, which includes the port-level state input corresponding to each forwarding, the selected target port, action probability, action mask, forwarding reward and trajectory termination result; (4c) Calculate the cumulative reward for each forward based on the reward for each forward in the complete forwarding trajectory; (4d) Input the port-level state corresponding to each forwarding into the value model to obtain the state value, and calculate the advantage function based on the cumulative reward and state value; (4e) Under the same action mask constraint, calculate the action probability of the current policy model and the policy model to be updated selecting the same target port, and calculate the probability ratio of the new and old policies; (4f) Calculate the PPO pruning target based on the dominance function and the probability ratio of the new and old strategies; (4g) Calculate the value loss based on the error between the state value and the cumulative return; (4h) Calculate the policy entropy based on the action probability distribution of the legitimate candidate ports; (4i) Calculate the joint training loss based on the PPO pruning target, value loss and policy entropy, and update the parameters of the graph neural network policy model and value model based on the joint training loss; (4j) Repeat (4b) to (4i) until the preset training conditions are met, and the trained graph neural network policy model is obtained.

6. The method according to claim 1, characterized in that, The step (5) involves obtaining the communication network status information of the route to be optimized in real time to construct a candidate port action mask, which includes: (5a) Determine the set of executable candidate ports based on the available status information of each candidate port of the current node; (5b) Generate an action mask based on the set of executable candidate ports and block unavailable candidate ports, wherein candidate ports that satisfy the forwarding constraints are marked as valid actions and candidate ports that do not satisfy the forwarding constraints are marked as invalid actions, so as to limit the action selection range of the policy model.

7. The method according to claim 1, characterized in that, In step (6), the trained policy model is deployed in the network environment, and a next-hop forwarding port is selected from the set of legal candidate ports, including: (6a) Calculate the status information of the candidate ports of the current node using the topology information of the communication network, the distance information between nodes, and the real-time status information of the ports; (6b) Input the state information and the global state information of the network into the trained graph neural policy model to obtain the action selection probability corresponding to each candidate port; (6c) Based on the action selection probability, select the next-hop forwarding port from the set of legal candidate ports after action mask filtering to form the routing forwarding path of the current data packet and realize routing decision.

8. An intelligent routing decision system based on graph neural networks and near-end policy optimization, characterized in that, include: The network status acquisition module is used to acquire the communication network status information of the route to be optimized, including the port status information of each candidate port of the current node, network topology information, destination node information, and global network status information. A port-level status input construction module is used to construct the port-level status input corresponding to the current node based on the communication network status information. The action mask generation module is used to generate action masks for candidate ports based on whether the candidate port actually exists, whether the corresponding link is valid, and whether the forwarding constraints from the current node to the destination node are met, and to block unusable candidate ports. The graph neural network policy model module is used to output the action probability corresponding to each legal candidate port of the current node based on the port-level state input and the candidate port action mask. The graph neural network value model module is used to output the state value corresponding to the current state based on the port-level state input; The near-end strategy optimization training module is used to train based on the complete forwarding trajectory and cumulative reward. State value Advantage function Probability ratio between new and old strategies PPO trimming target Value loss and policy entropy The parameters of the graph neural network strategy model module and value model module are trained and updated. The routing decision execution module is used to select a next-hop forwarding port from the legal candidate ports under the action mask constraint using the trained graph neural network policy model module, and forward the data packet through the next-hop forwarding port.

9. The system according to claim 8, characterized in that, The graph neural network strategy model module includes: The port feature encoding submodule is used to encode the port features of each candidate port of the current node to obtain the port embedding representation corresponding to each candidate port. ; The port feature aggregation submodule is used to represent the port embeddings of each candidate port. Aggregation is performed to obtain a context representation that characterizes the overall state of the current node's candidate port set. ; The strategy output submodule is used to embed the port representation of each candidate port. Contextual representation and network global status information Perform fusion and in the action mask Output the action probability corresponding to each legal candidate port under constraints. .

10. The system according to claim 8, characterized in that, The near-end policy optimization training module includes: The trajectory acquisition submodule is used to interact with the network simulation environment using the current policy model to acquire complete forwarding trajectories; The reward and advantage calculation submodule is used to calculate the forwarding reward based on the complete forwarding trajectory. Calculate cumulative return And based on cumulative returns The state value output by the value model Calculate the advantage function ; The policy probability ratio calculation submodule is used to calculate the probability ratio based on the current graph neural network policy model under the same action mask constraint. and the graph neural network policy model to be updated Output target port action probability Calculate the probability ratio of the new and old strategies. ; The joint loss construction submodule is used to construct the loss function based on the advantage function. Probability ratio between new and old strategies PPO trimming target Value loss and policy entropy Constructing joint training loss ; The parameter update submodule is used to update the parameters based on the joint training loss. Gradient calculations are performed on the parameters of the graph neural network policy model module and value model module, and the learning rate is set according to the preset policy model. and the learning rate of the preset value model Update the parameters.

Images