Dynamic routing optimization method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a dynamic routing optimization method and device, computer equipment, a readable storage medium and a program product. The method comprises the following steps: acquiring a topological space feature vector and a predicted time sequence feature according to a node load and a link time delay of a network topology, fusing the topological space feature vector and the predicted time sequence feature to obtain a target fusion feature, and according to the target fusion feature, obtaining a target fusion feature; determining a target routing action from all routing actions in the routing action space, executing a traffic forwarding operation corresponding to the target routing action, obtaining a network performance parameter after the traffic forwarding operation is executed, obtaining a target fusion reward value based on the network performance parameter and the state prediction parameter, and sending the target fusion reward value to the routing action space; and updating model parameters of the decision network model according to the target fusion reward value, and performing dynamic routing optimization based on the updated decision network model. By adopting the method, the communication reliability can be improved.

Description

Technical Field

[0001] This application relates to the field of network communication technology, and in particular to a dynamic routing optimization method, apparatus, computer equipment, readable storage medium, and program product. Background Technology

[0002] With the rapid development of technologies such as cloud computing and the Internet of Things, the frequent access and exit of mobile nodes in the network has become the norm, causing the network topology to be in a state of continuous dynamic adjustment. Furthermore, due to the drastic fluctuations in traffic, sudden traffic surges are easily triggered, posing a severe challenge to the network's ability to dynamically allocate bandwidth resources.

[0003] Traditional methods typically rely on pre-configured fixed rules for path selection. In scenarios with stable network topology and balanced traffic distribution, basic routing functions can be achieved. However, when faced with dynamic scenarios such as sudden traffic surges, fixed shortest path strategies can cause a large amount of data traffic to concentrate on a few critical links, which can easily lead to link congestion and fail to achieve long-term stable routing optimization effects, thus affecting communication reliability. Summary of the Invention

[0004] Therefore, it is necessary to provide a dynamic routing optimization method, apparatus, computer equipment, readable storage medium, and program product that can improve communication reliability in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a dynamic routing optimization method, including:

[0006] Based on the node load and link latency of the network topology, the topology spatial feature vector and the predicted time series feature are obtained, and the topology spatial feature vector and the predicted time series feature are fused to obtain the target fused feature.

[0007] Based on the target fusion characteristics, the target routing action is determined from all routing actions in the routing action space, and the traffic forwarding operation corresponding to the target routing action is executed; the routing action space is determined according to routing requirements.

[0008] Obtain network performance parameters after performing traffic forwarding operations, and obtain the target fusion reward value based on network performance parameters and state prediction parameters;

[0009] The model parameters of the decision network model are updated based on the target fusion reward value, and dynamic route optimization is performed based on the updated decision network model.

[0010] In one embodiment, the step of determining the target routing action from all routing actions in the routing action space based on the target fusion features includes:

[0011] The target fusion features are input into the decision network model to obtain the action value corresponding to each routing action in the routing action space.

[0012] Determine the probability threshold, and based on the probability threshold and action value, determine the target routing action from all routing actions.

[0013] In one embodiment, the step of obtaining the target fusion reward value based on network performance parameters and state prediction parameters includes:

[0014] Based on network performance parameters, obtain the instant reward value, and based on state prediction parameters, obtain the predicted reward value.

[0015] Obtain the network fluctuation coefficient based on network performance parameters;

[0016] When the network fluctuation coefficient is less than the fluctuation threshold, the immediate reward value and the predicted reward value are weighted and summed according to the first weight coefficient to obtain the target fusion reward value;

[0017] If the network fluctuation coefficient is not less than the fluctuation threshold, the immediate reward value and the predicted reward value are weighted and summed according to the second weight coefficient to obtain the target fusion reward value.

[0018] In one embodiment, the step of updating the model parameters of the decision network model based on the target fusion reward value includes:

[0019] The target fusion feature, target routing action, target fusion reward value, and target fusion feature after the traffic forwarding operation are combined to obtain an experience tuple, which is then stored in the replay buffer.

[0020] If the number of empirical tuples in the replay buffer exceeds a threshold, the target empirical tuple is obtained from the replay buffer and used as a training sample for the decision network model.

[0021] During the training of the decision network model using training samples, the model parameters of the decision network model are adjusted according to the difference between the current fusion reward value and the target fusion reward.

[0022] Once the target number of training iterations has been reached, the model parameters of the decision network model are updated based on the current model parameters.

[0023] In one embodiment, the topological space feature vector is extracted using a graph neural network; the method further includes:

[0024] When a new node is added to the network topology, the matrix dimension of the adjacency matrix of the graph neural network is updated according to the node load of the new node.

[0025] The updated adjacency matrix is ​​used as the input to the convolutional layer of the graph neural network, and the output is the updated topological space feature vector.

[0026] Dynamic route optimization based on updated topology space feature vectors.

[0027] In one embodiment, the predicted time series features are extracted via a long short-term memory network; the method further includes:

[0028] If the network volatility coefficient is not less than the volatility threshold, adjust the prediction step size of the Long Short-Term Memory network and obtain the updated prediction time series based on the adjusted Long Short-Term Memory network.

[0029] Adjust the weights corresponding to the predicted reward values ​​to obtain updated target fusion reward values;

[0030] Dynamic routing optimization is performed based on updated predicted time series and updated target fusion reward values.

[0031] Secondly, this application also provides a dynamic routing optimization apparatus, comprising:

[0032] The feature fusion module is used to obtain topological spatial feature vectors and predicted time series features based on node load and link latency of the network topology, and to fuse the topological spatial feature vectors and predicted time series features to obtain the target fused features.

[0033] The routing execution module is used to determine the target routing action from all routing actions in the routing action space based on the target fusion characteristics, and to execute the traffic forwarding operation corresponding to the target routing action; the routing action space is determined according to routing requirements.

[0034] The reward fusion module is used to obtain network performance parameters after performing traffic forwarding operations, and obtain the target fusion reward value based on the network performance parameters and state prediction parameters;

[0035] The routing optimization module is used to update the model parameters of the decision network model according to the target fusion reward value, and perform dynamic routing optimization based on the updated decision network model.

[0036] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method steps of any one of the first aspects.

[0037] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method steps of any one of the first aspects.

[0038] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the method steps of any one of the first aspects.

[0039] The aforementioned dynamic routing optimization method, apparatus, computer equipment, readable storage medium, and program product obtain topological spatial feature vectors and predicted time series features based on node load and link latency of the network topology. These features are then fused to obtain target fusion features. Based on these target fusion features, a target routing action is determined from all routing actions in the routing action space. The corresponding traffic forwarding operation is executed, and network performance parameters are obtained after the traffic forwarding operation. Based on the network performance parameters and state prediction parameters, a target fusion reward value is obtained. The model parameters of the decision network model are updated based on the target fusion reward value. Dynamic routing optimization is then performed based on the updated decision network model. This approach can accurately match future network states, improve the stability of routing decisions, and thus enhance communication reliability. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a diagram illustrating the application environment of the dynamic routing optimization method in one embodiment;

[0042] Figure 2 This is a flowchart illustrating a dynamic routing optimization method in one embodiment;

[0043] Figure 3 This is a flowchart illustrating the dynamic routing optimization method in another embodiment;

[0044] Figure 4 This is a structural block diagram of a dynamic routing optimization device in one embodiment;

[0045] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0047] The dynamic routing optimization method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated on server 104 or placed on a cloud or other network server. Terminal 102 is used to obtain topology spatial feature vectors and predicted time series features based on node load and link latency of the network topology, and fuses the topology spatial feature vectors and predicted time series features to obtain target fusion features. Based on the target fusion features, it determines the target routing action from all routing actions in the routing action space, executes the traffic forwarding operation corresponding to the target routing action, obtains network performance parameters after the traffic forwarding operation, obtains the target fusion reward value based on the network performance parameters and state prediction parameters, updates the model parameters of the decision network model based on the target fusion reward value, and performs dynamic route optimization based on the updated decision network model. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0048] In one exemplary embodiment, such as Figure 2 As shown, a dynamic routing optimization method is provided, which can be applied to... Figure 1 Taking terminal 102 as an example, the explanation includes the following steps 202 to 206. Wherein:

[0049] S202: Based on the node load and link latency of the network topology, obtain the topology spatial feature vector and the predicted time series feature, and fuse the topology spatial feature vector and the predicted time series feature to obtain the target fused feature.

[0050] Optionally, based on node load and link latency in the network topology, spatial features are extracted using a graph neural network. This automatically aggregates the attributes of each node (including load and latency) and their associations with neighboring nodes (such as link connectivity and bandwidth), outputting a spatial feature vector of node number × feature dimension (e.g., a 10×64-dimensional vector for a 10-node scenario). This supports real-time processing of dynamic node additions and subtractions. Based on historical network traffic data, a Long Short-Term Memory (LSTM) network learns traffic evolution patterns and outputs time-series features for future multiple steps. The LSTM network uses a gating mechanism to memorize historical load, packet loss rate, and other temporal patterns, predicting the network state for the next 3-5 time steps (e.g., node load and link packet loss rate in the next 3 seconds), outputting a time-series feature vector of prediction step size × node number × feature dimension. These two types of features are then concatenated into a three-dimensional tensor of prediction step size × node number × feature dimension. A 3×3×3 convolutional layer is then used to extract deep spatiotemporal correlation features, which are finally compressed into a fixed-dimensional target fusion feature, providing accurate input for subsequent decision-making.

[0051] S204: Based on the target fusion characteristics, determine the target routing action from all routing actions in the routing action space, and execute the traffic forwarding operation corresponding to the target routing action; the routing action space is determined according to routing requirements.

[0052] Optionally, based on actual routing requirements, a routing action space containing multiple core actions is preset, such as shortest path, load balancing, priority of urgent tasks, predictive stability, and fault-tolerant backup. The target fusion features are input into the dual-channel decision network model, which outputs the action value corresponding to each routing action in the action space, thereby determining the target routing action. The network gateway device executes the corresponding traffic forwarding logic based on the determined target routing action, completing a dynamic routing decision.

[0053] S206: Obtain network performance parameters after performing traffic forwarding operations, and obtain the target fusion reward value based on network performance parameters and state prediction parameters.

[0054] Optionally, after performing traffic forwarding operations, the current performance data collected by the real-time monitoring module, such as transmission latency, throughput, and packet loss rate, reflects the short-term performance of the current action. The state prediction parameters are the future multi-step network state prediction results output by the Long Short-Term Memory network, such as the link congestion probability (the probability that the link will soon be fully loaded) and path jitter rate (latency fluctuation amplitude) for the next 1-3 steps, reflecting the future stability trend of the current action. The target fusion reward value includes an immediate reward value and a predicted reward value. The immediate reward value is calculated based on network performance parameters, with the core logic being that the better the performance, the higher the reward. The predicted reward value is calculated based on the state prediction parameters, with the core logic being that the more stable the future, the higher the reward. The target fusion reward value is obtained by fusing the network performance parameters and the state prediction parameters.

[0055] S208: Update the model parameters of the decision network model according to the target fusion reward value, and perform dynamic route optimization based on the updated decision network model.

[0056] Optionally, the target fusion features, target routing actions, target fusion reward values, and new target fusion features after the actions are executed are grouped into experience tuples and stored in a fixed-capacity experience replay buffer to break the temporal correlation of data and achieve data reuse. When the number of experience tuples in the buffer exceeds a threshold, a random number of experience tuples are sampled as training samples to update the policy network parameters. After each round of policy network training is completed, the policy network parameters are synchronized to the target network. The updated decision network model will be used for the next round of route optimization, continuously optimizing the accuracy of route action selection, and ultimately achieving the goal of low oscillation, high stability, and low latency route optimization under dynamic networks.

[0057] In the aforementioned dynamic routing optimization method, topological spatial feature vectors and predicted time series features are obtained based on node load and link latency in the network topology. These features are then fused to obtain target fusion features. Based on these features, target routing actions are determined from all routing actions in the routing action space, and the corresponding traffic forwarding operation is executed. Network performance parameters after the traffic forwarding operation are obtained. Based on these performance parameters and state prediction parameters, a target fusion reward value is obtained. The model parameters of the decision network model are updated based on the target fusion reward value. Dynamic routing optimization is then performed based on the updated decision network model. This method can accurately match future network states, improve the stability of routing decisions, and thus enhance communication reliability.

[0058] In an exemplary embodiment, the step of determining a target routing action from all routing actions in the routing action space based on target fusion features includes: inputting the target fusion features into a decision network model to obtain the action value corresponding to each routing action in the routing action space; determining a probability threshold; and determining the target routing action from all routing actions based on the probability threshold and the action value.

[0059] Optionally, action value is a quantitative indicator of action reward output by the decision network model. It is used to evaluate the long-term cumulative reward that can be obtained after executing a certain routing action in the network state corresponding to the current target fusion feature. It integrates information from topological space features and prediction time series features, directly reflecting the quality of the action. The target fusion feature is input into the policy network. The policy network decouples the state value flow and the action advantage flow, and outputs the action value of all actions in the routing action space. For example, in a 10-node industrial IoT scenario, the routing action space contains 5 types of actions (shortest path, load balancing, emergency priority, prediction stability, and fault tolerance backup). The action values ​​(Q values) output by the policy network are as follows: Action 1 (Q=6.2), Action 2 (Q=7.5), Action 3 (Q=5.8), Action 4 (Q=8.72), and Action 5 (Q=6.9). Among them, Action 4 (prediction stability) has the highest Q value, which means that its long-term expected reward is the best in the current state.

[0060] Furthermore, the target routing action is determined based on a probability threshold. This probability threshold is the exploration rate in the ε-greedy strategy, assumed to be 0.1, corresponding to a fixed threshold of 90% utilization probability and 10% exploration probability, used to balance utilizing the known optimal path with exploring potentially better paths. If the 90% utilization probability is triggered, the routing action with the highest action value (Q-value) is directly selected as the target routing action. If the 10% exploration probability is triggered, the Q-value is ignored, and an action is randomly selected from the routing action space as the target routing action, with the aim of exploring potentially better future returns for actions that are not currently optimal.

[0061] In this embodiment, by inputting the target fusion features into the decision network model, the action value corresponding to each routing action in the routing action space is obtained, the probability threshold is determined, and the target routing action is determined from all routing actions based on the probability threshold and the action value. This can improve the decision accuracy and adaptability, thereby improving the long-term decision stability.

[0062] In an exemplary embodiment, the step of obtaining a target fusion reward value based on network performance parameters and state prediction parameters includes: obtaining an immediate reward value based on the network performance parameters and a predicted reward value based on the state prediction parameters; obtaining a network fluctuation coefficient based on the network performance parameters; if the network fluctuation coefficient is less than a fluctuation threshold, performing a weighted summation of the immediate reward value and the predicted reward value based on a first weighting coefficient to obtain the target fusion reward value; if the network fluctuation coefficient is not less than the fluctuation threshold, performing a weighted summation of the immediate reward value and the predicted reward value based on a second weighting coefficient to obtain the target fusion reward value.

[0063] Optionally, after executing the target routing action, the current transmission performance data collected by the network real-time monitoring module, including transmission latency, throughput, and current link load utilization, is used. Performance metrics are converted into reward scores using a preset quantification formula, with higher performance resulting in higher rewards. State prediction parameters are derived from the future multi-step network state prediction results output by the Long Short-Term Memory (LSTM) network, including future link congestion probability, path jitter rate, and future load growth trends. Future stability is converted into reward scores through threshold comparison and deviation quantification, with higher stability resulting in higher rewards. The network fluctuation coefficient is the core criterion for dynamic weighting, used to quantify network state stability. Network scenarios are divided by preset fluctuation thresholds, and differentiated weight coefficients are assigned to different scenarios to achieve a dynamic balance between short-term performance and long-term stability, corresponding to the adaptive weighting logic of the two-stage reward generator.

[0064] For example, if the network volatility coefficient is less than the volatility threshold, i.e., a low volatility scenario, the first weight coefficient is allocated according to the immediate reward (0.7) and the predicted reward (0.3), meaning that the weight of current short-term performance is higher than the weight of future long-term stability. If the network volatility coefficient is not less than the volatility threshold, i.e., a high volatility scenario, the second weight coefficient is adjusted to the immediate reward (0.5) and the predicted reward (0.5), meaning that the weight of current short-term performance is equal to the weight of future long-term stability, prioritizing future stability.

[0065] In this embodiment, by obtaining the instant reward value based on network performance parameters, the predicted reward value based on state prediction parameters, and the network fluctuation coefficient based on network performance parameters, and when the network fluctuation coefficient is less than the fluctuation threshold, the instant reward value and the predicted reward value are weighted and summed according to the first weight coefficient to obtain the target fusion reward value. When the network fluctuation coefficient is not less than the fluctuation threshold, the instant reward value and the predicted reward value are weighted and summed according to the second weight coefficient to obtain the target fusion reward value. This can improve decision accuracy and adaptability, thereby improving long-term decision stability.

[0066] In an exemplary embodiment, the step of updating the model parameters of the decision network model according to the target fusion reward value includes: combining the target fusion feature, the target routing action, the target fusion reward value, and the target fusion feature after performing a traffic forwarding operation to obtain an experience tuple, and storing the experience tuple in a replay buffer; if the number of experience tuples in the replay buffer exceeds a threshold, obtaining a target experience tuple from the replay buffer and using the target experience tuple as a training sample for the decision network model; during the training of the decision network model using the training sample, adjusting the model parameters of the decision network model according to the difference between the current fusion reward value and the target fusion reward; and updating the model parameters of the decision network model according to the current model parameters when the number of training iterations reaches a target number.

[0067] Optionally, experience tuples are the basic data units for model training. They need to completely record the entire decision-feedback chain information to ensure the effectiveness of training samples. The constructed experience tuples are stored in a replay buffer of a preset capacity. The buffer uses a first-in, first-out (FIFO) rule to manage data, ensuring that the stored experience can cover different network states to provide diverse samples for subsequent batch training. To avoid training instability caused by temporal correlation, training should only begin after the buffer has accumulated sufficient samples. A preset threshold for starting training in the buffer is set. When the sample size is insufficient, only experience is stored without training, avoiding overfitting due to insufficient samples. A preset number of experience tuples are randomly sampled from the replay buffer as target experience tuples. The core purpose of random sampling is to break the temporal correlation of experience, ensure a uniform distribution of training data, prevent the model from biased towards recent states and ignoring historically effective experience, and improve training stability.

[0068] Furthermore, the current state from the training samples is input into the policy network, which outputs the current Q-value when action 'a' is executed, i.e., the model's profit prediction for this empirical tuple. The new state is input into the target network, which outputs the Q-values ​​of all actions in the new state, and the maximum value is taken, i.e., the optimal expected profit of the new state. The target Q-value is calculated according to the reinforcement learning Bellman equation, and the mean squared error between the current Q-value and the target Q-value is calculated as the loss. The gradient of the loss function with respect to the policy network parameters is calculated using the Adam optimizer, and the parameters are updated in the direction of decreasing loss until the loss converges. To avoid training oscillations caused by frequent fluctuations in the target Q-value, the target network parameters need to be synchronized at fixed intervals. For example, every 10 training rounds is the target number of rounds. When the training rounds reach 10, all the current parameters of the policy network are completely copied to the target network, overwriting the original parameters of the target network, ensuring that the target Q-value of the target network is both timely and stable.

[0069] In this embodiment, by updating the model parameters of the decision network model according to the target fusion reward value, it is possible to ensure that the model gradually learns the optimal routing strategy, thereby improving the decision accuracy.

[0070] In an exemplary embodiment, the topology space feature vector is extracted through a graph neural network; the method further includes: when there is a new node in the network topology, updating the matrix dimension of the adjacency matrix of the graph neural network according to the node load of the new node; using the updated adjacency matrix as the input of the convolutional layer of the graph neural network, and outputting the updated topology space feature vector; and performing dynamic route optimization based on the updated topology space feature vector.

[0071] Optionally, the adjacency matrix is ​​the core input for the graph neural network to describe the relationships between nodes and links. Its dimension strictly matches the total number of network nodes. When adding a new node, the matrix dimension needs to be expanded and its content filled first. Assuming the network has N nodes before adding a node, the adjacency matrix is ​​an N×N square matrix. The element in the i-th row and j-th column of the matrix needs to simultaneously carry the node connectivity and node attribute association. Here, "0 / 1" indicates whether there is a direct link between node i and node j (1 for existence, 0 for non-existence). The element value (such as 0.8, 1.2) can further quantify the association attributes such as link latency and bandwidth utilization. At the same time, each row / column of the matrix also implicitly contains the load information of the corresponding node. When the network adds M nodes (such as adding 2 nodes in a 10-node scenario, M=2), the real-time topology detection module automatically obtains the attributes of the new nodes, namely the current load of each new node, and the association between the new node and the original nodes, i.e., which of the N original nodes have direct links with the new node, and the latency, bandwidth, and other attributes of the corresponding links. Based on the above information, the original N×N adjacency matrix is ​​expanded into a new matrix of (N+M)×(N+M). The newly added rows / columns from N+1 to N+M correspond to M new nodes. The elements in the rows / columns are filled according to connectivity and attribute rules. At the same time, the load information of the new nodes is incorporated into the element distribution of the corresponding rows / columns (e.g., nodes with high loads have higher mean values ​​for the elements in their corresponding rows).

[0072] Furthermore, the core advantage of graph neural networks (Graph Neural Networks) is their adaptive input dimension. The updated adjacency matrix can directly participate in feature extraction without adjusting the model structure, ensuring the continuity of topological space features. Graph Neural Networks extract topological features through a neighbor node information aggregation mechanism, automatically adjusting the neighbor aggregation range according to the dimension of the adjacency matrix, expanding from the original aggregation of N nodes to the aggregation of N+M nodes, without modifying the convolutional layer structure or retraining the model parameters. After 2-3 layers of graph convolution operations, the Graph Neural Network compresses the self-attribute and neighbor association features of each node into a fixed-dimensional feature vector, ultimately outputting an updated topological space feature vector. This vector contains both the feature updates of the original nodes and the features of the newly added nodes, and can be directly used for subsequent spatiotemporal joint fusion.

[0073] In this embodiment, when a new node is added to the network topology, the matrix dimension of the adjacency matrix of the graph neural network is updated according to the node load of the new node. The updated adjacency matrix is ​​used as the input of the convolutional layer of the graph neural network, and the updated topological space feature vector is output. Dynamic routing optimization based on the updated topological space feature vector can improve the real-time performance of decision-making and reduce decision latency.

[0074] In an exemplary embodiment, the predicted time series features are extracted through a long short-term memory network; the method further includes: adjusting the prediction step size of the long short-term memory network when the network fluctuation coefficient is not less than the fluctuation threshold, and obtaining an updated predicted time series based on the adjusted long short-term memory network; adjusting the weights corresponding to the predicted reward value to obtain an updated target fusion reward value; and performing dynamic route optimization based on the updated predicted time series and the updated target fusion reward value.

[0075] Optionally, the network fluctuation coefficient is calculated using the network load variance over multiple consecutive time points. When the fluctuation coefficient exceeds a threshold, it is identified as a high-fluctuation scenario, such as sudden traffic spikes, temporary link interference, or abrupt changes in load / latency. In high-fluctuation scenarios, the prediction step size is automatically shortened to 1 step (predicting only the network state for the next second). Since the historical traffic patterns have significantly reduced predictive value under high fluctuations, long-term predictions are prone to large deviations from actual values. Shortening the step size to 1 step minimizes the uncertainty of long-term predictions, ensuring that the deviation between the prediction results and the actual network state is controllable. The adjusted Long Short-Term Memory (LSTM) network, based on the latest historical load and packet loss rate data, re-outputs an updated prediction time series of 1 step × number of nodes × feature dimension. This includes key time-series features such as load predictions and link congestion probability predictions for each node within the next step, providing accurate short-term prediction basis for subsequent feature fusion and reward calculation.

[0076] Furthermore, in high-fluctuation scenarios, the weight of future stable returns needs to be increased to guide the model to prioritize routing actions with short-term predictable stability. For example, in low-fluctuation scenarios, the target fusion reward value is weighted by immediate reward (0.7) and predicted reward (0.3) to prioritize current transmission performance (such as throughput and low latency); in high-fluctuation scenarios, the weights are automatically adjusted to immediate reward (0.5) and predicted reward (0.5), meaning that the contribution of current performance gains and future stable returns to the final reward is equal. The core purpose is to prevent the model from choosing paths that are currently superior but will be congested in the future due to short-sighted pursuit of current performance. Then, an updated target fusion reward value is calculated, which focuses more on stability in the next step, guiding the model's decision-making to favor paths with no short-term congestion.

[0077] In this embodiment, by adjusting the prediction step size of the Long Short-Term Memory network when the network fluctuation coefficient is not less than the fluctuation threshold, and obtaining an updated prediction time series based on the adjusted Long Short-Term Memory network, adjusting the weights corresponding to the prediction reward value, and obtaining an updated target fusion reward value, dynamic routing optimization is performed based on the updated prediction time series and the updated target fusion reward value. This can reduce the prediction error rate in high-fluctuation scenarios, improve decision accuracy, and thus improve the long-term stability of decision-making.

[0078] In one exemplary embodiment, such as Figure 3As shown, a dynamic routing optimization method is provided, which includes the following steps:

[0079] (1) Spatiotemporal joint coding: Based on the node load and link delay of the network topology, the topology spatial feature vector and the predicted time series feature are obtained, and the topology spatial feature vector and the predicted time series feature are fused to obtain the target fused feature.

[0080] (2) Routing action decision: Input the target fusion features into the decision network model to obtain the action value corresponding to each routing action in the routing action space; determine the probability threshold, and determine the target routing action from all routing actions based on the probability threshold and action value.

[0081] (3) Execution of routing actions: Execute the traffic forwarding operation corresponding to the target routing action; the routing action space is determined according to the routing requirements.

[0082] (4) Two-stage reward fusion: Obtain network performance parameters after the traffic forwarding operation is performed. Based on the network performance parameters, obtain the instant reward value and the predicted reward value based on the state prediction parameters. Obtain the network fluctuation coefficient based on the network performance parameters. If the network fluctuation coefficient is less than the fluctuation threshold, the instant reward value and the predicted reward value are weighted and summed according to the first weight coefficient to obtain the target fusion reward value. If the network fluctuation coefficient is not less than the fluctuation threshold, the instant reward value and the predicted reward value are weighted and summed according to the second weight coefficient to obtain the target fusion reward value.

[0083] (5) Decision network update: Combine the target fusion features, target routing actions, target fusion reward value and target fusion features after performing traffic forwarding operation to obtain experience tuples, and store the experience tuples in the replay buffer; if the number of experience tuples in the replay buffer exceeds the threshold, obtain the target experience tuples from the replay buffer and use the target experience tuples as training samples for the decision network model; during the training of the decision network model using training samples, adjust the model parameters of the decision network model according to the difference between the current fusion reward value and the target fusion reward; if the number of training times reaches the target number, update the model parameters of the decision network model according to the current model parameters.

[0084] (6) Dynamic route optimization: Dynamic route optimization is performed based on the updated decision network model.

[0085] (7) Topology change response: When there are new nodes in the network topology, update the matrix dimension of the adjacency matrix of the graph neural network according to the node load of the new nodes; use the updated adjacency matrix as the input of the convolutional layer of the graph neural network, and output the updated topology space feature vector; perform dynamic route optimization based on the updated topology space feature vector.

[0086] (8) Network fluctuation adaptation: When the network fluctuation coefficient is not less than the fluctuation threshold, adjust the prediction step size of the long short-term memory network and obtain the updated prediction time series based on the adjusted long short-term memory network; adjust the weights corresponding to the prediction reward value to obtain the updated target fusion reward value; and perform dynamic route optimization based on the updated prediction time series and the updated target fusion reward value.

[0087] In this embodiment, topological spatial feature vectors and predicted time series features are obtained based on node load and link latency of the network topology. The topological spatial feature vectors and predicted time series features are then fused to obtain target fusion features. Based on the target fusion features, target routing actions are determined from all routing actions in the routing action space, and traffic forwarding operations corresponding to the target routing actions are executed. Network performance parameters after the traffic forwarding operations are obtained. Based on the network performance parameters and state prediction parameters, target fusion reward values ​​are obtained. The model parameters of the decision network model are updated based on the target fusion reward values. Dynamic routing optimization is performed based on the updated decision network model, which can accurately match future network states, improve the stability of routing decisions, and thus improve communication reliability.

[0088] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0089] Based on the same inventive concept, this application also provides a dynamic routing optimization apparatus for implementing the dynamic routing optimization method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more embodiments of the dynamic routing optimization apparatus provided below can be found in the limitations of the dynamic routing optimization method described above, and will not be repeated here.

[0090] In one exemplary embodiment, such as Figure 4As shown, a dynamic route optimization device is provided, including: a feature fusion module 10, a route execution module 20, a reward fusion module 30, and a route optimization module 40, wherein:

[0091] The feature fusion module 10 is used to obtain topological spatial feature vectors and predicted time series features based on the node load and link latency of the network topology, and to fuse the topological spatial feature vectors and predicted time series features to obtain the target fused features.

[0092] The routing execution module 20 is used to determine the target routing action from all routing actions in the routing action space based on the target fusion characteristics, and to execute the traffic forwarding operation corresponding to the target routing action; the routing action space is determined according to routing requirements.

[0093] The reward fusion module 30 is used to obtain network performance parameters after performing traffic forwarding operations, and obtain the target fusion reward value based on the network performance parameters and state prediction parameters.

[0094] The routing optimization module 40 is used to update the model parameters of the decision network model according to the target fusion reward value, and to perform dynamic routing optimization based on the updated decision network model.

[0095] In an exemplary embodiment, the routing execution module 20 is further configured to input the target fusion features into the decision network model, obtain the action value corresponding to each routing action in the routing action space, determine the probability threshold, and determine the target routing action from all routing actions based on the probability threshold and the action value.

[0096] In an exemplary embodiment, the reward fusion module 30 is further configured to obtain an instant reward value based on network performance parameters and a predicted reward value based on state prediction parameters; obtain a network fluctuation coefficient based on network performance parameters; if the network fluctuation coefficient is less than a fluctuation threshold, perform a weighted summation of the instant reward value and the predicted reward value based on a first weighting coefficient to obtain a target fused reward value; if the network fluctuation coefficient is not less than the fluctuation threshold, perform a weighted summation of the instant reward value and the predicted reward value based on a second weighting coefficient to obtain a target fused reward value.

[0097] In an exemplary embodiment, the routing optimization module 40 is further configured to combine the target fusion feature, the target routing action, the target fusion reward value, and the target fusion feature after performing the traffic forwarding operation to obtain an experience tuple, and store the experience tuple in the replay buffer; if the number of experience tuples in the replay buffer exceeds a threshold, obtain the target experience tuple from the replay buffer and use the target experience tuple as a training sample for the decision network model; during the training of the decision network model using the training sample, adjust the model parameters of the decision network model according to the difference between the current fusion reward value and the target fusion reward; and update the model parameters of the decision network model according to the current model parameters when the number of training iterations reaches the target number of iterations.

[0098] In an exemplary embodiment, the topology space feature vector is extracted through a graph neural network; the routing optimization module 40 is further configured to update the matrix dimension of the adjacency matrix of the graph neural network according to the node load of the new node when there is a new node in the network topology; use the updated adjacency matrix as the input of the graph neural network convolutional layer, and output the updated topology space feature vector; and perform dynamic routing optimization based on the updated topology space feature vector.

[0099] In an exemplary embodiment, the predicted time series features are extracted through a long short-term memory network; the routing optimization module 40 is further configured to adjust the prediction step size of the long short-term memory network when the network fluctuation coefficient is not less than the fluctuation threshold, and obtain an updated predicted time series based on the adjusted long short-term memory network; adjust the weights corresponding to the predicted reward value to obtain an updated target fusion reward value; and perform dynamic routing optimization based on the updated predicted time series and the updated target fusion reward value.

[0100] Each module in the aforementioned dynamic routing optimization device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0101] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements a dynamic routing optimization method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0102] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0103] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: obtaining a topology spatial feature vector and a predicted time series feature based on the node load and link latency of the network topology, and fusing the topology spatial feature vector and the predicted time series feature to obtain a target fused feature; determining a target routing action from all routing actions in the routing action space based on the target fused feature, and executing the traffic forwarding operation corresponding to the target routing action; the routing action space is determined according to routing requirements; obtaining network performance parameters after executing the traffic forwarding operation, and obtaining a target fused reward value based on the network performance parameters and state prediction parameters; updating the model parameters of the decision network model based on the target fused reward value, and performing dynamic route optimization based on the updated decision network model.

[0104] In one embodiment, the process of determining a target routing action from all routing actions in a routing action space based on target fusion features when the processor executes a computer program includes: inputting the target fusion features into a decision network model to obtain the action value corresponding to each routing action in the routing action space; determining a probability threshold; and determining the target routing action from all routing actions based on the probability threshold and the action value.

[0105] In one embodiment, the process of obtaining a target fusion reward value based on network performance parameters and state prediction parameters when the processor executes a computer program includes: obtaining an immediate reward value based on the network performance parameters and obtaining a predicted reward value based on the state prediction parameters; obtaining a network fluctuation coefficient based on the network performance parameters; if the network fluctuation coefficient is less than a fluctuation threshold, performing a weighted summation of the immediate reward value and the predicted reward value based on a first weighting coefficient to obtain the target fusion reward value; if the network fluctuation coefficient is not less than the fluctuation threshold, performing a weighted summation of the immediate reward value and the predicted reward value based on a second weighting coefficient to obtain the target fusion reward value.

[0106] In one embodiment, updating the model parameters of a decision network model based on a target fusion reward value when the processor executes a computer program includes: combining target fusion features, target routing actions, target fusion reward value, and target fusion features after performing a traffic forwarding operation to obtain an empirical tuple, and storing the empirical tuple in a replay buffer; if the number of empirical tuples in the replay buffer exceeds a threshold, retrieving a target empirical tuple from the replay buffer and using the target empirical tuple as a training sample for the decision network model; during the training of the decision network model using the training sample, adjusting the model parameters of the decision network model based on the difference between the current fusion reward value and the target fusion reward; and updating the model parameters of the decision network model based on the current model parameters when the number of training iterations reaches a target number.

[0107] In one embodiment, the topology space feature vector is extracted through a graph neural network; when the processor executes the computer program, it also implements the following steps: when there are new nodes in the network topology, update the matrix dimension of the adjacency matrix of the graph neural network according to the node load of the new nodes; use the updated adjacency matrix as the input of the convolutional layer of the graph neural network to obtain the updated topology space feature vector; and perform dynamic route optimization based on the updated topology space feature vector.

[0108] In one embodiment, the predicted time series features are extracted through a long short-term memory network; when the processor executes the computer program, it also performs the following steps: if the network fluctuation coefficient is not less than the fluctuation threshold, adjust the prediction step size of the long short-term memory network and obtain an updated predicted time series based on the adjusted long short-term memory network; adjust the weights corresponding to the predicted reward value to obtain an updated target fusion reward value; and perform dynamic route optimization based on the updated predicted time series and the updated target fusion reward value.

[0109] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program performs the following steps: obtaining a topology spatial feature vector and a predicted time series feature based on the node load and link latency of the network topology, and fusing the topology spatial feature vector and the predicted time series feature to obtain a target fused feature; determining a target routing action from all routing actions in the routing action space based on the target fused feature, and executing the traffic forwarding operation corresponding to the target routing action; the routing action space is determined according to routing requirements; obtaining network performance parameters after executing the traffic forwarding operation, and obtaining a target fused reward value based on the network performance parameters and state prediction parameters; updating the model parameters of the decision network model based on the target fused reward value, and performing dynamic route optimization based on the updated decision network model.

[0110] In one embodiment, when a computer program is executed by a processor, determining a target routing action from all routing actions in a routing action space based on target fusion features includes: inputting the target fusion features into a decision network model to obtain the action value corresponding to each routing action in the routing action space; determining a probability threshold; and determining the target routing action from all routing actions based on the probability threshold and the action value.

[0111] In one embodiment, the process of obtaining a target fusion reward value based on network performance parameters and state prediction parameters when the computer program is executed by a processor includes: obtaining an immediate reward value based on the network performance parameters and obtaining a predicted reward value based on the state prediction parameters; obtaining a network fluctuation coefficient based on the network performance parameters; if the network fluctuation coefficient is less than a fluctuation threshold, performing a weighted summation of the immediate reward value and the predicted reward value based on a first weighting coefficient to obtain the target fusion reward value; and if the network fluctuation coefficient is not less than the fluctuation threshold, performing a weighted summation of the immediate reward value and the predicted reward value based on a second weighting coefficient to obtain the target fusion reward value.

[0112] In one embodiment, the process of updating the model parameters of a decision network model based on a target fusion reward value when the computer program is executed by a processor includes: combining target fusion features, target routing actions, target fusion reward value, and target fusion features after performing a traffic forwarding operation to obtain an empirical tuple, and storing the empirical tuple in a replay buffer; if the number of empirical tuples in the replay buffer exceeds a threshold, retrieving a target empirical tuple from the replay buffer and using the target empirical tuple as a training sample for the decision network model; during the training of the decision network model using the training sample, adjusting the model parameters of the decision network model based on the difference between the current fusion reward value and the target fusion reward; and updating the model parameters of the decision network model based on the current model parameters when the number of training iterations reaches a target number.

[0113] In one embodiment, the topology space feature vector is extracted through a graph neural network; when the computer program is executed by the processor, it also performs the following steps: when there are new nodes in the network topology, update the matrix dimension of the adjacency matrix of the graph neural network according to the node load of the new nodes; use the updated adjacency matrix as the input of the convolutional layer of the graph neural network to obtain the updated topology space feature vector; and perform dynamic route optimization based on the updated topology space feature vector.

[0114] In one embodiment, the predicted time series features are extracted through a long short-term memory network; when the computer program is executed by the processor, it also performs the following steps: if the network fluctuation coefficient is not less than the fluctuation threshold, adjust the prediction step size of the long short-term memory network and obtain an updated predicted time series based on the adjusted long short-term memory network; adjust the weights corresponding to the predicted reward value to obtain an updated target fusion reward value; and perform dynamic route optimization based on the updated predicted time series and the updated target fusion reward value.

[0115] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps: obtaining a topology spatial feature vector and a predicted time series feature based on node load and link latency of the network topology, and fusing the topology spatial feature vector and the predicted time series feature to obtain a target fused feature; determining a target routing action from all routing actions in the routing action space based on the target fused feature, and executing the traffic forwarding operation corresponding to the target routing action; the routing action space is determined according to routing requirements; obtaining network performance parameters after executing the traffic forwarding operation, and obtaining a target fused reward value based on the network performance parameters and state prediction parameters; updating the model parameters of the decision network model based on the target fused reward value, and performing dynamic route optimization based on the updated decision network model.

[0116] In one embodiment, when a computer program is executed by a processor, determining a target routing action from all routing actions in a routing action space based on target fusion features includes: inputting the target fusion features into a decision network model to obtain the action value corresponding to each routing action in the routing action space; determining a probability threshold; and determining the target routing action from all routing actions based on the probability threshold and the action value.

[0117] In one embodiment, the process of obtaining a target fusion reward value based on network performance parameters and state prediction parameters when the computer program is executed by a processor includes: obtaining an immediate reward value based on the network performance parameters and obtaining a predicted reward value based on the state prediction parameters; obtaining a network fluctuation coefficient based on the network performance parameters; if the network fluctuation coefficient is less than a fluctuation threshold, performing a weighted summation of the immediate reward value and the predicted reward value based on a first weighting coefficient to obtain the target fusion reward value; and if the network fluctuation coefficient is not less than the fluctuation threshold, performing a weighted summation of the immediate reward value and the predicted reward value based on a second weighting coefficient to obtain the target fusion reward value.

[0118] In one embodiment, the process of updating the model parameters of a decision network model based on a target fusion reward value when the computer program is executed by a processor includes: combining target fusion features, target routing actions, target fusion reward value, and target fusion features after performing a traffic forwarding operation to obtain an empirical tuple, and storing the empirical tuple in a replay buffer; if the number of empirical tuples in the replay buffer exceeds a threshold, retrieving a target empirical tuple from the replay buffer and using the target empirical tuple as a training sample for the decision network model; during the training of the decision network model using the training sample, adjusting the model parameters of the decision network model based on the difference between the current fusion reward value and the target fusion reward; and updating the model parameters of the decision network model based on the current model parameters when the number of training iterations reaches a target number.

[0119] In one embodiment, the topology space feature vector is extracted through a graph neural network; when the computer program is executed by the processor, it also performs the following steps: when there are new nodes in the network topology, update the matrix dimension of the adjacency matrix of the graph neural network according to the node load of the new nodes; use the updated adjacency matrix as the input of the convolutional layer of the graph neural network to obtain the updated topology space feature vector; and perform dynamic route optimization based on the updated topology space feature vector.

[0120] In one embodiment, the predicted time series features are extracted through a long short-term memory network; when the computer program is executed by the processor, it also performs the following steps: if the network fluctuation coefficient is not less than the fluctuation threshold, adjust the prediction step size of the long short-term memory network and obtain an updated predicted time series based on the adjusted long short-term memory network; adjust the weights corresponding to the predicted reward value to obtain an updated target fusion reward value; and perform dynamic route optimization based on the updated predicted time series and the updated target fusion reward value.

[0121] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0122] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0123] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A dynamic routing optimization method, characterized in that, The method includes: Based on the node load and link latency of the network topology, obtain the topology spatial feature vector and the predicted time series feature, and fuse the topology spatial feature vector and the predicted time series feature to obtain the target fused feature; Based on the target fusion characteristics, a target routing action is determined from all routing actions in the routing action space, and the traffic forwarding operation corresponding to the target routing action is executed; the routing action space is determined according to routing requirements. Obtain network performance parameters after performing the traffic forwarding operation, and obtain the target fusion reward value based on the network performance parameters and state prediction parameters; The model parameters of the decision network model are updated based on the target fusion reward value, and dynamic route optimization is performed based on the updated decision network model.

2. The method according to claim 1, characterized in that, The step of determining the target routing action from all routing actions in the routing action space based on the target fusion feature includes: The target fusion features are input into the decision network model to obtain the action value corresponding to each routing action in the routing action space. Determine a probability threshold, and based on the probability threshold and the action value, determine the target routing action from all routing actions.

3. The method according to claim 1, characterized in that, The step of obtaining the target fusion reward value based on the network performance parameters and state prediction parameters includes: Based on the network performance parameters, obtain the instant reward value, and based on the state prediction parameters, obtain the predicted reward value; The network fluctuation coefficient is obtained based on the network performance parameters. If the network fluctuation coefficient is less than the fluctuation threshold, the instantaneous reward value and the predicted reward value are weighted and summed according to the first weight coefficient to obtain the target fusion reward value; If the network fluctuation coefficient is not less than the fluctuation threshold, the instantaneous reward value and the predicted reward value are weighted and summed according to the second weighting coefficient to obtain the target fusion reward value.

4. The method according to claim 1, characterized in that, The step of updating the model parameters of the decision network model based on the target fusion reward value includes: The target fusion feature, the target routing action, the target fusion reward value, and the target fusion feature after performing the traffic forwarding operation are combined to obtain an experience tuple, and the experience tuple is stored in the replay buffer. If the number of empirical tuples in the replay buffer exceeds a threshold, a target empirical tuple is obtained from the replay buffer and used as a training sample for the decision network model. During the training of the decision network model using the training samples, the model parameters of the decision network model are adjusted according to the difference between the current fusion reward value and the target fusion reward. Once the target number of training iterations has been reached, the model parameters of the decision network model are updated based on the current model parameters.

5. The method according to claim 1, characterized in that, The topological space feature vector is extracted using a graph neural network; the method further includes: When a new node is added to the network topology, the matrix dimension of the adjacency matrix of the graph neural network is updated according to the node load of the new node. The updated adjacency matrix is ​​used as the input to the convolutional layer of the graph neural network, and the updated topological space feature vector is output. Dynamic route optimization based on updated topology space feature vectors.

6. The method according to claim 3, characterized in that, The predicted time series features are extracted using a long short-term memory network; the method further includes: If the network fluctuation coefficient is not less than the fluctuation threshold, adjust the prediction step size of the long short-term memory network, and obtain an updated prediction time series based on the adjusted long short-term memory network. Adjust the weights corresponding to the predicted reward values ​​to obtain updated target fusion reward values; Dynamic routing optimization is performed based on updated predicted time series and updated target fusion reward values.

7. A dynamic routing optimization device, characterized in that, The device includes: The feature fusion module is used to obtain topological spatial feature vectors and predicted time series features based on node load and link latency of the network topology, and to fuse the topological spatial feature vectors and the predicted time series features to obtain target fused features. The routing execution module is used to determine the target routing action from all routing actions in the routing action space based on the target fusion characteristics, and to execute the traffic forwarding operation corresponding to the target routing action; the routing action space is determined according to routing requirements. The reward fusion module is used to obtain network performance parameters after the traffic forwarding operation is performed, and to obtain the target fusion reward value based on the network performance parameters and the state prediction parameters. The routing optimization module is used to update the model parameters of the decision network model according to the target fusion reward value, and perform dynamic routing optimization based on the updated decision network model.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images