Intelligent routing traffic management method and system based on reinforcement learning

Abstract

The invention discloses an intelligent routing flow management method and system based on reinforcement learning, and the method comprises the following steps: building a lightweight motion selection model at each network node, and mapping a communication link which can be used by the node into a plurality of forwarding options; the current network state is formed by collecting the delay, packet loss rate and utilization condition of each link in real time; and the system distributes priorities of different forwarding options by utilizing a probability strategy, and performs dynamic adjustment in combination with the change of the network state. And after each round of forwarding, the system updates and optimizes the strategy according to the communication effect, so that a routing decision mechanism which does not depend on central control and can be adaptively adjusted is realized. The method is suitable for a complex and frequently-changing network environment, and the overall transmission efficiency and the system stability are improved.

Description

Technical Field

[0001] This invention relates to the field of computer network technology, and in particular to an intelligent routing traffic management method and system based on reinforcement learning. Background Technology

[0002] With the rapid development of information technology and data communication infrastructure, network environments are constantly evolving towards large-scale, dynamic, and multi-link topologies. In such highly dynamic and complex networks, traditional routing algorithms face severe challenges in path selection and traffic scheduling. Especially in edge computing, IoT, and high-concurrency scenarios, how to implement intelligent routing strategies with adaptive capabilities has become a key issue in improving network performance and stability.

[0003] Existing technologies, such as static or model-based routing mechanisms like shortest path algorithms and prediction-based traffic engineering methods, typically rely on global network state information or pre-defined topology models. However, in real-world environments with frequent fluctuations in link states and dynamic changes in topology, they often exhibit problems like slow response times and inefficient resource allocation. Specifically, existing technologies have significant shortcomings in the following aspects:

[0004] 1. Slow convergence speed: Traditional algorithms need to wait for multiple rounds of state propagation and policy updates after network state changes, making it difficult to adapt to link fluctuations in a timely manner; 2. High average latency: Static or centralized path selection mechanisms often ignore the real-time status of local links, causing congested paths to be continuously selected; 3. Low bandwidth utilization: The lack of a real-time perception and feedback mechanism for dynamic link utilization leads to insufficient scheduling of high-quality links; 4. Reliance on centralized control and global modeling: Traditional mechanisms require collecting information from the entire network or building a unified model, which is not suitable for resource-constrained nodes and decentralized deployments; 5. Lack of adaptive adjustment mechanisms for unstable environments: Existing technologies generally cannot adjust the balance between exploration and utilization strategies according to real-time link status changes, and the strategies are prone to getting trapped in local optima.

[0005] Therefore, how to provide an intelligent routing traffic management method and system based on reinforcement learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose an intelligent routing traffic management method and system based on reinforcement learning. This invention adopts a mechanism that combines an improved multi-armed slot machine model with the Exp3 algorithm to construct an integrated distributed decision-making process that includes local state awareness, instant reward feedback, disturbance-guided sampling, and adaptive parameter adjustment. This enables link selection and data forwarding control in dynamic and complex network environments, and has the advantages of not requiring global modeling, strong self-learning ability, rapid response to link fluctuations, and flexible deployment.

[0007] According to an embodiment of the present invention, an intelligent routing traffic management method and system based on reinforcement learning includes the following steps: S1. Construct an improved multi-armed slot machine model at each network node. The improved multi-armed slot machine model maps each available link of the network node to multiple independent actions, forming a set of actions, and has the ability to perceive local network status and operate independently. S2. Collect hop delay parameters, packet loss rate parameters, and link utilization parameters through the state awareness module in the network node to generate the current network state vector; S3. Based on the Exp3 algorithm, each action in the action set is assigned an initial equal weight, and a probability distribution vector corresponding to the independent action is constructed by combining the network state vector. S4. Following the random weighted sampling mechanism of the Exp3 algorithm, select an independent action from the probability distribution vector and perform data forwarding through the corresponding link; S5. Construct an instant reward function that integrates hop latency, packet loss rate, and link utilization, based on the forwarding results. Calculate the immediate reward value for the current action; S6. Use the instant reward value to exponentially update the probability weight of the current forwarding action and correct the probability distribution vector for subsequent action selection. S7. When the state awareness module calculates the network state fluctuation trend based on multiple consecutive network state vectors, it calls the gating mechanism to adaptively adjust the exploration and utilization balance parameters in Exp3, thereby enhancing the policy's responsiveness to the dynamic environment.

[0008] Optionally, S1 specifically includes: S11. Construct an improved multi-armed slot machine model containing multiple independent actions within each network node. The improved multi-armed slot machine model maps all currently available links of the network node to multiple independent actions, forming an action set. S12. Set an initial selection probability for each independent action in the action set and construct the original selection probability distribution; S13. Collect historical state disturbance data related to the link corresponding to each independent action, calculate the disturbance degree index of the link, and obtain multiple disturbance degree values, which correspond to each independent action in the action set. S14. Based on the original selection probability and perturbation index, the perturbation fusion weight of each independent action is calculated internally in the improved multi-armed slot machine model. S15. Normalize the perturbation fusion weights to form an updated action selection distribution. The updated action selection distribution replaces the original action selection probability distribution and is used to control the action sampling process of the improved multi-armed slot machine model in the current round.

[0009] Optionally, S2 specifically includes: S21. Deploy a network status awareness module in each network node. The network status awareness module performs periodic status sampling operations on all available links connected to the current network node at preset time intervals. S22. The network status awareness module obtains the jump delay parameter of each available link by detecting the round-trip time of a network node sending data packets to neighboring nodes and receiving response confirmations. The jump delay parameter is used to characterize the transmission delay characteristics of data in the link. S23. The network status awareness module records the total number of data packets sent and the total number of response acknowledgment packets received for each available link in each sampling period, and calculates the packet loss rate parameter of the available link. The packet loss rate parameter is used to reflect the data reliability status of the link in that period. S24. Calculate the ratio of the actual data transmission rate to the maximum bandwidth capacity for each available link to obtain the link utilization parameter of the available link. The link utilization parameter is used to represent the degree of use of link resources. S25. During the sampling period, perform synchronous data consistency detection on the jump delay parameter, packet loss rate parameter and link utilization parameter of each available link, and perform state reliability judgment operation to include the three state parameters of available links that meet the preset judgment conditions into the state data set. S26. Encapsulate the hop delay parameters, packet loss rate parameters, and link utilization parameters of each available link determined by the state reliability judgment to construct the network state vector of the network node in the current round.

[0010] Optionally, S3 specifically includes: S31. Obtain an action set in each network node. The action set consists of multiple independent actions, and each independent action corresponds to an available link currently connected to the network node. S32. Combining the network state vector, evaluate the link state corresponding to each independent action in the action set, and select the set of feasible actions that meet the preset feasibility judgment conditions. S33. For each independent action in the set of possible actions, set an initial probability weight value. The initial probability weight values ​​are equally distributed within the set of possible actions, and the sum of the initial probability weight values ​​of each independent action is 1. S34. For independent actions in the action set that are not included in the action set, their initial probability weight values ​​are set to zero, thus obtaining an initial probability weight configuration set that includes all independent actions. S35. Based on the initial probability weight configuration set, construct the initial probability distribution vector of the Exp3 algorithm. The probability distribution vector satisfies the input requirements of the exponential update and sampling structure in the Exp3 algorithm.

[0011] Optionally, S4 specifically includes: S41. Obtain an initial probability distribution vector in each network node, wherein each initial probability value in the initial probability distribution vector corresponds to an independent action in the action set; S42. The network state awareness module deployed in the network node calculates the state fluctuation of each available link in the current round based on the sampled jump delay parameters, packet loss rate parameters and link utilization parameters, and generates a corresponding network state fluctuation guidance factor for each independent action according to the state fluctuation, thus forming a set of network state fluctuation guidance factors. S43. A perturbation weighting operation is performed in the network node, and the initial probability distribution vector and the set of network state fluctuation guidance factors are linearly combined according to a one-to-one correspondence to generate a perturbation probability distribution vector. The perturbation probability distribution vector is used to adjust the sampling priority of each independent action. S44. Normalize the perturbation probability values ​​of each independent action in the perturbation probability distribution vector, and construct a continuous and non-overlapping probability interval mapping structure as the basis for action selection in the current sampling round. S45. A random value between 0 and 1 is generated by the random sampling execution module inside the network node. The random value is mapped to the probability interval mapping structure to locate the corresponding independent action and determine the independent action as the target action of the current round. S46. Extract the target link information corresponding to the selected target action, and send the data packet to be forwarded from the current network node to the adjacent network node through the target link to complete a data forwarding operation.

[0012] Optionally, S5 specifically includes: S51. Construct an instant reward function in each network node. The instant reward function uses jump delay parameter, packet loss rate parameter and link utilization parameter as input variables to quantitatively evaluate the execution effect of the target action. S52. Normalize the jump delay parameter, packet loss rate parameter and link utilization parameter respectively to obtain standardized jump delay parameter, standardized packet loss rate parameter and standardized link utilization parameter, so as to unify the numerical range of different state parameters. S53. Based on the preset weighting coefficients, the standardized jump delay parameter, the standardized packet loss rate parameter, and the standardized link utilization parameter are weighted and combined to generate an instant reward function expression. The weighting coefficients are used to adjust the relative weight of each state parameter in the instant reward value calculation. S54. After the network node performs the packet forwarding operation corresponding to the target action, collect the actual observed values ​​of the jump delay parameter, packet loss rate parameter and link utilization parameter of the corresponding round and use the actual observed values ​​as the input of the instant reward function. S55. The instant reward calculation module in the network node evaluates the execution result of the current action based on the instant reward function and outputs the instant reward value corresponding to the action.

[0013] Optionally, S6 specifically includes: S61. Obtain the instant reward value corresponding to the target action in the current round in each network node; S62. In the network node, the reinforcement learning control module is invoked to perform an exponentially weighted update operation on the current probability weight of the target action according to the preset exponential learning rate parameter and the instantaneous reward value. The update operation adopts a multiplicative weight adjustment method, multiplying the original probability weight by the exponential function value. ,in The parameter is the exponential learning rate. The immediate reward value for the target action; S63. The network node summarizes the probability weight values ​​of all independent actions in the action set and unifies the updated probability weight of the target action with the unupdated probability weights of other independent actions into the same probability weight set. S64. The network nodes perform normalization processing on the probability weight set, so that the sum of the probability weight values ​​corresponding to all independent actions is equal to 1, and generate a corrected probability distribution vector.

[0014] Optionally, S7 specifically includes: S71. In each network node, continuously collect network state vectors within multiple sampling periods. The network state vectors include jump delay parameters, packet loss rate parameters, and link utilization parameters. S72. Set up a network state fluctuation analysis module in the network node, calculate the variance of the jump delay parameter, packet loss rate parameter and link utilization parameter within the time window, and generate the corresponding state fluctuation index set. S73. The jump delay state fluctuation index, packet loss rate state fluctuation index and link utilization state fluctuation index are weighted according to the preset weight coefficient to generate the comprehensive network state fluctuation value of the current network node. S74. Compare the overall network state fluctuation value with the preset dynamic response threshold in the network node. If the overall network state fluctuation value is greater than the dynamic response threshold, the gating adjustment module in the network node is triggered. S75. The gating adjustment module adjusts the exploration and exploitation balance parameters of the Exp3 algorithm in the reinforcement learning control module based on the change range of the comprehensive network state fluctuation value in the current round and in combination with the preset mapping relationship model. The exploration and exploitation balance parameters are used to control the probability perturbation range in the action selection process.

[0015] Optional, including the following modules: The network status awareness module is used to perform status sampling on each available link connected to the network node at preset time intervals, collect jump delay parameters, packet loss rate parameters and link utilization parameters, and construct the network status vector for the corresponding round. An improved multi-armed slot machine modeling module is used to map each available link of a network node into multiple independent actions, forming an action set, and to calculate the perturbation fusion weight of each independent action by combining historical state perturbation data, thereby generating an updated action selection distribution. The action set initialization module is used to filter the set of actionable actions based on the network state vector, set an initial equal probability weight for each independent action in the set of actionable actions, and construct an initial probability weight configuration set. The reinforcement learning control module is used to construct an initial probability distribution vector based on the initial probability weight configuration set and the Exp3 algorithm. After receiving the immediate reward value, it performs an exponential weighted update on the probability weight of the target action and normalizes the probability weight of all independent actions to generate a corrected probability distribution vector. When it receives the instruction from the gating adjustment module, it performs an adaptive adjustment of the exploration and exploitation balance parameters in the Exp3 algorithm. The random sampling and execution module is used to generate a perturbation probability distribution vector based on the corrected probability distribution vector and the set of network state fluctuation guidance factors, determine the target action using a random sampling mechanism, and perform packet forwarding operations through the corresponding link; The instant reward calculation module is used to collect the status parameters of the target link after the data forwarding is completed, construct the instant reward function, and calculate the instant reward value corresponding to the target action; The network state fluctuation analysis module is used to extract the changing trends of each state parameter based on the network state vectors of multiple consecutive rounds, and generate a network state fluctuation index. The gating adjustment module is used to determine whether the network state fluctuation exceeds a preset threshold. If it does, it sends an adjustment command to the reinforcement learning control module to dynamically correct the exploration and exploitation balance parameters in the Exp3 algorithm.

[0016] The beneficial effects of this invention are: (1) This invention constructs an improved multi-armed slot machine model in each network node and combines it with the Exp3 exponentially weighted exploration algorithm to achieve action sampling and weight updates, effectively adapting to non-stationary network topology and link state changes. It provides a fast-converging intelligent routing policy generation mechanism without relying on global modeling. By introducing a network state fluctuation guiding factor and a disturbance weighting mechanism, it achieves adaptive selection of data forwarding paths and improves the policy's responsiveness to network disturbances.

[0017] (2) The present invention constructs a reinforcement learning control module that includes functions such as state awareness, instant reward, exploration and utilization adjustment, and distributed decision-making. It can run independently on each node and achieve self-optimization of link strategy through high-frequency state monitoring and probability adjustment, thereby ensuring the system's latency control and stable transmission in complex network environments and improving the deployability and stability in resource-constrained scenarios.

[0018] (3) This invention establishes an instant reward mechanism that does not require complex path measurement. By integrating three indicators—hop delay, packet loss rate, and link utilization—it achieves lightweight performance evaluation, effectively improving the accuracy and real-time performance of learning feedback. Combined with a gating mechanism to automatically adjust the Exp3 exploration parameters, the system still has the ability to escape global policies when facing severe network fluctuations, effectively avoiding getting trapped in local optima, thereby significantly improving the system's robustness and long-term performance. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0020] Figure 1 The flowchart shows a method and system for intelligent routing traffic management based on reinforcement learning proposed in this invention. Figure 2 This is a flowchart of the construction and perturbation fusion weight calculation process for the improved multi-armed slot machine model in this invention; Figure 3 This is a schematic diagram of the action sampling and probability update mechanism based on the Exp3 algorithm in this invention. Detailed Implementation

[0021] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0022] refer to Figures 1-3 A method and system for intelligent routing traffic management based on reinforcement learning, comprising the following steps: S1. Construct an improved multi-armed slot machine model at each network node. The improved multi-armed slot machine model maps each available link of the network node to multiple independent actions, forming a set of actions, and has the ability to perceive local network status and operate independently. S2. Collect hop delay parameters, packet loss rate parameters, and link utilization parameters through the state awareness module in the network node to generate the current network state vector; S3. Based on the Exp3 algorithm, each action in the action set is assigned an initial equal weight, and a probability distribution vector corresponding to the independent action is constructed by combining the network state vector. S4. Following the random weighted sampling mechanism of the Exp3 algorithm, select an independent action from the probability distribution vector and perform data forwarding through the corresponding link; S5. Construct an instant reward function that integrates hop latency, packet loss rate, and link utilization, based on the forwarding results. Calculate the immediate reward value for the current action; S6. Use the instant reward value to exponentially update the probability weight of the current forwarding action and correct the probability distribution vector for subsequent action selection. S7. When the state awareness module calculates the network state fluctuation trend based on multiple consecutive network state vectors, it calls the gating mechanism to adaptively adjust the exploration and utilization balance parameters in Exp3, thereby enhancing the policy's responsiveness to the dynamic environment.

[0023] In this embodiment, S1 specifically includes: S11. Construct an improved multi-armed slot machine model containing multiple independent actions within each network node. The improved multi-armed slot machine model maps all currently available links of the network node to multiple independent actions, forming an action set. S12. Set an initial selection probability for each independent action in the action set and construct the original selection probability distribution; S13. Collect historical state disturbance data related to the link corresponding to each independent action, calculate the disturbance degree index of the link, and obtain multiple disturbance degree values, which correspond to each independent action in the action set. S14. Based on the original selection probability and perturbation index, the perturbation fusion weight of each independent action is calculated internally in the improved multi-armed slot machine model. S15. Normalize the perturbation fusion weights to form an updated action selection distribution. The updated action selection distribution replaces the original action selection probability distribution and is used to control the action sampling process of the improved multi-armed slot machine model in the current round.

[0024] This implementation constructs an improved multi-armed slot machine model within each network node, mapping each available link to an independent action. It then calculates a perturbation index by combining historical state perturbation data for each link. Based on the initial selection probability, perturbation fusion weights are introduced, and an updated action selection distribution is constructed, making the action sampling process more closely reflect the dynamic characteristics of the links. This mechanism improves the model's sensitivity to link state fluctuations and effectively avoids selecting unstable links in actual decision-making, thereby enhancing the stability and robustness of data forwarding path selection and significantly improving the adaptability and efficiency of reinforcement learning strategies in dynamic network environments.

[0025] In this embodiment, step S2 specifically includes: S21. Deploy a network status awareness module in each network node. The network status awareness module performs periodic status sampling operations on all available links connected to the current network node at preset time intervals. S22. The network status awareness module obtains the jump delay parameter of each available link by detecting the round-trip time of a network node sending data packets to neighboring nodes and receiving response confirmations. The jump delay parameter is used to characterize the transmission delay characteristics of data in the link. S23. The network status awareness module records the total number of data packets sent and the total number of response acknowledgment packets received for each available link in each sampling period, and calculates the packet loss rate parameter of the available link. The packet loss rate parameter is used to reflect the data reliability status of the link in that period. S24. Calculate the ratio of the actual data transmission rate to the maximum bandwidth capacity for each available link to obtain the link utilization parameter of the available link. The link utilization parameter is used to represent the degree of use of link resources. S25. During the sampling period, perform synchronous data consistency detection on the jump delay parameter, packet loss rate parameter and link utilization parameter of each available link, and perform state reliability judgment operation to include the three state parameters of available links that meet the preset judgment conditions into the state data set. S26. Encapsulate the hop delay parameters, packet loss rate parameters, and link utilization parameters of each available link determined by the state reliability judgment to construct the network state vector of the network node in the current round.

[0026] This implementation deploys a network state awareness module in each network node to periodically collect parameters such as hop delay, packet loss rate, and link utilization. This constructs a network state vector reflecting link performance, enabling real-time perception and quantitative representation of link operation status. Specifically, the system calculates hop delay using round-trip time, calculates packet loss rate based on packet sending and acknowledgment response records, and generates link utilization using bandwidth occupancy ratio. Furthermore, it introduces data consistency detection and reliability judgment mechanisms to filter valid state parameters and encapsulate the state vector. This method significantly enhances the ability to perceive link status, ensuring the accuracy and adaptability of action selection in subsequent decision-making processes, and providing a stable and reliable input foundation for reinforcement learning of intelligent routing strategies.

[0027] In this embodiment, S3 specifically includes: S31. Obtain an action set in each network node. The action set consists of multiple independent actions, and each independent action corresponds to an available link currently connected to the network node. S32. Combining the network state vector, evaluate the link state corresponding to each independent action in the action set, and select the set of feasible actions that meet the preset feasibility judgment conditions. S33. For each independent action in the set of possible actions, set an initial probability weight value. The initial probability weight values ​​are equally distributed within the set of possible actions, and the sum of the initial probability weight values ​​of each independent action is 1. S34. For independent actions in the action set that are not included in the action set, their initial probability weight values ​​are set to zero, thus obtaining an initial probability weight configuration set that includes all independent actions. S35. Based on the initial probability weight configuration set, construct the initial probability distribution vector of the Exp3 algorithm. The probability distribution vector satisfies the input requirements of the exponential update and sampling structure in the Exp3 algorithm.

[0028] This implementation method acquires independent actions corresponding to all currently available links at each network node, constructs an action set, and combines this with the network state vector to perform real-time evaluation of the link state, filtering out a set of feasible actions that meet the feasibility criteria. Then, it assigns equal initial probability weights to each independent action within this set, while setting the weights of inactive actions to zero, thus forming a complete initial probability weight configuration set and generating an initial probability distribution vector that meets the structural requirements of the Exp3 algorithm. This method effectively eliminates unavailable links in the initial stage of reinforcement learning, improves action selection efficiency and network adaptability, and provides an accurate and reasonable initial distribution foundation for subsequent learning sampling and policy iteration, thereby enhancing the convergence and stability of routing policies in dynamic networks.

[0029] In this embodiment, S4 specifically includes: S41. Obtain an initial probability distribution vector in each network node, wherein each initial probability value in the initial probability distribution vector corresponds to an independent action in the action set; S42. The network state awareness module deployed in the network node calculates the state fluctuation of each available link in the current round based on the sampled jump delay parameters, packet loss rate parameters and link utilization parameters, and generates a corresponding network state fluctuation guidance factor for each independent action according to the state fluctuation, thus forming a set of network state fluctuation guidance factors. S43. A perturbation weighting operation is performed in the network node, and the initial probability distribution vector and the set of network state fluctuation guidance factors are linearly combined according to a one-to-one correspondence to generate a perturbation probability distribution vector. The perturbation probability distribution vector is used to adjust the sampling priority of each independent action. S44. Normalize the perturbation probability values ​​of each independent action in the perturbation probability distribution vector, and construct a continuous and non-overlapping probability interval mapping structure as the basis for action selection in the current sampling round. S45. A random value between 0 and 1 is generated by the random sampling execution module inside the network node. The random value is mapped to the probability interval mapping structure to locate the corresponding independent action and determine the independent action as the target action of the current round. S46. Extract the target link information corresponding to the selected target action, and send the data packet to be forwarded from the current network node to the adjacent network node through the target link to complete a data forwarding operation.

[0030] This implementation introduces a network state fluctuation guidance factor into each network node based on the initial probability distribution vector and the link state fluctuation calculated by the network state awareness module. The priority of action sampling is adjusted through a perturbation weighting method to form a perturbation probability distribution vector. A probability interval mapping structure is constructed through normalization to realize action selection. Then, the forwarding path is determined by the random sampling execution module. This realizes the introduction of the dynamic fluctuation characteristics of the link state into the Exp3 sampling mechanism, which enhances the sensitivity and adaptability of action selection in complex network environments, thereby improving the robustness of data forwarding and load balancing efficiency.

[0031] In this embodiment, S5 specifically includes: S51. Construct an instant reward function in each network node. The instant reward function uses jump delay parameter, packet loss rate parameter and link utilization parameter as input variables to quantitatively evaluate the execution effect of the target action. S52. Normalize the jump delay parameter, packet loss rate parameter and link utilization parameter respectively to obtain standardized jump delay parameter, standardized packet loss rate parameter and standardized link utilization parameter, so as to unify the numerical range of different state parameters. S53. Based on the preset weighting coefficients, the standardized jump delay parameter, the standardized packet loss rate parameter, and the standardized link utilization parameter are weighted and combined to generate an instant reward function expression. The weighting coefficients are used to adjust the relative weight of each state parameter in the instant reward value calculation. S54. After the network node performs the packet forwarding operation corresponding to the target action, collect the actual observed values ​​of the jump delay parameter, packet loss rate parameter and link utilization parameter of the corresponding round and use the actual observed values ​​as the input of the instant reward function. S55. The instant reward calculation module in the network node evaluates the execution result of the current action based on the instant reward function and outputs the instant reward value corresponding to the action.

[0032] This implementation constructs an instant reward function based on hop delay, packet loss rate, and link utilization parameters in each network node to quantitatively evaluate the performance of data forwarding behavior. It also introduces parameter normalization and weighted combination mechanisms to achieve unified processing and dynamic adjustment of multi-dimensional link states. After each round of data forwarding, the method uses the actually collected link state parameters as input and outputs the instant reward value of the target action through the instant reward calculation module, effectively reflecting the quality of the current action decision. This implementation enhances the algorithm's sensitivity and responsiveness to differences in link performance, provides accurate feedback for subsequent probability weight updates, and improves the adaptability and optimization efficiency of routing strategies in dynamic network environments.

[0033] In this embodiment, S6 specifically includes: S61. Obtain the instant reward value corresponding to the target action in the current round in each network node; S62. In the network node, the reinforcement learning control module is invoked to perform an exponentially weighted update operation on the current probability weight of the target action according to the preset exponential learning rate parameter and the instantaneous reward value. The update operation adopts a multiplicative weight adjustment method, multiplying the original probability weight by the exponential function value. ,in The parameter is the exponential learning rate. The immediate reward value for the target action; S63. The network node summarizes the probability weight values ​​of all independent actions in the action set and unifies the updated probability weight of the target action with the unupdated probability weights of other independent actions into the same probability weight set. S64. The network nodes perform normalization processing on the probability weight set, so that the sum of the probability weight values ​​corresponding to all independent actions is equal to 1, and generate a corrected probability distribution vector.

[0034] This implementation acquires the instantaneous reward value of the target action at each network node and dynamically updates the probability weight of the target action using an exponentially weighted multiplication method, combined with a preset exponential learning rate parameter. Then, normalization is performed on the unified probability weight set to generate a corrected probability distribution vector, achieving real-time adaptive adjustment of the action sampling strategy. This mechanism enables the system to continuously optimize action selection based on historical forwarding results, maintaining superior path decision-making capabilities even in the face of dynamic network state changes. This improves data forwarding efficiency and stability, and enhances the overall network's self-learning and autonomous control capabilities.

[0035] In this embodiment, S7 specifically includes: S71. In each network node, continuously collect network state vectors within multiple sampling periods. The network state vectors include jump delay parameters, packet loss rate parameters, and link utilization parameters. S72. Set up a network state fluctuation analysis module in the network node, calculate the variance of the jump delay parameter, packet loss rate parameter and link utilization parameter within the time window, and generate the corresponding state fluctuation index set. S73. The jump delay state fluctuation index, packet loss rate state fluctuation index and link utilization state fluctuation index are weighted according to the preset weight coefficient to generate the comprehensive network state fluctuation value of the current network node. S74. Compare the overall network state fluctuation value with the preset dynamic response threshold in the network node. If the overall network state fluctuation value is greater than the dynamic response threshold, the gating adjustment module in the network node is triggered. S75. The gating adjustment module adjusts the exploration and exploitation balance parameters of the Exp3 algorithm in the reinforcement learning control module based on the change range of the comprehensive network state fluctuation value in the current round and in combination with the preset mapping relationship model. The exploration and exploitation balance parameters are used to control the probability perturbation range in the action selection process.

[0036] This implementation continuously collects network state vectors from each network node over multiple rounds. A variance analysis module is used to analyze the jump delay, packet loss rate, and link utilization parameters to obtain fluctuation indices for each state parameter. These indices are then weighted to generate a comprehensive network state volatility value. When this volatility value exceeds a preset dynamic response threshold, a gating adjustment module is triggered. Based on the fluctuation amplitude, the exploration and utilization balance parameters of the Exp3 algorithm in the reinforcement learning control module are adjusted. This dynamically adjusts the aggressiveness of the action selection strategy during perturbation sampling, enabling adaptive response to volatile network environments and improving the algorithm's decision-making stability and traffic management robustness in dynamic network environments.

[0037] This embodiment includes the following modules: The network status awareness module is used to perform status sampling on each available link connected to the network node at preset time intervals, collect jump delay parameters, packet loss rate parameters and link utilization parameters, and construct the network status vector for the corresponding round. An improved multi-armed slot machine modeling module is used to map each available link of a network node into multiple independent actions, forming an action set, and to calculate the perturbation fusion weight of each independent action by combining historical state perturbation data, thereby generating an updated action selection distribution. The action set initialization module is used to filter the set of actionable actions based on the network state vector, set an initial equal probability weight for each independent action in the set of actionable actions, and construct an initial probability weight configuration set. The reinforcement learning control module is used to construct an initial probability distribution vector based on the initial probability weight configuration set and the Exp3 algorithm. After receiving the immediate reward value, it performs an exponential weighted update on the probability weight of the target action and normalizes the probability weight of all independent actions to generate a corrected probability distribution vector. When it receives the instruction from the gating adjustment module, it performs an adaptive adjustment of the exploration and exploitation balance parameters in the Exp3 algorithm. The random sampling and execution module is used to generate a perturbation probability distribution vector based on the corrected probability distribution vector and the set of network state fluctuation guidance factors, determine the target action using a random sampling mechanism, and perform packet forwarding operations through the corresponding link; The instant reward calculation module is used to collect the status parameters of the target link after the data forwarding is completed, construct the instant reward function, and calculate the instant reward value corresponding to the target action; The network state fluctuation analysis module is used to extract the changing trends of each state parameter based on the network state vectors of multiple consecutive rounds, and generate a network state fluctuation index. The gating adjustment module is used to determine whether the network state fluctuation exceeds a preset threshold. If it does, it sends an adjustment command to the reinforcement learning control module to dynamically correct the exploration and exploitation balance parameters in the Exp3 algorithm.

[0038] Example 1: To verify the feasibility of this invention in practice, it was applied to the backbone routing cluster of a large-scale high-performance computing data center. This data center supports multiple research systems, handling an average of 200 million network forwarding requests daily. Its typical network environment is characterized by high link density, strong traffic dynamics, sensitivity to packet loss, and frequent path switching. For a long time, this data center has primarily relied on traditional static routing mechanisms such as OSPF and ECMP. When faced with sudden traffic disturbances or link jitter, these mechanisms suffer from slow convergence speeds, delayed policy updates, and unstable forwarding paths. This results in an average link utilization rate of less than 70% and an average latency exceeding 180ms during critical periods, severely restricting service performance and the experience of upper-layer systems.

[0039] In this embodiment, the system of the present invention is deployed in each core node and edge node of the data center. Each network node integrates the improved multi-armed slot machine model, state awareness module, Exp3 decision unit, gating mechanism, instant reward calculation module and reinforcement learning control module proposed in this invention, forming a fully distributed decentralized intelligent learning architecture that supports local state awareness, autonomous action selection and rapid policy evolution.

[0040] In actual operation, the system first constructs an improved multi-armed slot machine model within each network node, mapping all available links of the current node to independent actions and establishing an action set. The state awareness module collects parameters such as jump latency, link packet loss rate, and utilization rate in real time every 5 seconds to construct the network state vector of the current node. During the initialization phase, the system assigns equal initial weights to each action and constructs a probability distribution vector using the Exp3 algorithm. Subsequently, based on the current state fluctuation guidance factor, it performs perturbation sampling, selects the target action, and forwards data through the corresponding link.

[0041] Once data forwarding is complete, the system immediately constructs an instant reward function based on the collected actual observations (including link latency, packet loss, and utilization in the current round) and calculates the instant reward value for the target action. The reinforcement learning control module performs an exponential update on the action weights and normalizes the probability vector before proceeding to the next round of action selection. When the state awareness module continuously detects network state fluctuations exceeding a preset threshold, it triggers a gating adjustment mechanism, adjusting the exploration-utilization balance parameter in Exp3 according to the current network fluctuation intensity, thereby achieving agile response to high-fluctuation regions and improving policy escaping capabilities.

[0042] To quantitatively analyze the actual effectiveness of the system of this invention, network performance indicators were compared before and after deployment in the same data center. The experiment lasted for three months, and relevant data statistics were provided by the data center network monitoring platform. The comparison dimensions included policy convergence time, average link utilization, average network latency, path stability, packet loss rate, and algorithm processing overhead. Specific data are shown in Table 1.

[0043] Table 1. Performance Comparison of the Invention System and Traditional Routing Algorithms ; As shown in Table 1, the system of this invention achieves significant improvements in several core performance indicators. Specifically, in complex network environments, the average policy convergence time is reduced by 72.4%, significantly improving the routing response speed under dynamic topology changes; the average link utilization rate is increased by approximately 23.3 percentage points, significantly improving the problem of insufficient resource utilization; the average end-to-end latency is reduced by 43.1%, improving the real-time performance of service communication; during peak periods, path switching stability is enhanced, with the number of fluctuations decreasing from 32 times per hour to 7 times, significantly improving the overall stability of the system; and the packet loss rate is also reduced to less than 0.7%, ensuring the reliability of data transmission.

[0044] Although the intelligent strategy execution of this invention requires a certain computational overhead, it is kept within an acceptable range during actual operation, with the CPU utilization rate increasing by less than 3.5%. At the same time, the reinforcement learning strategy can support dynamic updates on a second-level basis, which significantly shortens the update cycle compared to the traditional static routing strategy, enhancing the system's timeliness and fine-grained control capabilities.

[0045] Furthermore, in a sudden link interruption experiment, the system node of this invention completed policy adjustment and path rerouting within 750ms after detecting link fluctuations, which is much faster than the recovery time of more than 2 seconds of traditional algorithms. This avoided service packet loss and interruption, and fully verified the adaptability of the gating adjustment mechanism to network disturbances.

[0046] In summary, this embodiment demonstrates the comprehensive advantages of the reinforcement learning-based intelligent routing traffic management system proposed in this invention in a high-intensity, highly dynamic data center backbone network, showcasing its advantages in policy adaptability, link utilization, real-time performance, and stability. The system possesses excellent scalability and deployment flexibility, making it suitable for various large-scale heterogeneous network environments. It can provide operators, research platforms, and others with efficient and intelligent traffic scheduling and management capabilities, solving the problems of poor adaptability to dynamic environments and significant performance bottlenecks in existing routing algorithms. It has broad practical application prospects and promotional value.

[0047] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for intelligent routing traffic management based on reinforcement learning, characterized in that, The method comprises the following steps: S1, constructing an improved multi-armed bandit model in each network node, the improved multi-armed bandit model mapping each available link of the network node into a plurality of independent actions to form an action set; S2, collecting a hop delay parameter, a packet loss rate parameter and a link utilization rate parameter through a state perception module in the network node to generate a current network state vector; S3, based on an Exp3 algorithm, assigning an initial equal weight to each action in the action set, and constructing a probability distribution vector corresponding to the independent actions in combination with the network state vector; S4, selecting an independent action from the probability distribution vector according to a random weighted sampling mechanism of the Exp3 algorithm, and performing data forwarding through the corresponding link; S5, constructing an instant reward function integrating the hop delay, the packet loss rate and the link utilization rate, and calculating an instant reward value of the current action according to the forwarding result; S6, updating the probability weight of the current forwarding action by the instant reward value, and correcting the probability distribution vector; S7, when the state perception module calculates a network state fluctuation trend based on a plurality of continuous network state vectors, calling a gating mechanism to adaptively adjust an exploration and utilization balance parameter in the Exp3. 2.The intelligent routing traffic management method based on reinforcement learning of claim 1, wherein, The S1 specifically comprises: S11, constructing an improved multi-armed bandit model containing a plurality of independent actions in each network node, the improved multi-armed bandit model mapping all available links of the network node into a plurality of independent actions to form an action set; S12, setting an initial selection probability for each independent action in the action set, and constructing an original selection probability distribution; S13, collecting historical state disturbance data related to the link corresponding to each independent action, calculating a disturbance degree index of the link, and obtaining a plurality of disturbance degree values corresponding to each independent action in the action set; S14, calculating a disturbance fusion weight of each independent action in the improved multi-armed bandit model based on the original selection probability and the disturbance degree index; S15, normalizing the disturbance fusion weight to form an updated action selection distribution. 3.The intelligent routing traffic management method based on reinforcement learning of claim 1, wherein, The S2 specifically comprises: S21, deploying a network state perception module in each network node, the network state perception module performing periodic state sampling operations on all available links connected to the current network node at a preset time interval; S22, the network state perception module acquiring a hop delay parameter of each available link by detecting a round trip time of sending a data packet by the network node to a neighboring node and receiving a response confirmation, the hop delay parameter being used to represent a transmission delay characteristic of data in the link; S23, the network state perception module recording a total number of data packets sent and a total number of response confirmation packets received of each available link in each sampling period to calculate a packet loss rate parameter of the available link, the packet loss rate parameter being used to reflect a data reliability condition of the link in the period; S24, calculating a ratio of an actual data transmission rate to a maximum bandwidth capacity of each available link to obtain a link utilization rate parameter of the available link, the link utilization rate parameter being used to represent a usage degree of the link resource; S25, synchronously detecting data consistency of the hop delay parameter, the packet loss rate parameter and the link utilization rate parameter of each available link in a sampling period, and performing state credibility determination operation, and adding the three state parameters of the available link meeting the preset determination condition into the state data set; S26, encapsulating the hop delay parameter, the packet loss rate parameter and the link utilization rate parameter of each available link determined by the state credibility determination, and constructing a network state vector of the current round of the network node. 4.The intelligent routing traffic management method based on reinforcement learning of claim 1, wherein, The S3 specifically comprises: S31, obtaining an action set in each network node, the action set being composed of a plurality of independent actions, each independent action corresponding to an available link currently connected by the network node; S32, evaluating the link state corresponding to each independent action in the action set in combination with the network state vector, and screening an actionable action set meeting a preset feasibility determination condition; S33, setting an initial probability weight value for each independent action in the actionable action set, the initial probability weight value being equally distributed in the actionable action set, and the sum of the initial probability weight values of the independent actions being 1; S34, setting the initial probability weight value of the independent action not included in the actionable action set in the action set as zero, and obtaining an initial probability weight configuration set containing all the independent actions; S35, constructing an initial probability distribution vector of the Exp3 algorithm based on the initial probability weight configuration set, the probability distribution vector meeting the input requirement of the exponential update and sampling structure in the Exp3 algorithm. 5.The intelligent routing traffic management method based on reinforcement learning of claim 1, wherein, The S4 specifically comprises: S41, obtaining an initial probability distribution vector in each network node, each initial probability value in the initial probability distribution vector corresponding to an independent action in the action set; S42, calculating the state fluctuation degree of each available link in the current round by the network state perception module deployed in the network node based on the hop delay parameter, the packet loss rate parameter and the link utilization rate parameter obtained by sampling, and generating a corresponding network state fluctuation guide factor for each independent action according to the state fluctuation degree, to form a network state fluctuation guide factor set; S43, performing perturbation weighting operation in the network node, linearly combining the initial probability distribution vector and the network state fluctuation guide factor set in a one-to-one correspondence to generate a perturbed probability distribution vector, the perturbed probability distribution vector being used to adjust the sampling priority of each independent action; S44, normalizing the perturbed probability values of each independent action in the perturbed probability distribution vector, and constructing a continuous and non-overlapping probability interval mapping structure as the basis for action selection in the current sampling round; S45, generating a random number between 0 and 1 by a random sampling execution module inside the network node, mapping the random number to the probability interval mapping structure, and locating the corresponding independent action to determine the independent action as the target action in the current round; S46, extracting target link information corresponding to the selected target action, and sending the current data packet to be forwarded from the current network node to the adjacent network node through the target link, to complete a data forwarding operation. 6.The intelligent routing traffic management method based on reinforcement learning of claim 1, wherein, The S5 specifically comprises: S51, constructing an immediate reward function in each network node, the immediate reward function taking a hop delay parameter, a packet loss rate parameter and a link utilization rate parameter as input variables, and being used for quantitatively evaluating an execution effect of a target action; S52, respectively performing normalization processing on the hop delay parameter, the packet loss rate parameter and the link utilization rate parameter, to obtain a standardized hop delay parameter, a standardized packet loss rate parameter and a standardized link utilization rate parameter, so as to unify numerical value intervals of different state parameters; S53, generating an expression of the immediate reward function by performing weighted combination on the standardized hop delay parameter, the standardized packet loss rate parameter and the standardized link utilization rate parameter according to a preset weighting coefficient, the weighting coefficient being used for adjusting relative weights of the state parameters in calculation of an immediate reward value; S54, after a data packet forwarding operation corresponding to the target action is executed in the network node, collecting actual observation values of the hop delay parameter, the packet loss rate parameter and the link utilization rate parameter of a corresponding link in a corresponding round, and taking the actual observation values as inputs of the immediate reward function; S55, evaluating, by an immediate reward calculation module in the network node, an execution result of a current action based on the immediate reward function, and outputting an immediate reward value corresponding to the action. 7.The intelligent routing traffic management method based on reinforcement learning of claim 1, wherein, The S6 specifically comprises: S61, obtaining, in each network node, an immediate reward value corresponding to a target action in a current round; S62, calling the reinforcement learning control module in the network node, performing exponential weighting update operation on the current probability weight of the target action according to the preset exponential learning rate parameter and the instant reward value, the update operation adopts multiplication weight adjustment mode, and the original probability weight is multiplied by the exponential function value wherein is the exponential learning rate parameter, is the instant reward value of the target action; S63, performing summary processing on probability weight values of all independent actions in an action set by the network node, and uniformly classifying a target action probability weight updated by an index and other independent action probability weights not updated into a same probability weight set; S64, performing normalization processing on the probability weight set by the network node, so that a sum of probability weight values corresponding to all independent actions is equal to 1, and generating a corrected probability distribution vector. 8.The intelligent routing traffic management method based on reinforcement learning of claim 1, wherein, The S7 specifically comprises: S71, continuously collecting, in each network node, network state vectors in a plurality of sampling periods, the network state vectors including the hop delay parameter, the packet loss rate parameter and the link utilization rate parameter; S72, setting a network state fluctuation analysis module in the network node, and respectively performing variance calculation on the hop delay parameter, the packet loss rate parameter and the link utilization rate parameter in a time window, to generate a corresponding state fluctuation index set; S73, performing weighted processing on the hop delay state fluctuation index, the packet loss rate state fluctuation index and the link utilization rate state fluctuation index according to a preset weight coefficient, to generate a comprehensive network state fluctuation degree value of the current network node; S74, comparing the comprehensive network state fluctuation degree value with a preset dynamic response threshold value in the network node, and if the comprehensive network state fluctuation degree value is greater than the dynamic response threshold value, triggering a gating adjustment module in the network node; S75, adjusting, by the gating adjustment module, an exploration and utilization balance parameter of an Exp3 algorithm in a reinforcement learning control module according to a variation amplitude of the comprehensive network state fluctuation degree value in a current round and in combination with a preset mapping relationship model, the exploration and utilization balance parameter being used for controlling a probability disturbance range in an action selection process.

9. A reinforcement learning based intelligent routing traffic management system characterized in that, The following modules are included: The network state perception module is configured to perform state sampling on each available link connected to the network node at a preset time interval, collect a hop delay parameter, a packet loss rate parameter, and a link utilization rate parameter, and construct a network state vector corresponding to a round; The improved multi-arm bandit modeling module is configured to map each available link of the network node to a plurality of independent actions respectively, construct an action set, and calculate a disturbance fusion weight of each independent action based on historical state disturbance data, to generate an updated action selection distribution; The action set initialization module is configured to filter an actionable action set based on the network state vector, set an initial equal-probability weight for each independent action in the actionable action set, and construct an initial probability weight configuration set; The reinforcement learning control module is configured to construct an initial probability distribution vector based on the initial probability weight configuration set and an Exp3 algorithm, perform exponential weighting update on a probability weight of a target action after receiving an immediate reward value, normalize the probability weights of all independent actions, generate a corrected probability distribution vector, and perform adaptive adjustment on an exploration-exploitation balance parameter in the Exp3 algorithm when receiving an instruction from the gating adjustment module; The random sampling and execution module is configured to generate a disturbed probability distribution vector based on the corrected probability distribution vector and a network state fluctuation factor set, determine a target action by using a random sampling mechanism, and perform a data packet forwarding operation through a corresponding link; The immediate reward calculation module is configured to collect state parameters of a target link after data forwarding is completed, construct an immediate reward function, and calculate an immediate reward value corresponding to the target action; The network state fluctuation analysis module is configured to extract a change trend of each state parameter based on network state vectors of a plurality of consecutive rounds, and generate a network state fluctuation degree index; The gating adjustment module is configured to determine whether the network state fluctuation degree exceeds a preset threshold, and send an adjustment instruction to the reinforcement learning control module to dynamically correct the exploration-exploitation balance parameter in the Exp3 algorithm if the network state fluctuation degree exceeds the preset threshold.

Images