Self-adaptive congestion control method and device based on time sequence prediction and reinforcement learning
By combining time-series prediction and reinforcement learning methods, the packet transmission interval is dynamically adjusted, which solves the instability problem of bandwidth changes in modern wireless network environments, achieves efficient congestion control, and improves transmission performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XIDIAN UNIV
- Filing Date
- 2025-12-24
- Publication Date
- 2026-05-05
AI Technical Summary
Existing congestion control algorithms lack forward-looking bandwidth awareness and proactive rate planning when facing highly dynamic bandwidth changes in modern wireless network environments, resulting in unstable transmission performance and an inability to effectively avoid congestion.
An adaptive congestion control method based on time-series prediction and reinforcement learning is adopted. By constructing a lightweight GRU neural network to predict the bandwidth gradient trend, and combining it with a deep reinforcement learning policy network, the packet sending interval is dynamically adjusted to achieve active detection or avoidance of network bandwidth fluctuations.
It improves network transmission stability and throughput, reduces queuing latency and packet loss, and increases bandwidth utilization by approximately 15% to 20%.
Smart Images
Figure CN121985373A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of communication technology, specifically relating to an adaptive congestion control method and apparatus based on time-series prediction and reinforcement learning. Background Technology
[0002] In modern wireless network environments such as 5G and Wi-Fi 6, link bandwidth exhibits highly dynamic and time-varying characteristics. This unstable network environment poses a severe challenge to transport layer congestion control algorithms, seriously affecting the transmission performance of high-throughput and low-latency services.
[0003] In existing technologies, traditional congestion control algorithms (such as Cubic and BBR) mainly rely on measurements of network states (such as maximum bandwidth and minimum round-trip time) over a past period to construct transmission models. Due to the lack of predictive ability for future states, they often exhibit significant "feedback lag" when facing bandwidth fluctuations caused by wireless channel fading. Existing deep learning-based optimization methods mostly focus on using models to predict network latency or packet loss probability, thereby optimizing the retransmission timeout threshold (RTO) or triggering fast retransmission. These methods are essentially "fault recovery" mechanisms, passively remedying congestion or packet loss after it occurs, failing to address the core issue of how to "actively plan" the transmission rate based on bandwidth change trends, and thus unable to avoid congestion at its source. Existing end-to-end reinforcement learning schemes typically allow the agent to directly output the specific value of the congestion window (CWND). This "black box" control method has too much action space and abandons the protection mechanisms of the underlying protocol stack, making it prone to causing drastic throughput fluctuations in complex network environments. Furthermore, the model is difficult to converge and lacks stability and robustness in dynamic environments.
[0004] Therefore, there is an urgent need to provide an adaptive congestion control method and device to address the shortcomings of existing technologies. Summary of the Invention
[0005] To address the aforementioned problems in the existing technology, this invention provides an adaptive congestion control method and apparatus based on time-series prediction and reinforcement learning. The technical problem to be solved by this invention is achieved through the following technical solution: In a first aspect, the present invention provides an adaptive congestion control method based on time-series prediction and reinforcement learning, comprising: The system collects network status data of the transport layer protocol stack in the network environment, constructs a sliding window state sequence with multiple time steps, and obtains a snapshot of the current network status. The network status data includes round-trip time, transmission throughput, current packet loss rate, and current congestion window size. A pre-trained gated recurrent unit neural network is used to process the sliding window state sequence to obtain a bandwidth gradient trend factor that characterizes the future bandwidth change rate and direction. Specifically, the reset gate module in the pre-trained gated recurrent unit neural network filters short-term random noise, and the update gate module in the pre-trained gated recurrent unit neural network extracts long-term dependency features. By concatenating the bandwidth gradient trend factor with the current network state snapshot, an enhanced state vector containing prior trend information is obtained. The enhanced state vector is processed using a pre-trained deep reinforcement learning policy network to obtain a continuous gain scaling factor; The BBR congestion control algorithm is used to obtain the reference pacing gain under the current state machine phase. Based on the reference pacing gain and the continuous gain scaling factor, the final pacing gain is calculated. The transmission interval of data packets in the network environment is dynamically adjusted according to the final pacing gain to achieve preemptive detection or defensive avoidance of network bandwidth fluctuations.
[0006] Secondly, the present invention also provides an adaptive congestion control device based on time-series prediction and reinforcement learning, comprising: The data acquisition module is used to collect network status data of the transport layer protocol stack in the network environment, construct a sliding window state sequence containing multiple time steps, and obtain a snapshot of the current network status; among which, the network status data includes round-trip time, transmission throughput, current packet loss rate, and current congestion window size; Data processing module one is used to process the sliding window state sequence using a pre-trained gated recurrent unit neural network to obtain the bandwidth gradient trend factor, which represents the rate and direction of future bandwidth change. Specifically, the reset gate module in the pre-trained gated recurrent unit neural network filters short-term random noise, and the update gate module in the pre-trained gated recurrent unit neural network extracts long-term dependency features. The feature concatenation model is used to concatenate the bandwidth gradient trend factor with the current network state snapshot to obtain an enhanced state vector containing prior trend information. Data processing module two is used to process the augmented state vector using a trained deep reinforcement learning policy network to obtain continuous gain scaling coefficients; The result acquisition module is used to obtain the reference pacing gain under the current state machine phase using the BBR congestion control algorithm, calculate the final pacing gain based on the reference pacing gain and the continuous gain scaling factor, and dynamically adjust the data packet transmission interval in the network environment based on the final pacing gain to achieve preemptive detection or defensive avoidance of network bandwidth fluctuations.
[0007] The beneficial effects of this invention are: This invention provides an adaptive congestion control method and apparatus based on time-series prediction and reinforcement learning. By constructing a joint closed-loop framework of "time-series bandwidth trend prediction + reinforcement learning decision fine-tuning + low-level gain parameter control", it improves the shortcomings of existing methods in that they cannot balance forward-looking bandwidth perception, active rate planning and control stability, and achieves a technological leap from "passive adaptation" to "active control".
[0008] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0009] Figure 1 This is a flowchart of an adaptive congestion control method based on time-series prediction and reinforcement learning provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of an adaptive congestion control method based on time-series prediction and reinforcement learning provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a gated recurrent unit neural network provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a deep reinforcement learning policy network provided in an embodiment of the present invention. Detailed Implementation
[0010] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.
[0011] This invention provides an adaptive congestion control method based on time-series prediction and reinforcement learning. By constructing a joint closed-loop framework of "time-series bandwidth trend prediction + reinforcement learning decision fine-tuning + low-level gain parameter control", it improves the shortcomings of existing methods in that they cannot balance forward-looking bandwidth perception, proactive rate planning and control stability, and achieves a technological leap from "passive adaptation" to "active control".
[0012] Specifically, the present invention aims to solve the following technical problems: First, this invention addresses the congestion accumulation problem caused by delayed perception in highly dynamic networks. In wireless network scenarios, bandwidth fluctuations often precede packet loss or increases in round-trip time (RTT). Existing traditional algorithms rely solely on historical measurements and cannot detect the "precursors" of bandwidth abrupt changes. When actual bandwidth suddenly drops, the sending end, lacking prediction, continues to send data at a higher estimated bandwidth, causing a momentary buffer overflow at network bottlenecks, leading to severe queuing delays and packet loss. This invention urgently aims to solve the problem of how to perceive bandwidth trends in advance before congestion occurs, breaking the deadlock of traditional algorithms that "measure but do not predict."
[0013] Second, existing solutions suffer from a lack of clear prediction objectives and transmission planning. Most existing solutions are limited to predicting packet loss to optimize retransmission (fault recovery), lacking explicit prediction of future bandwidth trends and failing to proactively plan transmission rates from the source. While simple time-series prediction models can sense bandwidth change trends, they cannot directly determine the control problem of "how much the transmission rate should be adjusted." Directly using predicted values as the transmission rate can lead to catastrophic consequences if predictions are inaccurate. Therefore, it is urgent to solve how to effectively translate future trend information into current control actions, i.e., to build a decision network that integrates predicted trends and current states, achieving a smooth mapping from "predictive data" to "control strategy," avoiding the limitations of single predictions and the blindness of pure reinforcement learning.
[0014] Please see Figure 1 and Figure 2 , Figure 1 This is a flowchart of an adaptive congestion control method based on time-series prediction and reinforcement learning provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of an adaptive congestion control method based on temporal prediction and reinforcement learning provided in an embodiment of the present invention. The adaptive congestion control method based on temporal prediction and reinforcement learning provided by the present invention includes: S101. Collect network status data of the transport layer protocol stack in the network environment, construct a sliding window state sequence containing multiple time steps, and obtain a snapshot of the current network status; wherein, the network status data includes round-trip time (RTT), delivery rate, current packet loss rate, and current congestion window size.
[0015] Specifically, in this embodiment, the network environment includes, but is not limited to, 5G / 6G millimeter-wave mobile communication networks, Wi-Fi 6 / 7 high-density wireless local area networks, and low-Earth orbit (LEO) satellite non-terrestrial networks; these network environments are characterized by high dynamic bandwidth fluctuations, large channel random noise, and severe round-trip delay jitter.
[0016] S102. The sliding window state sequence is processed using a trained gated recurrent unit (GRU) neural network to obtain a bandwidth gradient trend factor characterizing the future bandwidth change rate and direction. Optionally, the bandwidth gradient trend factor... A scalar normalized to the interval [0, 1]; wherein, short-term random noise is filtered by the reset gate module in the trained gated recurrent unit neural network, and long-term dependent features are extracted by the update gate module in the trained gated recurrent unit neural network.
[0017] Specifically, in this embodiment, please refer to Figure 3 , Figure 3This is a schematic diagram of a gated recurrent unit neural network provided in an embodiment of the present invention. To address the lag problem of the traditional BBR congestion control algorithm in non-steady-state networks, a lightweight GRU neural network is designed to predict the gradient of bandwidth changes rather than the absolute value. Predicting absolute bandwidth values is highly susceptible to interference from network noise (such as ACK aggregation), while predicting the trend direction has stronger robustness.
[0018] A pre-trained gated recurrent unit neural network is used to process the sliding window state sequence to obtain bandwidth gradient trend factors that characterize the future bandwidth change rate and direction, including: Sliding window state sequence The first in Feature vectors at each time step Normalization is performed, and the data is then input into the trained gated recurrent neural network for processing. The length of the sliding window is optional. ; Among them, the Feature vectors at each time step The normalized features include: normalized smooth round-trip time. (Smoothed RTT), normalized bottleneck bandwidth sample value (Bottleneck Bandwidth Sample), Current Packet Loss Rate (Loss Rate), Current Congestion Window Size (Congestion Window), first-order difference of round-trip delay (The rate of change of the delay) and the first difference of the bandwidth (Characterizing the trend of bandwidth increase or decrease); It should be noted that the difference term and This is a key technical feature of the present invention, which aims to explicitly enable the neural network to capture the dynamic changes in the network state, rather than focusing only on static values; Computational update gate (The decision on how much historical memory to retain) is represented as: ; in, This represents the Sigmoid function. This indicates updating the gate weights. Indicates the first The hidden state corresponding to each time step; Calculate the reset door (Deciding whether to ignore sudden noise) is expressed as: ; in, This indicates resetting the gate weight; Calculate candidate hidden state , is represented as: ; in, This represents the Tanh activation function. Represents the weight matrix. This represents element-wise multiplication, i.e., the Hadamard product, and represents the weighted control of historical information by the reset gate; Update hidden status , is represented as: ; A fully connected layer is used to process the hidden states corresponding to all time steps, and then the Tanh activation function is used to process the result of the fully connected layer to obtain the bandwidth gradient trend factor. , This represents an approximation of the first derivative of the bandwidth capacity over a future control cycle. Positive values indicate an increase in available bandwidth, while negative values indicate increased congestion.
[0019] S103. Concatenate the bandwidth gradient trend factor with the current network state snapshot to obtain an enhanced state vector containing prior trend information. .
[0020] Specifically, in this embodiment, the enhanced state vector is obtained. This achieves a fundamental upgrade in perception, moving from instantaneous static observation to temporal forward prediction.
[0021] S104. The trained deep reinforcement learning policy network is used to process the augmented state vector to obtain the continuous gain scaling coefficient.
[0022] Specifically, in this embodiment, a trained deep reinforcement learning policy network is used to process the augmented state vector to obtain continuous gain scaling coefficients, including: Based on the magnitude of the bandwidth gradient trend factor, the policy network in the trained deep reinforcement learning policy network is used to process the augmentation state vector to obtain the action. , Represents the augmented state vector. A function representing a deep reinforcement learning policy network. This represents the structure of a deep reinforcement learning policy network. This represents the set of trainable weight parameters in a deep reinforcement learning policy network. Actions Mapping to continuous coefficients ; The policy (Actor) network includes multiple fully connected modules, Tanh activation functions, and scaling modules. The fully connected modules include fully connected layers and ReLU activation functions.
[0023] Please see Figure 4 , Figure 4 This is a schematic diagram of a deep reinforcement learning policy network provided in an embodiment of the present invention. The Actor network uses a multi-layer fully connected structure superimposed with the ReLU nonlinear activation function to incorporate future bandwidth gradient factors. Deep representation extraction is performed on the enhanced features. During this process, No longer isolated numerical values, but rather crucial prior guiding signals, prompting the neural network to establish a strong correlation between future trend directions and current control gains in the latent space. When the prediction module outputs... When in the positive expansion region, the Actor network undergoes a nonlinear transformation, mapping high-dimensional features to continuous gain scaling factors greater than 1 in the action space. This drives the algorithm to perform preemptive probing; conversely, if The network outputs a value less than 1, indicating a potential congestion risk. The value triggers defensive avoidance logic.
[0024] Deep reinforcement learning policy networks also include a value (Actor) network, which is used when training deep reinforcement learning policy networks.
[0025] Critic network synchronous processing enhancement vector Evaluate the expected long-term reward in the current fusion state and output the state value. This is used to calculate the action advantage value and guide the parameter updates of the Actor network. Through this hierarchical and decoupled processing mechanism, the agent can effectively eliminate the "perception-decision" time lag existing in traditional algorithms, transforming ambiguous bandwidth fluctuation trends into precise pacing gain fine-tuning instructions. While ensuring the steady-state operation of the BBR state machine, it significantly improves the predictability and flexibility of the control strategy in complex dynamic network environments.
[0026] Optionally, the deep reinforcement learning policy network can be based on the proximal policy optimization (PPO) algorithm.
[0027] Furthermore, based on the magnitude of the bandwidth gradient trend factor, the policy network in the trained deep reinforcement learning policy network is used to process the augmented state vector, including: When the bandwidth gradient trend factor is greater than the positive threshold , The threshold represents the network bandwidth expansion trend. When the continuous gain scaling factor of the trained deep reinforcement learning strategy network output is greater than the first threshold, the BBR congestion control algorithm is triggered to execute the active detection logic. When the bandwidth gradient trend factor is less than the negative threshold This indicates that the network bandwidth is shrinking or there are signs of impending congestion. When the continuous gain scaling factor of the trained deep reinforcement learning strategy network output is less than the first threshold, the BBR congestion control algorithm is triggered to execute the active avoidance logic. When the absolute value of the bandwidth gradient trend factor is less than or equal to the positive threshold, This indicates that the network bandwidth is in a steady state. The continuous gain scaling factor of the trained deep reinforcement learning policy network output is equal to the first threshold, triggering the BBR congestion control algorithm to execute the baseline control logic.
[0028] For example, proactive probing: when the GRU neural network predicts that the bandwidth is about to be released ( Furthermore, when the BBR congestion control algorithm is in the "cruise" (Gain=1.0) phase, the deep reinforcement learning policy network outputs... (e.g., 1.2).
[0029] Effect: Actual gain becomes The system can immediately increase the transmission rate without waiting for the 8 RTT cycles to end, improving utilization by approximately 15% to 20%.
[0030] Defensive Backoff: When the GRU neural network predicts a worsening trend in RTT (Real-Time Tolerance) Even when the BBR congestion control algorithm is in the "probing" phase (Gain=1.25), the deep reinforcement learning policy network output... (e.g., 0.8).
[0031] Effect: Actual gain becomes The system forcibly canceled this aggressive probe, avoiding the injection of excessive data into the soon-to-be-congested link, thereby significantly reducing queuing latency.
[0032] Furthermore, in this embodiment, the reward function of the trained deep reinforcement learning policy network includes a smoothing penalty term. , is represented as: ; in, Indicates the throughput reward weight, Represents the normalized transmission throughput. Indicates the delay penalty weight. This represents the normalized round-trip time. Indicates the weight of packet loss penalty. This indicates the packet loss rate penalty. Indicates the weights of the motion smoothing constraint. Indicates the current time The continuous gain scaling factor, Indicates the previous moment The continuous gain scaling factor; As a smoothing penalty term, it is used to suppress drastic oscillations in the gain coefficient, ensure the continuity and stability of the congestion control strategy, and prevent retransmission timeouts in the underlying protocol stack caused by sudden changes in control signals.
[0033] Furthermore, in the training phase of this embodiment: a training scenario containing a 5G millimeter-wave high dynamic fading channel is constructed in a simulation environment (such as NS-3). A near-end policy optimization algorithm is used for training, and the reward function design focuses on penalizing high latency and action oscillations.
[0034] Inference phase: The trained model is lightweighted (pruned and quantized) and then embedded into the user-space protocol stack (such as QUIC). The inference frequency is synchronized with the RTT (approximately once every 20ms~100ms), and the computational overhead is less than 1% of the CPU utilization.
[0035] S105. The BBR congestion control algorithm is used to obtain the reference pacing gain (pacing_gain) under the current state machine phase. Based on the reference pacing gain and the continuous gain scaling factor, the final pacing gain is calculated, and the packet transmission interval (Pacing Rate) in the network environment is dynamically adjusted according to the final pacing gain to achieve preemptive detection or defensive avoidance of network bandwidth fluctuations.
[0036] Specifically, in this embodiment, the final pacing gain is calculated based on the reference pacing gain and the continuous gain scaling factor. ,include: The final pacing gain is calculated and expressed as: ; ; in, This represents the pre-defined gain cyclic array used by the BBR congestion control algorithm during the bandwidth probing phase. Optionally, in the standard BBR congestion control algorithm, the gain cyclic array can be [1.25, 0.75, 1, 1, 1, 1, 1, 1]. Indicates the current state machine phase index. This indicates that the BBR congestion control algorithm is in effect at the current time. A defined baseline pacing gain, optional. , This represents the continuous gain scaling factor.
[0037] Furthermore, the packet transmission interval in the network environment is dynamically adjusted based on the final pacing gain, including: When the BBR congestion control algorithm executes the active probing logic, it shortens the packet transmission interval in the network environment; When the BBR congestion control algorithm executes the active avoidance logic, it extends the packet transmission interval in the network environment.
[0038] It should be noted that the current state machine refers to the current running stage of the BBR congestion control algorithm, which mainly includes: Startup (start-up phase), exponentially increasing probe bandwidth, baseline gain; Drain (draining phase), draining the queue caused by the start-up phase, baseline gain; ProbeBW (bandwidth probe phase), steady-state operation phase, cyclically using the gain array [1.25, 0.75, 1, 1, 1, 1, 1, 1]; ProbeRTT (delay probe phase), sending at extremely low rates to measure pure RTT, baseline gain (but limited at the congestion window level).
[0039] It should be noted that existing BBR congestion control algorithms rely on fixed state machine cycles for bandwidth probing, resulting in a lag of at least 1 to 8 RTTs in response to network changes. This invention introduces a trend prediction variable into the GRU. This enables the control logic to react 50-100ms before packet loss occurs, achieving a leap from "feedback control" to "feedforward control".
[0040] Compared to end-to-end congestion control (which addresses stability), existing reinforcement learning algorithms (such as Aurora and PCC-RL) directly output the transmission rate from the neural network, lacking a safety boundary and prone to performance degradation in scenarios outside the training set. This invention employs a parameter scaling scheme, limiting the gain to a fraction of the baseline gain of the BBR congestion control algorithm. Within the range.
[0041] Compared to the LSTM prediction scheme (which addresses engineering implementation), the GRU network parameters used in this invention are reduced by approximately 25%, converge faster, and are more suitable for deployment in the transport layer protocol stack of mobile terminals or IoT devices with limited computing resources, thus meeting the low latency requirements for real-time packet processing.
[0042] In summary, the adaptive congestion control method based on time-series prediction and reinforcement learning provided by this invention has the following beneficial effects: First, this invention proposes a lightweight bandwidth trend sensing network for GRU.
[0043] This invention constructs a lightweight time-series prediction module based on gated recurrent units (GRUs). This module does not aim for an exact fit to the absolute value of future bandwidth, but rather focuses on capturing the evolution trend of network states. By encoding real-time collected historical network state sequences (including RTT, throughput, packet loss rate, etc.) into a latent space, and utilizing the GRU's unique reset and update gate mechanisms, it can effectively filter random noise in the network while preserving long-term dependency characteristics. Finally, the module outputs the bottleneck bandwidth change gradient (i.e., the relative trend value) within a specific future time step, providing the system with a forward-looking signal that "bandwidth is about to climb" or "bandwidth is about to drop sharply."
[0044] Second, a prediction-driven reinforcement learning fusion decision architecture.
[0045] To address the limitations of single prediction in direct control and the convergence difficulties of pure reinforcement learning, this invention designs a fusion decision architecture that integrates temporal prediction information into deep reinforcement learning (DRL). In this architecture, the "future bandwidth trend" output by the prediction module is no longer isolated information but rather a forward-looking prior knowledge. This prior knowledge is then fused with the current network observation snapshot in the feature dimension to construct an enhanced state space. Based on this fused state, the agent can comprehensively judge whether to adopt an aggressive or conservative strategy under the current network load and considering future bandwidth trends.
[0046] Third, the gain parameter fine-tuning decision mechanism.
[0047] To avoid the instability caused by end-to-end control, this invention employs an indirect control strategy of "parameter fine-tuning." Instead of directly setting the congestion window size, the agent outputs a continuous scaling factor on the pacing gain—a key parameter of the underlying BBR algorithm. Based on the agent's decisions, the system dynamically calculates the final gain value. When the predicted bandwidth is trending upwards and the current network is not saturated, a coefficient greater than 1 is output to increase the gain, guiding aggressive BBR probing; when the predicted bandwidth is trending downwards, a coefficient less than 1 is output to decrease the gain, guiding early BBR convergence. This mechanism preserves the state machine protection logic of the BBR algorithm itself, ensuring the system's robustness under extreme conditions.
[0048] Based on the same inventive concept, this invention also provides an adaptive congestion control device based on time-series prediction and reinforcement learning, used to implement the adaptive congestion control method based on time-series prediction and reinforcement learning provided in the above embodiments of this invention. Embodiments of the method can be referred to above and will not be repeated here. The device includes: The data acquisition module is used to collect network status data of the transport layer protocol stack in the network environment, construct a sliding window state sequence containing multiple time steps, and obtain a snapshot of the current network status; among which, the network status data includes round-trip time, transmission throughput, current packet loss rate, and current congestion window size; Data processing module one is used to process the sliding window state sequence using a pre-trained gated recurrent unit neural network to obtain the bandwidth gradient trend factor, which represents the rate and direction of future bandwidth change. Specifically, the reset gate module in the pre-trained gated recurrent unit neural network filters short-term random noise, and the update gate module in the pre-trained gated recurrent unit neural network extracts long-term dependency features. The feature concatenation model is used to concatenate the bandwidth gradient trend factor with the current network state snapshot to obtain an enhanced state vector containing prior trend information. Data processing module two is used to process the augmented state vector using a trained deep reinforcement learning policy network to obtain continuous gain scaling coefficients; The result acquisition module is used to obtain the reference pacing gain under the current state machine phase using the BBR congestion control algorithm, calculate the final pacing gain based on the reference pacing gain and the continuous gain scaling factor, and dynamically adjust the data packet transmission interval in the network environment based on the final pacing gain to achieve preemptive detection or defensive avoidance of network bandwidth fluctuations.
[0049] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that an article or device comprising a list of elements includes not only those elements but also other elements not expressly listed. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or device comprising said element. Terms such as "connected" or "linked" are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect. The orientations or positional relationships indicated by terms such as "upper," "lower," "left," and "right" are based on the orientations or positional relationships shown in the accompanying drawings and are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as limiting the invention.
[0050] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.
[0051] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.
Claims
1. An adaptive congestion control method based on time-series prediction and reinforcement learning, characterized in that, include: The system collects network status data of the transport layer protocol stack in the network environment, constructs a sliding window state sequence with multiple time steps, and obtains a snapshot of the current network status; wherein, the network status data includes round-trip time, transmission throughput, current packet loss rate, and current congestion window size; The sliding window state sequence is processed using a pre-trained gated recurrent unit neural network to obtain a bandwidth gradient trend factor that characterizes the future bandwidth change rate and direction. Short-term random noise is filtered through the reset gate module in the pre-trained gated recurrent unit neural network, and long-term dependency features are extracted through the update gate module in the pre-trained gated recurrent unit neural network. The bandwidth gradient trend factor is concatenated with the current network state snapshot to obtain an enhanced state vector containing prior trend information. The enhanced state vector is processed using a trained deep reinforcement learning policy network to obtain a continuous gain scaling factor; The BBR congestion control algorithm is used to obtain the reference pacing gain under the current state machine phase. Based on the reference pacing gain and the continuous gain scaling factor, the final pacing gain is calculated. The transmission interval of data packets in the network environment is dynamically adjusted according to the final pacing gain to achieve preemptive detection or defensive avoidance of network bandwidth fluctuations.
2. The adaptive congestion control method based on time-series prediction and reinforcement learning according to claim 1, characterized in that, The sliding window state sequence is processed using a trained gated recurrent unit neural network to obtain a bandwidth gradient trend factor characterizing the future bandwidth change rate and direction, including: The sliding window state sequence The first in Feature vectors at each time step Normalization is performed, and the data is then input into the trained gated recurrent neural network for further processing. Computational update gate , is represented as: ； in, This represents the Sigmoid function. This indicates updating the gate weights. Indicates the first The hidden state corresponding to each time step; Calculate the reset door , is represented as: ； in, This indicates resetting the gate weight; Calculate candidate hidden state , is represented as: ； in, This represents the Tanh activation function. Represents the weight matrix. This indicates element-wise multiplication; Update hidden status , is represented as: ； The hidden states corresponding to all time steps are processed by a fully connected layer, and the result of the fully connected layer processing is processed by the Tanh activation function to obtain the bandwidth gradient trend factor.
3. The adaptive congestion control method based on time-series prediction and reinforcement learning according to claim 2, characterized in that, The first Feature vectors at each time step The features after normalization include: Normalized smooth round-trip delay Normalized bottleneck bandwidth sample value Current packet loss rate Current congestion window size First-order difference of round-trip time delay First-order difference of bandwidth .
4. The adaptive congestion control method based on time-series prediction and reinforcement learning according to claim 1, characterized in that, The enhanced state vector is processed using a trained deep reinforcement learning policy network to obtain continuous gain scaling coefficients, including: Based on the magnitude of the bandwidth gradient trend factor, the enhanced state vector is processed using the policy network in the trained deep reinforcement learning policy network to obtain the action. , Represents the augmented state vector. A function representing a deep reinforcement learning policy network. This represents the structure of a deep reinforcement learning policy network. This represents the set of trainable weight parameters in a deep reinforcement learning policy network. The action Mapping to continuous coefficients ; The policy network includes multiple fully connected modules, a Tanh activation function, and a scaling module. The fully connected modules include fully connected layers and a ReLU activation function.
5. The adaptive congestion control method based on time-series prediction and reinforcement learning according to claim 4, characterized in that, Based on the magnitude of the bandwidth gradient trend factor, the enhanced state vector is processed using the policy network in the trained deep reinforcement learning policy network, including: When the bandwidth gradient trend factor is greater than the positive threshold, the continuous gain scaling factor output by the trained deep reinforcement learning policy network is greater than the first threshold, triggering the BBR congestion control algorithm to execute the active detection logic. When the bandwidth gradient trend factor is less than the negative threshold, the continuous gain scaling factor output by the trained deep reinforcement learning policy network is less than the first threshold, triggering the BBR congestion control algorithm to execute active avoidance logic. When the absolute value of the bandwidth gradient trend factor is less than or equal to the positive threshold, the continuous gain scaling factor output by the trained deep reinforcement learning policy network is equal to the first threshold, triggering the BBR congestion control algorithm to execute the baseline control logic.
6. The adaptive congestion control method based on time-series prediction and reinforcement learning according to claim 5, characterized in that, Dynamically adjusting the data packet transmission interval in the network environment based on the final pacing gain includes: When the BBR congestion control algorithm executes the active probing logic, it shortens the data packet transmission interval in the network environment; When the BBR congestion control algorithm executes the active avoidance logic, it extends the data packet transmission interval in the network environment.
7. The adaptive congestion control method based on time-series prediction and reinforcement learning according to claim 4, characterized in that, The reward function of the trained deep reinforcement learning policy network includes a smoothing penalty term. , is represented as: ； in, Indicates the throughput reward weight, Represents the normalized transmission throughput. Indicates the delay penalty weight. This represents the normalized round-trip time. Indicates the weight of packet loss penalty. This indicates the packet loss rate penalty. Indicates the weights of the motion smoothing constraint. Indicates the current time The continuous gain scaling factor, Indicates the previous moment The continuous gain scaling factor.
8. The adaptive congestion control method based on time-series prediction and reinforcement learning according to claim 1, characterized in that, Calculate the final pacing gain based on the reference pacing gain and the continuous gain scaling factor. ,include: The final pacing gain is calculated and expressed as: ； ； in, This represents the pre-defined gain loop array of the BBR congestion control algorithm during the bandwidth probing phase. Indicates the current state machine phase index. This indicates that the BBR congestion control algorithm is in effect at the current time. A defined baseline pacing gain, This represents the continuous gain scaling factor.
9. An adaptive congestion control device based on temporal prediction and reinforcement learning, characterized in that, include: The data acquisition module is used to collect network status data of the transport layer protocol stack in the network environment, construct a sliding window state sequence containing multiple time steps, and obtain a snapshot of the current network status; wherein, the network status data includes round-trip time, transmission throughput, current packet loss rate, and current congestion window size; Data processing module one is used to process the sliding window state sequence using a trained gated recurrent unit neural network to obtain a bandwidth gradient trend factor that characterizes the future bandwidth change rate and direction; wherein, short-term random noise is filtered through the reset gate module in the trained gated recurrent unit neural network, and long-term dependency features are extracted through the update gate module in the trained gated recurrent unit neural network. A feature concatenation model is used to concatenate the bandwidth gradient trend factor with the current network state snapshot to obtain an enhanced state vector containing prior trend information. Data processing module two is used to process the augmented state vector using a trained deep reinforcement learning policy network to obtain continuous gain scaling coefficients; The result acquisition module is used to obtain the reference pacing gain under the current state machine phase using the BBR congestion control algorithm, calculate the final pacing gain based on the reference pacing gain and the continuous gain scaling factor, and dynamically adjust the data packet transmission interval in the network environment based on the final pacing gain to achieve preemptive detection or defensive avoidance of network bandwidth fluctuations.