Reinforcement learning congestion control method for time sequence modeling based on Transform

Abstract

The invention relates to a reinforcement learning congestion control method for time sequence modeling based on Transform, and belongs to the technical field of communication networks and artificial intelligence, and the method comprises the following specific steps: step 1, an NS3 end carries out network simulation, and generates simulated network data; 2, the generated simulated network data are sent to a simulated TCP-RL protocol, and the TCP-RL protocol simulates sending, delay, packet loss and ACK return of a data packet; step 3, a congestion control RL environment of the python end; receiving simulated network data sent by the NS3 end, converting the simulated network data into observation data, and sending the observation data to the reinforcement learning agent; and step 4, the reinforcement learning agent at the Python end analyzes the received observation data to form a historical sequence, dynamically pays attention to the real-time condition of data transmission, and returns a congestion control result of analysis decision to the simulation network environment of the NS3 end as an action in reinforcement learning. Compared with a traditional algorithm, the congestion control effect is better and more stable.

Description

A reinforcement learning-based congestion control method for time series modeling based on Transformer Technical Field

[0001] This invention belongs to the fields of communication networks and artificial intelligence technology, and relates to a reinforcement learning congestion control method based on Transformer for time-series modeling. Background Technology

[0002] Congestion control technology is a mechanism used in computer networks to prevent excessive data from flooding the network simultaneously, which can lead to latency, packet loss, and performance degradation. It monitors network load and dynamically adjusts the transmission rate to maintain efficient and stable data transmission. Common methods include slow start and congestion avoidance.

[0003] Traditional congestion control methods have evolved over time to develop their own interpretations of congestion signals and rate adjustment rules, but these rules are essentially "fixed structures" based on experience. For example, BBR attempts to proactively maintain the optimal transmission rate by modeling bandwidth and round-trip time (RTT). Fixed-rule strategies struggle to accommodate all environmental conditions and can even lead to network instability, unfair resource contention, and a decrease in overall throughput efficiency.

[0004] The connection between reinforcement learning and network congestion control lies primarily in "decision optimization." Traditional congestion control algorithms rely on manually designed rules, while reinforcement learning, through continuous interaction with the network environment, automatically learns the optimal transmission strategy based on feedback (such as throughput, latency, and packet loss rate). It can dynamically adjust the congestion window or transmission rate under complex and variable network conditions, achieving more efficient resource utilization and more stable transmission performance. Compared to fixed algorithms, reinforcement learning methods are more adaptable and perform better in scenarios such as mobile networks.

[0005] Existing simple reinforcement learning methods for network congestion control perform slightly better or worse than traditional algorithms. Therefore, there is an urgent need to propose a more optimized reinforcement learning method to improve its network adaptability, thereby enabling more efficient network congestion control in complex and ever-changing network environments. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention aims to propose a reinforcement learning-based network congestion control method based on Transformer-based temporal modeling. This method addresses the challenge of dynamically adjusting the congestion window or transmission rate under complex and variable network conditions, making it difficult to achieve more efficient resource utilization and more stable transmission performance under traditional congestion control methods. This invention provides superior and more stable congestion control compared to traditional algorithms.

[0007] This invention establishes an efficient reinforcement learning network congestion control method adaptable to complex network environments by replacing the classic PPO algorithm of traditional reinforcement learning with the simple neural network MLP, and then replacing the MLP with the more innovative Transformer architecture for temporal modeling. Specifically, the traditional MLP multilayer perceptron is replaced with an improved Transformer as the feature extractor. Transformers excel at handling sequential data. In network congestion control scenarios, historical network states (such as latency, packet loss rate, bandwidth, etc.) constitute a time series. Transformers can capture the dependencies and long-term dependencies within these time series, while MLPs can only handle fixed-size inputs and have a weaker perception of temporal order. The self-attention mechanism of Transformers allows the model to dynamically focus on the most relevant parts of the input sequence. This means that when making decisions, the model can "selectively" focus on network state information that has the greatest impact on the current decision over a past period, rather than treating all historical information indiscriminately. This is especially advantageous when handling longer sequences. Furthermore, this module does not directly adopt the standard Transformer module, but incorporates several targeted improvements to enhance stability during RL training and adaptability to network characteristics, such as Xavier initialization, residual connections and LayerNorm, and network state-aware modulation. Additionally, this invention utilizes more refined PPO parameters and optimizer configurations.

[0008] Therefore, this invention can efficiently and dynamically adjust network congestion in complex network environments, achieving efficient resource utilization, low packet loss rate and high information confirmation rate during data transportation, while also ensuring the stability of reinforcement learning model performance and adapting to different network environments.

[0009] This invention proposes a reinforcement learning congestion control method based on Transformer-based temporal modeling. Addressing the shortcomings of traditional reinforcement learning congestion control methods in dynamic adjustment of congestion windows and transmission rates, as well as low resource utilization in complex network environments such as low bandwidth and high latency, this invention proposes a novel optimization method. The main difference lies in replacing the classic PPO+MLP combination reinforcement learning method with a more innovative PPO+Transformer reinforcement learning method for network congestion control. Basic MLPs can only see the current step's observations and cannot utilize historical changes in network state for congestion control. In contrast, the improved Transformer model in this invention can maintain a 64-step historical observation sequence and, through Transformer encoding, can learn: RTT rising trends, queue accumulation processes, bandwidth change patterns, and precursory signals before packet loss, enabling the model to make more stable and forward-looking congestion control decisions. Furthermore, several stabilization designs for Transformer are added, along with training stabilization mechanisms. Additionally, the PPO algorithm undergoes more refined parameter tuning and optimizer configuration.

[0010] The technical solution of this invention is as follows: a reinforcement learning congestion control method based on Transformer for time-series modeling, comprising an NS3 simulation environment process (NS3 client) and a Python process (Python client), wherein the NS3 client and the Python client communicate using the inter-process communication mechanism provided by the ZeroMQ library; the specific steps are as follows: Step 1: The NS3 client performs network simulation to generate simulated network data; Step 2: The generated simulated network data is sent to a simulated TCP-RL protocol, which simulates packet sending, delay, packet loss, and ACK return; Step 3: The congestion control RL environment on the Python client receives the simulated network data sent from the NS3 client, converts it into observation data, and sends it to the reinforcement learning agent; Step 4: The reinforcement learning agent on the Python client... Analyze the received observation data to form a historical sequence, dynamically monitor the real-time data transmission situation, and return the congestion control results of the analysis and decision-making as actions in reinforcement learning to the simulation network environment at the NS3 end. The actions in reinforcement learning include the current congestion control window size, bytes in flight, throughput, RTT round-trip time, RTT change rate, packet loss rate, and information acknowledgment signal rate; Step 5: The actions in reinforcement learning returned by the Python end are first passed to the reinforcement learning environment on the Python end, and then sent to the NS3 end through the inter-process communication mechanism. The network environment at the NS3 end adjusts the network data transmission according to the feedback data of the reinforcement learning agent, thereby performing congestion control, and feeds back the network control results to the agent end in the form of reward; Step 6: Iterative training: Repeat steps 2 to 5 until the preset training termination condition is reached.

[0011] According to a preferred embodiment of the present invention, in step 3, receiving simulated network data sent from the NS3 terminal and converting it into observation data includes: super().step() is responsible for interacting with the NS3 terminal, transmitting the actions taken by the agent to the NS3 terminal, and executing a time step in the NS3 terminal; after the NS3 terminal completes the execution of a time step, it returns the current network state information to the Python terminal through the ZeroMQ library; the original network state information is stored in the obs variable; the transform_obs(self, obs) method receives the original observation data obs returned by super().step(), the original observation data obs includes the network connection ID, i.e., socket ID, slow start threshold ssThresh, congestion window size cWnd, number of bytes sent currently bytesTX, number of bytes successfully received bytesRx, round-trip time rtt, and processes the original observation data obs to extract the features required by the agent, and finally forms the agent's observation space.

[0012] According to a preferred embodiment of the present invention, the intelligent agent includes a multi-head attention mechanism, a robust Transformer encoder block, an enhanced TCP congestion control feature extractor, and a stable TCP congestion control Actor-Critic policy module; the specific process is as follows: Step 4.1: Input feature construction and embedding mapping process; let the reinforcement learning environment start from time t. The stable, controllable, and generalizable state representation extracted from the output raw network observation data is as follows: ;in, Maintain a length of The historical sequence constitutes the input tensor. : ;in, Indicates batch size. This represents the length of the historical time step; the original input sequence is fed into the enhanced TCP congestion control feature extractor module for signal preprocessing, and low-dimensional, heterogeneous network indicators are mapped to a unified high-dimensional feature space through an embedded mapping function. The mapping relationship... Represented as: ;in, This represents an embedding mapping network that includes linear mapping, layer normalization, and nonlinear activation functions, and outputs features. Step 4.2: Stabilizing the multi-head attention feature modeling process; Let the input features be... The query matrix, key matrix, and value matrix are constructed using linear projection, and are represented as follows: ;in, The parameter matrix is ​​a linear transformation matrix; the single-head attention score for multi-head attention is calculated as follows: ;in, This represents the feature dimension of each attention head. Indicates the first The first query and the first Attention score between each key, representing the degree of attention; Indicates the first A query vector, derived from the input Through linear transformation get; Indicates the first There are _ key vectors, with dimension _ . , by input Through linear transformation get; , To represent different numerical values; Represents the dot product. The attention score is normalized using a fixed scaling factor; a gating mechanism is added, where the Attention output is approved, specifically using the formula: g = ( ),in, (⋅) refers to the sigmoid function, which maps element-wise to (0, 1). The process involves adding residual connections and layer normalization, where output = LayerNorm(output + query), and output represents the current state, query represents the adjustment given by historical experience, and LN represents the layer normalization operation. Subsequently, a softmax operation is performed on the attention scores to obtain the attention weight matrix, which is then weighted and summed with the corresponding value vectors to form the context feature representation. The multi-head attention output is recombined through a linear mapping and added to the input features via residual connections. Finally, after layer normalization, the attention layer output is obtained. The calculation process is expressed as follows: Where LN represents the layer normalization operation, and MSA represents the stable multi-head attention computation; therefore, the stable multi-head attention mechanism is generally expressed as: SMHA(x) = LN(Z+ ( )⊙(softmax( )V)); where SMHA represents a stable multi-head attention mechanism, LN represents layer normalization operation, and Z is the input feature sequence, For Hadamard product; Step 4.3: Feedforward network and network state-aware modulation mechanism; Obtain the output of the attention layer Then, it is further fed into a robust Transformer encoder block for nonlinear transformation. The feedforward network consists of two linear mapping layers and a nonlinear activation function. The linear mapping layers include a linear layer + ReLU and a linear layer + Tanh. The nonlinear activation function is the GELU activation of the feedforward network. The output is... Represented as: A network state-aware modulation mechanism is introduced in the output stage of the feedforward network. This mechanism generates modulation factors based on the attention output. : ;in, This represents a modulation network composed of linear transformations, where the Tanh function is used to constrain the range of values ​​for the modulated signal. Finally, the output characteristics of the feedforward network are dynamically adjusted using a combined additive and multiplicative modulation method, expressed as: The modulated features and attention layer output are then processed through residual connections and layer normalization to obtain the final output of a single Transformer encoder block. Step 4.4: Stacking of Transformer Feature Encoder Layers; Assume the output of the Transformer feature encoder is... for: ;in, For batch size, The length of the time series. For the feature channel dimension; Step 4.5: Temporal feature fusion and output representation construction; The Transformer feature encoder performs fusion processing on the output features, specifically, extracting end-of-sequence features and average features of the entire sequence respectively: ;in, This represents the temporal feature tensor after multi-layer robust Transformer block encoding, where L represents the sequence length. This indicates the current state of the high-level network. Indicating historical trend characteristics, and The features are concatenated along the feature dimension and then fed into the feature fusion module to obtain the final feature representation. : Final output features The data is fed into the Actor-Critic policy network to guide TCP congestion control decisions; where B represents the batch size, and represents the number of TCP connections / environments processed in parallel. Indicates the number of feature channels.

[0013] According to a preferred embodiment of the present invention, the reinforcement learning agent forms a historical sequence; including: in the enhanced TCP congestion control feature extractor, state_buffer refers to a state buffer used to store the observation history of the most recent seq_len time steps, saving the observations of the most recent seq_len time steps. During each forward propagation, the new observations are normalized and transformed into (batch_size, 1, input_dim) through unsqueeze(1), where batch_size refers to the number of samples processed at one time during the current training or inference; input_dim refers to the feature dimension of each time step observation, i.e., the 7-dimensional vector output by transform_obs; concatenated to the state buffer, the shape of state_buffer after concatenation is still (batch_size, seq_len, input_dim), always saving the latest seq_len time steps; the last dimension, the oldest step is discarded, resulting in state_buffer always saving the latest historical sequence, i.e., the formed historical sequence; the agent makes an action response by learning a historical sequence; including: historical sequence (batch_size, seq_len, ... The input_dim is fed into the feature extraction layer of the policy network to extract temporal features; the Actor network outputs action probabilities or specific actions based on the feature vectors; the Critic network evaluates the value function of the current state; the output action is sent to the NS3 terminal, which adjusts the network based on the action and returns new observations and rewards.

[0014] According to a preferred embodiment of the present invention, fixed temperature scaling is used, as shown in the following formula: ;in, Indicates the first The first query and the first Attention score between each key, representing the degree of attention; Indicates the first Query vectors, with dimensions of _ , by input Through linear transformation get; Indicates the first There are _ key vectors, with dimension _ . , by input Through linear transformation get; , To represent different values, Represents the dot product. Represent the dimension; perform Xavier initialization, input sequence. Each token is linearly projected as ,in ; The input feature matrix represents the feature vector at each position in the sequence; Batch size is the number of sequence samples processed in each training / inference iteration. `seq_len` is the sequence length and the number of positions in each sample. , , This is the Query, Key, and Value matrix after linear projection; The linear transformation weight matrix of the query is of size . , used to project the input into the Query space; Let be the linear transformation weight matrix of Key, with a size equal to . The same applies to projecting the input into the Key space; Let be the linear transformation weight matrix of Value, with a size equal to . The same applies to projecting the input into the Value space.

[0015] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the reinforcement learning congestion control method based on Transformer for time-series modeling described above.

[0016] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the reinforcement learning congestion control method based on Transformer for time-series modeling described above.

[0017] The beneficial effects of this invention are as follows: This invention enhances the feature extraction capability of reinforcement learning through the Transformer architecture, proposing an intelligent decision-making method for network congestion control. In feature extraction, a multi-head attention mechanism is introduced to perform temporal modeling of multi-dimensional network states, effectively capturing the dynamic evolution of congestion states. In network architecture design, a lightweight Transformer encoder block is adopted, reducing computational overhead while maintaining sensitivity to network fluctuations. Regarding training stability, gating mechanisms and other techniques are used to avoid policy oscillations during reinforcement learning training. Experiments show that this method achieves increased throughput and reduced packet loss rate compared to traditional congestion control algorithms. This invention can adapt to different network load scenarios, significantly improving network resource utilization while ensuring fairness, providing a deployable intelligent congestion control solution for high-performance network environments such as 6G and data centers. Attached Figure Description

[0018] Figure 1 is a flowchart illustrating a reinforcement learning congestion control method based on Transformer for temporal modeling according to the present invention; Figure 2 is a block diagram of the agent according to the present invention; Figure 3 is a comparison diagram of ACK acknowledgment signals under different congestion control methods according to an embodiment of the present invention; Figure 4 is a comparison diagram of packet loss rates under different congestion control methods according to an embodiment of the present invention; Figure 5 is a reward diagram of the reinforcement learning network congestion control method using Transformer for temporal modeling according to an embodiment of the present invention. Detailed Implementation

[0019] The present invention will be further defined below with reference to the accompanying drawings and embodiments, but is not limited thereto.

[0020] Terminology Explanation: 1. NS3 Simulation Environment Process: NS3 (Network Simulator 3) is an open-source network simulator widely used in academia and industry. It can simulate various network topologies, protocol stacks, and traffic models. This invention simulates a real network environment on the NS3 side and interacts with the intelligent agent on the reinforcement learning side.

[0021] 2. In simulation-based learning scenarios, the Python process, specifically the simulation environment (network simulator NS3), is responsible for simulating the behavior of a real network environment and providing feedback to the agent. The agent, in turn, is responsible for learning the optimal policy. These components typically reside in different processes to achieve decoupling and efficient execution. The NS3 process simulates the real environment, while the Python process reacts based on observations of the environment.

[0022] 3. ZeroMQ library, a high-performance, lightweight asynchronous message queue library, is an embedded, multi-threaded, and portable message queue system. It provides a set of socket APIs that can be used to build distributed or concurrent applications. In the technical solution described in this invention, the ZeroMQ library is used as a communication middleware to facilitate communication between the NS3 simulation environment process (usually a child process written in C++) and the Python machine learning training process (usually a parent process written in Python). ZeroMQ provides socket abstractions for various communication modes (such as request-response REQ / REP, publish-subscribe PUB / SUB, push-pull PUSH / PULL, etc.). This allows components written in different processes or languages ​​to connect and exchange messages in a standardized way, without needing to worry about the underlying operating system IPC details.

[0023] 4. TCP-RL Protocol: TCP-RL is a communication protocol implementation responsible for intelligently and dynamically determining the congestion window size for data transmission to optimize communication performance under various network conditions. This protocol can work in conjunction with existing transport layer services and continuously learns and adapts from network feedback through a monitoring interval mechanism.

[0024] 5. ACK, Acknowledgement, is a crucial communication mechanism. An ACK is a message sent by the receiver to the sender to indicate that the receiver has successfully received one or more data packets previously sent by the sender.

[0025] 6. RL, RL (Reinforcement Learning) represents reinforcement learning.

[0026] 7. RTT (Round-Trip Time) refers to the total time elapsed from when the sender transmits a data packet until the sender receives an acknowledgment packet from the receiver. This time includes transmission delay, processing delay, queuing delay, and other factors, and is an important indicator of network performance.

[0027] 8. super().step() completes the process of sending control actions to ns-3 and receiving the raw network state from ns-3.

[0028] 9. The transform_obs(self, obs) method transforms the original network state variables returned by ns-3 into low-dimensional, physically interpretable state vectors suitable for PPO+Transformer learning.

[0029] Example 1: A reinforcement learning congestion control method based on Transformer for time-series modeling, as shown in Figure 1, is used to improve network data transmission efficiency and increase resource utilization under bandwidth-constrained and high-latency network conditions. Ultimately, it constructs a congestion control agent capable of maintaining stable and efficient decision-making in complex, dynamic, and noisy network environments. This network congestion control method includes an NS3 simulation environment process (NS3 client) and a Python process (Python client). The NS3 client and the Python client communicate using the inter-process communication mechanism provided by the ZeroMQ library. The specific steps are as follows:

[0030] Step 1: The NS3 terminal performs network simulation, generating simulated network data. This simulated network data includes network status parameters such as socketID (network connection ID), ssThresh (slow start threshold), cWnd (congestion window size), bytesTX (number of bytes currently sent), bytesRx (number of bytes successfully received), and RTT (round-trip time). Step 2: The generated simulated network data is sent to a simulated TCP-RL protocol. The TCP-RL protocol simulates packet transmission, delay, packet loss, and ACK return. Step 3: A congestion control RL environment is implemented on the Python side. This environment receives the simulated network data from the NS3 terminal, transforms it into observation data, and sends it to the reinforcement learning agent. Step 4: The reinforcement learning agent on the Python side analyzes the received 7-dimensional observation data, forming a historical sequence. It dynamically monitors the real-time data transmission situation and incorporates the analysis and decision-making regarding congestion control. The results are returned to the simulation network environment of the NS3 terminal as actions in reinforcement learning. These actions include the current congestion control window size, bytes in flight (number of bytes sent but not received), throughput, RTT round-trip time, RTT rate of change, packet loss rate, and acknowledgment rate. Step 5: The actions returned by the Python terminal are first passed to the reinforcement learning environment in Python, and then sent to the NS3 terminal via inter-process communication. The network environment of the NS3 terminal adjusts network data transmission based on the feedback data from the reinforcement learning agent, thereby performing congestion control, and feeding back the network control results to the agent terminal in the form of a reward. This invention also innovates and improves the reward mechanism by adding a throughput zeroing safety mechanism, making the reward mechanism more reasonable and enabling the model to focus more on the most relevant parts of the input sequence. Step 6: Iterative training: Repeat steps 2 to 5 until the preset training termination condition is met.

[0031] Example 2 is a reinforcement learning congestion control method based on Transformer for time-series modeling as described in Example 1, with the difference being: In step 3, the simulated network data sent from the NS3 terminal is received and transformed into observation data; this includes: super().step() is responsible for interacting with the NS3 terminal, passing the action taken by the agent (here, the target congestion window processed by transform_action(action)) to the NS3 terminal, and executing a time step in the NS3 terminal; after the NS3 terminal completes the execution of a time step, it returns the current network state information to the Python terminal through the ZeroMQ library; the original network state information (such as socket ID, i.e., network connection ID, ssThresh slow start threshold, etc.) is stored in the obs variable; the transform_obs(self, obs) method receives the original observation data obs returned by super().step(), and the original observation data obs includes the network connection ID, i.e., socket ID. Network status data such as ID, slow start threshold (ssThresh), congestion window size (cWnd), current number of bytes sent (bytesTX), number of bytes successfully received (bytesRx), and round-trip time (RTT) are analyzed. The raw observation data (obs) is then processed to extract the features needed by the agent, ultimately forming the agent's observation space. This invention selects the seven most important dimensions from the raw data for parsing, accumulation, and calculation. The final observation data includes: current window size, throughput, round-trip time, packet loss rate, RTT fluctuation, number of bytes in transit, and ACK reception rate.

[0032] This invention constructs a congestion control agent for TCP congestion control tasks, used for temporal modeling and high-level feature abstraction of continuous network state sequences. As shown in Figure 2, the agent includes a multi-head attention mechanism, a robust Transformer encoder block, an enhanced TCP congestion control feature extractor, and a stable TCP congestion control Actor-Critic policy module; through hierarchical design, it achieves accurate modeling of network congestion evolution patterns. The agent employs residual connections and layer normalization mechanisms throughout its sub-modules to ensure the stability of gradient propagation during long-term online training and to avoid model degradation due to drastic changes in state distribution. The specific process is as follows:

[0033] Step 4.1: Input feature construction and embedding mapping process; Let the reinforcement learning environment start from time t. The stable, controllable, and generalizable state representation extracted from the output raw network observation data is as follows: ;in, This corresponds to the seven core network metrics in TCP congestion control. To characterize the temporal correlation of network states, the system maintains a segment of length... The historical sequence constitutes the input tensor. :

[0034] ;in, Indicates batch size. This represents the length of the historical time step; the original input sequence is fed into the enhanced TCP congestion control feature extractor module for signal preprocessing, and low-dimensional, heterogeneous network indicators are mapped to a unified high-dimensional feature space through an embedded mapping function. The mapping relationship... Represented as: ;in, This represents an embedding mapping network that includes linear mapping, layer normalization, and nonlinear activation functions, and outputs features. This process achieves scale uniformity and noise suppression of the original network signal, providing a stable input for subsequent time series modeling.

[0035] Step 4.2: Stabilize the multi-head attention feature modeling process; to extract the network state dependencies in the time dimension, the Transformer feature encoder embeds features. Based on this, a stable multi-head attention mechanism is introduced. Let the input features be... ;

[0036] The query matrix, key matrix, and value matrix are constructed using linear projection, and are represented as follows: ;in, The parameter matrix is ​​a linear transformation matrix; the single-head attention score for multi-head attention is calculated as follows: ;in, This represents the feature dimension of each attention head. Indicates the first The first query and the first Attention score between each key, representing the degree of attention; Indicates the first Each query vector is derived from the input. Through linear transformation get; Indicates the first Each key vector has a dimension of _Key_. , by input Through linear transformation get; , To represent different numerical values; Represents the dot product. The attention score is normalized using a fixed scaling factor to represent the dimension, thus preventing numerical instability of the attention weights during training. A fixed temperature parameter is incorporated into this process to ensure stable network output.

[0037] Adding a gating mechanism, where the output of Attention is approved, specifically using the formula: g = ( ),in, (⋅) refers to the sigmoid function, which maps element-wise to (0, 1). The process involves adding residual connections and layer normalization, where output = LayerNorm(output + query), and output represents the current state, query represents the adjustment given by historical experience, and LN represents the layer normalization operation. Subsequently, a softmax operation is performed on the attention scores to obtain the attention weight matrix, which is then weighted and summed with the corresponding value vectors to form the context feature representation. The multi-head attention output is recombined through a linear mapping and added to the input features via residual connections. Finally, after layer normalization, the attention layer output is obtained. The calculation process is expressed as follows: Where LN represents the layer normalization operation, and MSA represents the stable multi-head attention computation; therefore, the stable multi-head attention mechanism is generally expressed as: SMHA(x) = LN(Z+ ( )⊙(softmax( )V)); where SMHA represents a stable multi-head attention mechanism, LN represents layer normalization operation, and Z is the input feature sequence, For Hadamard product; Step 4.3: Feedforward network and network state-aware modulation mechanism; Obtain the output of the attention layer Then, it is further fed into a robust Transformer encoder block for nonlinear transformation. The feedforward network consists of two linear mapping layers and a nonlinear activation function. The linear mapping layers include a linear layer + ReLU and a linear layer + Tanh. The nonlinear activation function is the GELU activation of the feedforward network. The output is... Represented as: To enhance the model's adaptability to changes in network congestion, this invention introduces a network state-aware modulation mechanism in the feedforward network output stage. This mechanism generates modulation factors based on attention output. : ;in, This represents a modulation network composed of linear transformations, where the Tanh function is used to constrain the range of values ​​for the modulated signal. Finally, the output characteristics of the feedforward network are dynamically adjusted using a combined additive and multiplicative modulation method, expressed as: This design enables the model to automatically suppress aggressive features when network congestion intensifies and amplify throughput-oriented features when the network is idle, thereby achieving adaptive adjustment of TCP behavior. The modulated features and attention layer output are then processed again through residual connections and layer normalization to obtain the final output of a single Transformer encoder block. Step 4.4: Layered stacking of the Transformer feature encoder; the stabilized multi-head attention module and the network state-aware modulation module constitute a complete robust Transformer encoding block; multiple Transformer encoding blocks are stacked according to the same structure to form a complete Transformer feature encoder, used to extract high-order temporal features from the network state sequence layer by layer; let the output of the Transformer feature encoder be... for: ;in, For batch size, The length of the time series. For the feature channel dimension; Step 4.5: Temporal feature fusion and output representation construction; To simultaneously capture the current network state and long-term congestion trends, the Transformer feature encoder fuses the output features, specifically extracting end-of-sequence features and the average features of the entire sequence: ;in, This represents the temporal feature tensor encoded by multiple robust Transformer blocks, where L represents the sequence length, which is specified as 64 in this invention. This indicates the current state of the high-level network. Indicating historical trend characteristics (long-term information), and The features are concatenated along the feature dimension and then fed into the feature fusion module to obtain the final feature representation. : Final output features The data is fed into the Actor-Critic policy network (a stable TCP congestion control Actor-Critic policy module) to guide TCP congestion control decisions; where B represents the batch size, indicating the number of TCP connections / environments processed in parallel. Indicates the number of feature channels.

[0038] After network state observations are input into the policy network, the system first uses a temporal Transformer feature encoder to jointly model the current and historical network states, generating a stable high-level state representation. Subsequently, the Actor network outputs TCP congestion control actions based on this state representation, while the Critic network evaluates the long-term reward under this state. The two work together to optimize within the PPO framework, thereby achieving online learning and adaptive adjustment of the TCP congestion control policy. The above are the main tasks of the stable TCP congestion control Actor-Critic policy module.

[0039] The reinforcement learning agent forms a historical sequence, including: in the enhanced TCP congestion control feature extractor, there is a state buffer that stores the raw network data input from the NS3 network environment. The state_buffer is a state buffer used to store the observation history of the most recent seq_len (here set to 64) time steps. It saves the observations of the most recent seq_len (in this invention, it is set to 64) time steps. During each forward propagation, the new observations (the observation state obtained from the network environment (NS3) at the current time step, i.e., the network metrics returned by the environment) are normalized and transformed into (batch_size, 1, input_dim) by unsqueeze(1) (matching the input requirements of the Transformer time series model). Batch_size refers to the number of samples processed at one time during the current training or inference. Input_dim refers to the feature dimension of each time step observation, i.e., the 7-dimensional vector output by transform_obs. It is concatenated to the state buffer. After concatenation, the shape of state_buffer is still (batch_size, seq_len, input_dim), and the latest seq_len is always saved. Each time step; this invention specifies that seq_len is 64, so that the agent can make decisions using the network state of the past 64 steps.

[0040] In the last dimension, the oldest step is discarded, resulting in the state_buffer always storing the latest historical sequence, i.e., the formed historical sequence. The agent learns from this historical sequence to make action responses. This includes: the historical sequence (batch_size, seq_len, input_dim) is fed into the feature extraction layer (Transformer) of the policy network to extract temporal features; the Actor network outputs action probabilities or specific actions (such as congestion window increments) based on the feature vectors; the Critic network evaluates the value function of the current state; the output action is sent to the NS3 terminal, which adjusts the network based on the action and returns new observations. And rewards; when reinforcement learning agents receive and process observation data, the methods also include: introducing the Transformer structure for network control signal modeling, capturing long-term dependencies between TCP states by observing sent data over a long period of time, and increasing the historical observation buffer; for example, observing queue packet loss by detecting RTT round-trip time and then readjusting the congestion window, which are time series chain changes that basic MLP cannot handle at all; attention mechanisms focus on sudden spikes (e.g., instantaneous queue growth); improve stable control capabilities; the self-attention mechanism of the Transformer structure allows reinforcement learning agents to dynamically focus on the most relevant parts of the input sequence; for example, the importance of RTT, loss, and throughput varies at different time points, and the attention of the Transformer is more flexible; using more refined PPO parameters and optimizer configurations, creating custom policies to achieve deep integration of the Transformer extractor with the PPO actor / critic, and using more stable optimizer hyperparameters (AdamW).

[0041] Reinforcement learning agents have also improved in ensuring model stability. Methods include using fixed-temperature scaling to avoid gradient explosion, as shown in the following formula: ;in, Indicates the first The first query and the first Attention score between each key, representing the degree of attention; Indicates the first Each query vector has a dimension of . , by input Through linear transformation get; Indicates the first Each key vector has a dimension of _Key_. , by input Through linear transformation get; , To represent different values, Represents the dot product. Represent the dimension; perform Xavier initialization, input sequence. Each token is linearly projected as ,in ; The input feature matrix represents the feature vector at each position in the sequence; The batch size is the number of sequence samples processed in each training / inference iteration. `seq_len` is the sequence length and the number of positions in each sample. , , This is the Query, Key, and Value matrix after linear projection; The linear transformation weight matrix of the query is of size . , used to project the input into the Query space; Let be the linear transformation weight matrix of Key, with a size equal to . The same applies to projecting the input onto the key space; Let be the linear transformation weight matrix of Value, with a size equal to . The same applies; it is used to project the input onto the Value space.

[0042] A lightweight gating mechanism has been added to allow the model to automatically adjust the intensity of attention output.

[0043] When reinforcement learning agents perform network state-aware feedforward modulation, they extract network state information by not directly inputting the raw network parameters into the model. Instead, through specialized preprocessing and latent feature engineering, the model gains a deep understanding of the combined meaning of these parameters and their trends over time, thus perceiving the true network state. This understanding of the network state is transformed into a dynamic modulation signal, capable of responding differently to subtle changes in the network state. This modulation signal is then applied to key computational components within the model (such as the feedforward layer output). Through a combination of additive and multiplicative methods, this modulation signal can dynamically and precisely calibrate the amplitude and characteristics of the model's output.

[0044] Interaction between the NS3 and Python environments utilizes a richer observation space, resulting in a more rational structure and clearer physical meaning. A suitable arrangement of observations is beneficial for model learning. A throughput-zeroing safety mechanism has been added to the reward system, making the reward mechanism more reasonable and enabling the model to focus more on the most relevant parts of the input sequence. The observed data undergoes consistent processing, with more rigorous feedback logic.

[0045] The network congestion control results of the reinforcement learning congestion control method based on Transformer for temporal modeling proposed in this embodiment are shown in Figures 3 and 4. As can be seen from Figures 3 and 4, the congestion control method using the Transformer architecture outperforms the basic MLP method and traditional methods in terms of both ACK acknowledgment signal performance and packet loss rate. Furthermore, as the number of rounds of environmental state data sent by the NS3 terminal increases and the learnable data of the reinforcement learning agent continues to grow, the reward value fed back from the NS3 environment terminal to the agent generally shows an upward trend, as shown in Figure 5. Therefore, the reinforcement learning congestion control method based on Transformer for temporal modeling proposed in this invention can effectively improve network congestion control performance.

[0046] Example 3: A computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the reinforcement learning congestion control method based on Transformer for time-series modeling as described in Example 1 or 2.

[0047] Example 4: A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the reinforcement learning congestion control method based on Transformer for time-series modeling as described in Example 1 or 2.

Claims

1. A reinforcement learning congestion control method based on Transformer for temporal modeling, characterized in that, It includes an NS3 simulation environment process (NS3 client) and a Python process (Python client). The NS3 client and the Python client communicate using the inter-process communication mechanism provided by the ZeroMQ library. The specific steps are as follows: Step 1: The NS3 client performs network simulation and generates simulated network data; Step 2: The generated simulated network data is sent to a simulated TCP-RL protocol. The TCP-RL protocol simulates the sending, delay, packet loss, and ACK return of data packets. Step 3: The congestion control RL environment on the Python side receives simulated network data sent from the NS3 side, transforms it into observation data, and sends it to the reinforcement learning agent. Step 4: The reinforcement learning agent on the Python side analyzes the received observation data, forming a historical sequence, dynamically monitoring the real-time data transmission, and returns the congestion control results of the analysis and decision-making as actions in reinforcement learning to the simulated network environment on the NS3 side. These actions include the current congestion control window size, bytes in flight, throughput, RTT round-trip time, RTT change rate, packet loss rate, and acknowledgment rate. Step 5: The actions returned from the Python side are first passed to the reinforcement learning environment on the Python side, and then sent to the NS3 side through inter-process communication. The network environment on the NS3 side adjusts network data transmission based on the feedback data from the reinforcement learning agent, thereby performing congestion control, and feeds back the network control results to the agent as a reward. Step 6: Iterative training: Repeat steps 2 to 5 until the preset training termination condition is met.

2. The reinforcement learning congestion control method based on Transformer for temporal modeling as described in claim 1, characterized in that, In step 3, the simulated network data sent from the NS3 terminal is received and transformed into observation data. This includes: super().step() interacting with the NS3 terminal, transmitting the actions taken by the agent to the NS3 terminal, and executing a time step in the NS3 terminal; after the NS3 terminal completes the execution of a time step, it returns the current network state information to the Python terminal through the ZeroMQ library; the original network state information is stored in the obs variable; the transform_obs(self, obs) method receives the original observation data obs returned by super().step(). The original observation data obs includes the network connection ID (socket ID), the slow start threshold ssThresh, the congestion window size cWnd, the number of bytes sent (bytesTX), the number of bytes successfully received (bytesRx), and the round-trip time (rtt). The original observation data obs is then processed to extract the features needed by the agent, ultimately forming the agent's observation space.

3. The reinforcement learning congestion control method based on Transformer for temporal modeling as described in claim 1, characterized in that, The intelligent agent includes a multi-head attention mechanism, a robust Transformer encoder block, an enhanced TCP congestion control feature extractor, and a stable TCP congestion control Actor-Critic policy module; the specific process is as follows: Step 4.1: Input feature construction and embedding mapping process; let the reinforcement learning environment start from time . The stable, controllable, and generalizable state representation extracted from the output raw network observation data is as follows: ;in, Maintain a length of The historical sequence constitutes the input tensor. ： ;in, Indicates batch size. This represents the length of the historical time step; the original input sequence is fed into the enhanced TCP congestion control feature extractor module for signal preprocessing, and low-dimensional, heterogeneous network indicators are mapped to a unified high-dimensional feature space through an embedded mapping function. The mapping relationship... Represented as: ;in, This represents an embedding mapping network that includes linear mapping, layer normalization, and nonlinear activation functions, and outputs features. Step 4.2: Stabilizing the multi-head attention feature modeling process; Let the input features be... The query matrix, key matrix, and value matrix are constructed using linear projection, and are represented as follows: ;in, The parameter matrix is ​​a linear transformation matrix; the single-head attention score for multi-head attention is calculated as follows: ;in, This represents the feature dimension of each attention head. Indicates the first The first query and the first Attention score between each key, representing the degree of attention; Indicates the first A query vector, derived from the input Through linear transformation get; Indicates the first There are _ key vectors, with dimension _ . , by input Through linear transformation get; ， To represent different numerical values; Represents the dot product. The attention score is normalized using a fixed scaling factor; a gating mechanism is added, where the Attention output is approved, specifically using the formula: g = ( ),in, (⋅) refers to the sigmoid function, which maps element-wise to (0, 1). The process involves adding residual connections and layer normalization, where output = LayerNorm(output + query), and output represents the current state, query represents the adjustment given by historical experience, and LN represents the layer normalization operation. Subsequently, a softmax operation is performed on the attention scores to obtain the attention weight matrix, which is then weighted and summed with the corresponding value vectors to form the context feature representation. The multi-head attention output is recombined through a linear mapping and added to the input features via residual connections. Finally, after layer normalization, the attention layer output is obtained. The calculation process is expressed as follows: Where LN represents the layer normalization operation, and MSA represents the stable multi-head attention computation; therefore, the stable multi-head attention mechanism is generally expressed as: SMHA(x) = LN(Z+ ( )⊙(softmax( )V)); where SMHA represents a stable multi-head attention mechanism, LN represents layer normalization operation, and Z is the input feature sequence, For Hadamard product; Step 4.3: Feedforward network and network state-aware modulation mechanism; Obtain the output of the attention layer Then, it is further fed into a robust Transformer encoder block for nonlinear transformation. The feedforward network consists of two linear mapping layers and a nonlinear activation function. The linear mapping layers include a linear layer + ReLU and a linear layer + Tanh. The nonlinear activation function is the GELU activation of the feedforward network. The output is... Represented as: A network state-aware modulation mechanism is introduced in the output stage of the feedforward network. This mechanism generates modulation factors based on the attention output. ： ;in, This represents a modulation network composed of linear transformations, where the Tanh function is used to constrain the range of values ​​for the modulated signal. Finally, the output characteristics of the feedforward network are dynamically adjusted using a combined additive and multiplicative modulation method, expressed as: The modulated features and attention layer output are then processed through residual connections and layer normalization to obtain the final output of a single Transformer encoder block. Step 4.4: Stacking of Transformer Feature Encoder Layers; Assume the output of the Transformer feature encoder is... for: ;in, For batch size, The length of the time series. For the feature channel dimension; Step 4.5: Temporal feature fusion and output representation construction; The Transformer feature encoder performs fusion processing on the output features, specifically, extracting end-of-sequence features and average features of the entire sequence respectively: ;in, This represents the temporal feature tensor after multi-layer robust Transformer block encoding, where L represents the sequence length. This indicates the current state of the high-level network. Indicating historical trend characteristics, and The features are concatenated along the feature dimension and then fed into the feature fusion module to obtain the final feature representation. ： Final output features The data is fed into the Actor-Critic policy network to guide TCP congestion control decisions; where B represents the batch size, and represents the number of TCP connections / environments processed in parallel. Indicates the number of feature channels.

4. The reinforcement learning congestion control method based on Transformer for temporal modeling according to claim 1, characterized in that, The reinforcement learning agent forms a historical sequence; including: in the enhanced TCP congestion control feature extractor, state_buffer refers to a state buffer used to store the observation history of the most recent seq_len time steps. It saves the observations of the most recent seq_len time steps. During each forward propagation, the new observations are normalized and transformed into (batch_size, 1, input_dim) through unsqueeze(1). batch_size refers to the number of samples processed at one time during the current training or inference. input_dim refers to the feature dimension of the observation at each time step, that is, the 7-dimensional vector output by transform_obs. It is concatenated to the state buffer. After concatenation, the shape of state_buffer is still (batch_size, seq_len, input_dim), and the latest seq_len time steps are always saved. In the last dimension, the oldest step is discarded. As a result, state_buffer always saves the latest historical sequence, that is, the formed historical sequence. The agent makes an action response by learning a historical sequence; including: historical sequence (batch_size, seq_len, input_dim) The feature extraction layer of the policy network extracts temporal features; the Actor network outputs action probabilities or specific actions based on the feature vectors; the Critic network evaluates the value function of the current state; the output action is sent to the NS3 terminal, which adjusts the network based on the action and returns new observations and rewards.

5. A reinforcement learning congestion control method based on Transformer for temporal modeling as described in any one of claims 1-4, characterized in that, Using fixed-temperature scaling, the formula is as follows: ;in, Indicates the first The first query and the first Attention score between each key, representing the degree of attention; Indicates the first Query vectors, with dimensions of . , by input Through linear transformation get; Indicates the first There are _ key vectors, with dimension _ . , by input Through linear transformation get; ， To represent different values, Represents the dot product. Represent the dimension; perform Xavier initialization, input sequence. Each token is linearly projected as ,in ； The input feature matrix represents the feature vector at each position in the sequence; Batch size is the number of sequence samples processed in each training / inference iteration. `seq_len` is the sequence length and the number of positions in each sample. 、 、 This is the Query, Key, and Value matrix after linear projection; The linear transformation weight matrix of the query is of size . , used to project the input into the Query space; Let be the linear transformation weight matrix of Key, with a size equal to . The same applies to projecting the input into the Key space; Let be the linear transformation weight matrix of Value, with a size equal to . The same applies to projecting the input into the Value space.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the reinforcement learning congestion control method based on Transformer for temporal modeling as described in any one of claims 1-5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the reinforcement learning congestion control method based on Transformer for temporal modeling as described in any one of claims 1-5.

Images