A network congestion control framework and method based on adaptive piecewise learning
By using an adaptive segmented learning method, the state space parameters of network congestion control are updated in real time, which solves the failure problem of deep reinforcement learning when facing changing network environments and improves the performance of congestion control.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-29
- Publication Date
- 2026-03-17
AI Technical Summary
Existing congestion control methods based on deep reinforcement learning are prone to failure when faced with changing network environments, failing to effectively adapt to changes in the network environment and resulting in performance loss.
An adaptive segmented learning method is adopted to transform network information into sample pairs, and the model parameters in the state space are updated in real time through the adaptive segmented learning module to form new decision-making strategies and quickly adapt to changes in the network environment.
Through adaptive segmented learning, strategies can be quickly adjusted for changes in the network environment, reducing performance loss and improving the overall performance of congestion control algorithms.
Smart Images
Figure CN116633859B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer network transport layer optimization, specifically, it relates to a network congestion control framework and method based on adaptive segmented learning. Background Technology
[0002] As a fundamental module in computer networks, congestion control frameworks play a crucial role in maximizing network resource utilization and have long been a subject of intense research in both academia and industry. Since the discovery of the "congestion collapse" phenomenon in the 1980s, numerous congestion control methods have been proposed, and to date, 15 different methods have been integrated into the Linux kernel protocol stack. However, due to the diversity of network scenarios and the unpredictability of network events in links, designing an efficient and universal congestion control method remains a significant challenge. Current research proposes using machine learning techniques to replace traditional rule-based congestion control methods, leveraging the powerful fitting capabilities of deep neural networks to enable congestion control methods to support diverse network scenarios.
[0003] Most existing learning-based congestion control methods rely on deep reinforcement learning. These methods first model the congestion control problem as a sequential decision problem, defining its state space, action space, and reward function. By continuously interacting with the environment, a reinforcement learning agent is trained to maximize the cumulative reward. In practical deployment, the offline-trained model is loaded and allowed to interact with the network, without further model updates. A problem with this approach is that the training environment often fails to fully cover all runtime features. Therefore, when the runtime environment changes and deviates from the training samples, the previously trained agent may malfunction. The best solution to this is to continuously update the model, enabling the reinforcement learning agent to adapt to changing network environments.
[0004] Function fitting methods and table-based algorithms are two learning strategies for dealing with changing operating environments. However, both strategies have certain problems. Function fitting methods pre-train on possible environments and assume that environmental changes follow certain underlying rules. Although this type of method can handle complex problems with continuous high-dimensional state and action spaces, it requires comprehensive tuning of its approximate function. In deep reinforcement learning, this typically manifests as a large neural network, and it often suffers from inefficient repetitive learning when facing changing environments. Table-based algorithms can detect changes through model-based methods and can quickly adjust the Q-value of each state-action pair using the characteristics of table-based algorithms. However, this type of method cannot be applied to problems with continuous state and action spaces. Summary of the Invention
[0005] To address the problems existing in the prior art, this invention provides a network congestion control framework and method based on adaptive segmented learning. By introducing an adaptive segmented learning method into the congestion control framework, it is possible to quickly learn the environmental changes in the continuous action space and state space without relearning all the parameters of the entire prediction model. This enables the reinforcement learning model to adapt to changes in the network environment and reduces the performance loss caused by the failure of the reinforcement learning model.
[0006] To achieve the above objectives, the present invention adopts the following technical solution: a network congestion control framework based on adaptive segmented learning, comprising: a congestion control module and an adaptive segmented learning module. The congestion control module is used to adjust the network transmission rate according to network information and convert the network information into sample pairs before inputting them into the adaptive segmented learning module. The adaptive segmented learning module updates the model parameters of the state space corresponding to the current sample pair in real time according to the sample pairs, forms a new decision strategy, and updates the congestion control module through the new decision strategy.
[0007] Furthermore, the congestion control module includes a monitoring module, a reinforcement learning agent module, and a control module connected in sequence; the monitoring module is used to collect network information and convert the network information into state vectors and reward values; the reinforcement learning agent module is used to convert the state vectors and reward values into actions; and the control module adjusts the network transmission rate according to the actions.
[0008] Furthermore, the adaptive segmented learning module includes a replay buffer module, a state segmentation module, and a function fitting module connected in sequence. The replay buffer module stores sample pairs composed of the state vector, reward value, action value, and the state vector at the next moment from the congestion control module. The state segmentation module divides the replay buffer into sub-state spaces. The function fitting module is used to perform function fitting on the divided sub-state spaces, update the sub-state space parameters, and form a new decision strategy.
[0009] Furthermore, the function fitting module is connected to the reinforcement learning agent module, and the function fitting module updates the latest decision policy to the reinforcement learning agent module.
[0010] Furthermore, the present invention also provides a congestion control method based on the adaptive segmented learning-based network congestion control framework, specifically including the following steps:
[0011] Step 1: Start the online congestion control module, collect network information, adjust the network transmission rate, convert the network information into sample pairs and store them in the replay buffer, and record the number of rounds of the sample.
[0012] Step 2: Start the adaptive segmented learning module, update the sub-state space parameters based on the sample pairs in the replay buffer, and form a new decision strategy.
[0013] Furthermore, step 1 includes the following sub-steps:
[0014] Step 1.1: In the i-th round, at each monitoring interval t, collect the current network state and convert it into the current state value s through the monitoring module. i,t and reward value rew i,t ;
[0015] Step 1.2: Set the current state value s i,t The input is fed into the reinforcement learning agent module to obtain the Q-value corresponding to the current action, and the action with the largest Q-value is taken as the best action a. i,t Set the current state value s i,t Optimal action value a i,t Reward value (rew) i,t Maximum Q value q i,t The state value s at the next moment i,t+1 Form sample pairs and store them in the replay buffer;
[0016] Step 1.3: Based on action value a i,t Compared to the original rate send_rate base Adjust the sending rate Apply the send rate (send_rate) to the network system.
[0017] Furthermore, step 2 includes the following sub-steps:
[0018] Step 2.1: When each new sample pair is placed into the replay buffer, determine the corresponding state space. Is the number of sample pairs greater than the state space threshold? If so, the corresponding state space Divide the state into two non-overlapping sub-state spaces. Use the samples in each sub-state space to fit the Q-value function to the sub-state space, update the sub-state space parameters, and form a new decision strategy; otherwise, proceed to step 2.2.
[0019] Step 2.2, if For the state space Perform Q-value function fitting, update state space parameters, and form a new decision strategy;
[0020] Where δ represents the growth factor, N represents the initial threshold of the state space, and l k This indicates the number of times the current state space has been partitioned. Representing the state space The number of sample pairs that have not yet been used for fitting. Representing the state space The number of sample pairs already used for fitting is given, and λ represents the fitting threshold.
[0021] Furthermore, the process of partitioning the sub-state space in step 2.1 is as follows: by finding a dimension d among the various dimensions of the state space. * , so that in dimension d * After partitioning the state space, the sum of the total errors between the Q-value functions corresponding to the two sub-state spaces and the samples is minimized.
[0022]
[0023] st
[0024]
[0025]
[0026] in, Let e represent the set of sample pairs in sub-state space k. i,t This represents a sample pair at round number i and time t. This represents all sample pairs in sub-state space k in dimension d. j The mean of the above, Both represent the value function of the current state space, i n Indicates the current round number. Let these represent the upper and lower bounds of the d-th dimension of the sub-state space, respectively. Indicates the current state value s i,t The dth j The values of the dimension, μ and τ are both hyperparameters, μ is set to 2 and τ is set to 0.99.
[0027] Furthermore, the fitted Q-value function is specifically as follows:
[0028]
[0029] Among them, f w This represents the Q-value fitting function for the current sample pair in the state space, where w represents the parameters. Let e represent the set of sample pairs in sub-state space k. i,t Let i represent the sample pair at round number i and time t. n This represents the current round number, β|w|2 represents the regularization term, and τ represents the hyperparameter, which is set to 0.99.
[0030] Furthermore, the parameter update process for the sub-state space and the state space is as follows:
[0031]
[0032] Among them, rew i,t Let represent the reward value recorded in the sample pair, where γ and α are hyperparameters, α = 1 and γ = 0.99; a represents the action value, f w This represents the Q-value fitting function for the current sample pair in the state space.
[0033] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention first applies an adaptive segmented learning algorithm to network congestion control. The adaptive segmented learning algorithm can quickly relearn the corresponding network structure of changing parts in a continuous environment. By dynamically dividing the state space into multiple subspaces and fitting a Q-value function to each subspace, it adjusts the Q-values of certain action-states without affecting other action-states, and it also has the ability to handle continuous state and action spaces. Applying this invention to the practical field of network congestion control, a system prototype was designed to deploy and run the adaptive segmented learning algorithm. Utilizing the algorithm's ability to quickly relearn, the negative effects of learning-based algorithms are reduced, and the overall performance of the congestion control algorithm is improved. Attached Figure Description
[0034] Figure 1 This is a block diagram of the network congestion control framework based on adaptive segmented learning of the present invention;
[0035] Figure 2 This is a schematic diagram of the state space partitioning in this invention. Detailed Implementation
[0036] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings.
[0037] like Figure 1 This is a block diagram of the network congestion control framework based on adaptive segmented learning of the present invention. The framework includes a congestion control module and an adaptive segmented learning module. The congestion control module adjusts the network transmission rate based on network information and converts the network information into sample pairs, which are then input into the adaptive segmented learning module. The adaptive segmented learning module updates the model parameters of the state space corresponding to the current sample pair in real time based on the sample pairs, forming a new decision strategy, and updates the congestion control module using the new decision strategy. The present invention utilizes the rapid relearning capability of the adaptive segmented learning module to reduce the negative effects of learning-based algorithms and improve the overall performance of the congestion control algorithm.
[0038] In this invention, the congestion control module includes a monitoring module, a reinforcement learning agent module, and a control module connected in sequence. The monitoring module is used to collect network information and convert the network information into state vectors and reward values. The reinforcement learning agent module is used to convert the state vectors and reward values into actions. The control module adjusts the network transmission rate according to the actions.
[0039] To combine the advantages of function fitting methods in handling continuous states, action spaces, and block-based updates using table-based algorithms, the adaptive segmented learning module employs a state segmentation method. As the training phase progresses, the initial state space is continuously divided into finer-grained subspaces. Figure 2 In this architecture, the leaf nodes represent all current state spaces, and each node has a corresponding Q-value fitting function. Whenever the environment changes, only the Q-value functions of the already fitted nodes in the changed state space are adjusted, without changing the Q-value functions of other nodes. The adaptive segmented learning module in this invention includes a replay buffer module, a state segmentation module, and a function fitting module connected in sequence. The replay buffer module stores sample pairs composed of the state vector, reward value, action value, and the state vector at the next moment from the congestion control module. The state segmentation module divides the replay buffer into sub-state spaces. The function fitting module performs function fitting on the divided sub-state spaces, updates the sub-state space parameters, and forms a new decision strategy. The function fitting module is connected to the reinforcement learning agent module, updating the latest decision strategy to the reinforcement learning agent module, enabling the reinforcement learning agent module to quickly complete the update and allowing the congestion control strategy to adapt quickly to network changes, reducing performance loss during operation.
[0040] This invention also provides a congestion control method based on an adaptive segmented learning network congestion control framework, specifically including the following steps:
[0041] Step 1: Activate the online congestion control module, collect network information, adjust the network transmission rate, convert the network information into sample pairs, store them in the replay buffer, and record the number of rounds for each sample; specifically, this includes the following sub-steps:
[0042] Step 1.1: In the i-th round, at each monitoring interval t, collect the current network state and convert it into the current state value s through the monitoring module. i,t and reward value rew i,t ;
[0043] Step 1.2: Set the current state value s i,t The input is fed into the reinforcement learning agent module to obtain the Q-value corresponding to the current action, and the action with the largest Q-value is taken as the best action a. i,t Set the current state value s i,t Optimal action value a i,t Reward value (rew) i,t Maximum Q value q i,t The state value s at the next moment i,t+1 Form sample pairs and store them in the replay buffer;
[0044] Step 1.3: Based on action value a i,t Compared to the original rate send_rate base Adjust the sending rate Apply the send rate (send_rate) to the network system.
[0045] Step 2: Activate the adaptive segmented learning module to update the sub-state space parameters based on the sample pairs in the replay buffer, forming a new decision strategy; specifically including the following sub-steps:
[0046] Step 2.1: When each new sample pair is placed into the replay buffer, determine the corresponding state space. Is the number of sample pairs greater than the state space threshold? If so, it means state space The corresponding state space has been fully explored. Divide the state into two non-overlapping sub-state spaces. Use the samples in each sub-state space to fit the Q-value function to the sub-state space, update the sub-state space parameters, and form a new decision strategy; otherwise, proceed to step 2.2.
[0047] The process of partitioning the state space in this invention is as follows: by finding a dimension d among the various dimensions of the state space. * , so that in dimension d *After partitioning the state space, the sum of the total errors between the Q-value functions corresponding to the two sub-state spaces and the samples is minimized.
[0048]
[0049] The distance from a certain boundary cannot be too close, otherwise it indicates that the state space partitioning is insufficient, resulting in excessive differences in the size of the sub-state spaces. Therefore, the following must be satisfied:
[0050]
[0051]
[0052] in, Let e represent the set of sample pairs in sub-state space k. i,t This represents a sample pair at round number i and time t. This represents all sample pairs in sub-state space k in dimension d. j The mean of the above, Both represent the value function of the current state space, i n Indicates the current round number. Let these represent the upper and lower bounds of the d-th dimension of the sub-state space, respectively. Indicates the current state value s i,t The dth j The dimensions, μ and τ are both hyperparameters. μ is set to 2 and τ is set to 0.99. τ will combine the round number of the current sample pair and give it different weights so that the sample pairs that are closer to the current one have a higher weight in the fitting.
[0053] Step 2.2, if For the state space Perform Q-value function fitting, update state space parameters, and form a new decision strategy;
[0054] Where δ represents the growth factor, N represents the initial threshold of the state space, and l k This indicates the number of times the current state space has been partitioned. Representing the state space The number of sample pairs that have not yet been used for fitting. Representing the state space The number of sample pairs already used for fitting is given, and λ represents the fitting threshold, which is set to 0.2.
[0055] The Q-value function fitted in this invention is specifically as follows:
[0056]
[0057] Among them, f wThis represents the Q-value fitting function for the current sample pair in the state space, where w represents the parameters. Let e represent the set of sample pairs in sub-state space k. i,t Let i represent the sample pair at round number i and time t. n This represents the current round number, β|w|2 represents the regularization term, and τ represents the hyperparameter, which is set to 0.99.
[0058] The sub-state space and the parameter update process of the state space in this invention are as follows:
[0059]
[0060] Among them, rew i,t Let represent the reward value recorded in the sample pair, where γ and α are hyperparameters, α = 1 and γ = 0.99; a represents the action value, f w This represents the Q-value fitting function for the current sample pair in the state space.
[0061] This invention is the first to integrate the ideas of function-fitting-based reinforcement learning and table-based reinforcement learning, proposing an adaptive state space partitioning method. This allows the reinforcement learning agent to retrain without adjusting the fitters for all state spaces, updating only the changed parts, thus achieving efficient and fast retraining. Applying this invention to congestion control scenarios addresses the problem of easily changing network environments in such environments.
[0062] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principle of the present invention should be considered within the scope of protection of the present invention.
Claims
1. A congestion control method of a network congestion control framework based on adaptive piecewise learning, characterized by, The network congestion control framework based on adaptive segmented learning comprises a congestion control module and an adaptive segmented learning module, the congestion control module is used for adjusting network sending rate according to network information and converting the network information into sample pairs and inputting the sample pairs into the adaptive segmented learning module; the adaptive segmented learning module updates model parameters of a state space corresponding to the sample pairs in real time according to the sample pairs, forms a new decision strategy, and updates the congestion control module through the new decision strategy; The adaptive segmented learning module comprises a replay buffer module, a state segmentation module and a function fitting module connected in sequence; the replay buffer module stores sample pairs composed of state vectors, reward values, action values and state vectors at next time in the congestion control module; the state segmentation module divides the replay buffer into sub-state spaces; the function fitting module is used for fitting functions of the divided sub-state spaces, updating parameters of the sub-state spaces, and forming a new decision strategy; The congestion control method of the network congestion control framework based on adaptive segmented learning comprises the following steps: Step 1: starting an online congestion control module, collecting network information, adjusting network sending rate, and storing sample pairs after converting the network information into the sample pairs into a replay buffer and recording the round number of the sample; Step 2: starting an adaptive segmented learning module, updating parameters of sub-state spaces according to the sample pairs in the replay buffer, and forming a new decision strategy; comprising the following sub-steps: Step 2.1, when each new sample pair is put into the replay buffer, judge whether the number of sample pairs in the corresponding state space is greater than the state space threshold If yes, divide the corresponding state space into two disjoint sub-state spaces, use the sample pairs under each sub-state space to fit the Q value function of the sub-state space, update the parameters of the sub-state space, and form a new decision strategy; otherwise, execute step 2.2; Step 2.2, if Perform Q-value function fitting, update state space parameters, form a new decision policy Perform Q-value function fitting, update state space parameters, form a new decision policy wherein δ denotes a speed-up factor, N denotes an initial threshold of state space, l k denotes the number of times of partitioning of the current state space, denotes the number of pairs of samples in the state space that have not been used for fitting, denotes the number of pairs of samples in the state space that have been used for fitting, and λ denotes a fitting threshold.
2. The congestion control method of the adaptive segment-based learning based network congestion control framework according to claim 1, wherein, Step 1 comprises the following sub-steps: Step 1.
1. At each monitoring interval t in the i-th round, collect the current network state and transform it into a current state value s by the monitoring module i,t and a reward value rew i,t ; Step 1.2, the current state value s i,t Input into the reinforcement learning agent module, get the corresponding Q value under the current action, take the action with the maximum Q value as the best action a i,t , the current state value s i,t , the best action value a i,t , the reward value rew i,y , the maximum Q value q i,t , the next time state value s i,t+1 Form a sample pair and store it in the replay buffer; Step 1.
3. Adjusting the sending rate based on the action value a i,t With the original rate send_rate base Adjusting the sending rate Applying the sending rate send_rate to the network system.
3. The congestion control method of the adaptive segment-based learning based network congestion control framework according to claim 1, wherein, The partitioning process of the sub-state space in Step 2.1 is as follows: find a dimension d in each dimension of the state space * such that the state space is partitioned in dimension d * After the state space is partitioned in dimension d , the sum of the total errors between the Q value functions and samples of the two sub-state spaces is minimum. where, denotes the set of sample pairs under the sub-state space k, e i,t denotes the sample pair under the i-th round number and time t, denotes the mean of all sample pairs under the sub-state space k in the d j dimension, both denote the value function of the current state space, i n denotes the current round number, respectively denote the upper and lower bounds of the d-th dimension of the sub-state space k, denotes the value of the d i,t dimension of the current state s j , μ and τ are both hyperparameters, μ is set to 2 and τ is set to 0.
99.
4. The congestion control method of adaptive segment-based learning based network congestion control framework according to claim 1, wherein, The fitted Q value function is specifically: where f w represents the Q-value fitting function under the state space to which the current sample pair belongs, w represents parameters, represents the sample pair set under the sub-state space k, e i,t represents the sample pair under the i-th round number and time t, i n represents the current round number, β|w|2represents a regularization term, and τ represents a hyperparameter, and τ is set to 0.
99.
5. The congestion control method of adaptive segment-based learning based network congestion control framework according to claim 1, wherein, The parameter updating process of the sub-state space and the state space is: wherein, rew i,t represents the reward value recorded in the sample pair, γ, α are hyperparameters, α = 1, γ = 0.99; a represents the action value, f w represents the Q value fitting function under the state space to which the current sample pair belongs.
6. The congestion control method of adaptive segment-based learning based network congestion control framework according to claim 1, wherein, The congestion control module comprises a monitoring module, a reinforcement learning agent module and a control module connected in sequence; the monitoring module is used for network information collection and converting the network information into state vectors and reward values; the reinforcement learning agent module is used for converting the state vectors and the reward values into actions; and the control module adjusts network sending rate according to the actions.
7. The congestion control method of adaptive segment-based learning based network congestion control framework according to claim 1, wherein, The function fitting module is connected with the reinforcement learning agent module, and the function fitting module updates the latest decision strategy into the reinforcement learning agent module.