Task difference oriented congestion control scheduling method and system based on deep reinforcement learning

Abstract

The application discloses a congestion control scheduling method and system based on deep reinforcement learning and oriented to task differences, which comprises the following steps: network topology configuration is performed; an application layer task flow is generated; network protocol stack running configuration is performed; a customized congestion control module is realized; data collection is performed; reward construction is performed; reinforcement learning training is performed; an action space is designed according to the scheduling strategy difference of different task flows; the congestion window adjustment is mapped to an action set according to the focus of each task on performance indicators, so that the intelligent agent can output corresponding scheduling strategies according to the task flow difference, thereby realizing adaptive congestion control oriented to the task difference. The application has high configurability and real mapping, and each module designed can correspond to the related network function in the Linux kernel. The data structure and processing logic collected in the system are also based on the realization in the real operating system kernel, and therefore have strong real feasibility.

Description

Technical Field

[0001] This invention relates to the field of computer network technology, specifically to a congestion control scheduling method and system based on deep reinforcement learning and oriented towards task differences. Background Technology

[0002] In computer networks, congestion control (CC) is a core mechanism for ensuring network transmission performance and improving resource utilization. Traditional congestion control methods are mostly based on static strategies, and their design principle usually involves directly mapping packet-level events (such as packet loss, latency changes, etc.) to specific control responses. This type of "hard-wired" method is difficult to achieve optimal performance when facing dynamically changing network environments and diverse application scenarios.

[0003] In recent years, with the development of artificial intelligence technology, congestion control methods based on reinforcement learning (RL) have gradually attracted researchers' attention. These methods continuously learn strategies through interaction with the environment to adapt to complex and ever-changing network states and diverse application requirements. However, the application of reinforcement learning in network transmission is still in its early stages and urgently needs further research and exploration.

[0004] Existing reinforcement learning congestion control methods are mainly divided into two categories: one is a hybrid approach, which integrates traditional TCP congestion control mechanisms into RL policies, attempting to balance stability and learning ability; the other is a pure learning approach, which does not rely on manually designed control rules at all, but directly generates optimization policies through learning mechanisms. The latter typically employs reinforcement learning algorithms such as Q-Learning, Actor-Critic, and Proximal Policy Optimization (PPO), possessing strong adaptability and potential performance ceilings.

[0005] Industry experience has shown that most congestion control schemes perform well in certain specific scenarios, but their performance drops significantly in others. Some studies have used large-scale custom server clusters for large-scale training to cover more application scenarios, but such methods are demanding in terms of computing power and deployment costs, making them difficult to promote widely. Summary of the Invention

[0006] The purpose of this invention is to provide a congestion control scheduling method and system based on deep reinforcement learning and oriented towards task differences, so as to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A task-differentiated congestion control scheduling method based on deep reinforcement learning includes:

[0009] Step 1: Configure the network topology;

[0010] Step 2: Generate application layer task flow;

[0011] Step 3: Configure the network protocol stack for operation;

[0012] Step 4: Implement a customized congestion control module;

[0013] Step 5: Collect data;

[0014] Step 6: Construct rewards;

[0015] Step 7: Perform reinforcement learning training, input the collected state information into the deep Q-learning agent, combine the ε-greedy policy to select actions, use the experience replay mechanism to improve sample utilization, and improve training stability through the dual network structure of policy network and target network.

[0016] Step 8: Design the action space based on the differences in scheduling strategies for different task flows. Based on the performance focus of each task, adjust and map the congestion window to a set of actions, so that the agent can output corresponding scheduling strategies for the differences in task flows, thereby realizing adaptive congestion control based on task differences.

[0017] Further, step 1 includes:

[0018] Construct a network topology containing a variable number of transmitters and receivers;

[0019] Provide a queue disk option on the router of the bottleneck link to test the performance of the congestion control algorithm under different queuing mechanisms;

[0020] Each sender and receiver is equipped with a standard TCP / IP protocol stack.

[0021] Further, step 2 includes:

[0022] Step 2.1: Configure a traffic generation mechanism at the sending end, and achieve timing control of the data stream by setting the time period parameters for data transmission and static state;

[0023] Step 2.2: Introduce a constant distribution or exponential distribution to the time period parameter to simulate continuous and intermittent data streams;

[0024] Step 2.3: Utilize the traffic generation mechanism to flexibly construct task flows with different business behavior patterns in the simulation environment to meet the needs of scheduling strategy training and performance evaluation based on task differences.

[0025] Further, step 3 includes:

[0026] Step 3.1: Deploy a complete TCP / IP protocol stack on each simulation node;

[0027] Step 3.2: Create and integrate a transport layer protocol management module inside the simulation node to uniformly manage the creation, maintenance and resource release of all TCP sockets on the simulation node;

[0028] Step 3.3: Encapsulate a transmission control block at the transport layer. The transmission control block contains statistical information, and the key parameters used include congestion window, slow start threshold, maximum segment size, and round-trip delay estimation.

[0029] Further, step 4 includes:

[0030] Step 4.1: Build an extensible and customized congestion control module based on the congestion control abstract interface in the network simulator. The structure of this congestion control abstract interface corresponds to the TCP congestion control strategy implementation interface in the Linux kernel in terms of architecture.

[0031] Step 4.2 involves refactoring the core functions in the congestion control abstract interface, specifically including:

[0032] Step 4.2.1: After receiving the acknowledgment message, dynamically adjust the congestion window according to the control actions output by the agent;

[0033] Step 4.2.2: During the confirmation message processing stage, collect two network state information items, round-trip time delay and packet loss, and use them for state construction and scheduling decisions.

[0034] Further, step 5 includes:

[0035] Step 5.1: The ns-3 simulation script periodically triggers the data acquisition process using a fixed time slot T as the time unit.

[0036] Step 5.2: Within each fixed time slot, the simulation system extracts the current network operating state from the transmission control structure and constructs the state input required for reinforcement learning; the constructed state input information includes:

[0037] The average interval between data packet transmission events;

[0038] Average interval between acknowledgment message arrivals;

[0039] The ratio of average round-trip time to optimal round-trip time;

[0040] The average amount of data currently in transit;

[0041] The state information is collected by the ns-3 simulation script in each scheduling cycle of the congestion control scheduling system and transmitted to the agent in the reinforcement learning module.

[0042] Further, step 6 includes:

[0043] In the congestion control scheduling system, the ns-3 simulation script collects real-time statistical information based on the transmission control block corresponding to each task flow during each scheduling cycle, and constructs a utility function in combination with the task type to calculate the instantaneous reward value of the reinforcement learning model.

[0044] For control command streams and video streams in PLC control and remote monitoring application scenarios, the following utility functions are defined:

[0045] ,

[0046] Where U represents the utility value, Delay represents the average delay, Jitter represents jitter, and α and β are the weighting coefficients for delay and jitter, respectively.

[0047] For sensor data reporting and large file / update streams, construct the following utility function:

[0048] ,

[0049] Where throughput represents throughput, lossRate represents packet loss rate, and γ and δ are the relative weights for controlling throughput and packet loss rate, respectively.

[0050] A reward definition method is proposed, which uses the difference in utility function values ​​between two consecutive scheduling cycles as the reward signal. The calculation method is as follows:

[0051] ,

[0052] Where R represents the reward function value, U t U is the utility function value calculated at time t. t-T Let t be the utility function value at time tT.

[0053] Further, step 7 includes:

[0054] In the congestion control scheduling system, the state information collected by the ns-3 network simulator in each scheduling cycle is input to the agent; the agent is trained and makes decisions using the deep Q-learning algorithm, and the core structure of the agent consists of a policy network and a target network.

[0055] During training, the agent samples actions using an ε-greedy policy: when selecting a non-random action, the agent feeds the current state into the policy network and uses the Q-value of the network output as the basis for estimating the value of each action in the current state.

[0056] By introducing experience replay, the agent caches historical data generated during the interaction process in a replay buffer. During training, the policy network model will sample batch data from this buffer for updating the neural network parameters.

[0057] A dual-network mechanism is adopted: the policy network is used to calculate the current Q value and make action decisions, and the target network is used to estimate the target Q value; the agent copies the parameters of the policy network to the target network every fixed number of steps.

[0058] During the specific training process, the policy network updates the model parameters based on historical data from the experience replay. The update of the neural network parameters uses the following gradient descent formula: the value of the neural network parameter in the i-th update is equal to the value of the neural network parameter in the previous iteration minus the product of the learning rate and the gradient of the loss function with respect to the neural network parameter.

[0059] Further, step 8 includes:

[0060] a. Control command flow action design, including:

[0061] Action space settings: Design small-step, symmetrical congestion window adjustment actions, as well as lazy actions that maintain the current window;

[0062] b. Data reporting flow design, including:

[0063] Action space settings: Prioritize providing more positive actions or exponentially growing actions, while reducing the frequency of negative actions;

[0064] c. Large file / update stream action design, including:

[0065] Action space settings: Sensing link congestion by measuring round-trip delay gradient and proactively relinquishing bandwidth;

[0066] Behavioral characteristics: It actively transmits when the network is idle, and quickly reduces the rate when it detects an increase in latency;

[0067] Coordinated adjustment strategy: Introduce a “coordinated adjustment” mechanism, using attenuation actions that are symmetrical to the action space of the data reporting stream. That is, when the data reporting stream contains multiple increase actions, the large file / update stream will introduce multiple “window reduction” actions accordingly.

[0068] d. Video stream motion design, including:

[0069] Action space settings: Use a set of fine-grained congestion windows to add or remove actions, with each action representing a minor adjustment;

[0070] Congestion window adjustment is performed, including:

[0071] Timing of execution: The actual adjustment operation is carried out upon receipt of confirmation information;

[0072] Adjustment basis: Update the congestion window according to the previously determined control strategy parameters;

[0073] The role of reinforcement learning: The control instructions output by the reinforcement learning module do not directly modify the kernel window value, but serve as the basis for adjustment in the subsequent confirmation and feedback processing stage, ensuring that the control process conforms to the standard transmission control protocol workflow.

[0074] This invention also provides a task-differentiated congestion control scheduling system based on deep reinforcement learning, used to implement the congestion control scheduling method described above, comprising:

[0075] The network simulation topology configuration module constructs a network topology based on the ns-3 network simulator, including the sender, receiver, and intermediate routers, and integrates an interface encapsulation mechanism for interacting with the external reinforcement learning environment in terms of state, action, and reward.

[0076] The customized congestion control module is used to implement user-customizable congestion control logic based on the congestion control abstract class, enabling the replacement, extension and dynamic adjustment of TCP congestion control strategies and their runtime behavior.

[0077] The environment encapsulation module is used to construct network environment instances required during reinforcement learning training. It achieves information interaction and policy iteration updates between the simulated environment and the reinforcement learning agent by redefining the environment interface functions.

[0078] The DQN strategy module is used to implement a deep Q network decision model based on the TCP protocol.

[0079] The DQN training and deployment module is used to implement the training and deployment interaction process between the reinforcement learning agent and the simulation environment. The process includes: obtaining the current network state, predicting and outputting action instructions based on the network state, obtaining the corresponding reward value, and storing the state, action, reward and next state information into the experience cache; then using the data in the experience cache to update and optimize the policy network.

[0080] The performance metrics and task definition module is used to define network performance metrics and the attributes of task-differentiated flows. The performance metrics and task definition module defines differentiated scheduling objectives based on task attributes and is used for policy training and evaluation.

[0081] Compared with existing technologies, the advantages of this invention are as follows: Based on the ns-3 network simulator, this invention possesses high configurability and real-world mappability. Each designed module corresponds directly to relevant network functions in the Linux kernel. The data structures and processing logic collected in the system are also based on implementations within a real operating system kernel, thus demonstrating strong practical feasibility. Through this system, customized algorithm design and verification can be achieved at low cost. The system can dynamically train and adjust scheduling strategies for different types of network tasks, considering their performance indicators and scheduling priorities. Furthermore, the lightweight network and modular design enhance the feasibility of practical deployment and engineering migration capabilities, giving it broad application prospects and practical value. Attached Figure Description

[0082] Figure 1 This is a flowchart of a task-differentiated congestion control scheduling method based on deep reinforcement learning according to the present invention.

[0083] Figure 2 This is a schematic diagram of the training process of the present invention.

[0084] Figure 3 This is the UML class diagram of the customized congestion control module of the present invention.

[0085] Figure 4 This is a UML class diagram of the customized reinforcement learning environment interaction interface of the present invention. Detailed Implementation

[0086] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0087] Please see Figure 1 and Figure 2 A task-differentiated congestion control scheduling method based on deep reinforcement learning includes:

[0088] Step 1, Network Topology Configuration, including:

[0089] Construct a network topology containing a variable number of senders and receivers to build task flows of different quantities and types.

[0090] Provide a queue disk option on the router of the bottleneck link to test the performance of the congestion control algorithm under different queuing mechanisms.

[0091] Each sender and receiver is equipped with a standard TCP / IP protocol stack.

[0092] Step 2, application layer task flow generation, including:

[0093] Step 2.1: Configure a traffic generation mechanism at the sending end, and achieve timing control of the data stream by setting the time period parameters for data transmission and static state;

[0094] Step 2.2: Introduce different characteristics such as constant distribution or exponential distribution to the above time period parameters to simulate continuous data streams (such as video streams and large file transfers) and intermittent data streams (such as control commands or periodic reporting tasks).

[0095] Step 2.3: Utilize the traffic generation mechanism to flexibly construct task flows with different business behavior patterns in the simulation environment to meet the needs of scheduling strategy training and performance evaluation based on task differences.

[0096] Step 3, Network protocol stack runtime configuration, including:

[0097] Step 3.1: Deploy a complete TCP / IP protocol stack on each simulation node;

[0098] Step 3.2: Create and integrate a transport layer protocol management module inside the simulation node to uniformly manage the creation, maintenance and resource release of all TCP sockets on the simulation node;

[0099] Step 3.3: Encapsulate a Transmission Control Block (TCB) at the transport layer. The TCB contains key parameters such as congestion window, slow start threshold, maximum segment size, round-trip time estimation, and congestion state control, which serve as the core data source for collecting operational status information and statistical indicators in this invention.

[0100] Step 4, implementation of the congestion control module, including:

[0101] Step 4.1: Build an extensible and customized congestion control module based on the congestion control abstract interface in the network simulator. The structure of this congestion control abstract interface corresponds to the TCP congestion control strategy implementation interface in the Linux kernel in terms of architecture.

[0102] Step 4.2 involves refactoring the core functions in the congestion control abstract interface, specifically including:

[0103] Step 4.2.1: After receiving the acknowledgment message, dynamically adjust the congestion window according to the control actions output by the agent;

[0104] Step 4.2.2: During the confirmation message processing stage, network state information such as round-trip delay and packet loss is collected and used for state construction and scheduling decisions.

[0105] Step 5, the data acquisition process, includes:

[0106] Step 5.1: The ns-3 simulation script periodically triggers the data acquisition process using a fixed time slot T as the time unit.

[0107] Step 5.2: Within each fixed time slot, the simulation system extracts the current network operating state from the transmission control structure and constructs the state input required for reinforcement learning; the constructed state input information includes:

[0108] The average interval between data packet transmission events;

[0109] Average interval between acknowledgment message arrivals;

[0110] The ratio of average round-trip time to optimal round-trip time;

[0111] The average amount of data currently in transit;

[0112] The state information is collected by the ns-3 simulation script in each scheduling cycle of the congestion control scheduling system and transmitted to the agent in the reinforcement learning module.

[0113] Step 6, the reward construction process, includes:

[0114] In the constructed congestion control scheduling system, the ns-3 simulation script collects real-time statistical information based on the Transmission Control Block (TCB) corresponding to each task flow within each scheduling cycle (i.e., time slice T), and constructs a utility function in combination with the task type to calculate the immediate reward value of the reinforcement learning model.

[0115] For control command streams and video streams in application scenarios such as PLC control and remote monitoring, they are highly sensitive to network latency and jitter. Therefore, this invention defines the following utility function to characterize the scheduling objectives of control command streams and video streams:

[0116] ,

[0117] Where U represents the utility value, Delay represents the average delay, and Jitter is approximated by the gradient change of round-trip delay (RTT) within consecutive time slots. α and β are the weighting coefficients for delay and jitter, respectively. For control command streams, minimizing transmission delay is prioritized, so a larger α value can be set. For video streams, more attention is paid to playback smoothness and buffer stability, so the weight of β can be appropriately increased to emphasize the suppression of jitter.

[0118] For tasks such as sensor data reporting and large file / update streams, the main metrics the system focuses on are throughput and loss rate. This invention constructs the following utility function accordingly:

[0119] ,

[0120] Where throughput represents throughput, lossRate represents packet loss rate, and γ and δ are the relative weights for controlling throughput and packet loss rate, respectively.

[0121] Using utility function values ​​directly as reward input to reinforcement learning models can lead to policy learning bias because the network environment itself is dynamically changing. For example, when a high-volume user is added to the system, the utility value of the original task flow may decrease even if it maintains the same behavior.

[0122] Based on this, the present invention proposes a reward definition method based on the change in utility value, that is, using the difference in utility function values ​​of two consecutive scheduling cycles as the reward signal, and the calculation method is as follows:

[0123] ,

[0124] Where R represents the reward function value, U t U is the utility function value calculated at time t. t-T For time tT, that is, the utility function value of the time slot preceding time t.

[0125] Step 7, Reinforce the learning and training process, including:

[0126] In the congestion control scheduling system, the state information collected by the ns-3 network simulator in each scheduling cycle is input into the agent in the reinforcement learning module; the agent is trained and makes decisions using the Deep Q-Learning (DQN) algorithm, and the core structure of the agent consists of a policy network and a target network.

[0127] During training, the agent samples actions using an ε-greedy policy: it selects random actions with a certain probability ε to increase exploration, and selects actions that maximize the Q value to utilize the learned policy at other times. When selecting non-random actions, the agent feeds the current state into the policy network and uses the Q value output by the network as the basis for estimating the value of each action in the current state.

[0128] To improve sample utilization efficiency and model convergence stability, this invention introduces experience replay. The agent caches historical data (i.e., transitions) generated during the interaction process in a replay buffer. This historical data includes fields such as the current state, the action performed, the reward obtained, the next state, and whether it has terminated (done). During training, the network model will sample batch data from this buffer for updating the neural network parameters.

[0129] To avoid drastic fluctuations in the target value during training that could lead to learning instability, this invention employs a dual-network mechanism: a policy network is used for calculating the current Q-value and making action decisions, while the target network is used for estimating the target Q-value; the agent copies the parameters of the policy network to the target network every fixed number of steps, thereby effectively improving training stability.

[0130] During the actual training process, the policy network updates its model parameters based on historical data from the experience replay. The update of the neural network parameters uses the following gradient descent formula:

[0131] ,

[0132] in, These are the parameters of the neural network. It's the learning rate. Represents the loss function For parameters The gradient.

[0133] loss function The Mean Squared Error (MSE) is defined as follows:

[0134] ,

[0135] in, To determine the desired target Q value, Indicates the current state of the policy network. Next action The estimated value.

[0136] The calculation method is as follows:

[0137] ,

[0138] in, The current reward is represented by γ, the discount factor is represented by γ, and Q represents the output of the target network. Indicates the next state In the middle, select all possible actions. The action with the highest value is selected, and "done" indicates whether it is a terminated state (true for terminated, false for not terminated). The future value is estimated by passing the next state to the target network and taking the maximum Q-value.

[0139] To improve the robustness and efficiency of model training, this invention adopts the Adam optimization algorithm in the selection of optimizers and introduces a gradient pruning mechanism to suppress the gradient explosion problem, thereby ensuring the stability and convergence of the neural network during the training process.

[0140] Step 8: Design the action space based on the differences in scheduling strategies for different task flows, including:

[0141] Control command flow action design includes:

[0142] Application scenario description: Control command flow is typically used in industrial automation control, PLC systems, etc., and is characterized by strong periodicity and high sensitivity to time delay and jitter.

[0143] Action space settings: Design small-step, symmetrical congestion window adjustment actions, such as ±1MSS, ±0.5MSS, as well as lazy actions that maintain the current window;

[0144] Design goal: To reduce transmission bursts caused by sudden window changes, thereby controlling latency fluctuations and jitter.

[0145] The data reporting flow action design includes:

[0146] Application scenario description: The data reporting stream is used for industrial sensor data uploading, periodic status reporting, etc. The data volume is moderate, the real-time requirements are moderate, and the transmission integrity and link utilization efficiency are sensitive.

[0147] Action space settings: Prioritize providing more positive actions (such as +1MSS, +2MSS) or exponentially growing actions, while reducing the frequency of negative (window shrinking) actions;

[0148] Design goal: To maximize the utilization of available links while ensuring network stability.

[0149] Large file / update stream action design, including:

[0150] Application scenario description: Transmitting firmware update packages, system logs, or offline data files, where real-time requirements are not high, but the data volume is large;

[0151] Scheduling objective: Maximize link utilization without impacting the performance of foreground tasks;

[0152] Action space settings: Drawing on the basic idea of ​​LEDBAT, link congestion is detected by measuring the RTT delay gradient, and bandwidth is actively relinquished;

[0153] Behavioral characteristics: It actively transmits when the network is idle, and quickly reduces the rate when it detects an increase in latency;

[0154] Coordinated adjustment strategy: Introduce a "coordinated adjustment" mechanism, using attenuation actions that are symmetrical to the action space of the data reporting stream. That is, when the data reporting stream contains multiple increase actions, the large file / update stream will introduce multiple "window reduction" actions.

[0155] Video stream motion design, including:

[0156] Application scenario description: Video streams need to dynamically adjust the transmission rate to adapt to changes in network conditions, while avoiding buffer exhaustion and rebuffering, thus ensuring playback continuity and picture quality;

[0157] Action space settings: Use a set of fine-grained congestion windows to increase or decrease actions, each action representing a tiny adjustment; for example, ±1 / 8, ±1 / 4, ±1 / 2 MSS;

[0158] Design goal: To provide a smooth bitrate adjustment mechanism to ensure continuous and stable transmission rate and avoid bitrate oscillations or buffer exhaustion caused by drastic fluctuations.

[0159] Congestion window adjustment is performed, including:

[0160] Timing of execution: The actual adjustment operation is carried out upon receipt of confirmation information;

[0161] Adjustment basis: Update the congestion window according to the previously determined control strategy parameters;

[0162] The role of reinforcement learning: The control instructions output by the reinforcement learning module do not directly modify the kernel window value, but serve as the basis for adjustment in the subsequent confirmation and feedback processing stage, ensuring that the control process conforms to the standard transmission control protocol workflow.

[0163] This invention also provides a task-differentiated congestion control scheduling system based on deep reinforcement learning, used to implement the congestion control scheduling method described above, comprising:

[0164] The network simulation topology configuration module is a computer program. Its specific implementation is described in steps 1-3 of the congestion control scheduling method above. It constructs a network topology based on the ns-3 network simulator, which includes a sender, a receiver, and intermediate routers. It supports configuring the queue management mechanism, link parameters, error model, sending flow parameters, TCP parameters, and pluggable congestion control algorithms of the intermediate routers. Furthermore, it integrates an interface encapsulation mechanism for interacting with the external reinforcement learning environment in terms of state, action, and reward.

[0165] The customized congestion control module is a computer program. Its specific implementation is described in step 4 of the congestion control scheduling method above. It implements user-customizable congestion control logic based on the congestion control abstract class. By redefining at least one core function in the abstract class, it realizes the replacement, expansion and dynamic adjustment of the TCP congestion control strategy and its runtime behavior to adapt to the scheduling needs of different types of task flows.

[0166] The environment encapsulation module is a computer program, and its specific implementation is described in steps 5-7 of the congestion control scheduling method above. It constructs the network environment instance required in the reinforcement learning training process. The environment encapsulation module is implemented based on a general environment interface encapsulation class. By redefining the environment interface function, it constructs an interaction interface including state space, action space, and reward calculation, thereby realizing information interaction and policy iteration updates between the simulated environment and the reinforcement learning agent.

[0167] The DQN strategy module is a computer program. Its specific implementation is described in step 7 of the congestion control scheduling method above. It implements a deep Q-network (DQN) decision model based on the TCP protocol, including some sub-modules and core algorithm parts, such as action sampling, action prediction, loss function calculation, and gradient descent, which are used to train the congestion control strategy network.

[0168] The DQN training and deployment module is a computer program. Its specific implementation is described in step 7 of the congestion control scheduling method above. It completes the training and deployment interaction process between the reinforcement learning agent and the simulation environment. The process includes: obtaining the current network state from the simulation system, predicting and outputting action instructions based on the network state, executing the action instructions in the simulation environment to obtain the corresponding reward value, and storing the state, action, reward, and next state information in the experience cache; and then using the data in the experience cache to update and optimize the policy network.

[0169] The performance metrics and task definition module is a computer program. Its specific implementation is described in step 8 of the congestion control scheduling method above. It defines network performance metrics and the attributes of task-differentiated flows. The performance metrics and task definition module defines differentiated scheduling objectives based on task attributes and uses them for policy training and evaluation.

[0170] Please see Figure 3The customized congestion control module, based on the congestion control strategy extension interface, realizes the custom extension of the congestion control strategy by redefining the core functions of slow start threshold setting, window growth control, acknowledgment response processing, and congestion state change notification. The module further integrates an encapsulation mechanism for reinforcement learning interfaces, supporting interactive operations of state acquisition, action execution, and reward calculation. The interaction process is implemented based on a periodic time-triggered mechanism, and the system defines a fixed time slot T to schedule the update frequency of state sampling and strategy execution.

[0171] Please see Figure 4 The environment encapsulation module enables interactive communication with the reinforcement learning agent by redefining the general environment interface. This includes defining the action space, the observation space, acquiring current state information, determining environment termination, generating reward feedback, providing additional information, and executing agent actions. The module also includes a data collection mechanism that binds customized event callback functions to send and receive event hooks in the transmission protocol stack to capture data packet sending and receiving behavior and calculate relevant timing statistics. Furthermore, the module integrates congestion control logic with the interface bridging function of the training environment and can be extended to implement auxiliary calculation methods to enrich state representation, thereby improving the stability and accuracy of model training.

[0172] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A task-difference-oriented congestion control scheduling method based on deep reinforcement learning, characterized in that, Comprise: Step 1, network topology configuration is carried out; Step 2, application layer task flow is generated; Step 3, network protocol stack running configuration is carried out; Step 4, customized congestion control module is realized; Step 5, data collection is carried out; Step 6, reward construction is carried out; Step 7, reinforcement learning training is carried out, the collected state information is input into the deep Q learning agent, the action selection is carried out combined with the epsilon-greedy strategy, the sample utilization rate is improved by using the experience replay mechanism, and the training stability is improved by using the double network structure of the policy network and the target network; Step 8, according to the difference of the scheduling strategy of different task flows, the action space is designed, the congestion window adjustment is mapped to the action set according to the focus of each task on the performance index, so that the agent can output the corresponding scheduling strategy according to the task flow difference, so as to realize the adaptive congestion control facing the task difference, including: a. Control command flow action design, including: Action space setting: design small step, symmetric congestion window adjustment action, and inert action of keeping current window; b. Data reporting flow action design, including: Action space setting: prefer to provide more positive actions or exponential growth actions, while reducing the frequency of negative actions; c. Large file / update flow action design, including: Action space setting: measure the round-trip delay gradient to perceive link congestion and actively give up bandwidth; Behavior characteristics: actively transmit when the network is idle, and quickly reduce the rate when detecting delay rise; Cooperative regulation strategy: introduce "cooperative regulation" mechanism, use symmetric decay actions with data reporting flow action space, that is, when there are multiple increase actions in data reporting flow, multiple "window reduction" actions are introduced for large file / update flow; d. Video flow action design, including: Action space setting: use a set of fine-grained congestion window increase and decrease actions, each action represents a slight adjustment; Congestion window adjustment execution, including: Execution time: actual adjustment operation is carried out when receiving confirmation information; Adjustment basis: update the congestion window according to the previously determined control policy parameters; Reinforcement learning effect: the control instruction output by the reinforcement learning module does not directly modify the kernel window value, but serves as the basis for adjustment in the subsequent confirmation feedback processing stage, ensuring that the control process conforms to the standard transmission control protocol workflow.

2. The method of claim 1, wherein, The step 1 comprises: Building a network topology containing an indefinite number of sending ends and receiving ends; Providing queue selection options on the routers of the bottleneck link for testing the performance of the congestion control algorithm under different queue mechanisms; Each sending end and receiving end is equipped with a standard TCP / IP protocol stack.

3. The method of claim 2, wherein, The step 2 comprises: Step 2.1, configure the traffic generation mechanism at the sending end, control the timing of data flow by setting the time period parameters of data transmission and idle state; Step 2.2, introduce constant distribution or exponential distribution to the time period parameters to simulate continuous data flow and intermittent data flow; Step 2.3, use the traffic generation mechanism to flexibly build task flows with different service behavior patterns in the simulation environment to meet the needs of task difference-oriented scheduling strategy training and performance evaluation.

4. The method of claim 1, wherein, The step 3 comprises: Step 3.1, deploying a complete TCP / IP protocol stack on each simulation node; Step 3.2, creating and integrating a transport layer protocol management module inside the simulation node for unified management of the creation, maintenance and resource release of all TCP sockets on the simulation node; Step 3.3, encapsulating a transmission control block in the transport layer, which contains statistical information, and the key parameters used include multiple types of congestion window, slow start threshold, maximum packet size, round-trip delay estimation.

5. The method of claim 1, wherein, The step 4 includes: Step 4.1, constructing an extensible customized congestion control module based on the congestion control abstract interface in the network simulator, and the structure of the congestion control abstract interface corresponds to the implementation interface of the TCP congestion control strategy in the Linux kernel in terms of architecture; Step 4.2, reconstructing the core functions in the congestion control abstract interface, specifically including: Step 4.2.1, after receiving an acknowledgement packet, dynamically adjusting the congestion window according to the control action output by the agent; Step 4.2.2, in the acknowledgement packet processing stage, collecting network state information such as round-trip delay and packet loss for state construction and scheduling decision.

6. The method of claim 1, wherein, The step 5 includes: Step 5.1, the ns-3 simulation script triggers the data collection process periodically with a fixed time slot T as the time unit; Step 5.2, in each fixed time slot, the simulation system extracts the current network running state from the transmission control structure to construct the state input required for reinforcement learning; the constructed state input information includes: Average interval of packet sending events; Average interval of acknowledgement packet arrival; Ratio between average round-trip delay and optimal round-trip delay; Average value of current in-transit data volume; The state information is collected by the ns-3 simulation script in each scheduling period of the congestion control scheduling system and passed to the agent in the reinforcement learning module.

7. The method of claim 6, wherein the method is based on deep reinforcement learning. The step 6 includes: In the congestion control scheduling system, the ns-3 simulation script collects real-time statistical information from the transmission control block corresponding to each task flow in each scheduling period, and constructs an utility function based on the task type for calculating the immediate reward value of the reinforcement learning model; For control command flow and video flow in the application scenario of PLC control and remote monitoring, the following utility function is defined: ， Where U represents the utility value, Delay represents the average delay, Jitter represents the jitter, and α and β are the weight coefficients of delay and jitter, respectively; For sensor data reporting and large file / update flow, the following utility function is constructed: ， Where throughout represents the throughput, lossRate represents the packet loss rate, and γ and δ are the relative weights of controlling throughput and packet loss rate, respectively; A reward definition method is proposed, which uses the difference between the utility function values of two consecutive scheduling periods as the reward signal, and the calculation method is as follows: ， where R represents the reward function value, U t the utility function value calculated at time t t-T the utility function value at time t-T. 8.The method of claim 1, wherein, The step 7 includes: In the congestion control scheduling system, the state information collected by the ns-3 network simulator in each scheduling period is input to the agent; the agent uses the deep Q learning algorithm for training and decision-making, and the core structure of the agent is composed of a policy network and a target network; During the training process, the agent samples actions through an ε-greedy policy: when a non-random action is selected, the agent inputs the current state into the policy network and takes the Q value output by the network as the value estimate of each action in the current state; Experience replay is introduced: the agent caches historical data generated during the interaction process in a replay buffer, and the policy network model updates the neural network parameters using batch data sampled from the buffer during training; A double network mechanism is adopted: the policy network is used for current Q value calculation and action decision, and the target network is used for target Q value estimation; the agent copies the parameters of the policy network to the target network every fixed number of steps; During the specific training process, the policy network updates the policy network model parameters based on the historical data in the experience replay, and the neural network parameter update uses the following gradient descent formula: the value of the neural network parameter at the ith update is equal to the value of the neural network parameter at the last iteration minus the product of the learning rate and the gradient of the loss function with respect to the neural network parameter.

9. A deep reinforcement learning based task-difference oriented congestion control scheduling system for implementing the congestion control scheduling method according to any one of claims 1-8, characterized in that, Comprise: A network simulation topology configuration module, which is based on the ns-3 network simulator to build a network topology containing a sending end, a receiving end and intermediate routers, and integrates an interface encapsulation mechanism for interacting with an external reinforcement learning environment in terms of state, action and reward; A customized congestion control module, which is used to implement user-customizable congestion control logic based on a congestion control abstract class, and to replace, extend and dynamically adjust the runtime behavior of TCP congestion control strategies; An environment encapsulation module, which is used to build network environment instances required during the reinforcement learning training process, and to realize information interaction and policy iteration update between the simulation environment and the reinforcement learning agent by redefining environment interface functions; A DQN policy module, which is used to implement a deep Q network decision model based on the TCP protocol; A DQN training and deployment module, which is used to implement the training and deployment interaction process between the reinforcement learning agent and the simulation environment, including: obtaining the current network state, predicting and outputting action instructions based on the network state, obtaining the corresponding reward value, and storing the state, action, reward and next state information in the experience cache; then updating and optimizing the policy network using data in the experience cache; A performance indicator and task definition module, which is used to define network performance indicators and the attributes of task differentiated flow, and the performance indicator and task definition module defines differentiated scheduling targets according to task attributes and is used for policy training and evaluation.

Images