A traffic scheduling method and system based on dynamic priority

Abstract

The present invention discloses a traffic scheduling method and system based on dynamic priority. Specifically, the method comprises the following steps: the sending end calculates the remaining time and priority of each flow based on the SDN network status information; the intelligent decision-making layer server uses the MADDPG-Q algorithm to optimize the queue threshold to obtain a centralized training model; the control layer SDN controller reports the network status information to the decision-making layer server for model training and then transmits it to the switch; the forwarding layer programmable switch obtains multi-level feedback queue thresholds through the model and local information, assigns different priority queues to flows of different priorities, and completes flow priority scheduling; at the same time, data packets of different packet priorities in the same flow priority interval queue are sorted, and the position of the data packet in the queue is adjusted when the flow priority changes, thereby realizing dynamic priority queue scheduling. The present invention reduces the system's average flow completion time and can better adapt to complex distributed network applications.

Description

Technical Field

[0001] The present invention relates to the field of computer network technology, and in particular to a flow scheduling method and system based on dynamic priority. Background Art

[0002] With the development of information technology, data volumes are increasing, and more and more applications are migrating to data centers. Common distributed computing frameworks such as Spark, MapReduce, and Dryad, as well as distributed storage frameworks such as Ceph and BigQuery, are increasingly being hosted in data centers. By interconnecting computing and storage resources through high-speed data center networks, distributed applications in data centers enjoy higher performance, greater reliability, and more flexible management. While highly interconnected data center infrastructure improves application performance, these applications also present new challenges to data center networks. Handling the massive volume and complex characteristics of network flows has become a pressing challenge.

[0003] Complex, large-scale models trained with unprecedented amounts of training data offer high accuracy and strong expressiveness, effectively approximating the decision boundaries of many difficult problems that conventional methods struggle to solve. At the same time, big data and large-scale models present new technical challenges for artificial intelligence. A growing number of scholars and research institutions are investigating distributed machine learning techniques to better leverage large-scale data to train more accurate large models. Distributed training's iterative algorithms, high-frequency parameter communication, and large amounts of data, coupled with the fact that computing power is far outpacing network transmission capacity, have led to communication costs becoming a performance bottleneck for distributed machine learning platforms.

[0004] The goal of flow scheduling is generally to ensure fairness among flows within each service class, maximize the throughput of bandwidth-demanding flows, and minimize the completion time of delay-sensitive flows. Research on the traffic characteristics of data center networks has revealed that data flows vary widely in size and follow a heavy-tailed distribution. In general, large flows are bandwidth-sensitive (elephant flows), while small flows are delay-sensitive (rat flows). Setting different priorities for different data flows and scheduling them accordingly during network transmission is crucial for the efficient use of network resources.

[0005] The number of services and traffic distribution within a network constantly fluctuate, necessitating frequent algorithm parameter adjustments. Due to the random nature of traffic and routing strategies, coupled with the shallow buffering of data center network switches, switch queues can randomly become congested. In other words, queue congestion can occur on any switch at any time. Queue congestion can cause packets to experience significant queuing delays, increasing flow completion times. Traditional queue scheduling algorithms struggle to achieve fine-grained flow control due to the high labor costs and cumbersome configuration associated with frequent static configuration. This makes static parameter configuration unsuitable for dynamic network traffic demands. Summary of the Invention

[0006] The purpose of the present invention is to provide a traffic scheduling method and system based on dynamic priority, thereby overcoming the cumbersome limitation of queue thresholds in traditional traffic scheduling that require manual configuration, and being able to adapt to different network traffic distribution conditions.

[0007] The technical solution to achieve the purpose of the present invention is: a traffic scheduling method based on dynamic priority, which is provided with a sending end, an intelligent decision-making layer server, a control layer SDN controller, a forwarding layer programmable switch and a receiving end. The specific method is as follows:

[0008] Step 1: At the sending end, the remaining time is calculated by dividing the remaining flow size by the available bandwidth, and the priority of the flow is obtained by comparing it with the multi-level feedback queue threshold;

[0009] Step 2: The intelligent decision-making layer server uses the MADDPG-Q algorithm to adaptively calculate and optimize the queue thresholds for the multi-level feedback queues, and obtains a centrally trained multi-level feedback queue threshold update model.

[0010] Step 3: The control layer SDN controller connects the underlying hardware devices and the intelligent decision layer. The controller obtains the global network status and provides the necessary information for the intelligent decision layer server to calculate the optimization threshold. At the same time, the multi-level feedback queue threshold update model trained by the intelligent decision layer server is sent to the forwarding layer programmable switch.

[0011] Step 4: The forwarding layer programmable switch uses a multi-level feedback queue as a priority queue and manually sets the initial threshold of the multi-level feedback queue. After calculating the priority of the sender, it matches flows of different priorities to different priority queues to complete flow priority scheduling. At the same time, it sorts packets of different packet priorities in the same flow priority interval queue and adjusts the position of the packet in the queue when the flow priority changes, thus achieving packet-level scheduling.

[0012] Step 5. The receiving end receives data packets: When congestion occurs, the receiving end receives marked data packets and marks the returned ACK. When the TCP sending end receives ACK, it calculates the percentage of marked ACKs corresponding to the last window and estimates the degree of network congestion.

[0013] A dynamic priority-based traffic scheduling system implements the dynamic priority-based traffic scheduling method. The system specifically includes a sending end, an intelligent decision-making layer server, a control layer SDN controller, a forwarding layer programmable switch, and a receiving end, wherein:

[0014] The sender calculates the remaining time by dividing the remaining flow size by the available bandwidth and compares it with the multi-level feedback queue threshold to obtain the flow priority.

[0015] The intelligent decision-making layer server uses the MADDPG-Q algorithm to adaptively calculate and optimize the queue thresholds for multi-level feedback queues, resulting in a centrally trained multi-level feedback queue threshold update model.

[0016] The SDN controller at the control layer connects the underlying hardware devices and the intelligent decision layer. The controller obtains the global network status and provides the necessary information for the intelligent decision layer server to calculate the optimization threshold. At the same time, it sends the multi-level feedback queue threshold update model trained by the intelligent decision layer server to the forwarding layer programmable switch.

[0017] The forwarding layer programmable switch uses a multi-level feedback queue as a priority queue, with manually set initial thresholds for the multi-level feedback queue. After calculating the priority of the sender, it matches flows of different priorities to different priority queues, completing flow priority scheduling. It also sorts packets of different packet priorities within the same flow priority interval queue and adjusts the position of the packet in the queue when the flow priority changes, achieving packet-level scheduling.

[0018] The receiving end is used to receive data packets: when congestion occurs, the receiving end marks the returned ACK when receiving the marked data packets; when the TCP sending end receives the ACK, it calculates the percentage of marked ACK corresponding to the last window and estimates the degree of network congestion.

[0019] A mobile terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the dynamic priority-based traffic scheduling method is implemented.

[0020] Compared with the prior art, the present invention has the following significant advantages:

[0021] (1) Using the shortest remaining time first algorithm, the proposed method can simply and effectively obtain the remaining time of the flow. Compared with the simulated shortest job first algorithm, the link bandwidth factor is taken into account, which improves the real-time performance of the algorithm and is more suitable for real environments.

[0022] (2) Based on the multi-agent deep reinforcement learning method, the MADDPG-Q algorithm based on the SDN intelligent network architecture is proposed. It adaptively determines the threshold of the multi-level feedback queue, which can adapt to the complex and changing network traffic, further reduce FCT, and improve system throughput.

[0023] (3) Centralized training and distributed execution: When a controller fails, the MADDPG-Q algorithm can use the computing power of the source switch or router to execute the algorithm. It does not require too many computing resources. Each agent only needs to know the local observation information (such as link load, bandwidth, flow transmission time, etc.) to make the optimal decision, saving the interaction delay between the agent and the controller.

[0024] (4) Combine flow-level scheduling with packet-level scheduling, sort packets with the same flow priority but different packet priorities in a single priority queue, and extract packets into the corresponding priority queue for processing when the flow priority changes or when retransmitted packets are detected, thereby reducing the waiting time of packets in the queue. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a system framework diagram.

[0026] Figure 2 It is the overall flow chart of the system.

[0027] Figure 3 This is the flowchart for calculating the shortest remaining time.

[0028] Figure 4 It is the threshold model update flowchart.

[0029] Figure 5 It is a flow chart of packet scheduling.

[0030] Figure 6 This is a diagram of the centralized training process of the threshold model.

[0031] Figure 7 It is a diagram of the distributed execution process of the threshold model. DETAILED DESCRIPTION

[0032] The present invention provides a traffic scheduling method based on dynamic priority, which is provided with a sending end, an intelligent decision-making layer server, a control layer SDN controller, a forwarding layer programmable switch and a receiving end. The specific method is as follows:

[0033] Step 1: At the sending end, the remaining time is calculated by dividing the remaining flow size by the available bandwidth, and the priority of the flow is obtained by comparing it with the multi-level feedback queue threshold;

[0034] Step 2: The intelligent decision-making layer server uses the MADDPG-Q algorithm to adaptively calculate and optimize the queue thresholds for the multi-level feedback queues, and obtains a centrally trained multi-level feedback queue threshold update model.

[0035] Step 3: The control layer SDN controller connects the underlying hardware devices and the intelligent decision layer. The controller obtains the global network status and provides the necessary information for the intelligent decision layer server to calculate the optimization threshold. At the same time, the multi-level feedback queue threshold update model trained by the intelligent decision layer server is sent to the forwarding layer programmable switch.

[0036] Step 4: The forwarding layer programmable switch uses a multi-level feedback queue as a priority queue and manually sets the initial threshold of the multi-level feedback queue. After calculating the priority of the sender, it matches flows of different priorities to different priority queues to complete flow priority scheduling. At the same time, it sorts packets of different packet priorities in the same flow priority interval queue and adjusts the position of the packet in the queue when the flow priority changes, thus achieving packet-level scheduling.

[0037] Step 5. The receiving end receives data packets: When congestion occurs, the receiving end receives marked data packets and marks the returned ACK. When the TCP sending end receives ACK, it calculates the percentage of marked ACKs corresponding to the last window and estimates the degree of network congestion.

[0038] As a specific example, in step 1, the remaining time is calculated by dividing the remaining flow size by the available bandwidth at the sending end, and the priority of the flow is obtained by comparing it with the multi-level feedback queue threshold, as follows:

[0039] Step 1.1, replace the remaining bytes with the bytes that have been sent;

[0040] We define short flows as flows with a size less than 100KB. Most short flows have only one packet. For short flows, the number of bytes sent and the number of bytes remaining are small numbers, and the difference between them is also small. Therefore, the number of bytes sent increases over time and approaches the approximate value of the remaining flow size during the short lifetime of the short flow.

[0041] For long flows, a flow with a size greater than 100KB is defined as a long flow. The size of a long flow can reach GB. Similarly, the bytes that have been sent are used to replace the remaining bytes.

[0042] Since short flows are unlikely to send a large number of bytes, it's unlikely that they will be misclassified as long flows. In contrast, long flows may be misclassified as short flows very early in their lifecycles. However, the impact on other actual short flows is negligible, as the number of long flows is much smaller than that of short flows, and the misclassification lasts only for a short flow's normal lifecycle, after which the long flow is demoted to a low-priority queue due to reaching the multi-level feedback queue threshold. Therefore, while there is a significant difference between the number of bytes sent by a long flow and the number of bytes remaining, using already sent bytes instead of remaining bytes does not cause long flows to affect the scheduling of short flows.

[0043] Step 1.2: Estimate available bandwidth using link capacity and congestion function.

[0044] Step 1.3: Use the ratio of the number of bytes sent to the available bandwidth as the remaining time to achieve the shortest remaining time scheduling;

[0045] Step 1.4: Obtain the priority of the flow by comparing the shortest remaining time of the flow with the switch multi-level feedback queue threshold.

[0046] As a specific example, the available bandwidth is estimated using the link capacity and congestion function in step 1.2 as follows:

[0047] When the difference between the number of bytes sent and the number of remaining bytes of a short flow is small, and the long flow is mistakenly identified as a short flow and does not affect the normal short flow scheduling, the number of bytes sent is used as the estimated remaining size. When delivering the data packet, the TCP sender calculates the number of bytes sent, size a , and the remaining size of the stream is size s =size a ;

[0048] If the queue length exceeds a threshold, the forwarding layer programmable switch will mark the data packet. When the TCP receiver receives the marked data packet, it will also mark the returned ACK. When the TCP sender receives the ACK, it will calculate the percentage of marked ACKs corresponding to the last window and estimate the degree of network congestion as follows:

[0049]

[0050] CEND←CWND×(1-∝ / 2)

[0051] Where ∝ is an estimate of the network congestion range, 0<∝<1; is the weight given to the new sample relative to the previously estimated sample, F is the fraction or percentage of packets corresponding to the last window marked ACK, and CWND is the size of the congestion window;

[0052] ∝ is used to estimate the available bandwidth. The smaller the value of ∝, the smaller the proportion of marked packets, the shorter the time the queue length exceeds the threshold, and therefore the lower the network congestion;

[0053] Therefore, the estimated available bandwidth is:

[0054] B←L×(1-∝)

[0055] Where B is the estimated available bandwidth and L is the link capacity.

[0056] As a specific example, in step 1.3, the ratio of the number of bytes sent to the available bandwidth is used as the remaining time to implement the shortest remaining time scheduling. The formula is:

[0057] t s =size s / B

[0058] Among them, t s The minimum remaining time of the flow.

[0059] As a specific example, in step 2, the intelligent decision-making layer server uses the MADDPG-Q algorithm to adaptively calculate and optimize the queue thresholds for the multi-level feedback queues, and obtains a centrally trained multi-level feedback queue threshold update model, as follows:

[0060] Step 2.1, set the initial threshold of the multi-level feedback queue;

[0061] Step 2.2: Obtain network status information from the control layer SDN controller;

[0062] Step 2.3: The global network status information and flow history information are processed through the network analysis platform to obtain the source / destination IP, source / destination port number and transmission protocol, FCT, flow size, and link available bandwidth as input to start centralized training of the deep neural network, and obtain the multi-level feedback queue threshold update model;

[0063] Step 2.4: Send the centrally trained multi-level feedback queue threshold update model to the forwarding layer programmable switch, and repeat steps 2.2 to 2.4 at set intervals for updating.

[0064] As a specific example, in step 2.3, the multi-level feedback queue threshold update model training adopts the multi-level feedback queue threshold update algorithm based on the MADDPG reinforcement learning concept, namely the MADDPG-Q algorithm, as follows:

[0065] Step 2.3.1. Consider the switch as an agent, an environment with m agents, π=(π1,π2,…,π m) represents the strategy of m agents, θ=(θ1,θ2,…,θ m ) represents the strategy parameters of m agents;

[0066] Step 2.3.2, design the state space, which consists of the observation information of each agent s = (o1, o2, o3, ..., o m ), o m represents the mth flow. In the model, the state is represented as the set of all completed flows in the entire network at the current time step. Each flow is identified by a 5-tuple: source IP, destination IP, source port number, destination port number, and transport protocol. In addition, the FCT and the time spent on the link are recorded as flow attributes. Each flow has a total of 7 features. The bandwidth of the link where the flow is located is also included.

[0067] Step 2.3.3: Design the action space and calculate a set of MLFQ thresholds for each agent. Where t represents time, i represents the queue number, and m represents the agent number;

[0068] Step 2.3.4. Design a reward function, set a separate reward value for each agent, and model the reward as the ratio between the objective functions of two consecutive time steps: r t =T t / T t-1 , which indicates whether the previous operation caused a lower average FCT or reduced the overall performance; the average throughput of each completed flow is but

[0069]

[0070] in Represents the set of all completed streams;

[0071] Step 2.3.5: The intelligent decision-making layer server executes the MADDPG-Q algorithm training model. The specific algorithm is as follows:

[0072] Algorithm 1 MADDPG-Q for m agents

[0073] ① Initialize the online parameters of each agent actor network and the online parameters of the critic network, and initialize the state s;

[0074] ②For t=1 to MAX-EPISODE do

[0075] For each agent, choose action a m =μ m (o m )+N t , where om Represents the observed state of the agent, N t Represents the random noise added by exploration, μ m represents a deterministic strategy;

[0076] ③ Each agent performs the corresponding action a m , and the corresponding reward value r=(r1,r2,r3,r4,…,r m ), enter the next state s′, s←s′;

[0077] ④ Store (s, a, r, s′) in reply buffer B s , where a=(a1,a2,…,a m );

[0078] For agent M=1to m do

[0079] From B s Mini-batch sampling is performed in, and the sampling size is T s , each sample is (s i ,a i ,r i ,s′ i ) calculate the corresponding where y i is the set of target strategies, γ is the discount factor, γ∈(0,1], Represents the state-action function of the mth agent. The Q of different agents is independent, so different agents can set reward value functions with different structures;

[0080] ⑤ Update the critic network through back propagation of the loss function Loss, where Loss is defined as

[0081]

[0082] Where T is a time step;

[0083] ⑥ Update the actor network through the policy gradient formula. The calculation formula of the policy gradient is

[0084]

[0085] End for

[0086] ⑦Update the target network parameters of each agent, τ represents the learning rate

[0087] θ′ m =τθ m +(1-τ)θ′ m

[0088] End for

[0089] As a specific example, in step 3, the controller obtains the global network status, including network topology, link status, and traffic information, where the traffic information includes the ID corresponding to each flow, the five-tuple of the flow, the total number of transmissions, the number of completed transmissions, the transmission start time, the last feedback time, and the completion time.

[0090] As a specific example, step 4 is implemented by the following steps:

[0091] Step 4.1: Set the initial threshold for the switch queue during initialization.

[0092] Step 4.2: The new flow starts to be transmitted. The sender sets the priority of the new flow to the highest priority and marks each flow with an ID. The data packets corresponding to the flow enter the highest priority queue of the switch for transmission.

[0093] Step 4.3: Report network status information and traffic information to the control layer controller;

[0094] Step 4.4: Obtain a multi-level feedback queue threshold optimization model and perform threshold optimization calculation based on local information;

[0095] Step 4.5: For flows whose priority changes, that is, when the multi-level feedback queue threshold is reached or the multi-level feedback queue threshold is updated, the untransmitted data packets of the flow are marked, and the untransmitted data packets of the flow enter the new priority queue when they enter the queue;

[0096] Step 4.6: If high-priority transmissions continue to be queued, low-priority transmissions will be blocked, leading to a starvation state. The sender records the start time of the stream. For any stream that has not received feedback within 10ms, a control message is sent to increase the network transmission priority of the corresponding stream to avoid starvation.

[0097] Step 4.7: In the same priority queue, packets of different flows have the same flow priority, but different packet priorities, that is, different specific remaining times within the same remaining time threshold. By comparing with the priority queue threshold, the specific packet priority is calculated with a 20% difference between the two thresholds. Therefore, using PIFO technology, when a packet enters the queue, the packet priority is compared with the existing packets in the queue and inserted into the corresponding position obtained by the packet priority comparison and sorting. A packet with a higher priority means that the flow priority is the same but the transmission order in the same queue is higher.

[0098] Step 4.8: When the flow priority changes, that is, when the multi-level feedback queue threshold is reached or the multi-level feedback queue threshold is updated, the flow ID is traversed to find the data packets of the flow that are still waiting for transmission in the priority queue, and the data packets are put into the corresponding priority queue;

[0099] Step 4.9: Mark the retransmitted data packet. When a retransmission mark is detected in a data packet, the priority of the retransmitted data packet is adjusted and placed in a higher-level queue for priority transmission.

[0100] The present invention also provides a dynamic priority-based traffic scheduling system, which implements the dynamic priority-based traffic scheduling method. The system specifically includes a sending end, an intelligent decision-making layer server, a control layer SDN controller, a forwarding layer programmable switch, and a receiving end, wherein:

[0101] The sender calculates the remaining time by dividing the remaining flow size by the available bandwidth and compares it with the multi-level feedback queue threshold to obtain the flow priority.

[0102] The intelligent decision-making layer server uses the MADDPG-Q algorithm to adaptively calculate and optimize the queue thresholds for multi-level feedback queues, resulting in a centrally trained multi-level feedback queue threshold update model.

[0103] The SDN controller at the control layer connects the underlying hardware devices and the intelligent decision layer. The controller obtains the global network status and provides the necessary information for the intelligent decision layer server to calculate the optimization threshold. At the same time, it sends the multi-level feedback queue threshold update model trained by the intelligent decision layer server to the forwarding layer programmable switch.

[0104] The forwarding layer programmable switch uses a multi-level feedback queue as a priority queue, with manually set initial thresholds for the multi-level feedback queue. After calculating the priority of the sender, it matches flows of different priorities to different priority queues, completing flow priority scheduling. It also sorts packets of different packet priorities within the same flow priority interval queue and adjusts the position of the packet in the queue when the flow priority changes, achieving packet-level scheduling.

[0105] The receiving end is used to receive data packets: when congestion occurs, the receiving end marks the returned ACK when receiving the marked data packets; when the TCP sending end receives the ACK, it calculates the percentage of marked ACK corresponding to the last window and estimates the degree of network congestion.

[0106] The present invention also provides a mobile terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the dynamic priority-based traffic scheduling method when executing the program.

[0107] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0108] Example 1

[0109] Combined with Table 1, Figures 1 to 7 This embodiment provides a method and system for traffic scheduling based on dynamic priority. The meanings of the symbols involved are shown in Table 1:

[0110] Table 1 Symbol meaning

[0111]

[0112]

[0113] The test employs a leaf-spine network topology, commonly used in traditional data center network testing. The core layer has four spine switches, each interconnected to the eight leaf switches at the access layer via 40Gbps links. Each leaf switch then connects to 16 end devices via 10Gbps links. This architecture avoids the high latency and overload on core switches that occur in traditional multi-layer architectures due to the need for data to pass through multiple layers.

[0114] 1. Sending end: The sending end calculates the remaining time by dividing the remaining flow size by the available bandwidth, and obtains the flow priority by comparing it with the multi-level feedback queue threshold. The specific steps are as follows:

[0115] 1) Replace the remaining bytes with the bytes that have been sent;

[0116] 2) Estimate available bandwidth using link capacity and congestion functions;

[0117] 3) Use the ratio of the number of bytes sent to the available bandwidth as the remaining time to achieve the shortest remaining time scheduling.

[0118] 4) The priority of the flow is obtained by comparing the shortest remaining time of the flow with the switch multi-level feedback queue threshold.

[0119] For short flows, we generally define flows with a size less than 100KB as short flows. Most short flows have only one packet, and the number of bytes sent and the number of bytes remaining are both small numbers, with a small difference between them. Therefore, the number of bytes sent increases over time and approaches the approximate value of the remaining flow size within the short lifespan of the short flow.

[0120] For long flows, flows with a size greater than 100KB are generally defined as long flows. Long flows can reach sizes of GB and are processed using the same method described above. Since short flows cannot send a large number of bytes, it is impossible to mistakenly classify short flows as long flows. In contrast, long flows may be mistakenly classified as short flows at the very beginning of their life cycle. However, the impact on other actual short flows is negligible because the number of long flows is much smaller than that of short flows, and the duration of the misclassification is at most one normal life cycle of a short flow, after which the long flow will be reduced to a low priority queue due to reaching the multi-level feedback queue threshold. Therefore, although there is a significant difference between the number of bytes sent by long flows and the number of remaining bytes, the method of using bytes that have been sent instead of remaining bytes will not cause long flows to affect the scheduling of short flows.

[0121] When the difference between the number of bytes sent and the number of remaining bytes of a short flow is small and the long flow is mistakenly identified as a short flow and does not affect the normal short flow scheduling, the number of bytes sent can be used as the estimated remaining size, and when delivering the data packet, the TCP sender can easily calculate the number of bytes sent. a , and the remaining size of the stream is size s =size a .

[0122] If the queue length exceeds a threshold, the switch will mark the packet. When the TCP receiver receives the marked packet, it will also mark the returned ACK. When the TCP sender receives the ACK, it will calculate the percentage of marked ACKs corresponding to the last window and estimate the degree of network congestion as follows:

[0123]

[0124] CWND←CWND×(1-∝ / 2)

[0125] where α(0<α<1) is an estimate of the network congestion range, (usually set to 0.0625) is the weight given to new samples relative to previously estimated samples, F is the fraction (or percentage) of packets marked ack that correspond to the last window of packets, and CWND is the size of the congestion window.

[0126] α is suitable for estimating available bandwidth because it reflects the degree of network congestion. A smaller value of α indicates a smaller proportion of marked packets, a shorter period of time when the queue length exceeds the threshold, and therefore less network congestion.

[0127] Therefore, the estimated available bandwidth is:

[0128] B←L×(1-∝)

[0129] Where B is the estimated available bandwidth and L is the link capacity.

[0130] The estimated shortest remaining time of the flow can be obtained as t s =size s / B. The remaining time of the flow is then compared with the threshold of the multi-level feedback queue to obtain the priority of the flow.

[0131] 2. Intelligent Decision-Making Server: The intelligent decision-making server uses the MADDPG-Q algorithm to adaptively calculate and optimize the queue thresholds for multi-level feedback queues. This is achieved through the following steps:

[0132] 1) Set the initial threshold of the multi-level feedback queue;

[0133] 2) Obtain network status information from the control layer SDN controller;

[0134] 3) The intelligent analysis platform processes global network status information and historical flow information to obtain data as input for centralized training of deep neural networks;

[0135] 4) The model after centralized training is distributed and updated at regular intervals.

[0136] The specific multi-level feedback queue threshold update model training process is as follows:

[0137] Generally speaking, an SDN-based intelligent network architecture consists of the forwarding layer, control layer, and intelligent decision-making layer, from bottom to top. The forwarding layer, composed of software-programmable switches, primarily forwards and processes packets, communicating with the SDN controller via southbound interfaces. The control layer primarily connects the underlying hardware devices with the intelligent decision-making layer and obtains global network status, such as network topology and link conditions. As the core of the intelligent network architecture, the intelligent decision-making layer uses reinforcement learning to learn a control policy. The control layer collects state information from all sessions and reports it to the intelligent decision-making layer for centralized training. (The intelligent decision-making layer utilizes historical and real-time network information collected by the network analysis platform, processes this information, and uses it as input for a deep neural network algorithm.) Once training is complete, the decision-making layer deploys the trained model to the network's edge switches (agents). Each agent only needs to know its own local observations to make optimal decisions.

[0138] Since MADDPG is well-suited for deployment in intelligent network architectures, we propose a multi-level feedback queue threshold update algorithm, MADDPG-Q, based on the MADDPG reinforcement learning concept. Considering a switch as an intelligent agent, we consider an environment with m intelligent agents, π = (π1, π2, ..., π m ) represents the strategy of m agents, θ=(θ1,θ2,…,θ m) represents the policy parameters of m agents. For the accumulation of m agents, the expected reward can be written as:

[0139]

[0140] Using a random strategy, the policy gradient can be written as:

[0141]

[0142] Represents the state-action function of the mth agent. The Q of different agents is independent, so different agents can set reward value functions with different structures.

[0143] Furthermore, the agent adopts the deterministic strategy μ m In the case of , the gradient formula is expanded using the chain rule to:

[0144]

[0145] The above gradient formula uses the Experience-Reply technique, where B s Represents a Buffer container, which contains several pieces of information, each of which is represented by a four-tuple (s, a, r, s′).

[0146] s=(o1,o2,…,o m ),a=(a1,a2,…,a m ),r=(r1,r2,…,r m )

[0147] o m 、a m 、r m Represent the observed state, action, and reward value of the mth agent respectively. The actor network is updated using the gradient descent method using the above gradient. The critic network can be updated by backpropagation using the loss function, where the loss function is:

[0148]

[0149] in: It is a collection of target strategies.

[0150] The entire multi-agent training process is similar to the single-agent training process. After training is completed, the actor can take corresponding actions based on its own observed state.

[0151] The cumulative density function of the time a flow spends on a link is expressed as F(x), so F(x) represents the probability that the time a flow spends is not greater than x. Let L i Indicates that a given flow enters different queues Q i The number of packets, where i = 1, ..., K, α i represents the optimal threshold of the i-th queue, so

[0152] E[L i ]≤(α i -α i-1 )(1-F(α i-1 ))

[0153] Denote the flow arrival rate as λ, then to queue Q i The packet arrival rate is λ i =λE[L i ]. The service rate of a queue depends on whether all queues with higher priority are empty. Therefore, P1 (highest priority) has capacity u1=u, where u is the service rate of the link. The idle rate of Q1 is (1-ρ1), where ρ i =λ i / u i Q i Therefore, the service rate of Q2 is u2=(1-ρ1)u, because if ρ1 is empty, its service rate is u (full link capacity). So we can get ρ0=0. Therefore, T i =1 / (u i -λ i ) is the queue Q i The average delay of is the average time spent in the i-th queue. i-1 , α i ), it will experience delays in queues of different priorities up to the i-th queue. Let i Max(x) is the queue number of the minimum degradation threshold greater than x. Therefore, the upper bound T(x) of the average FCT of a flow with a time of x is:

[0154] Let g i =F(α i )-F(α i-1 ) indicates the time required to spend [α i-1 , α i ) is the percentage of the flow. Therefore, g i is the gap between two consecutive thresholds. i Equivalently expressing α i , the FCT minimization problem can be expressed as:

[0155]

[0156] where g i ≥0, i=1,....,K-1.

[0157] First, the state space, action space and reward value function are designed, and then the MADDPG-Q algorithm is designed based on the intelligent network architecture.

[0158] 1) State space: consists of the observation information of each agent s = (o1, o2, o3, ..., o m ), o m represents the mth flow. In the model, the state is represented as the set of all completed flows in the entire network at the current time step. Each flow is identified by its 5-tuple: source / destination IP, source / destination port number, and transport protocol. The FCT and flow size are also recorded as flow attributes, for a total of seven features per flow, along with the estimated link available bandwidth.

[0159] 2) Action space: a set of MLFQ thresholds calculated by each agent Where t represents time, i represents the queue number, and m represents the agent number.

[0160] 3) Reward value function: Set a separate reward value for each agent and model the reward as the ratio between the objective functions of two consecutive time steps: r t =T t / T t-1 It indicates whether the previous operation caused the lower average FCT or it reduced the overall performance. The average throughput of each completed flow is

[0161]

[0162] in Represents the set of all completed streams.

[0163] Algorithm 1 MADDPG-Q for m agents

[0164] 1. Initialize the online parameters of each agent actor network and the online parameters of the critic network, and initialize the state s;

[0165] 2. For t=1 to MAX-EPISODE do

[0166] For each agent, choose action a m =μ m (o m )+N t , where om Represents the observed state of the agent, N t represents the random noise added by exploration;

[0167] 3. Each agent performs the corresponding action a m , and the corresponding reward value r=(r1,r2,r3,r4,…,r m ), enter the next state s′, s←s′;

[0168] 4. Store (s, a, r, s′) in reply buffer B s , where a=(a1,a2,…,a m );

[0169] For agent M=1 to m do

[0170] From B s Mini-batch sampling is performed in, and the sampling size is T s , each sample is (s i ,a i ,r i ,s′ i ) calculate the corresponding Where γ is the discount factor, γ∈(0,1];

[0171] 5. Update the critic network through the loss function loss, where Loss is defined as

[0172]

[0173] 6. Update the actor network using the policy gradient formula. The policy gradient formula is

[0174]

[0175] End for

[0176] 7. Update the target network parameters of each agent. The learning rate τ is usually set to 0.01

[0177] θ′ m =τθ m +(1-τ)θ′ m

[0178] End for

[0179] MADDPG-Q employs a centralized training and decentralized execution architecture consisting of multiple actor-critic networks. During training, each agent's actor network receives local observation state information, while the agent's critic network receives its own observation state information as well as additional information (the actions of other agents) for centralized global training. Once model training is complete, the actors only need to interact with the environment to determine the optimal action.

[0180] 3. Control layer SDN controller: The control layer SDN controller connects the underlying hardware devices and the intelligent decision layer. The controller obtains the global network status, such as network topology, link status, and traffic information (including the ID corresponding to each flow, the flow quintuple, the total number of transmissions, the number of completed transmissions, the transmission start time, the last feedback time, and the completion time), etc., to provide the required information for the intelligent decision layer to calculate the optimization threshold, and at the same time send the trained model to the switch.

[0181] 4. Forwarding layer programmable switch: The forwarding layer programmable switch uses multi-level feedback queues as priority queues, manually sets the initial thresholds of the multi-level feedback queues, and adaptively updates the thresholds through the distributed model and local information. After priority calculation for the sending end, it matches flows of different priorities into different priority queues to complete flow priority scheduling. At the same time, it sorts data packets of different packet priorities in the same flow priority interval queue, and adjusts the position of the data packet in the queue when the flow priority changes to achieve packet-level scheduling.

[0182] Flow priority scheduling is achieved through the following steps:

[0183] 1) Initial thresholds are set for the switch queue during initialization;

[0184] 2) When a new flow starts to be transmitted, the sender sets the priority of the new flow to the highest priority and marks each flow with an ID. The data packet corresponding to the flow enters the highest priority queue of the switch for transmission;

[0185] 3) Report network status information and traffic information to the control layer controller;

[0186] 4) The switch obtains the multi-level feedback queue threshold optimization model and combines local information to perform threshold optimization calculation;

[0187] 5) For flows whose priority changes (i.e., when the multi-level feedback queue threshold is reached or the multi-level feedback queue threshold is updated, resulting in a priority change), the untransmitted data packets of the flow are marked, and the untransmitted data packets of the flow enter the new priority queue when they enter the queue;

[0188] 6) If high-priority transmissions continue to queue, low-priority transmissions will be blocked, leading to a starvation state. The sender records the start time of the stream and, for any stream that has not received feedback within 10ms, sends a control message to increase the network transmission priority of the corresponding stream to avoid starvation.

[0189] 7) In the same priority queue, packets of different flows have the same flow priority, but different packet priorities, that is, the specific remaining time within the same remaining time threshold is different. By comparing with the priority queue threshold, the specific packet priority is calculated with a 20% difference between the two thresholds. Therefore, through PIFO technology, when a packet enters the queue, the packet priority is compared with the existing packets in the queue and inserted into the corresponding position obtained by the packet priority comparison and sorting. The packet with a higher priority means that the flow priority is the same but the transmission order in the same queue is higher;

[0190] 8) When the flow priority changes, that is, when the multi-level feedback queue threshold is reached or the multi-level feedback queue threshold is updated, the flow ID is traversed to find the data packets still waiting for transmission in the priority queue, and the data packets are put into the corresponding priority queue;

[0191] 9) Mark the retransmitted data packets. When a retransmission mark is detected in a data packet, the priority of the retransmitted data packet is adjusted and placed in a higher-level queue for priority transmission.

[0192] When the flow priority changes, the Modify(ID, PRIORITY) interface is called through a control message to dynamically adjust the priority scheduling algorithm. The control message contains three fields: an 8-bit MsgType field (bool type), which identifies whether the IP packet is a control message; a 32-bit ID field, which identifies the ID of the data packet and transmission stream; and a 32-bit PRIORITY field, which indicates the modified priority. In a real environment, these modifications can be implemented by modifying the optional field in the IPV6 protocol header. In the simulator, control messages can be implemented by simply adding additional fields to the IP data structure.

[0193] When each link receives a data packet for transmission, it first checks the MsgType field in the IP header of the packet. If it is not a control message, it continues with subsequent packet processing. If the MsgType field in the IP header indicates that the packet being processed is a control message, it extracts the 32-bit ID and 32-bit PRIORITY information from the IP header and calls the Modify interface of the queue corresponding to the current link. After the Modify operation returns, the control message is sent to the next link or node. Because control messages have an independent top priority, they can be sent directly to the next link or node without waiting in the queue. Therefore, control messages sent during the transmission process can promptly modify the priority of data packets along the entire network path from the sender to the receiver.

[0194] If the data packet is not a control message, when the data packet enters the queue, the packet priority pPRIORITY field in the packet header is checked and compared with the packet priority in the packet header of the data packet in the current queue, and the data packet is inserted into the corresponding position; if a retransmission flag is detected, the priority is changed and it is added to the end of the higher priority queue to implement priority scheduling of data packets in a single-priority queue, further reducing the waiting time of data packets with shorter remaining time in the queue.

[0195] 5. Receiver: When congestion occurs, the receiver will mark the returned ACK when it receives a marked packet. When the TCP sender receives the ACK, it will calculate the percentage of marked ACKs corresponding to the last window and estimate the degree of network congestion.

[0196] In summary, the present invention implements priority-based flow-level scheduling for data flows with varying demands and urgency, while also implementing dynamic priority-based packet-level scheduling for packets waiting in queues. By combining flow-level and packet-level scheduling, the present invention further reduces packet waiting time in the network and the system's average flow completion time (FCT). This further addresses the shortcomings of traditional traffic scheduling methods based on static queue thresholds, making it more adaptable to complex distributed network applications.

Claims

1. A traffic scheduling method based on dynamic priority, characterized in that: The configuration includes a sending end, an intelligent decision-making layer server, a control layer SDN controller, a forwarding layer programmable switch, and a receiving end. The specific method is as follows: Step 1: At the sending end, the remaining time is calculated by dividing the remaining flow size by the available bandwidth, and the priority of the flow is obtained by comparing it with the multi-level feedback queue threshold; Step 2: The intelligent decision-making layer server uses the MADDPG-Q algorithm to adaptively calculate and optimize the queue thresholds for the multi-level feedback queues, and obtains a centrally trained multi-level feedback queue threshold update model. Step 3: The control layer SDN controller connects the underlying hardware devices and the intelligent decision layer. The controller obtains the global network status and provides the necessary information for the intelligent decision layer server to calculate the optimization threshold. At the same time, the multi-level feedback queue threshold update model trained by the intelligent decision layer server is sent to the forwarding layer programmable switch. Step 4: The forwarding layer programmable switch uses a multi-level feedback queue as a priority queue and manually sets the initial threshold of the multi-level feedback queue. After calculating the priority of the sender, it matches flows of different priorities to different priority queues to complete flow priority scheduling. At the same time, it sorts packets of different packet priorities in the same flow priority interval queue and adjusts the position of the packet in the queue when the flow priority changes, thus achieving packet-level scheduling. Step 5: The receiving end receives data packets: When congestion occurs, the receiving end receives marked data packets and marks the returned ACK. When the TCP sending end receives the ACK, it calculates the percentage of marked ACKs corresponding to the last window and estimates the degree of network congestion. The MADDPG-Q algorithm described in step 2 is as follows: Step 2.3.

1. Consider the switch as an agent, an environment with m agents, π=(π1, π2, ..., π m ) represents the strategy of m agents, θ=(θ1, θ2,…,θ m ) represents the strategy parameters of m agents; Step 2.3.2, design the state space, which consists of the observation information of each agent s=(o1, o2, o3, …, o m ), o m represents the mth flow. In the model, the state is represented as the set of all completed flows in the entire network at the current time step. Each flow is identified by a 5-tuple: source IP, destination IP, source port number, destination port number, and transport protocol. In addition, the FCT and the time spent on the link are recorded as flow attributes. Each flow has a total of 7 features. The bandwidth of the link where the flow is located is also included. Step 2.3.3: Design the action space and calculate a set of MLFQ thresholds for each agent. ; Where t represents time, i represents the queue number, and m represents the agent number; Step 2.3.

4. Design a reward function, set a separate reward value for each agent, and model the reward as the ratio between the objective functions of two consecutive time steps: r t =T t / T t-1 , which indicates whether the previous operation caused a lower average FCT or reduced the overall performance; the average throughput of each completed flow is ;but ； in Represents the set of all completed streams; Step 2.3.5: The intelligent decision-making layer server executes the MADDPG-Q algorithm training model. The specific algorithm is as follows: Algorithm 1 MADDPG-Q for m agents ① Initialize the online parameters of each agent actor network and the online parameters of the critic network, and initialize the state s; ②For t=1 to MAX-EPISODE do For each agent, choose action a m =μ m (o m )+N t , where o m Represents the observed state of the agent, N t Represents the random noise added by exploration, μ m represents a deterministic strategy; ③ Each agent performs the corresponding action a m , and the corresponding reward value r=(r1, r2, r3, r4, …, r m ), enter the next state s′, s←s′; ④Store (s, a, r, s′) in reply buffer B s , where a=(a1, a2, …, a m ); For agent M=1 to m do From B s Mini-batch sampling is performed in, and the sampling size is T s , each sample is (s i , a i , r i , s ′i ) calculate the corresponding , where y i is the set of target strategies, γ is the discount factor, γ∈(0, 1], Represents the state-action function of the mth agent. The Q of different agents is independent, so different agents can set reward value functions with different structures; ⑤ Update the critic network through back propagation of the loss function Loss, where Loss is defined as ； Where T is a time step; ⑥ Update the actor network through the policy gradient formula. The calculation formula of the policy gradient is ； End for ⑦Update the target network parameters of each agent, Represents the learning rate ； End for.

2. The traffic scheduling method based on dynamic priority according to claim 1, characterized in that: In step 1, the remaining time is calculated by dividing the remaining flow size by the available bandwidth at the sending end. The priority of the flow is obtained by comparing it with the multi-level feedback queue threshold, as follows: Step 1.1, replace the remaining bytes with the bytes that have been sent; Step 1.2: Estimate available bandwidth using link capacity and congestion function. Step 1.3: Use the ratio of the number of bytes sent to the available bandwidth as the remaining time to achieve the shortest remaining time scheduling; Step 1.4: Obtain the priority of the flow by comparing the shortest remaining time of the flow with the switch multi-level feedback queue threshold.

3. The traffic scheduling method based on dynamic priority according to claim 2, characterized in that: Step 1.2 estimates the available bandwidth using link capacity and congestion function as follows: Using the number of bytes sent as the estimated remaining size, when delivering the packet, the TCP sender calculates the number of bytes sent as size a , and the remaining size of the stream is size s =size a ; If the queue length exceeds a threshold, the forwarding layer programmable switch will mark the data packet. When the TCP receiver receives the marked data packet, it will also mark the returned ACK. When the TCP sender receives the ACK, it will calculate the percentage of marked ACKs corresponding to the last window and estimate the degree of network congestion as follows: ； ； in is an estimate of the network congestion range, ; is the weight given to the new sample relative to the previously estimated sample, is the fraction or percentage of packets marked ACK corresponding to the last window, is the size of the congestion window; Used to estimate available bandwidth, The smaller the value, the smaller the proportion of data packets marked, the shorter the time the queue length exceeds the threshold, and therefore the lower the network congestion; Therefore, the estimated available bandwidth is: ； Where B is the estimated available bandwidth and L is the link capacity.

4. The traffic scheduling method based on dynamic priority according to claim 3, characterized in that: In step 1.3, the ratio of the number of bytes sent to the available bandwidth is used as the remaining time to achieve the shortest remaining time scheduling. The formula is: t s =size s / B Among them, t s The minimum remaining time of the flow.

5. The traffic scheduling method based on dynamic priority according to claim 1, characterized in that: In step 2, the intelligent decision-making layer server uses the MADDPG-Q algorithm to adaptively calculate and optimize the queue thresholds for the multi-level feedback queues, and obtains a centrally trained multi-level feedback queue threshold update model, as follows: Step 2.1, set the initial threshold of the multi-level feedback queue; Step 2.2: Obtain network status information from the control layer SDN controller; Step 2.3: The global network status information and flow history information are processed through the network analysis platform to obtain the source / destination IP, source / destination port number and transmission protocol, FCT, flow size, and link available bandwidth as input to start centralized training of the deep neural network, and obtain the multi-level feedback queue threshold update model; Step 2.4: Send the centrally trained multi-level feedback queue threshold update model to the forwarding layer programmable switch, and repeat steps 2.2 to 2.4 at set intervals for updating.

6. The traffic scheduling method based on dynamic priority according to claim 5, characterized in that: In step 2.3, the multi-level feedback queue threshold update model training adopts the multi-level feedback queue threshold update algorithm based on the MADDPG reinforcement learning idea.

7. The traffic scheduling method based on dynamic priority according to claim 1, characterized in that: In step 3, the controller obtains the global network status, including network topology, link status, and traffic information. The traffic information includes the ID corresponding to each flow, the five-tuple of the flow, the total number of transmissions, the number of completed transmissions, the start time of the transmission, the last feedback time, and the completion time.

8. The traffic scheduling method based on dynamic priority according to claim 1, characterized in that: Step 4 is achieved by the following steps: Step 4.1: Set the initial threshold for the switch queue during initialization. Step 4.2: The new flow starts to be transmitted. The sender sets the priority of the new flow to the highest priority and marks each flow with an ID. The data packets corresponding to the flow enter the highest priority queue of the switch for transmission. Step 4.3: Report network status information and traffic information to the control layer controller; Step 4.4: Obtain a multi-level feedback queue threshold optimization model and perform threshold optimization calculation based on local information; Step 4.5: For flows whose priority changes, that is, when the multi-level feedback queue threshold is reached or the multi-level feedback queue threshold is updated, the untransmitted data packets of the flow are marked, and the untransmitted data packets of the flow enter the new priority queue when they enter the queue; Step 4.6: If high-priority transmissions continue to be queued, low-priority transmissions will be blocked, leading to a starvation state. The sender records the start time of the stream. For any stream that has not received feedback within 10ms, a control message is sent to increase the network transmission priority of the corresponding stream to avoid starvation. Step 4.7: In the same priority queue, packets from different flows have the same flow priority, but different packet priorities. That is, the specific remaining time within the same remaining time threshold is different. By comparing with the priority queue threshold, the specific packet priority is calculated with a 20% difference between the two thresholds. Therefore, using PIFO technology, when a packet enters the queue, the packet priority is compared with the existing packets in the queue and inserted into the corresponding position obtained by the packet priority comparison and sorting. A packet with a higher priority indicates that the flow priority is the same but the transmission order in the same queue is higher. Step 4.8: When the flow priority changes, that is, when the multi-level feedback queue threshold is reached or the multi-level feedback queue threshold is updated, the flow ID is traversed to find the data packets of the flow that are still waiting for transmission in the priority queue, and the data packets are put into the corresponding priority queue; Step 4.9: Mark the retransmitted data packet. When a retransmission mark is detected in a data packet, the priority of the retransmitted data packet is adjusted and placed in a higher-level queue for priority transmission.

9. A dynamic priority traffic scheduling system, characterized in that: The system implements the dynamic priority-based traffic scheduling method according to any one of claims 1 to 8. The system specifically includes a sending end, an intelligent decision-making layer server, a control layer SDN controller, a forwarding layer programmable switch, and a receiving end, wherein: The sender calculates the remaining time by dividing the remaining flow size by the available bandwidth and compares it with the multi-level feedback queue threshold to obtain the flow priority. The intelligent decision-making layer server uses the MADDPG-Q algorithm to adaptively calculate and optimize the queue thresholds for multi-level feedback queues, resulting in a centrally trained multi-level feedback queue threshold update model. The SDN controller at the control layer connects the underlying hardware devices and the intelligent decision layer. The controller obtains the global network status and provides the necessary information for the intelligent decision layer server to calculate the optimization threshold. At the same time, it sends the multi-level feedback queue threshold update model trained by the intelligent decision layer server to the forwarding layer programmable switch. The forwarding layer programmable switch uses a multi-level feedback queue as a priority queue, with manually set initial thresholds for the multi-level feedback queue. After calculating the priority of the sender, it matches flows of different priorities to different priority queues, completing flow priority scheduling. It also sorts packets of different packet priorities within the same flow priority interval queue and adjusts the position of the packet in the queue when the flow priority changes, achieving packet-level scheduling. The receiving end is used to receive data packets: when congestion occurs, the receiving end marks the returned ACK when receiving the marked data packets; when the TCP sending end receives the ACK, it calculates the percentage of marked ACK corresponding to the last window and estimates the degree of network congestion.

10. A mobile terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the dynamic priority-based traffic scheduling method according to any one of claims 1 to 8 is implemented.

Images