Group selection scheduling method based on PerDQN-TAS protection band

By abstracting the protection band problem in the TSN network into a backpack problem and using the PerDQN-TAS network, the weighted reward function of packet length and priority is solved, and the problem of low protection band bandwidth utilization in the existing technology is achieved efficiently Bandwidth resource utilization and priority scheduling.

CN118945118APending Publication Date: 2024-11-12CHANGZHOU UNIV +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410989920.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The bandwidth utilization of the protection band in the existing TSN network is low, resulting in waste of resources, and it is difficult for existing methods to effectively schedule appropriate packets to improve bandwidth utilization.

Method used

Using PerDQN-TAS network, by abstracting the protection belt problem into a backpack problem with priority constraints, using the weighted reward function of group length and priority, the agent is trained to adopt the optimal behavior in different states, and optimize the grouping selection to improve the Bandwidth utilization and cumulative priority.

Benefits of technology

The bandwidth utilization rate has always been maintained above 91%, and gradually increased with the increase of the initial residual capacity of the protection belt, and finally reached a bandwidth utilization rate of 8.1%. At the same time, the cumulative priority reached 8.98, which significantly improved the resource utilization efficiency of the protection belt. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118945118A_ABST
    Figure CN118945118A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of TSN networks, in particular to a PerDQN-TAS-based protection band packet selection scheduling method, which comprises the following steps of: setting packet length, packet priority and residual capacity of a protection band; initializing a learning rate, a discount factor, an exploration rate, an experience playback pool size and a reward weight coefficient; updating the reward function; solving the priority of a certain experience, and solving the probability of the certain experience; taking the current state St as input, outputting an optimal action at, executing according to the selected action, and obtaining a reward Rt + 1 and a state St + 1; and (St, at, Rt + 1, St + 1) is stored in an experience pool, data is sampled from the experience pool, parameters of the Q network are updated by using the sampled data through a loss function, and a PerDQN-TAS network is obtained. According to the invention, the problem of low transmission efficiency of the TSN network caused by incapability of effectively scheduling appropriate packets to fully utilize the bandwidth of the guard band in the prior art is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of TSN network technology, and in particular to a PerDQN-TAS protection band grouping selection scheduling method. Background Art

[0002] The research directions of TSN mainly include time synchronization, traffic scheduling, and network configuration; Time-aware shaping scheduling technology (TAS) supports data streams with multiple priorities and schedules them in order of priority. However, at the beginning of the TT (Time Triggered, TT) flow time window, low-priority packets that are being transmitted but not completed may occupy the link.

[0003] To solve this problem, the IEEE 802.1Qbv standard introduces a guard band before the time window of each TT flow, stipulating that within the guard band, the scheduler will not schedule any new packets except the packets being transmitted on the link; however, while ensuring determinism, the guard band reduces the bandwidth utilization of the network, resulting in a waste of bandwidth resources.

[0004] For the design and optimization of TSN (time-sensitive network) traffic scheduling algorithms, the IEEE802.1Qbu and IEEE802.3br standards proposed a frame preemption mechanism. The frame preemption mechanism achieves preemptive forwarding of high-priority TT packets by interrupting the transmission of low-priority packets. However, the frame preemption mechanism relies on a complex underlying structure and cannot efficiently achieve the goal of improving the utilization of the protection band. It performs well in processing intermittent traffic, but performs relatively poorly under medium to high periodic traffic loads.

[0005] The paper Performance comparison of IEEE 802.1TSN time aware shaper (TAS) and asynchronous traffic shaper (ATS) proposes an adaptive adjustment mechanism for TAS gating. The results show that under appropriate configuration (such as accurate and precise gating plan), TAS can usually meet the specified delay bounds of the two scheduling types; while ATS performs well in processing intermittent traffic, but performs relatively poorly under medium to high periodic traffic loads.

[0006] Heilmann F et al. proposed an approach to improve bandwidth utilization in tsnnetworks by physically setting up two different FIFO sub-queues for each non-TT queue, which are divided into caching long packets and short packets, which alleviates the guard band problem to a certain extent; the literature Packet-size aware scheduling algorithms in guardband for time sensitive networking proposed the PAS algorithm, which abstracted the guard band problem into a priority constrained knapsack problem (PCKP) for solution. Summary of the invention

[0007] In view of the shortcomings of the existing methods, the present invention constructs a PerDQN-TAS network (Prioritized ExperienceReplay DQN TAC) to solve the problem that the existing methods cannot effectively schedule appropriate packets to fully utilize the guard band bandwidth, resulting in low TSN network transmission efficiency.

[0008] The technical solution adopted by the present invention is: the PerDQN-TAS protection band group selection scheduling method includes the following steps:

[0009] Step 1: Generate several data according to the FIFO packet queue, and set the packet length, packet priority and remaining capacity of the protection band;

[0010] As a preferred implementation of the present invention, the grouping priorities obey discrete uniform distribution.

[0011] As a preferred implementation of the present invention, the packet length obeys uniform distribution and Caida distribution.

[0012] Step 2: Initialize the learning rate, discount factor, exploration rate, experience replay pool size, and reward weight coefficient;

[0013] As a preferred implementation of the present invention, step 2 also includes: initializing the priority constraint, assuming that the packet buffer has n FIFO queues, the current remaining capacity r does not exceed the maximum packet length among the n, and the length l of each packet to be scheduled is initialized according to the remaining capacity. i,j To constrain.

[0014] Step 3: Update the reward function according to the group length, group priority and reward weight coefficient of the current state;

[0015] As a preferred embodiment of the present invention, the formula of the reward function is:

[0016] R t =βlt +(1-β)ω t (2)

[0017] Among them, l t is the packet length of the current state, ω t is the grouping priority of the current state, and β is the reward weight coefficient.

[0018] Step 4: Solve the priority of a certain experience based on the TD error and the exploration rate, and solve the probability of a certain experience based on the priority of a certain experience;

[0019] As a preferred embodiment of the present invention, the formula of probability of a certain experience is:

[0020]

[0021] Among them, δ t is the TD error, ε is the exploration rate, α is the learning rate, N is the experience replay pool size, and i is the i-th experience.

[0022] Step 5: Set the current state S t As input, the optimal action a is given by the output of the Q network t , get reward R t+1 and state S t+1 ; will (S t ,a t ,R t+1 ,S t+1 ) is stored in the experience pool, data is sampled from the experience pool, and the parameters of the Q network are updated using the sampled data through the loss function. When the maximum number of iterations is met, the PerDQN-TAS model is obtained;

[0023] As a preferred implementation manner of the present invention, it also includes: using bandwidth utilization and cumulative priority to evaluate the performance of the PerDQN-TAS network.

[0024] As a preferred embodiment of the present invention, a PerDQN-TAS protection band group selection scheduling system includes: a memory for storing instructions executable by a processor; a processor for executing instructions to implement a PerDQN-TAS protection band group selection scheduling method.

[0025] As a preferred embodiment of the present invention, a computer readable medium stores a computer program code, and when the computer program code is executed by a processor, the computer program code implements a PerDQN-TAS protection band group selection scheduling method.

[0026] Beneficial effects of the present invention:

[0027] 1. By introducing two packet length and packet priority reward values ​​and performing weighted summation, the PerDQN-TAS model is trained to learn the optimal behavior to be taken in different states, making full use of the available bandwidth in the guard band while improving the cumulative priority;

[0028] 2. Compared with the existing TAS guard band method, the bandwidth utilization of PerDQN-TAS always remains above 91%, and gradually increases with the increase of the initial remaining capacity of the guard band, eventually reaching 8.1%; at the same time, the cumulative priority continues to improve, eventually reaching 8.98. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is the PerDQN-TAS logic diagram of the present invention;

[0030] Figure 2 are examples of guard band utilization under different methods;

[0031] Figure 3 This is a comparison chart of reward values ​​on different distributions;

[0032] Figure 4 is the bandwidth utilization on a uniform distribution;

[0033] Figure 5 is the bandwidth utilization on the Caida distribution;

[0034] Figure 6 is the cumulative priority over the uniform distribution;

[0035] Figure 7 is the cumulative priority on the Caida distribution. DETAILED DESCRIPTION

[0036] The present invention is further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner, and therefore it only shows the components related to the present invention.

[0037] like Figure 1 As shown, the PerDQN-TAS protection band group selection scheduling method includes the following steps:

[0038] First, we construct a Markov decision process (MDP) and define the state S t 、Behavior a t , Reward R t and state transition S t ; Secondly, the weighted sum of group length and priority is used as the reward function; Thirdly, by initializing the agent and the environment, the environment is reset for training; Thirdly, the Q network (value network) is updated according to the experience replay mechanism, and the agent gradually learns how to make optimal decisions in a dynamic environment.

[0039] Step 1: Abstract the guard band problem into a prioritized knapsack problem. The prioritized knapsack problem can be formalized as a Markov decision process (MDP). Each group has a group length l. i and group priority ω i Two properties, the remaining capacity r of the guard band is equivalent to the total capacity of the backpack;

[0040] By selecting the most appropriate group to fill the guard band, the bandwidth utilization is maximized and the cumulative priority of the selected group is also the highest, thus ensuring that bandwidth resources are not wasted to the greatest extent.

[0041] Data preprocessing, generating a number of data according to the generation method of the FIFO packet queue, in this embodiment, 9986 data; wherein, ω i Obeying discrete uniform distribution, l i It obeys uniform distribution and divides the data into training set and test set according to the ratio of 7:3;

[0042] Step 2: Initialize the agent and the environment. The agent is understood as all flows of different priorities, and the environment includes states, actions, rewards, and state transfers. Initialization parameters include: learning rate α, which is used to control the step size of the model weight adjustment at each parameter update, and determines the speed at which the algorithm updates the model weights when minimizing the loss function; discount factor γ, which is used to weigh the importance of current rewards and future rewards; exploration rate ε, which is used to balance the proportion of exploring new strategies and using existing strategies in reinforcement learning, and usually decreases gradually with the training process; experience replay pool size N, which is used to store the state, action, reward, and next state experience experienced by the agent in the environment, and randomly extract small batches of experience during training to update the model to improve the stability and efficiency of training; reward weight coefficient β, which is used to adjust the proportion of the contribution of these two factors to the total reward when the packet length and priority are weighted and summed in the reward function; residual capacity r, which refers to the bandwidth capacity remaining in the guard band in the current TSN network after the data packet is placed.

[0043] It also includes: assuming that the packet buffer has n FIFO queues, and the current remaining capacity r does not exceed the maximum packet length among the n; assuming that all packets have been pre-buffered in their FIFO queues; for the jth packet in queue i, l i,j and ω i,j Represents its packet length and priority respectively; where i∈{1,2,…,n}, j∈{1,2,…,n}, n i represents the number of packets cached in queue i; packets in the same queue must be dequeued in first-in-first-out order; the remaining capacity r can be used to schedule the number of packets scheduled in each queue x iIndicates; where x i ∈{0,1,…,n}, the length of each packet to be scheduled is recorded as l i,j , transformed into solving the following integer programming problem, the formula is:

[0044]

[0045] If the remaining capacity r corresponds to the volume of a backpack, the grouping length l corresponds to the volume of the item, and the grouping priority ω corresponds to the value of the item, the guard band problem can be abstracted into a backpack problem with priority constraints for solution.

[0046] Step 3: Update the reward function. For each state, the action is divided into two types: "select to put the group into the protection zone" and "do not select to put the group into the protection zone". Therefore, the size of the action space is 2. Set the reward value of the knapsack problem to three types. The first is the group length l, the second is the group priority ω, and the third reward function R t , β adds weights to the packet length and priority respectively, and the corresponding formula is:

[0047] R t =βl t +(1-β)ω t (2)

[0048] Among them, l t is the packet length of the current state, ω t is the grouping priority of the current state, and β is the reward weight coefficient.

[0049] Step 4: Create an experience pool for storing experience to store the current state S t 、Action a t , Reward R t , the probability formula for selecting the group to be scheduled can be calculated based on the priority of its experience. Assuming that an experience in the experience pool is i, then the priority of i is Then the formula for the probability P(i) of selecting experience i is as follows:

[0050]

[0051] Among them, N is the size of the experience replay pool and α is the learning rate.

[0052] Priority By TD error δ t Decision, δ tIt measures the difference between the value estimate in the current state and the value estimate after one-step state transfer plus the immediate reward, that is, the difference between the current Q value and the target Q value; the priority experience replay mechanism uses the TD error to determine the importance of each experience, so as to replay those experiences with larger TD errors more frequently to accelerate the learning process. The calculation formula is as follows:

[0053]

[0054] Therefore, the formula for the probability P(i) of selecting experience i can also be expressed as:

[0055]

[0056] Step 5: Update the Q network (DQN), use the small batch experience to update the current value network, and periodically update the reward and new state with the target value network; the loss function is used to measure the predicted Q value and the target Q value y i The difference between them is calculated by using the mean squared error (MSE) as the loss function for the Q network to optimize the parameters of the Q network. The MSE loss is calculated as follows:

[0057]

[0058] In the actual operation process, the agent will change the current state S t As input, select the best action a according to the output of the Q network t , and then execute the selected action to get the reward R t+1 and the new state S t+1 ; Then transfer this (S t ,a t ,R t+1 ,S t+1 ) is stored in the experience pool, data is sampled from the experience pool, and the sampled data is used to update the parameters of the Q network. When the network reaches the maximum iteration round, the trained PerDQN-TAS model is output for decision-making.

[0059] Step 6: Use bandwidth utilization and cumulative priority indicators to evaluate performance;

[0060] Bandwidth utilization η r , is the expected cumulative length of packets scheduled by the shaper when the initial remaining capacity is r. r The ratio of ] to r:

[0061]

[0062] Cumulative priority ω sum, is the expected cumulative priority of the scheduling group when the remaining capacity is r, and the formula is:

[0063] ω sum =E[ω1+…+ω T |l1+…+l T ≤r] (9)

[0064] Using cumulative priority rather than priority density facilitates the implementation of algorithms in a unified programmable framework.

[0065] Figure 2 is an example of the utilization of guard bands under different schemes, where packets a1 and a2 belong to time-triggered flows; packets b and c represent flows with lower priority than time-triggered flows, such as AVB flows or BE flows; a specific time window [t1, t2] is specifically allocated to time-triggered flows during scheduling; if guard bands are not used, the non-time-triggered flow packets being sent will occupy part of the time window of the time-triggered flow, increase the jitter of the time-triggered flow, and even trigger an abnormal chain reaction; the basic principle of the TAS algorithm is that as long as the highest priority packet can be transmitted within the remaining capacity, it will continue to be scheduled, otherwise, it will enter the guard band; the frame preemption mechanism is as follows Figure 2 As shown in Figure 3, packet b3 is cut into two segments b at time t1. 3l and b 3r , the latter part b 3r After all packets of TT flow a are transmitted, they are forwarded. However, the implementation complexity of frame preemption is too high. It is necessary to add a new link sublayer between the link layer and the physical layer to support the slicing and assembly of low-priority packets. At the same time, under the frame preemption mechanism, the protection band must at least ensure that the minimum non-preemptible packet can be transmitted in it, so the frame preemption mechanism is not very suitable. The principle of the fourth scheduling algorithm PAS is to continue scheduling as long as the highest priority packet can be transmitted within the remaining capacity.

[0066] The PerDQN-TAS method of the present invention tries to select the optimal packet scheduling by setting a reward mechanism, that is, to select the most suitable packet among all the packets to be scheduled to improve bandwidth utilization while maximizing the cumulative priority.

[0067] Experimental process:

[0068] The simulation was performed with the help of pytorch on a Windows 11 computer with AMD Ryzen75800H, RTX3060, and 16GB RAM; the simulation process is mainly divided into three parts: first, the system environment is set up, the PerDQN-TAS network is initialized and the parameters are set, then the problem is solved, and finally the performance of PerDQN-TAS is compared with other methods; the input of the scheduling algorithm is 256 first-in-first-out packet queues, and enough packets are generated for each queue; the packet priority ω is made to obey the discrete uniform distribution on {0, 1 / 255, 2 / 255,…, 1}; the packet length l adopts two distributions: 1. Uniform distribution, the value range is the set {84, 85,…, 1538}; 2. Caida distribution, according to the statistical results of packet length on the Caida website; the frequency distribution shown in Table 1 is generated.

[0069] Table 1 Distribution of Caida group length

[0070] Packet length interval frequency [84,138) 45% [138,1438) 15% [1438,1538) 20% 1538 20%

[0071] According to the generation method of the group queue, 9986 data are generated, and then the generated group queue is placed in the custom environment of reinforcement learning. The number of neurons in the input layer is 4, the number of neurons in the two hidden layers is 64 and 128 respectively, and the number of neurons in the output layer is 2. The parameter values ​​of the PerDQN-TAS model are defined as shown in Table 2:

[0072] Table 2 Parameter values

[0073] parameter Numeric Learning rate α 0.001 Discount factor γ 0.95 Exploration rate ε Initial value 1.0, minimum value 0.01, decay rate 0.995 Experience replay pool size N 1000 Batch size n 16 Weight coefficient β 0.1

[0074] Figure 3 The smoothing result of the reward value changing with the number of iterations under the condition that the initial residual capacity is 300 bytes is shown. The four curves represent the changes in the weighted reward values ​​of DQN and PerDQN-TAS under uniform distribution and Caida distribution, respectively, where the packet length and packet priority are weighted at 1:9. In the mid-term stage, it can be seen that the reward value of PerDQN-TAS grows significantly faster than that of DQN. When the number of training times exceeds 15,000, PerDQN-TAS gradually stabilizes in Caida distribution, the reward value tends to saturation, and the performance approaches the limit. The results show that PerDQN-TAS performs better than the DQN algorithm under both distributions.

[0075] Figure 4 , 5The theoretical and simulated bandwidth utilization curves of the four methods on uniform distribution and Caida distribution are shown as the initial residual capacity increases. As the initial residual capacity increases, the bandwidth utilization of the four methods also increases. Under uniform distribution, PerDQN-TAS shows extremely high bandwidth utilization, from about 85% when the initial residual capacity is small to nearly 95% at the maximum capacity, indicating that PerDQN-TAS can effectively utilize bandwidth resources under uniform distribution conditions. Under Caida distribution, the PerDQN-TAS algorithm still performs well, and the bandwidth utilization always remains above 80%, but there are certain fluctuations when the initial residual capacity is small.

[0076] Figure 6 , 7 The graph shows the cumulative priority curves of TAS, GPAS, DQN and PerDQN-TAS in the packet length distribution in uniform distribution and Caida distribution as the initial remaining capacity increases; the graph on the uniform distribution shows that as the initial remaining capacity increases, PerDQN-TAS has the highest cumulative priority and performs best, followed by the DQN algorithm; these two deep reinforcement learning algorithms can quickly increase their priority when processing large capacity, showing high scheduling efficiency; the graph on the Caida distribution also shows the advantage of PerDQN-TAS, which still maintains the highest cumulative priority when the initial remaining capacity increases, showing the best scheduling performance.

[0077] In summary, PerDQN-TAS has the fastest training speed and convergence speed, and also comprehensively considers the two indicators of bandwidth utilization and cumulative priority. Therefore, it can be concluded that PerDQN-TAS, which sets the reward value as the weighted sum of packet length and priority, is suitable for solving the problem of low TSN guard band utilization.

[0078] Aiming at the problem of bandwidth waste caused by guard band in IEEE 802.1Qbv, the present invention first compares the advantages and disadvantages of existing solutions and finds that the existing method handles the problem of guard band bandwidth resource waste; abstracts the guard band problem into a special knapsack problem PCKP, proposes PerDQN-TAS, considers two indicators of bandwidth utilization and cumulative priority by setting the weight of comprehensive packet length and priority, compares bandwidth utilization and cumulative priority, and the PerDQN-TAS of the present invention has significantly improved in improving bandwidth utilization and prioritizing scheduling of high priority packets.

[0079] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. Based on the PerDQN-TAS guard band group selection scheduling method, It is characterized in that The following steps are involved: Step 1: Generate several data according to the FIFO packet queue, and set the packet length, packet priority and remaining capacity of the protection band; Step 2: Initialize the learning rate, discount factor, exploration rate, experience replay pool size, and reward weight coefficient; Step 3: Update the reward function according to the group length, group priority and reward weight coefficient of the current state; Step 4: Solve the priority of a certain experience based on the TD error and the exploration rate, and solve the probability of a certain experience based on the priority of a certain experience; Step 5: Set the current state S t As input, the optimal action a is given by the output of the Q network t , get reward R t+1 and state S t+1 ; will (S t ,a t ,R t+1 ,S t+1 ) is stored in the experience pool, data is sampled from the experience pool, and the parameters of the Q network are updated using the sampled data through the loss function. When the maximum number of iterations is met, the PerDQN-TAS model is obtained.

2. The method for selecting and scheduling a group based on the PerDQN-TAS protection band according to claim 1, It is characterized in that The formula for the reward function is: R t =βl t +(1-β)ω t (2) Among them, l t is the packet length of the current state, ω t is the grouping priority of the current state, and β is the reward weight coefficient.

3. The PerDQN-TAS protection band grouping selection scheduling method according to claim 1, It is characterized in that The formula for the probability of a certain experience is: Among them, δ t is the TD error, ε is the exploration rate, α is the learning rate, N is the experience replay pool size, and i is the i-th experience.

4. The method for selecting and scheduling a group based on the PerDQN-TAS protection band according to claim 1, It is characterized in that Step 2 also includes: initializing the priority constraint, assuming that the packet buffer has n FIFO queues, the current remaining capacity r does not exceed the maximum packet length among the n, and the length l of each packet to be scheduled is set according to the remaining capacity. i,j To constrain.

5. The PerDQN-TAS protection band grouping selection scheduling method according to claim 1, It is characterized in that The grouping priority follows a discrete uniform distribution.

6. The PerDQN-TAS protection band grouping selection scheduling method according to claim 1, It is characterized in that The packet length follows uniform distribution and Caida distribution.

7. The PerDQN-TAS protection band grouping selection scheduling method according to claim 1, It is characterized in that Also includes: The performance of the PerDQN-TAS network is evaluated using bandwidth utilization and cumulative priority.

8. Based on PerDQN-TAS protection band group selection scheduling system, It is characterized in that include: a memory for storing instructions executable by a processor; A processor, configured to execute instructions to implement the PerDQN-TAS protection band grouping selection scheduling method according to any one of claims 1 to 7.

9. A computer readable medium storing computer program code, It is characterized in that When the computer program code is executed by a processor, the computer program code implements the PerDQN-TAS protection band group selection scheduling method according to any one of claims 1 to 7.