A Hybrid Traffic Resource Scheduling Method and Device for Time-Sensitive Networks

Abstract

The present invention provides a hybrid traffic resource scheduling method and apparatus applied to a time-sensitive network, belonging to the technical field of communication network traffic scheduling. This method uses the time-aware shaper mechanism and the cyclic queue forwarding mechanism to jointly perform traffic scheduling to meet the deterministic transmission requirements. On the premise of establishing deterministic transmission for time-sensitive flows, a multi-objective function for scheduling quality such as maximizing the scheduling success rate and bandwidth resource utilization of bandwidth flows is established, and a deep deterministic Q-network integrating a noise network is used to solve this multi-objective function to obtain the optimal time slot allocation decision for the flow to be scheduled. The present invention can flexibly integrate multiple objectives and quickly converge to a near-optimal solution in a large-scale search space, achieving a significant improvement in the scheduling quality of the time-sensitive network.

Description

Technical Field

[0001] The present invention belongs to the technical field of communication network traffic scheduling, and particularly relates to a hybrid traffic resource scheduling method and device applied to a time-sensitive network. Background Art

[0002] Low-latency deterministic networking is regarded as the primary requirement and challenge for the next-generation industrial Internet of Things. With the continuous improvement of data transmission rates, more intelligent industrial devices are connected through the network, and the potential demand for communication infrastructure is also increasing continuously. Its communication bandwidth and latency requirements are also constantly rising, and the existing network technologies can no longer meet the needs. As an Ethernet layer 2 network technology, Time-Sensitive Network (TSN) has good scalability and aims to support a standard-based real-time deterministic network to meet the requirements of low latency and low jitter for time-sensitive traffic. TSN integrates different scheduling mechanisms to provide an ultra-low-latency real-time deterministic network for time-critical services, such as Time-Aware Shaper (TAS), Cyclic Queuing and Forwarding (CQF), Asynchronous Traffic Shaper (ATS), and Credit-based Shaper (CBS). These shapers can be used alone or in combination. Among them, TAS is a key component in TSN to achieve deterministic low latency through a gating mechanism; CQF mainly solves the problem of bounded delay in transmission; ATS avoids using global clock synchronization but can still provide real-time guarantees by reshaping the traffic at each hop to reduce traffic burstiness; CBS is a commonly used traffic shaper that schedules traffic based on a credit mechanism. And the scheduling problem of time-sensitive flows is an NP-hard problem and remains a core challenge in TSN.

[0003] Many constraint programming-based methods and heuristic methods have been proposed in the existing literature to solve the TSN scheduling problem for time-sensitive services. However, the number of constraints may quickly expand with the number of scheduled flows, especially when scheduling multiple different types of flows, which exacerbates this situation and hinders the solver from solving the problem within a practical and acceptable running time. Heuristic methods rely on fixed rules and are difficult to handle dynamically changing or unseen traffic patterns. Summary of the Invention

[0004] The purpose of the present invention is to propose a hybrid traffic resource scheduling method and device applied to a time-sensitive network, which is used to optimize the allocation of time slots to solve the complex problem of hybrid traffic scheduling in TSN.

[0005] To achieve the above object, the present invention is implemented through the following technical solutions:

[0006] In a first aspect, the present invention provides a hybrid traffic resource scheduling method applied to a time-sensitive network, including:

[0007] Obtain the characteristic information of the flows to be scheduled in the time-sensitive network, and establish a traffic model based on the characteristic information; the flows to be scheduled include time-sensitive flows and bandwidth flows;

[0008] Configure a hybrid traffic scheduling model based on the traffic model; the hybrid traffic scheduling model jointly performs traffic scheduling based on the time-aware shaper mechanism and the cyclic queue forwarding mechanism;

[0009] On the premise of satisfying the deterministic transmission of time-sensitive flows, with the goal of maximizing the scheduling success rate and bandwidth utilization of bandwidth flows, transform the hybrid traffic scheduling model into a multi-objective optimization problem;

[0010] Model the multi-objective optimization problem as a Markov decision process;

[0011] Solve the Markov decision process to obtain the optimal time slot allocation result of the flows to be scheduled.

[0012] Preferably, the obtaining the characteristic information of the flows to be scheduled in the time-sensitive network and establishing a traffic model based on the characteristic information includes:

[0013] ,

[0014] wherein, is the set of flows to be scheduled, including time-sensitive flows and bandwidth flows, is an element in , is the flow sequence number, is the total number of flows to be scheduled, is the flow identifier, is the flow transmission period, is the flow deadline, is the jitter of the flow, is the flow packet size, is the number of flow packets.

[0015] Preferably, the configuring a hybrid traffic scheduling model based on the traffic model includes:

[0016] During the hybrid traffic scheduling process, configure the time-sensitive flows to adopt the time-aware shaper mechanism and be given the highest priority, and configure the bandwidth flows to adopt the cyclic queue forwarding mechanism and be given the second highest priority.

[0017] Preferably, on the premise of satisfying the deterministic transmission of time-sensitive flows, aiming to maximize the scheduling success rate and bandwidth utilization of bandwidth flows, the hybrid traffic scheduling model is transformed into a multi-objective optimization problem, including:

[0018] The multi-objective optimization problem is expressed as:

[0019] ,

[0020] where, , , and represent weights, represents the scheduling success rate of the flow, represents the bandwidth flow the total delay from generation to sending completion, represents the bandwidth flow the deadline, represents the gated total scheduling period, represents the number of time slots in the gated total scheduling period, represents the minimum scheduling time slot, represents the time slot the total size of the data packets within, represents the time slot the maximum capacity, represents the time slot the size of the time-sensitive flow data packets within, represents the time-sensitive flow the total delay from generation to sending completion, represents the time-sensitive flow the deadline, represents the bandwidth flow the jitter, represents the bandwidth flow the maximum allowable jitter, represents the flow the jitter, represents the flow the maximum allowable jitter, represents the flow the packet loss, represents the maximum allowable number of packet losses, represents the th packet of the flow represents an indicator function whose value is 1 when the data packet is injected into the time slot and 0 otherwise, represents the number of flow data packets of the flow , represents the time-sensitive flow The size of the flow data packet, represents the link bandwidth, represents the flow transmission period, is the modulo symbol, represents the maximum capacity of each queue in the cyclic queuing and forwarding mechanism, represents the network error compensation, represents the time slot allocation decision, represents the flow injection time slot.

[0021] Preferably, the gated total scheduling period is set to the least common multiple of the time-sensitive flow gated scheduling period and the bandwidth flow gated scheduling period;

[0022] The time-sensitive flow gated scheduling period is the least common multiple of the time-sensitive flow transmission periods;

[0023] The bandwidth flow gated scheduling period is twice the minimum scheduling time slot.

[0024] Preferably, modeling the multi-objective optimization problem as a Markov decision process includes:

[0025] Setting the state parameters of the Markov decision process as: , where represents the state at the th interaction between the agent and the environment, represents the characteristic information of the flow to be scheduled at the th interaction between the agent and the environment, represents the remaining capacity of each time slot within a gated total scheduling period at the th interaction between the agent and the environment;

[0026] Setting the action parameters of the Markov decision process as , and the action parameter represents the action selected at the th interaction between the agent and the environment, and the action is the time slot allocation decision of the multi-objective optimization problem;

[0027] Setting the reward function of the Markov decision process as:

[0028] ,

[0029] where, is the reward obtained after taking the action , is a positive constant value, is a negative constant value, is the delay reward, is the jitter reward, is the resource utilization reward.

[0030] Preferably, solving the Markov decision process to obtain the optimal time slot allocation result of the flow to be scheduled includes:

[0031] Integrating a noise network into the confrontation network to construct a deep noise Q network;

[0032] Set the action value function of the deep noise Q network as:

[0033] ,

[0034] where, is the action value function of the deep noise Q network, is the state value neural network integrating the noise network, is the advantage neural network integrating the noise network, and are the mean and standard deviation of the neural network, and are the random noises of the state value neural network and the advantage neural network, is to take the mean of the advantage neural network integrating the noise network, is the action set;

[0035] Based on the deep noise Q network, solve the Markov decision process to obtain the optimal time slot allocation result of the flow to be scheduled.

[0036] In a second aspect, the present invention provides a hybrid traffic resource scheduling device applied to a time-sensitive network, which is used to implement the above-mentioned hybrid traffic resource scheduling method applied to a time-sensitive network. The device includes:

[0037] A model construction module, configured to obtain the feature information of the flow to be scheduled in the time-sensitive network, and establish a traffic model based on the feature information; the flow to be scheduled includes a time-sensitive flow and a bandwidth flow;

[0038] A model configuration module, configured to configure a hybrid traffic scheduling model based on the traffic model; the hybrid traffic scheduling model jointly performs traffic scheduling based on the time-aware shaper mechanism and the cyclic queuing and forwarding mechanism;

[0039] A model conversion module, configured to convert the hybrid traffic scheduling model into a multi-objective optimization problem with the goal of maximizing the scheduling success rate and bandwidth utilization rate of the bandwidth flow on the premise of satisfying the deterministic transmission of the time-sensitive flow;

[0040] A problem modeling module, configured to model the multi-objective optimization problem as a Markov decision process;

[0041] An optimization scheduling module, configured to solve the Markov decision process to obtain an optimal time slot allocation result for the to-be-scheduled flow.

[0042] In a third aspect, the present invention provides a computer-readable storage medium storing a computer program, and the computer program is adapted to be loaded and executed by a processor to perform the hybrid traffic resource scheduling method applied to a time-sensitive network as described above.

[0043] In a fourth aspect, the present invention provides a computer device, including: a processor and a computer-readable storage medium;

[0044] A processor, adapted to execute a computer program;

[0045] A computer-readable storage medium storing a computer program, and when the computer program is executed by the processor, the hybrid traffic resource scheduling method applied to a time-sensitive network as described above is implemented.

[0046] By adopting the above technical means, the beneficial effects achieved by the present invention are as follows:

[0047] The present invention provides a hybrid traffic resource scheduling method applied to a time-sensitive network. Through the collaborative scheduling of the TAS mechanism and the CQF mechanism, while ensuring the deterministic transmission (low latency, low jitter) of time-sensitive flows, the bandwidth resource allocation is optimized by using the CQF periodic queue characteristics, significantly improving the scheduling success rate and bandwidth utilization rate of bandwidth flows (such as video surveillance data), and solving the problem that it is difficult for traditional single mechanisms to balance real-time performance and resource efficiency. By establishing a multi-objective function, indicators such as the scheduling success rate, latency, jitter, and resource utilization rate are jointly optimized, and the conflicts between various objectives are dynamically balanced through deep reinforcement learning. Compared with the single-objective optimization strategy, the comprehensive scheduling quality is significantly improved. By introducing parametric noise into the time slot resource scheduling algorithm based on the deep noisy Q-network to enhance the exploration ability, it can quickly converge to a near-optimal solution in a super-large time slot search space and adapt to dynamic network topology changes, meeting the TSN scheduling requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of the port architecture in a time-sensitive network switch in an embodiment of the present invention;

[0049] Figure 2 It is a schematic flowchart of the hybrid traffic resource scheduling method applied to a time-sensitive network provided by an embodiment of the present invention;

[0050] Figure 3 It is a schematic diagram of the deep noisy Q-network architecture provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] To make the objectives, technical solutions, and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but do not limit the present invention.

[0052] Here, it should also be noted that in order to avoid obscuring the present invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the present invention are shown in the drawings, while other details less related to the present invention are omitted.

[0053] It should be emphasized that the term "comprising / including" when used herein refers to the presence of features, elements, steps, or components, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0054] It should be emphasized here that the step marks mentioned hereinafter do not limit the sequence of the steps. Instead, it should be understood that the steps can be executed in the order mentioned in the embodiments, or different from the order in the embodiments, or several steps can be executed simultaneously.

[0055] The present invention comprehensively considers the service requirements of time-sensitive flows and bandwidth flows in a time-sensitive network. Among them, time-sensitive flows transmit time-sensitive information such as transmission control information and synchronization information. These flows are few in number and have short packet lengths, but have high requirements for latency and jitter. The transmission process should be as wait-free as possible. Therefore, the time-sensitive flow queue should adopt the TAS mechanism and be given the highest priority. Bandwidth flows are large-bandwidth periodic flows that transmit non-critical information in the network. They have a large sampling period and low requirements for latency and jitter, but often have a large number and long data. Therefore, the bandwidth flow queue should adopt the CQF mechanism and be given the second highest priority. Best-effort flows are given the lowest priority.

[0056] In order to achieve the mixed scheduling of time-sensitive flows and bandwidth flows, while ensuring the deterministic transmission of time-sensitive flows and improving the scheduling success rate of bandwidth flows, the present invention uses the TAS mechanism and the CQF mechanism in combination for traffic scheduling. The port architecture in the time-sensitive network switch is as Figure 1 shown. When the input flow passes through the switching fabric, the flow filter forwards the flow to the corresponding queue according to the priority. The queue exit is controlled by the gating list. When multiple gates are opened simultaneously, the priority selector forwards the data packets to the output port in descending order of queue priority.

[0057] Based on the above port architecture in the time-sensitive network switch, the first embodiment of the present invention provides a hybrid traffic resource scheduling method applied to a time-sensitive network. See Figure 2 , and the specific steps are as follows:

[0058] Step 1: Obtain the characteristic information of the flows to be scheduled in the time-sensitive network and establish a traffic model.

[0059] In the scenarios where time-sensitive network applications are used, the transmitted traffic needs to have bounded end-to-end delay, jitter, and packet loss. The deterministic transmission guarantee in the time-sensitive network is to reserve bandwidth resources for the traffic in advance. For time-sensitive flows and bandwidth flow services, the traffic information can be obtained in advance, and based on this, the temporal resource allocation strategy for transmission can be determined. Based on this, the traffic models for time-sensitive flows and bandwidth flows can be shown as follows:

[0060] ,

[0061] where, is the set of flows to be scheduled, including time-sensitive flows and bandwidth flows, is the flow sequence number, is the total number of flows to be scheduled, is the flow identifier, is the flow transmission period, is the flow deadline, is the jitter of the flow, is the size of the flow data packet, is the number of flow data packets. The best-effort flows in the network have no fixed period. In this embodiment, the specific parameters of the best-effort flows are not considered, and only their priority is defined as the lowest priority.

[0062] Step 2: Based on the traffic model in Step 1, configure a hybrid traffic scheduling model and design the constraints of the hybrid traffic scheduling model, specifically as follows.

[0063] In this embodiment, the TAS mechanism and the CQF mechanism are jointly used for traffic scheduling to meet the deterministic transmission requirements. Since multiple different types of traffic are involved in scheduling, when using a hybrid traffic scheduler, it is also necessary to constrain the gating, scheduling time slots, flow transmission, and flow service requirements:

[0064] (1) Traffic scheduling gating constraint: Since the time-sensitive flow uses the time-aware shaping mechanism TAS, its gating scheduling period is represented by the least common multiple (LCM) of the time-sensitive flow transmission periods as follows:

[0065] ,

[0066] where, represents the time-sensitive flow, represents the gating scheduling period of the time-sensitive flow, is the least common multiple, is the set of flows to be scheduled in the time-sensitive flow transmission period.

[0067] Since the bandwidth flow adopts the traffic shaping mechanism CQF of cyclic queuing and forwarding, the bandwidth flow gating scheduling period is twice the minimum scheduling time slot, that is .

[0068] The total gating scheduling period is the least common multiple of the time-sensitive flow and the bandwidth flow gating scheduling period, which is expressed as follows:

[0069] ,

[0070] The sequence of gate events in the gating list is defined as an ordered entry of an octet string. When all eight octets of the octet with a length of eight are 1 (i.e., ), the maximum value of the gating scheduling period will be obtained . Therefore, it is necessary to satisfy that cannot be greater than : .

[0071] (2) Minimum scheduling time slot Constraint: According to the definition, a gating scheduling period should be divided into several time slots, and the duration of each time slot is . Therefore, the number of time slots in each total gating scheduling period is . Let be the greatest common divisor (GCD) of the transmission periods of the time-sensitive flow and the bandwidth flow:

[0072] , , ,

[0073] Among them, is the transmission period of the time-sensitive flow in the set of flows to be scheduled , is the transmission period of the bandwidth flow in the set of flows to be scheduled , is the greatest common divisor.

[0074] Since is a multiple of the scheduling time slot , that is: Among them is a positive integer. Therefore, both the total gating scheduling period and the flow transmission period should be divisible by :

[0075] ,

[0076] ,

[0077] Among them, represents the set of flows to be scheduled in the flow transmission period, is the modulo operator.

[0078] To ensure the secure, stable and low-latency transmission of time-sensitive flows, the time-sensitive flow packets need to be all transmitted within the same time slot and, according to the gating technology of the time-aware shaper, it is necessary to satisfy:

[0079] ,

[0080] Among them, represents the set of flows to be scheduled in the time-sensitive flow packet size, is the link bandwidth.

[0081] According to the problem of the cyclic queuing and forwarding mechanism it adopts, the bandwidth flow needs to satisfy the ping-pong rule. The packets injected into the switch in the previous time slot need to be all forwarded in the next time slot, so it should follow:

[0082] ,

[0083] Among them is the link bandwidth, is the maximum capacity of each queue of the cyclic queuing and forwarding mechanism, is the network error compensation, which is used to compensate for the errors caused by time synchronization and transmission jitter, etc.

[0084] (3) Flow transmission constraint: Calculate the time resources occupied by each time slot within the current gating total scheduling period, that is:

[0085] ,

[0086] Among them is the time slot total packet size within, is the set of flows to be scheduled in the flow the th packet, is an indicator function. When the packet is injected into the time slot its value is 1, otherwise 0, represents the set of flows to be scheduled in the flow packet size, represents the set of flows to be scheduled The middle stream The number of streaming data packets

[0087] In addition, each data packet can be injected into at most one time slot, that is:

[0088] ,

[0089] Due to the store-and-forward mechanism of CQF, all data packets stored in the transmit buffer need to be sent before the start of the next time slot. Therefore, the total size of data packets within each time slot cannot exceed its maximum capacity, that is:

[0090] ,

[0091] where is the maximum capacity of time slot

[0092] (4)Constraints on streaming service requirements:

[0093] For time-sensitive streams, since time-sensitive streams have the highest priority and use the TAS queue scheduling, the remaining queues can send data only when the ST queue is empty. Therefore, to ensure that all streams in the network can be scheduled, it is necessary to ensure that within any unit time slot the size of time-sensitive stream data packets cannot exceed its maximum capacity, that is:

[0094] ,

[0095] where represents the size of time-sensitive stream data packets within time slot

[0096] And the total delay from the generation to the completion of sending of time-sensitive stream data packets needs to be less than or equal to the deadline of this stream, that is:

[0097] ,

[0098] where represents the total delay from the generation to the completion of sending of time-sensitive stream , represents the deadline of time-sensitive stream , represents the size of the stream data packets of time-sensitive stream

[0099] For bandwidth streams, since they use CQF scheduling, their delay includes the transmission delays of time-sensitive streams and other bandwidth streams with higher priority than , the transmission delays of other bandwidth streams that enter the queue first, the switch buffer delay, and the current ​​​Transmission delay of flow data packets. The bandwidth flow data packets also need to ensure that the total time delay from generation to transmission completion of the data packets is less than or equal to the deadline of the flow, that is:

[0100] ,

[0101] wherein, represents the total time delay from generation to transmission completion of the bandwidth flow , represents the deadline of the bandwidth flow , represents the flow sequence number with a higher priority than the bandwidth flow , represents the flow sequence number of the flow with the same priority as the bandwidth flow but entering the queue earlier represents the packet size of other bandwidth flows with a higher priority than the bandwidth flow , represents the packet size of other bandwidth flows with the same priority as the bandwidth flow but entering the queue earlier represents that the data packet is injected into the switch at time slot and sent within time slot .

[0102] In addition to the time delay, the jitter and packet loss of the transmission flow are also important parameters for deterministic transmission, that is:

[0103] ,

[0104] wherein, represents the jitter of flow , represents the packet loss of flow , represents the delay of the th packet of flow , represents the total delay of all packets of flow , represents the total number of packets of flow , represents the number of packets of flow successfully injected into time slot . Only when the requirements of time delay, jitter and packet loss are met simultaneously can the deterministic transmission of time-sensitive services be guaranteed.

[0105] Step 3: Formulate a multi-objective optimization problem based on the hybrid traffic scheduling model and service requirements.

[0106] In this embodiment, the hybrid scheduling problem of time-sensitive flows and bandwidth flows is formulated as an optimization problem. The goal is to maximize the scheduling success rate of bandwidth flows on the premise of meeting the deterministic transmission of time-sensitive flows.

[0107] In addition, this embodiment also adds the average total delay, jitter, and bandwidth resource utilization to the optimization goal to formulate a multi-objective optimization problem, which is expressed as follows:

[0108] ,

[0109]

[0110] ,

[0111] where, represents the jitter of bandwidth flow , represents the maximum allowable jitter of bandwidth flow , represents the jitter of flow , represents the maximum allowable jitter of flow , represents the maximum allowable number of packet losses, represents the time slot allocation decision, represents the injection time slot of flow , , , and represent weights, represents the scheduling success rate of the flow, that is, the ratio of the number of successfully scheduled flows to the total number of all flows.

[0112] In this embodiment, the scheduling success rate, delay, jitter, packet loss, and bandwidth resource utilization are jointly optimized to efficiently implement the hybrid traffic scheduling of time-sensitive flows and SR. Therefore, the weight coefficient ( ) is to balance the importance among several objective components. The constraint conditions include scheduling gating constraints, scheduling time slot constraints, flow transmission constraints, and flow service requirements. These constraints ensure that the traffic service can meet the traffic demand and provide deterministic transmission support for the traffic service, where is a very small constant.

[0113] Step 4: Model the multi-objective optimization problem formulated in Step 3 as a Markov decision process and set the parameters of the Markov decision process.

[0114] In this embodiment, the formulated multi-objective optimization problem is to design a dynamic resource scheduling strategy that maximizes the network's scheduling ability while satisfying the constraint conditions. This strategy belongs to the Markov decision process. To solve this scheduling problem and optimize the traffic scheduling performance, the multi-objective optimization problem of resource scheduling is re-modeled as a Markov decision process problem. The parameters of the Markov decision process include state, action, and reward. Among them, the state represents the current traffic information and network state, the action represents the current time slot resource allocation decision, and the reward represents the reward feedback in the scheduling decision.

[0115] State: , where is the state at the -th interaction between the agent and the environment, represents the characteristic information of the flow to be scheduled at the -th interaction between the agent and the environment, represents the remaining capacity of each time slot within a gated total scheduling period at the -th interaction between the agent and the environment.

[0116] Action: Based on the state and policy, the agent can make a time slot allocation decision to determine which time slots to schedule for transmitting the flow . Each action element is an integer value and indicates the time slot into which the traffic should be injected. Its definition is as follows:

[0117] ,

[0118] where, is the action selected at the -th interaction between the agent and the environment, is the action set, is the action in the action set, is the state, is the random noise, and each of its elements is randomly drawn independently from the standard normal distribution . and are parameters that are randomly initialized and learned from experience. is the action value function incorporating random noise.

[0119] Reward: After taking the action , a reward will be obtained to evaluate the quality of this action in the state .

[0120] In this embodiment, while optimizing the scheduling success rate, with the goal of reducing latency, jitter, and resource utilization, the reward function is set as:

[0121] ,

[0122] Among them, is a positive constant value, is the delay reward, is the jitter reward, is the resource utilization reward. When the scheduling fails, by setting as a negative constant value to punish improper decisions. Through the obtained rewards, the parameterized policy can be optimized in the training phase. In the Markov decision process, the goal of the agent is to find the optimal time slot allocation policy that can maximize the cumulative discounted reward so as to maximize the action value function . According to the Bellman equation, the action value function can be calculated as:

[0123] ,

[0124] Among them, represents the expectation, represents the discount factor, represents using the policy to control the interaction between the agent and the environment times, represents the agent's interaction with the environment for the th time, represents the reward for the agent's interaction with the environment for the th time using the policy represents the value function of the state and the action when using the policy

[0125] Step 5: Use the time slot resource scheduling algorithm based on the Deep Noisy Q-Network (DNQN) to solve the Markov decision process in Step 4 above to obtain the optimal time slot allocation decision.

[0126] In this embodiment, the time slot resource scheduling algorithm based on DNQN is used to solve the Markov decision process, and the deep noisy Q-network architecture is as Figure 3 shown. In this algorithm, double Q-learning is used to overcome the problem of decision overestimation, and prioritized experience replay is used to accelerate convergence to obtain a higher average return. In the neural network structure, the dueling network (Dueling Network) is adopted and the noise network is incorporated to improve the neural network structure of the Deep Q-Network (DQN).

[0127] The dueling network consists of two neural networks. One neural network is denoted as which is an approximation of the optimal advantage function, where is the advantage network parameters, and another neural network is denoted as is an approximation of the optimal state-value function, where is the state-value network parameters. Incorporating the noise network means adding noise to the network parameters , where and and are the mean and standard deviation, which are the parameters of the neural network, randomly initialized at the beginning and then learning from experience is the random noise, randomly sampled from the standard normal distribution .

[0128] When outputting, the flows of the two networks are combined to produce an approximate optimal action-value function, that is:

[0129] ,

[0130] where is the action-value function of the deep noise Q-network is the state-value neural network incorporating the noise network is the advantage neural network incorporating the noise network and are the random noises of the state-value neural network and the advantage neural network is to take the mean of the neural network to solve the non-uniqueness problem and make the parameters of the neural network and not change randomly.

[0131] Through this improvement, the value function and the advantage function can be estimated separately, so that the influence of both the state and the decision on the Q-value estimation can be considered simultaneously, and the correct decision can be identified more quickly during decision evaluation.

[0132] In this embodiment, the specific steps of the slot resource scheduling algorithm based on DNQN are as follows:

[0133] (1) Initialize the main network , the target network and the experience replay pool with size , where represents the noise of the target network and represent the mean and standard deviation of the neural network of the target network.

[0134] (2) Directly select the action through the main network, explore using the randomness of the noise in DNQN, and the obtained Stored in the experience replay pool, Indicates the state at the th interaction of the main network.

[0135] (3)Using prioritized experience replay, extract a quadruple from the experience replay pool and denote it as .

[0136] (4)Generate using the standard normal distribution, perform forward propagation on DNQN, and obtain: , where represents the predicted value of the main network at the th interaction, represents the action value function of the main network, and are the mean and standard deviation of the current neural network of the main network.

[0137] (5)Use DNQN to select the optimal action: .

[0138] (6)Generate using the standard normal distribution and calculate the value using the target network: , where represents the predicted value of the target network at the th interaction, represents the action value function of the target network, and represent the state and action at the th interaction, and are the mean and standard deviation of the current neural network of the target network.

[0139] (7)Calculate the Temporal Difference (TD) target and TD error:

[0140] , ,

[0141] where represents the TD target, represents the TD error, is the discount factor.

[0142] (8)Update the parameters of DNQN using gradient descent: ,

[0143] where and represent the updated parameters of DNQN, and are the learning rates, and Indicates the gradient.

[0144] (9) Update the parameters of the target network using weighted average:

[0145] ,

[0146] Where and are the parameters of the target network after update, is a hyperparameter that needs to be manually adjusted.

[0147] (10) Return to step (2) and repeat the above steps until the maximum number of training steps is reached.

[0148] Based on the same inventive concept, the second embodiment of the present invention provides a hybrid traffic resource scheduling device applied to a time-sensitive network, which is used to implement the hybrid traffic resource scheduling method applied to the time-sensitive network in the above embodiment. The device includes:

[0149] A model construction module, configured to obtain the feature information of the flows to be scheduled in the time-sensitive network, and establish a traffic model based on the feature information; the flows to be scheduled include time-sensitive flows and bandwidth flows;

[0150] A model configuration module, configured to configure a hybrid traffic scheduling model based on the traffic model; the hybrid traffic scheduling model jointly performs traffic scheduling based on the time-aware shaper mechanism and the cyclic queue forwarding mechanism;

[0151] A model conversion module, configured to convert the hybrid traffic scheduling model into a multi-objective optimization problem with the goal of maximizing the scheduling success rate and bandwidth utilization rate of the bandwidth flow on the premise of satisfying the deterministic transmission of the time-sensitive flow;

[0152] A problem modeling module, configured to model the multi-objective optimization problem as a Markov decision process;

[0153] An optimization scheduling module, configured to solve the Markov decision process to obtain the optimal time slot allocation result of the flows to be scheduled.

[0154] It should be noted that the device embodiment corresponds to the above method embodiment, and the implementation manners of the above method embodiment are all applicable to the device embodiment and can achieve the same or similar technical effects, so they will not be elaborated here.

[0155] Based on the same inventive concept, the third embodiment of the present invention provides a computer-readable storage medium, which stores a computer program, and the computer program is suitable for being loaded and executed by a processor to perform the hybrid traffic resource scheduling method applied to a time-sensitive network as in the above embodiment.

[0156] Based on the same inventive concept, the fourth embodiment of the present invention provides a computer device, including: a processor and a computer-readable storage medium;

[0157] The processor is adapted to execute a computer program;

[0158] The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the hybrid traffic resource scheduling method applied to the time-sensitive network as described in the above embodiments.

[0159] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0160] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of processes and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0161] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured product including an instruction device, and the instruction device implements the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0162] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide means for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1Steps of functions specified in one or more boxes.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: the specific implementation manners of the present invention can still be modified or equivalently replaced, and any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the protection scope of the claims of the present invention.

Claims

1. A hybrid traffic resource scheduling method applied to a time-sensitive network, characterized in that Including: Obtain the characteristic information of the flows to be scheduled in the time-sensitive network, and establish a traffic model based on the characteristic information; the flows to be scheduled include time-sensitive flows and bandwidth flows; Configure a hybrid traffic scheduling model based on the traffic model; the hybrid traffic scheduling model jointly performs traffic scheduling based on the time-aware shaper mechanism and the cyclic queue forwarding mechanism; On the premise of satisfying the deterministic transmission of time-sensitive flows, with the goal of maximizing the scheduling success rate and bandwidth utilization rate of bandwidth flows, transform the hybrid traffic scheduling model into a multi-objective optimization problem, and the multi-objective optimization problem is expressed as: Among them, β1, β2, β3, and β4 represent weights, F represents the set of flows to be scheduled, including time-sensitive flows and bandwidth flows, and f i is an element in F, i is the flow sequence number, n is the number of time-sensitive flows in the set of flows to be scheduled F, S represents the scheduling success rate of the flow, represents the bandwidth flow the total delay from generation to sending completion, represents the bandwidth flow the deadline of, SC Gate represents the gated total scheduling period, N slot represents the number of time slots in the gated total scheduling period, T slot represents the minimum scheduling time slot, C s represents the size of the total data packets in time slot s, represents the maximum capacity of time slot s, represents the size of the time-sensitive flow data packets in time slot s, represents the time-sensitive flow f i ST the total delay from generation to sending completion, f i ST [D] represents the deadline of the time-sensitive flow f i ST represents the bandwidth flow the jitter of, represents the bandwidth flow the maximum allowable jitter, represents the flow f i the jitter of, f i [Jitter] represents the flow f i the maximum allowable jitter, represents the flow f i the packet loss of, ψ represents the maximum allowable number of packet losses, represents the flow f i the x-th data packet of, represents an indicator function, whose value is 1 when the data packet is injected into time slot s and 0 otherwise, f i [p num represents the number of flow data packets of the flow f i i ST [p size represents the size of the flow data packets of the time-sensitive flow f i ST i [cycle] represents the transmission cycle of the flow f i max ​​​​represents the maximum capacity of each queue in the cyclic queuing and forwarding mechanism, and δ represents the network error compensation, represents the time slot allocation decision, α fi ∈ [1, N slot represents the injection time slot of flow f i ; Model the multi-objective optimization problem as a Markov decision process, including: Set the state parameters of the Markov decision process as: s t = {f t , Φ t}, where s t represents the state at the t-th interaction between the agent and the environment, f t represents the characteristic information of the flow to be scheduled at the t-th interaction between the agent and the environment, and Φ t represents the remaining capacity of each time slot within a gating total scheduling period at the t-th interaction between the agent and the environment; Set the action parameter of the Markov decision process to a t , where the action parameter a t represents the action selected by the agent and the environment during the t-th interaction, and the action is the time slot allocation decision of the multi-objective optimization problem; Set the reward function of the Markov decision process as: where r t is the reward obtained after taking action a t , Γ succ is a positive constant value, Γ fail is a negative constant value, R delay is the delay reward, R jitter is the jitter reward, R until is the resource utilization reward; Solve the Markov decision process to obtain the optimal time slot allocation result of the flows to be scheduled.

2. The hybrid traffic resource scheduling method applied to a time-sensitive network according to claim 1, wherein The obtaining the characteristic information of the flows to be scheduled in the time-sensitive network and establishing a traffic model based on the characteristic information includes: Among them, N is the total number of flows to be scheduled, id is the flow identifier, cycle is the flow transmission period, D is the flow deadline, jitter is the jitter of the flow, p size is the size of the flow data packet, p num is the number of flow data packets.

3. The hybrid traffic resource scheduling method for a time-sensitive network according to claim 2, wherein The configuring a hybrid traffic scheduling model based on the traffic model includes: In the hybrid traffic scheduling process, configure the time-sensitive flows to adopt the time-aware shaper mechanism and assign the highest priority, and configure the bandwidth flows to adopt the cyclic queue forwarding mechanism and assign the secondary priority.

4. A hybrid traffic resource scheduling method applied to a time-sensitive network according to claim 3, characterized in that The gated total scheduling period is set as the least common multiple of the gated scheduling period of time-sensitive flows and the gated scheduling period of bandwidth flows; The gated scheduling period of time-sensitive flows is the least common multiple of the transmission periods of time-sensitive flows; The gated scheduling period of bandwidth flows is twice the minimum scheduling time slot.

5. The hybrid traffic resource scheduling method for a time-sensitive network according to claim 4, wherein The solving the Markov decision process to obtain the optimal time slot allocation result of the flows to be scheduled includes: Integrate a noisy network into the confrontation network to construct a deep noisy Q-network; Set the action value function of the deep noisy Q-network as: Among them, Q(s t , a t , ξ; μ, σ) is the action-value function of the deep noisy Q-network, V(s t , ξ V ; μ, σ) is the state-value neural network of the noise-embedded network, D(s t , a t , ξ D ; μ, σ) is the advantage neural network of the noise-embedded network, μ and σ are the mean and standard deviation of the neural network, ξ V and ξ D are the random noises of the state-value neural network and the advantage neural network, is to take the mean of the advantage neural network of the noise-embedded network, and A is the action set; Based on the deep noisy Q-network, solve the Markov decision process to obtain the optimal time slot allocation result of the flows to be scheduled.

6. A hybrid traffic resource scheduling device applied to a time-sensitive network, characterized in that A device for implementing the hybrid traffic resource scheduling method for a time-sensitive network according to any one of claims 1 to 5, the device includes: A model construction module, configured to obtain the characteristic information of the flows to be scheduled in the time-sensitive network, and establish a traffic model based on the characteristic information; the flows to be scheduled include time-sensitive flows and bandwidth flows; A model configuration module, configured to configure a hybrid traffic scheduling model based on the traffic model; the hybrid traffic scheduling model jointly performs traffic scheduling based on the time-aware shaper mechanism and the cyclic queue forwarding mechanism; A model transformation module, configured to transform the hybrid traffic scheduling model into a multi-objective optimization problem on the premise of satisfying the deterministic transmission of time-sensitive flows, with the goal of maximizing the scheduling success rate and bandwidth utilization rate of bandwidth flows; A problem modeling module, configured to model the multi-objective optimization problem as a Markov decision process; An optimization scheduling module, configured to solve the Markov decision process to obtain the optimal time slot allocation result of the flows to be scheduled.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded and executed by a processor to perform the hybrid traffic resource scheduling method for a time-sensitive network according to any one of claims 1 to 5.

8. A computer device, characterized in that, Comprising: A processor and a computer-readable storage medium; The processor is adapted to execute a computer program; The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the hybrid traffic resource scheduling method applied to a time-sensitive network according to any one of claims 1 to 5.

Images