Mixed flow resource scheduling method and device applied to time-sensitive network

Abstract

The invention provides a mixed flow resource scheduling method and device applied to a time-sensitive network, and belongs to the technical field of communication network flow scheduling. According to the method, a time-aware shaper mechanism and a circular queuing forwarding mechanism are combined to carry out flow scheduling to meet the deterministic transmission requirement, and a multi-objective function for maximizing the scheduling success rate of bandwidth flow, the bandwidth resource utilization rate and other scheduling quality is established on the premise of meeting time-sensitive flow deterministic transmission. And solving the multi-objective function by adopting a deep noise Q network fused with a noise network to obtain an optimal time slot allocation decision of the to-be-scheduled flow. According to the method, multiple targets can be flexibly integrated and quickly converged to a near-optimal solution in a large-scale search space, and the scheduling quality of the time-sensitive network is obviously improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of communication network traffic scheduling, and in particular relates to a hybrid traffic resource scheduling method and device applied to a time-sensitive network. Background Art

[0002] Low-latency deterministic networking is regarded as the primary requirement and challenge for the next-generation industrial Internet of Things. With the continuous improvement of data transmission rates, more intelligent industrial devices are connected through the network, and the potential demand for communication infrastructure is also increasing continuously. Its communication bandwidth and latency requirements are also constantly rising, and existing network technologies can no longer meet them. As an Ethernet layer 2 network technology, Time-Sensitive Network (TSN) has good scalability and aims to support standard-based real-time deterministic networks to meet the requirements of low latency and low jitter for time-sensitive traffic. TSN integrates different scheduling mechanisms to provide an ultra-low latency real-time deterministic network for time-critical services, such as Time-Aware Shaper (TAS), Cyclic Queuing and Forwarding (CQF), Asynchronous Traffic Shaper (ATS), and Credit-based Shaper (CBS). These shapers can be used alone or in combination. Among them, TAS is a key component in TSN to achieve deterministic low latency through a gating mechanism; CQF mainly solves the problem of bounded delay in transmission; ATS avoids the use of global clock synchronization but can still provide real-time guarantees by reshaping the traffic at each hop to reduce traffic bursts; CBS is a commonly used traffic shaper that schedules traffic based on a credit mechanism. And the scheduling problem of time-sensitive flows is an NP-hard problem and remains a core challenge in TSN.

[0003] Many constraint programming-based methods and heuristic methods have been proposed in the existing literature to solve the TSN scheduling problem for time-sensitive services. However, the number of constraints may increase rapidly with the number of scheduled flows, especially when scheduling multiple different types of flows, which exacerbates this situation and hinders the solver from solving the problem within a practical and acceptable running time. Heuristic methods rely on fixed rules and are difficult to handle dynamically changing or unseen traffic patterns. Summary of the Invention

[0004] The purpose of the present invention is to propose a hybrid traffic resource scheduling method and device applied to a time-sensitive network, which is used to optimize the allocation of time slots to solve the complex problem of hybrid traffic scheduling in TSN.

[0005] To achieve the above object, the present invention is implemented through the following technical solutions:

[0006] In a first aspect, the present invention provides a hybrid traffic resource scheduling method applied to a time-sensitive network, including:

[0007] Obtain the characteristic information of the flows to be scheduled in the time-sensitive network, and establish a traffic model based on the characteristic information; the flows to be scheduled include time-sensitive flows and bandwidth flows;

[0008] Configure a hybrid traffic scheduling model based on the traffic model; the hybrid traffic scheduling model jointly performs traffic scheduling based on the time-aware shaper mechanism and the cyclic queue forwarding mechanism;

[0009] On the premise of satisfying the deterministic transmission of time-sensitive flows, with the goal of maximizing the scheduling success rate and bandwidth utilization of bandwidth flows, transform the hybrid traffic scheduling model into a multi-objective optimization problem;

[0010] Model the multi-objective optimization problem as a Markov decision process;

[0011] Solve the Markov decision process to obtain the optimal time slot allocation result of the flows to be scheduled.

[0012] Preferably, the obtaining the characteristic information of the flows to be scheduled in the time-sensitive network and establishing a traffic model based on the characteristic information includes:

[0013] ,

[0014] wherein, is the set of flows to be scheduled, including time-sensitive flows and bandwidth flows, is an element in , is the flow sequence number, is the total number of flows to be scheduled, is the flow identifier, is the flow transmission period, is the flow deadline, is the jitter of the flow, is the flow packet size, is the number of flow packets.

[0015] Preferably, the configuring a hybrid traffic scheduling model based on the traffic model includes:

[0016] In the hybrid traffic scheduling process, configure the time-sensitive flows to adopt the time-aware shaper mechanism and be given the highest priority, and configure the bandwidth flows to adopt the cyclic queue forwarding mechanism and be given the second highest priority.

[0017] Preferably, on the premise of satisfying the deterministic transmission of time-sensitive flows, aiming at maximizing the scheduling success rate and bandwidth utilization of bandwidth flows, the hybrid traffic scheduling model is transformed into a multi-objective optimization problem, including:

[0018] The multi-objective optimization problem is expressed as:

[0019] ,

[0020] where, 、 、 and represent weights, represents the scheduling success rate of the flow, represents the bandwidth flow the total delay from generation to sending completion, represents the bandwidth flow the deadline of, represents the gated total scheduling period, represents the number of time slots in the gated total scheduling period, represents the minimum scheduling time slot, represents the time slot the total size of the data packets within, represents the time slot the maximum capacity of, represents the time slot the size of the time-sensitive flow data packets within, represents the time-sensitive flow the total delay from generation to sending completion, represents the time-sensitive flow the deadline of, represents the bandwidth flow the jitter of, represents the bandwidth flow the maximum allowable jitter of, represents the flow the jitter of, represents the flow the maximum allowable jitter of, represents the flow the packet loss of, represents the maximum allowable number of packet losses, represents the flow the th data packet of, represents an indicator function, whose value is 1 when the data packet is injected into the time slot and 0 otherwise, represents the number of flow data packets of the flow , represents the time-sensitive flow The size of the flow data packet, represents the link bandwidth, represents the flow transmission period, is the modulo symbol, represents the maximum capacity of each queue in the cyclic queuing and forwarding mechanism, represents network error compensation, represents the time slot allocation decision, represents the flow injection time slot.

[0021] Preferably, the gated total scheduling period is set to the least common multiple of the time-sensitive flow gated scheduling period and the bandwidth flow gated scheduling period;

[0022] The time-sensitive flow gated scheduling period is the least common multiple of the time-sensitive flow transmission periods;

[0023] The bandwidth flow gated scheduling period is twice the minimum scheduling time slot.

[0024] Preferably, modeling the multi-objective optimization problem as a Markov decision process includes:

[0025] Setting the state parameters of the Markov decision process as: , where represents the state at the th interaction between the agent and the environment, represents the characteristic information of the flow to be scheduled at the th interaction between the agent and the environment, represents the remaining capacity of each time slot within a gated total scheduling period at the th interaction between the agent and the environment;

[0026] Setting the action parameters of the Markov decision process as , and the action parameter represents the action selected at the th interaction between the agent and the environment, and the action is the time slot allocation decision of the multi-objective optimization problem;

[0027] Setting the reward function of the Markov decision process as:

[0028] ,

[0029] where, is the reward obtained after taking the action , is a positive constant value, is a negative constant value, is the delay reward, is the jitter reward, is the resource utilization reward.

[0030] Preferably, solving the Markov decision process to obtain the optimal time slot allocation result of the flow to be scheduled includes:

[0031] Integrating a noise network into the confrontation network to construct a deep noise Q network;

[0032] Set the action value function of the deep noise Q network as:

[0033] ,

[0034] where, is the action value function of the deep noise Q network, is the state value neural network integrating the noise network, is the advantage neural network integrating the noise network, and are the mean and standard deviation of the neural network, and are the random noises of the state value neural network and the advantage neural network, is to take the mean of the advantage neural network integrating the noise network, is the action set;

[0035] Based on the deep noise Q network, solve the Markov decision process to obtain the optimal time slot allocation result of the flow to be scheduled.

[0036] In a second aspect, the present invention provides a hybrid traffic resource scheduling device applied to a time-sensitive network, which is used to implement the above-mentioned hybrid traffic resource scheduling method applied to a time-sensitive network. The device includes:

[0037] A model construction module, configured to obtain the feature information of the flow to be scheduled in the time-sensitive network, and establish a traffic model based on the feature information; the flow to be scheduled includes a time-sensitive flow and a bandwidth flow;

[0038] A model configuration module, configured to configure a hybrid traffic scheduling model based on the traffic model; the hybrid traffic scheduling model jointly performs traffic scheduling based on the time-aware shaper mechanism and the cyclic queue forwarding mechanism;

[0039] A model conversion module, configured to convert the hybrid traffic scheduling model into a multi-objective optimization problem with the goal of maximizing the scheduling success rate and bandwidth utilization rate of the bandwidth flow on the premise of satisfying the deterministic transmission of the time-sensitive flow;

[0040] A problem modeling module, configured to model the multi-objective optimization problem as a Markov decision process;

[0041] An optimization scheduling module, which is used to solve the Markov decision process to obtain the optimal time slot allocation result of the flow to be scheduled.

[0042] In a third aspect, the present invention provides a computer-readable storage medium storing a computer program, which is adapted to be loaded and executed by a processor to perform the hybrid traffic resource scheduling method applied to a time-sensitive network as described above.

[0043] In a fourth aspect, the present invention provides a computer device, including: a processor and a computer-readable storage medium;

[0044] A processor adapted to execute a computer program;

[0045] A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the hybrid traffic resource scheduling method applied to a time-sensitive network as described above.

[0046] By adopting the above technical means, the beneficial effects achieved by the present invention are as follows:

[0047] The present invention provides a hybrid traffic resource scheduling method applied to a time-sensitive network. Through the collaborative scheduling of the TAS mechanism and the CQF mechanism, while ensuring the deterministic transmission (low latency, low jitter) of time-sensitive flows, the bandwidth resource allocation is optimized by using the characteristics of the CQF periodic queue, significantly improving the scheduling success rate and bandwidth utilization rate of bandwidth flows (such as video surveillance data), and solving the problem that it is difficult for traditional single mechanisms to balance real-time performance and resource efficiency. By establishing a multi-objective function, indicators such as the scheduling success rate, latency, jitter, and resource utilization rate are jointly optimized, and the conflicts between various objectives are dynamically weighed through deep reinforcement learning. Compared with the single-objective optimization strategy, the comprehensive scheduling quality is significantly improved. By introducing parameterized noise into the time slot resource scheduling algorithm based on the deep noisy Q-network, the exploration ability is enhanced, and it can quickly converge to a near-optimal solution in a super-large time slot search space and adapt to dynamic network topology changes to meet the TSN scheduling requirements. Description of the Drawings

[0048] Figure 1 It is a schematic diagram of the port architecture in a time-sensitive network switch in an embodiment of the present invention;

[0049] Figure 2 It is a schematic flowchart of the hybrid traffic resource scheduling method applied to a time-sensitive network provided by an embodiment of the present invention;

[0050] Figure 3 It is a schematic diagram of the deep noisy Q-network architecture provided by an embodiment of the present invention. Detailed Embodiments

[0051] To make the objectives, technical solutions and advantages of the present invention more clear and understandable, the present invention will be further described in detail below in conjunction with the embodiments and the accompanying drawings. Herein, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but not to limit the present invention.

[0052] Herein, it should also be noted that in order to avoid obscuring the present invention due to unnecessary details, only the structures and / or processing steps closely related to the solution according to the present invention are shown in the drawings, while other details less related to the present invention are omitted.

[0053] It should be emphasized that the term "comprising / including" when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0054] It should be emphasized here that the step marks mentioned hereinafter are not intended to limit the order of the steps. Instead, it should be understood that the steps can be executed in the order mentioned in the embodiments, or different from the order in the embodiments, or several steps can be executed simultaneously.

[0055] The present invention comprehensively considers the service requirements of time-sensitive flows and bandwidth flows in a time-sensitive network. Among them, time-sensitive flows transmit time-sensitive information such as transmission control information and synchronization information. These flows are few in number and short in packet length, but have high requirements for delay and jitter. The transmission process should be as wait-free as possible. Therefore, the time-sensitive flow queue should adopt the TAS mechanism and be given the highest priority. Bandwidth flows are large-bandwidth periodic flows that transmit non-critical information in the network. Their sampling period is large and their requirements for delay and jitter are low, but they are often large in number and long in data. Therefore, the bandwidth flow queue should adopt the CQF mechanism and be given the second highest priority. Best-effort flows are given the lowest priority.

[0056] In order to achieve the mixed scheduling of time-sensitive flows and bandwidth flows, while ensuring the deterministic transmission of time-sensitive flows and improving the scheduling success rate of bandwidth flows, the present invention adopts the TAS mechanism and the CQF mechanism in combination for traffic scheduling. The port architecture in the time-sensitive network switch is as Figure 1 shown. When the input flow passes through the switching fabric, the flow filter forwards the flow to the corresponding queue according to the priority. The queue exit is controlled by the gating list. When multiple gates are opened simultaneously, the priority selector forwards the data packets to the output port in order of queue priority from high to low.

[0057] Based on the above port architecture in the time-sensitive network switch, the first embodiment of the present invention provides a hybrid traffic resource scheduling method applied to a time-sensitive network. See Figure 2 , and the specific steps are as follows:

[0058] Step 1: Obtain the characteristic information of the flows to be scheduled in the time-sensitive network and establish a traffic model.

[0059] In the scenarios of time-sensitive network applications, the transmitted traffic needs to have bounded end-to-end delay, jitter, and packet loss. The deterministic transmission guarantee in the time-sensitive network is to reserve bandwidth resources in advance for the traffic. For time-sensitive flows and bandwidth flows, the traffic information can be obtained in advance, and based on this, the temporal resource allocation strategy for transmission can be determined. Based on this, the traffic models for time-sensitive flows and bandwidth flows can be shown as follows:

[0060] ,

[0061] where, is the set of flows to be scheduled, including time-sensitive flows and bandwidth flows, is the flow sequence number, is the total number of flows to be scheduled, is the flow identifier, is the flow transmission period, is the flow deadline, is the jitter of the flow, is the flow packet size, is the number of flow packets. The best-effort flows in the network have no fixed period, and in this embodiment, the specific parameters of the best-effort flows are not considered, and only their priority is defined as the lowest priority.

[0062] Step 2: Based on the traffic model in Step 1, configure a hybrid traffic scheduling model and design the constraints of the hybrid traffic scheduling model, specifically as follows.

[0063] In this embodiment, the TAS mechanism and the CQF mechanism are jointly used for traffic scheduling to meet the deterministic transmission requirements. Since multiple different types of traffic are involved in scheduling, when using a hybrid traffic scheduler, it is also necessary to constrain the gating, scheduling time slots, flow transmission, and flow service requirements:

[0064] (1) Traffic scheduling gating constraint: Since the time-sensitive flow uses the time-aware shaping mechanism TAS, its gating scheduling period is represented by the least common multiple (LCM) of the time-sensitive flow transmission periods as follows:

[0065] ,

[0066] where, represents the time-sensitive flow, represents the gating scheduling period of the time-sensitive flow, is the least common multiple, is the set of flows to be scheduled in the time-sensitive flow transmission period.

[0067] Since the bandwidth flow adopts the traffic shaping mechanism CQF of cyclic queuing and forwarding, the bandwidth flow gating scheduling period is twice the minimum scheduling time slot, that is .

[0068] The total gating scheduling period is the least common multiple of the time-sensitive flow and the bandwidth flow gating scheduling period, which is expressed as follows:

[0069] ,

[0070] The sequence of gate events in the gating list is defined as an ordered entry of an octet string. When all eight octets of the octet with a length of eight are 1 (i.e., ), the maximum value of the gating scheduling period will be obtained . Therefore, it is necessary to satisfy that cannot be greater than : .

[0071] (2) Minimum scheduling time slot Constraint: According to the definition, a gating scheduling period should be divided into several time slots, and the duration of each time slot is . Therefore, the number of time slots in each total gating scheduling period is . Let be the greatest common divisor (GCD) of the transmission periods of the time-sensitive flow and the bandwidth flow:

[0072] , , ,

[0073] Among them, is the transmission period of the time-sensitive flow in the set of flows to be scheduled , is the transmission period of the bandwidth flow in the set of flows to be scheduled , is the greatest common divisor.

[0074] Since is a multiple of the scheduling time slot , that is: Among them is a positive integer. Therefore, both the total gating scheduling period and the flow transmission period should be divisible by :

[0075] ,

[0076] ,

[0077] Among them, represents the set of flows to be scheduled in the flow transmission period, is the modulo operator.

[0078] To ensure the safe, stable, and low-latency transmission of time-sensitive flows, the time-sensitive flow packets need to be completely transmitted within the same time slot and, according to the gating technology of the time-aware shaper, it is necessary to satisfy:

[0079] ,

[0080] Among them, represents the set of flows to be scheduled in the time-sensitive flows packet size of the flow, is the link bandwidth.

[0081] The bandwidth flow needs to satisfy the ping-pong rule according to the problem of the cyclic queuing and forwarding mechanism it adopts. The packets injected into the switch in the previous time slot need to be completely forwarded in the next time slot, so it should follow:

[0082] ,

[0083] Among them is the link bandwidth, is the maximum capacity of each queue in the cyclic queuing and forwarding mechanism, is the network error compensation, which is used to compensate for errors caused by time synchronization and transmission jitter, etc.

[0084] (3) Flow transmission constraint: Calculate the time resources occupied by each time slot within the current gating total scheduling period, that is:

[0085] ,

[0086] Among them is the time slot total packet size within, is the set of flows to be scheduled in the flow the th packet, is an indicator function. When the packet is injected into the time slot its value is 1, otherwise it is 0, represents the set of flows to be scheduled in the flow packet size of the flow, represents the set of flows to be scheduled The number of flow data packets in the middle stream of the stream.

[0087] In addition, each data packet can be injected into at most one time slot, that is:

[0088] ,

[0089] Because of the store-and-forward mechanism of CQF, all data packets stored in the transmit buffer need to be sent before the start of the next time slot. Therefore, the total size of data packets within each time slot cannot exceed its maximum capacity, that is:

[0090] ,

[0091] where is the maximum capacity of time slot of the stream.

[0092] (4)Flow service demand constraints:

[0093] For time-sensitive flows, because time-sensitive flows have the highest priority and use the TAS queue scheduling, so when and only when the ST queue is empty, the other queues can send. Therefore, to ensure that all flows in the network can be scheduled, it is necessary to ensure that within any unit time slot the size of time-sensitive flow data packets cannot exceed its maximum capacity, that is:

[0094] ,

[0095] where represents the size of time-sensitive flow data packets within time slot of the stream.

[0096] And the total delay from the generation to the completion of sending of time-sensitive flow data packets needs to be less than or equal to the deadline of the flow, that is:

[0097] ,

[0098] where represents the total delay from the generation to the completion of sending of time-sensitive flow of the stream, represents the deadline of time-sensitive flow of the stream, represents the size of flow data packets of time-sensitive flow of the stream.

[0099] For bandwidth flows, since they use CQF scheduling, their delay includes the transmission delays of time-sensitive flows and other bandwidth flows with higher priority than of the stream, the transmission delays of other bandwidth flows that enter the queue first, the switch buffer delay, and the current The transmission delay of the flow data packet. The bandwidth flow data packet must also ensure that the total time delay from generation to completion of transmission of the data packet is less than or equal to the deadline of the flow, that is:

[0100] ,

[0101] wherein, represents the total time delay from generation to completion of transmission of the bandwidth flow , represents the deadline of the bandwidth flow , represents the flow sequence number with a higher priority than the bandwidth flow , represents the flow sequence number with the same priority as the bandwidth flow but entering the queue earlier represents the packet size of other bandwidth flows with a higher priority than the bandwidth flow , represents the packet size of other bandwidth flows with the same priority as the bandwidth flow but entering the queue earlier represents that the data packet is injected into the switch at time slot and sent within time slot .

[0102] In addition to the time delay, the jitter and packet loss of the transmission flow are also important parameters for deterministic transmission, that is:

[0103] ,

[0104] wherein, represents the jitter of the flow , represents the packet loss of the flow , represents the delay of the th packet of the flow , represents the total delay of all packets of the flow , represents the total number of packets of the flow , represents the number of packets of the flow successfully injected into time slot . Only by simultaneously meeting the requirements of time delay, jitter and packet loss can the deterministic transmission of time-sensitive services be guaranteed.

[0105] Step 3: Formulate a multi-objective optimization problem based on the hybrid traffic scheduling model and service requirements.

[0106] In this embodiment, the hybrid scheduling problem of time-sensitive flows and bandwidth flows is formulated as an optimization problem. The goal is to maximize the scheduling success rate of bandwidth flows on the premise of meeting the deterministic transmission of time-sensitive flows.

[0107] In addition, this embodiment also adds the average total delay, jitter, and bandwidth resource utilization to the optimization goal to formulate a multi-objective optimization problem, which is expressed as follows:

[0108] ,

[0109]

[0110] ,

[0111] Among them, represents the jitter of the bandwidth flow , represents the maximum allowable jitter of the bandwidth flow , represents the jitter of the flow , represents the maximum allowable jitter of the flow , represents the maximum allowable number of packet losses, represents the time slot allocation decision, represents the injection time slot of the flow , , , and represent weights, represents the scheduling success rate of the flow, that is, the ratio of the number of successfully scheduled flows to the total number of all flows.

[0112] In this embodiment, the scheduling success rate, delay, jitter, packet loss, and bandwidth resource utilization are jointly optimized to efficiently implement the hybrid traffic scheduling of time-sensitive flows and SR. Therefore, the weight coefficient ( ) is to balance the importance among several objective components. The constraint conditions include scheduling gating constraints, scheduling time slot constraints, flow transmission constraints, and flow service requirements. These constraints ensure that the traffic service can meet the traffic demand and provide deterministic transmission support for the traffic service, where is a very small constant.

[0113] Step 4: Model the multi-objective optimization problem formulated in Step 3 as a Markov decision process and set the parameters of the Markov decision process.

[0114] In this embodiment, the formulated multi-objective optimization problem is to design a dynamic resource scheduling strategy that maximizes the network's scheduling ability while satisfying the constraints. This strategy belongs to the Markov decision process. To solve this scheduling problem and optimize the traffic scheduling performance, the multi-objective optimization problem of resource scheduling is re-modeled as a Markov decision process problem. The parameters of the Markov decision process include state, action, and reward. Among them, the state represents the current traffic information and network state, the action represents the current time slot resource allocation decision, and the reward represents the reward feedback in the scheduling decision.

[0115] State: , where is the state at the th interaction between the agent and the environment, represents the characteristic information of the flow to be scheduled at the th interaction between the agent and the environment, represents the remaining capacity of each time slot within a gated total scheduling period at the th interaction between the agent and the environment.

[0116] Action: Based on the state and policy, the agent can make a time slot allocation decision to determine which time slots to schedule the transmission flow . Each action element is an integer value and indicates the time slot into which the traffic should be injected. Its definition is as follows:

[0117] ,

[0118] where, is the action selected at the th interaction between the agent and the environment, is the action set, is the action in the action set, is the state, is the random noise, and each of its elements is randomly drawn independently from the standard normal distribution . and are parameters that are randomly initialized and learned from experience. is the action value function incorporating random noise.

[0119] Reward: After taking the action , a reward will be obtained to evaluate the quality of this action in the state .

[0120] In this embodiment, while optimizing the scheduling success rate, with the goal of reducing latency, jitter, and resource utilization, the reward function is set as:

[0121] ,

[0122] Among them, is a positive constant value, is the delay reward, is the jitter reward, is the resource utilization reward. When the scheduling fails, by setting as a negative constant value to punish improper decisions. Through the obtained rewards, the parametric policy can be optimized in the training phase. In the Markov decision process, the goal of the agent is to find the optimal time slot allocation policy that can maximize the cumulative discounted reward so as to maximize the action value function . According to the Bellman equation, the action value function can be calculated as:

[0123] ,

[0124] Among them, represents the expectation, represents the discount factor, represents using the policy to control the interaction between the agent and the environment times, represents the -th interaction reward between the agent and the environment, represents using the policy to control the interaction between the agent and the environment times, the value function of the state and the action .

[0125] Step Five: Use the time slot resource scheduling algorithm based on the Deep Noisy Q-Network (DNQN) to solve the Markov decision process in Step Four above to obtain the optimal time slot allocation decision.

[0126] In this embodiment, the time slot resource scheduling algorithm based on DNQN is used to solve the Markov decision process, and the deep noisy Q-network architecture is as Figure 3 shown. In this algorithm, double Q-learning is used to overcome the problem of decision overestimation, and prioritized experience replay is used to accelerate convergence to obtain a higher average return. In the neural network structure, the dueling network is adopted and the noise network is incorporated to improve the neural network structure of the Deep Q-Network (DQN).

[0127] The dueling network consists of two neural networks. One neural network is denoted as which is an approximation of the optimal advantage function, where is the advantage network parameters, and another neural network is denoted as is an approximation of the optimal state-value function, where is the state-value network parameters. Incorporating the noise network means adding noise to the network parameters where and are the mean and standard deviation, which are parameters of the neural network, randomly initialized at the beginning and then learned from experience, and is the random noise, randomly sampled from the standard normal distribution .

[0128] When outputting, the flows of the two networks are combined to produce an approximate optimal action-value function, that is:

[0129] ,

[0130] where is the action-value function of the deep noise Q-network, is the state-value neural network incorporating the noise network, is the advantage neural network incorporating the noise network, and are the random noises of the state-value neural network and the advantage neural network, is to take the mean of the neural network to solve the non-uniqueness problem and prevent the parameters of the neural networks and from changing randomly.

[0131] Through this improvement, the value function and the advantage function can be estimated separately, so that the influence of both the state and the decision on the Q-value estimation can be considered simultaneously, and the correct decision can be identified more quickly during decision evaluation.

[0132] In this embodiment, the specific steps of the slot resource scheduling algorithm based on DNQN are as follows:

[0133] (1) Initialize the main network , the target network and the experience replay pool of size , where represents the noise of the target network, and represent the mean and standard deviation of the neural network of the target network.

[0134] (2) Directly select the action through the main network, explore using the randomness of the noise in DNQN, and the obtained Stored in the experience replay pool, Indicating the state at the th interaction of the main network.

[0135] (3) Using prioritized experience replay, extract a quadruple from the experience replay pool and denote it as .

[0136] (4) Generate using the standard normal distribution, perform forward propagation on the DNQN, and obtain: , where represents the predicted value of the main network at the th interaction, represents the action-value function of the main network, and are the mean and standard deviation of the current neural network of the main network.

[0137] (5) Use the DNQN to select the optimal action: .

[0138] (6) Generate using the standard normal distribution and calculate the value using the target network: , where represents the predicted value of the target network at the th interaction, represents the action-value function of the target network, and represent the state and action at the th interaction, and are the mean and standard deviation of the current neural network of the target network.

[0139] (7) Calculate the Temporal Difference (TD) target and TD error:

[0140] , ,

[0141] where represents the TD target, represents the TD error, is the discount factor.

[0142] (8) Update the parameters of the DNQN using gradient descent: ,

[0143] where and represent the updated parameters of the DNQN, and are the learning rates, and Indicates the gradient.

[0144] (9) Update the parameters of the target network using weighted average:

[0145] ,

[0146] where and are the parameters after the target network is updated, is a hyperparameter that needs to be manually adjusted.

[0147] (10) Return to step (2) and repeat the above steps until the maximum number of training steps is reached.

[0148] Based on the same inventive concept, the second embodiment of the present invention provides a hybrid traffic resource scheduling device applied to a time-sensitive network, which is used to implement the hybrid traffic resource scheduling method applied to the time-sensitive network in the above embodiment. The device includes:

[0149] A model construction module, configured to obtain the feature information of the flow to be scheduled in the time-sensitive network, and establish a traffic model based on the feature information; the flow to be scheduled includes a time-sensitive flow and a bandwidth flow;

[0150] A model configuration module, configured to configure a hybrid traffic scheduling model based on the traffic model; the hybrid traffic scheduling model jointly performs traffic scheduling based on the time-aware shaper mechanism and the cyclic queue forwarding mechanism;

[0151] A model conversion module, configured to convert the hybrid traffic scheduling model into a multi-objective optimization problem with the goal of maximizing the scheduling success rate and bandwidth utilization rate of the bandwidth flow on the premise of satisfying the deterministic transmission of the time-sensitive flow;

[0152] A problem modeling module, configured to model the multi-objective optimization problem as a Markov decision process;

[0153] An optimization scheduling module, configured to solve the Markov decision process to obtain the optimal time slot allocation result of the flow to be scheduled.

[0154] It should be noted that the device embodiment is corresponding to the above method embodiment. The implementation manners of the above method embodiment are all applicable to the device embodiment and can achieve the same or similar technical effects, so they will not be elaborated here.

[0155] Based on the same inventive concept, the third embodiment of the present invention provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded and executed by a processor to perform the hybrid traffic resource scheduling method applied to the time-sensitive network in the above embodiment.

[0156] Based on the same inventive concept, the fourth embodiment of the present invention provides a computer device, including: a processor and a computer-readable storage medium;

[0157] The processor is adapted to execute a computer program;

[0158] The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the hybrid traffic resource scheduling method for time-sensitive networks as described in the above embodiments.

[0159] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0160] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the specified functions in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0161] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured product including an instruction device, and the instruction device implements the specified functions in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0162] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide for implementing the specified functions in one Figure 1 one flow or multiple flows and / or blocks Figure 1Steps of the functions specified in one or more boxes.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: the specific implementation manners of the present invention can still be modified or equivalently replaced, and any modification or equivalent replacement without departing from the spirit and scope of the present invention shall be covered by the protection scope of the claims of the present invention.

Claims

1. A hybrid traffic resource scheduling method applied to a time-sensitive network, characterized in that: include: Acquire characteristic information of a flow to be scheduled in a time-sensitive network, and establish a traffic model based on the characteristic information; the flow to be scheduled includes a time-sensitive flow and a bandwidth flow; A hybrid traffic scheduling model is configured based on the traffic model; the hybrid traffic scheduling model performs traffic scheduling based on a time-aware shaper mechanism and a round-robin queuing forwarding mechanism; Under the premise of satisfying the deterministic transmission of time-sensitive flows, the hybrid traffic scheduling model is transformed into a multi-objective optimization problem with the goal of maximizing the scheduling success rate and bandwidth utilization of bandwidth flows; Modeling the multi-objective optimization problem as a Markov decision process; The Markov decision process is solved to obtain the optimal time slot allocation result of the flow to be scheduled.

2. The hybrid traffic resource scheduling method for a time-sensitive network according to claim 1, characterized in that: The obtaining characteristic information of the flow to be scheduled in the time-sensitive network and establishing a traffic model based on the characteristic information includes: ， in, is the set of flows to be scheduled, including time-sensitive flows and bandwidth flows. for The elements in is the stream number, is the total number of flows to be scheduled, is the stream identifier, is the streaming period, is the flow cutoff time, For the jitter of the flow, is the flow packet size, is the number of flow packets.

3. The hybrid traffic resource scheduling method for a time-sensitive network according to claim 2, characterized in that: The configuring a hybrid traffic scheduling model based on the traffic model includes: In the hybrid traffic scheduling process, the time-sensitive flow is configured to adopt a time-aware shaper mechanism and is given the highest priority, and the bandwidth flow is configured to adopt a round-robin queuing forwarding mechanism and is given a second priority.

4. The hybrid traffic resource scheduling method for a time-sensitive network according to claim 3, characterized in that: Under the premise of satisfying the deterministic transmission of time-sensitive flows, the hybrid traffic scheduling model is transformed into a multi-objective optimization problem with the goal of maximizing the scheduling success rate and bandwidth utilization of bandwidth flows, including: The multi-objective optimization problem is expressed as: ， in, , , and represents the weight, Indicates the scheduling success rate of the flow, Indicates bandwidth flow The total delay from generation to completion of transmission, Indicates bandwidth flow The deadline, represents the total gating scheduling period, represents the number of time slots in the total scheduling cycle of the gate, represents the minimum scheduling time slot, Indicates time slot The total size of the data packet, Indicates time slot The maximum capacity of Indicates time slot The size of the packets within the time-sensitive stream, Representing time-sensitive flows The total delay from generation to completion of transmission, Representing time-sensitive flows The deadline, Indicates bandwidth flow The jitter, Indicates bandwidth flow The maximum allowed jitter, Representation Flow The jitter, Representation Flow The maximum allowed jitter, Representation Flow Packet loss, Indicates the maximum number of packet losses allowed. Representation Flow No. Data packets, Represents an indicator function. When the data packet Injection Slot Its value is 1 when , otherwise it is 0. Representation Flow The number of flow packets, Representing time-sensitive flows The flow packet size, Indicates the link bandwidth, Representation Flow The transmission cycle, is the remainder symbol, Indicates the maximum capacity of each queue in the round-robin queuing forwarding mechanism, represents network error compensation, represents the time slot allocation decision, Representation Flow injection time slot.

5. The hybrid traffic resource scheduling method for a time-sensitive network according to claim 4, characterized in that: The total gating scheduling period is set to the least common multiple of the time-sensitive flow gating scheduling period and the bandwidth flow gating scheduling period; The time-sensitive flow gating scheduling period is the least common multiple of the time-sensitive flow transmission period; The bandwidth flow gating scheduling period is twice the minimum scheduling time slot.

6. The hybrid traffic resource scheduling method for a time-sensitive network according to claim 4, characterized in that: Modeling the multi-objective optimization problem as a Markov decision process includes: Set the state parameters of the Markov decision process to: ,in Represents the relationship between the agent and the environment The state of the interaction, Represents the relationship between the agent and the environment The characteristic information of the flow to be scheduled during the interaction. Represents the relationship between the agent and the environment The remaining capacity of each time slot in a gated total scheduling cycle during the interaction; Set the action parameter of the Markov decision process to , the action parameters Represents the relationship between the agent and the environment an action selected during the second interaction, wherein the action is a time slot allocation decision for the multi-objective optimization problem; The reward function of the Markov decision process is set to: ， in, To take action After the reward, is a positive constant value, is a negative constant value, For delayed rewards, For the jitter reward, Reward for resource utilization.

7. The hybrid traffic resource scheduling method for a time-sensitive network according to claim 6, characterized in that: Solving the Markov decision process to obtain an optimal time slot allocation result for the flow to be scheduled includes: Integrate the noise network into the duel network to build a deep noise Q network; The action value function of the deep noise Q network is set to: ， in, is the action-value function of the deep noisy Q-network, is a state-value neural network integrated with a noise network, is a dominant neural network integrated with a noisy network, and are the mean and standard deviation of the neural network, and is the random noise of the state-value neural network and the advantage neural network, is the mean of the dominant neural network integrated into the noise network, is a collection of actions; The Markov decision process is solved based on the deep noise Q network to obtain an optimal time slot allocation result for the flow to be scheduled.

8. A hybrid traffic resource scheduling device applied to a time-sensitive network, characterized in that: The device is used to implement the hybrid traffic resource scheduling method applied to a time-sensitive network as described in any one of claims 1 to 7, and comprises: A model building module, used to obtain characteristic information of a flow to be scheduled in a time-sensitive network, and to establish a traffic model based on the characteristic information; the flow to be scheduled includes a time-sensitive flow and a bandwidth flow; A model configuration module, configured to configure a hybrid traffic scheduling model based on the traffic model; the hybrid traffic scheduling model performs traffic scheduling based on a time-aware shaper mechanism and a round-robin queuing forwarding mechanism; A model conversion module is used to convert the hybrid traffic scheduling model into a multi-objective optimization problem with the goal of maximizing the scheduling success rate and bandwidth utilization of bandwidth flows under the premise of satisfying deterministic transmission of time-sensitive flows; A problem modeling module, used for modeling the multi-objective optimization problem as a Markov decision process; The optimization scheduling module is used to solve the Markov decision process to obtain the optimal time slot allocation result of the flow to be scheduled.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the hybrid traffic resource scheduling method applied to a time-sensitive network as described in any one of claims 1 to 7.

10. A computer device, characterized in that: include: a processor and a computer readable storage medium; a processor adapted to execute a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the hybrid traffic resource scheduling method applied to a time-sensitive network as described in any one of claims 1 to 7 is implemented.

Images