A training method for traffic flow scheduling model in delay-sensitive networks

Abstract

The present invention relates to a training method for a service flow scheduling model based on a delay-sensitive network, belonging to the field of delay-sensitive network technology, and solving the problem of low service flow scheduling efficiency in the prior art. The method includes the following steps: reserving time slot resources for burst flows in each time slot; constructing a state space and action space of reinforcement learning based on a multi-queue cyclic forwarding mechanism according to the time slot information after reserving the time slot resources; determining the optimization target and optimization constraints for service flow scheduling; the optimization target is to maximize the number of service flow scheduling and maximize resource load balancing; constructing a deep reinforcement learning network model, and training the deep reinforcement learning network model based on real-time service flow information of the scheduling period, the optimization target, and the optimization constraints to obtain a service flow scheduling model for the delay-sensitive network. This improves the network's scheduling efficiency and scheduling capability for heterogeneous service traffic.

Description

Technical Field

[0001] The present invention relates to the technical field of delay-sensitive networks, and in particular to a training method for a service flow scheduling model of a delay-sensitive network. Background Art

[0002] With the increasing number of applications with real-time requirements, the end-to-end latency and reliability requirements for real-time business flow transmission are becoming increasingly stringent. Therefore, solving the problem of scheduling heterogeneous business flows in delay-sensitive networks (Temporal Segment Networks, TSN) while ensuring the needs of real-time application business flows is an urgent problem that needs to be solved.

[0003] Existing business flow resource allocation methods are complex and inefficient. Summary of the Invention

[0004] In view of the above analysis, an embodiment of the present invention aims to provide a training method for a service flow scheduling model in a delay-sensitive network, so as to solve the problem of low efficiency in existing service flow scheduling.

[0005] In one aspect, an embodiment of the present invention provides a method for training a service flow scheduling model for a delay-sensitive network, comprising the following steps:

[0006] Time slot resources are reserved for burst flows in each time slot. The state space and action space of reinforcement learning are constructed based on the multi-queue cyclic forwarding mechanism according to the time slot information after the time slot resources are reserved.

[0007] Determine the optimization goal and optimization constraints of business flow scheduling; the optimization goal is to maximize the number of business flow scheduling and maximize resource load balancing;

[0008] A deep reinforcement learning network model is constructed, and the deep reinforcement learning network model is trained based on the real-time business flow information of the scheduling period, the optimization goal, and the optimization constraints to obtain a business flow scheduling model for the delay-sensitive network.

[0009] Based on the further improvement of the above method, the deep reinforcement learning network model includes a prediction network and a target network; the prediction network and the target network have the same structure and both adopt a residual neural network structure.

[0010] Based on a further improvement of the above method, the residual neural network includes a plurality of residual blocks connected in sequence;

[0011] The residual blocks include dimension-upgraded residual blocks and basic residual blocks, and the dimension-upgraded residual blocks and the basic residual blocks are alternately arranged.

[0012] Based on the further improvement of the above method, the optimization constraints include: burst flow receiving time slot length constraint, time slot capacity constraint, and end-to-end bounded low delay, jitter and zero packet loss constraints for each real-time business flow transmitted in the network.

[0013] Based on the further improvement of the above method, the time slot capacity constraint is expressed as:

[0014]

[0015] in, Indicates the number of real-time business flows within the scheduling period. Indicates the total number of data packets sent by the i-th real-time service flow in a scheduling cycle, size i Indicates the data packet size of the i-th real-time service flow, Represents the indicator function. If the mth data packet of the i-th service flow is successfully scheduled in the ξth time slot, Otherwise C ξ Indicates the maximum capacity of the ξth time slot excluding the reserved time slot resources.

[0016] Based on the further improvement of the above method, each real-time service flow is subject to the end-to-end bounded low latency, jitter and zero packet loss constraints in the network transmission, which can be expressed as:

[0017]

[0018] in, It represents the queue offset value allocated to the i-th real-time service flow on the n-th switch, T represents the time slot length, and N represents the number of switches. represents the end-to-end delay requirement of the i-th real-time service flow, It represents the delay required for the jth data packet of the i-th real-time service flow to be sent from the source to the destination. represents the jitter requirement of the i-th real-time service flow, Indicates the total number of data packets sent by the i-th real-time service flow in a scheduling cycle, represents the indicator function, and λ represents the number of time slots in the scheduling period.

[0019] Based on the further improvement of the above method, the state space and action space of reinforcement learning are constructed based on the multi-queue cyclic forwarding mechanism according to the time slot information after reserving the time slot resources, including:

[0020] The state space includes the current network state information after reserving time slot resources, real-time service flow information, and the offset value of each queue in each time slot; the offset value of each queue of each switch in each time slot is determined based on a multi-queue cyclic forwarding mechanism;

[0021] The action space includes time slots and offset values ​​that can be allocated to real-time service flows.

[0022] Based on a further improvement of the above method, the offset value of each queue of each switch in each time slot is determined based on multi-queue round-robin forwarding, including:

[0023] Determine a sending queue of each switch in each time slot according to the time slot sequence number and set an offset value of the sending queue;

[0024] The other queues are determined as receiving queues, and the offset values ​​of the other queues are calculated based on the distances from the other queues to the sending queue and the offset value of the sending queue.

[0025] Based on the further improvement of the above method, the following formula is used to calculate the offset value of other queues:

[0026]

[0027] Among them, ψ base Indicates the offset value of the sending queue, ψ j represents the offset value of the jth queue, K represents the total number of queues, and ξ represents the time slot number.

[0028] Based on the further improvement of the above method, the deep reinforcement learning model is trained in the following way:

[0029] S31. For each round of training, the first real-time service flow is used as the current real-time service flow;

[0030] S32. Construct a current state and input it into the prediction network, predict the Q values ​​corresponding to the current real-time service flow when it is allocated at different time slots and offset values, and determine a current action based on the prediction results; the current state includes the offset value of each queue of each switch in each time slot, current network state information, and flow information of the current real-time service flow; the current action includes the time slot and offset value corresponding to the current real-time service flow;

[0031] S33. Determine whether the optimization constraint condition is satisfied based on the time slot and offset value corresponding to the current real-time service flow. If not, scheduling of the current real-time service flow fails. If satisfied, scheduling of the current real-time service flow succeeds. Generate experience samples based on the current state and current action and store them in an experience sample library. The experience sample library is used to update parameters of the prediction network and the target network.

[0032] S34. If the current real-time service flow scheduling fails, the time slot resource utilization is updated to the initial resource utilization, the current round of training ends, and step S35 is executed; if the current real-time service flow scheduling succeeds, the next real-time service flow is used as the current real-time service flow, and step S32 is returned; if there is no next real-time service flow, step S35 is executed;

[0033] S35. If the number of training rounds reaches the preset number, the training is terminated to obtain a service flow scheduling model for the delay-sensitive network. Otherwise, the next round of training is performed.

[0034] Compared with the existing technology, the present invention ensures the reliability of burst flow transmission by reserving time slot resources for burst flow in each time slot, constructs the state space and action space of reinforcement learning based on the multi-queue cyclic forwarding mechanism, and determines the optimization target and optimization constraint conditions of service flow scheduling. Then, the deep reinforcement model is trained based on the real-time service flow information of the scheduling period, so as to quickly obtain the service flow scheduling model with the maximum number of scheduled service flows and load balancing. Then, based on the service flow scheduling model, the time slot resource allocation plan for each real-time service flow can be efficiently obtained while ensuring the reliability of the burst flow, thereby improving the network's scheduling capability for heterogeneous service traffic.

[0035] In the present invention, the above-mentioned technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of the present invention will be described in the following description, and some advantages will become apparent from the description or be learned through practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. Throughout the drawings, the same reference symbols denote the same components.

[0037] Figure 1 This is a flowchart of a method for training a service flow scheduling model for a delay-sensitive network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0039] A specific embodiment of the present invention discloses a training method for a service flow scheduling model of a delay-sensitive network, such as Figure 1 As shown, the following steps are included:

[0040] S1. Reserve time slot resources for burst flow in each time slot; construct the state space and action space of reinforcement learning based on the multi-queue cyclic forwarding mechanism according to the time slot information after reserving time slot resources;

[0041] S2. Determine the optimization goal and optimization constraints for business flow scheduling; the optimization goal is to maximize the number of business flow scheduling and maximize resource load balancing;

[0042] S3. Build a deep reinforcement learning network model, and train the deep reinforcement learning network model based on the real-time business flow information of the scheduling space, the optimization goal, and the optimization constraints to obtain a business flow scheduling model for the delay-sensitive network.

[0043] During implementation, the Time-Sensitive Network (TSN) utilizes a fully centralized network control architecture. Its control plane consists of a global controller, which houses the centralized user configuration (CUC), the centralized network configuration (CNC), and a database. The CUC provides a web interface for user interaction and collects information about registered terminals, TSN switches, and real-time application traffic. The CNC centrally controls the data plane's traffic flow scheduling, distributes configuration information, and performs the necessary calculations for planned traffic flows. The database connects the CUC and CNC, recording and storing process data. The data plane consists of terminals and TSN switches.

[0044] During implementation, each switch node is equipped with eight queues for sending and receiving traffic flows. The first K queues are used for cyclic forwarding of burst flows and real-time application traffic flows, while the remaining queues are used to store best-effort flows. This invention imposes no restrictions on the number of switch nodes in the network. 3 ≤ K ≤ 8.

[0045] Compared with the prior art, the training method for a service flow scheduling model for a delay-sensitive network provided in this embodiment ensures the reliability of burst flow transmission by reserving time slot resources for burst flows in each time slot. It constructs a state space and action space for reinforcement learning based on a multi-queue cyclic forwarding mechanism, determines the optimization goal and optimization constraints for service flow scheduling, and then trains a deep reinforcement model based on real-time service flow information in the scheduling period, thereby quickly obtaining a service flow scheduling model with the maximum number of scheduled service flows and load balancing. Based on the service flow scheduling model, a time slot resource allocation plan for each real-time service flow can be efficiently obtained while ensuring the reliability of the burst flow, thereby improving the network's scheduling capability for heterogeneous service traffic.

[0046] During implementation, time slot resources may be reserved for burst flows in advance in each time slot of a scheduling period according to the distribution status of historical burst flows, so as to accommodate the transmission of burst flows in the time slot.

[0047] The state space of reinforcement learning includes the current network state information after reserving time slot resources, real-time service flow information, and the offset value of each queue in each time slot.

[0048] The network status information includes the network topology, link rate and resource utilization of each time slot.

[0049] During implementation, in order to avoid packet loss dead zones within the receiving time slot and ensure that service flows from different queues do not conflict with each other on the same sending interface, a multi-queue round-robin forwarding mechanism is used to determine the offset value of each queue in each time slot.

[0050] Specifically, in step S1, determining the offset value of each queue of each switch in each time slot based on multi-queue cyclic forwarding includes:

[0051] S11. Determine a sending queue of each switch in each time slot according to the time slot sequence number and set an offset value of the sending queue;

[0052] S12: Determine the other queues as receiving queues, and calculate offset values ​​of the other queues based on the distances from the other queues to the sending queue and the offset value of the sending queue.

[0053] During implementation, the transmit queue for each switch is first determined within each time slot. For example, for the ξth time slot, the ξ%Kth queue on the switch is used as the transmit queue for burst and real-time traffic, while the remaining queues are used as receive queues for burst and real-time traffic. This ensures that within any time slot, only one queue on a switch node is used to transmit traffic, while all other queues are used to receive traffic. This prevents traffic from different queues from conflicting on the same transmit interface. ξ = 0, ... λ - 1, where λ represents the number of time slots in the scheduling cycle.

[0054] At the same time, the queues on the switch nodes are circulated as sending queues and receiving queues. The queue type is not specified and changes with time slots, thereby avoiding dead zones that affect the reliability of business flows.

[0055] When implementing, set the offset value of the send queue ψ base =0.

[0056] Use the following formula to calculate the offset value of other queues:

[0057]

[0058] Among them, ψ base Indicates the offset value of the sending queue, ψ j represents the offset value of the jth queue, K represents the total number of queues, and ξ represents the time slot number.

[0059] During implementation, the offset value of the sending queue is used as a benchmark, and the offset values ​​of other queues are calculated using formula (1), that is, the offset value of the receiving queue is calculated. The offset value of other queues is the distance from the queue to the sending queue. The offset value is cyclically increased by 1, and the offset value of each queue in different time slots is different. By calculating the offset value of the queue, the corresponding relationship between the queue and the service flow is established by assigning a queue offset value to each service flow. For bursty flows, as long as the assigned offset value is greater than 1, the dead zone introduced by the traditional two-queue cyclic forwarding mechanism in each time slot can be eliminated, and the legal time slot reception range can be expanded. For each real-time application service flow, it is easier to be managed by the central controller, and it is only necessary to ensure that the assigned offset value is greater than or equal to 0.

[0060] Specifically, the flow information of each real-time service flow includes the ID number, flow period, frame size, source address, destination address, end-to-end delay requirement, jitter requirement, and reliability requirement. It should be noted that for a fully centralized network control architecture, the real-time service flow information within each scheduling cycle is known. The action space includes the time slots and offset values ​​that can be allocated to real-time service flows, and the offset values ​​correspond to queues. The offset value of each queue in each time slot is different, that is, the distance between the queue and the sending queue is different. It can be considered that the waiting time for the queue to become the sending queue is different. The real-time service flow is associated with the queue through the offset value, making the scheduling more flexible.

[0061] For real-time business flows within the scheduling period, the current network status information, the offset value of each queue of each switch in each time slot, and the flow information of the real-time business flow are input into the business flow scheduling model as the state of reinforcement learning to obtain the resource allocation result of the real-time business flow, that is, the time slot and offset value allocated to the real-time business flow.

[0062] It should be noted that the jump path of the real-time service flow on the switch is fixed. After allocating a real-time service flow, the current time slot resource utilization of each queue needs to be updated.

[0063] Then, the optimization goal and optimization constraint conditions of the service flow scheduling are determined. Specifically, the optimization goal of the service flow scheduling of the present invention is to maximize the number of service flow scheduling and maximize resource load balancing.

[0064] Specifically, the goal of resource allocation in the present invention is to maximize the number of schedulable service flows and the load balancing degree of time slot resources. The optimization goal can be expressed as:

[0065]

[0066] in, represents the benefit of allocating the i-th real-time service flow to the ξ-th time slot, represents the number of real-time service flows in the scheduling period, λ represents the number of time slots in the scheduling period, Φ ξ (f i ) represents the indicator function of whether the i-th real-time service flow is successfully scheduled in the ξ-th time slot, which is 1 if the scheduling is successful, otherwise it is 0.

[0067] Among them, the income Among them, α and β are weights, A(f i ,λ) represents the scheduling quantity benefit of the i-th real-time service flow, Represents the load balancing function.

[0068]

[0069] Among them, k ξ represents the resource utilization rate of the ξth time slot, represents the average resource occupancy of all time slots, Indicates the total number of data packets sent by the i-th real-time service flow in a scheduling cycle, Represents the indicator function. If the mth data packet of the i-th real-time service flow is successfully scheduled in the ξth time slot, Otherwise

[0070] The rewards for deep reinforcement learning are related to the optimization goal. Based on the above analysis, the rewards include the number of service flow scheduling and load balancing. Specifically, the reward value is calculated using the following formula:

[0071]

[0072] Among them, γ a and γ b represents the weight coefficient, Indicates the difference in the number of business flow scheduling in two consecutive steps, Indicates that the load balancing capability of two consecutive service flow scheduling steps is poor.

[0073] To speed up the convergence of the algorithm, the reward for the number of service flow scheduling is the difference between the number of service flow scheduling in the previous and next steps. The following formula is used to calculate the difference between the number of service flow scheduling in two consecutive steps:

[0074]

[0075] Among them, χ t represents the number of real-time business flows scheduled in step t, Indicates the number of real-time business flows within the scheduling period, X t-1 It represents the number of real-time service flows scheduled in step t-1, and μ represents the weight coefficient.

[0076] To speed up the convergence of the algorithm, the load balancing reward is the difference between the load balancing capabilities of two consecutive business flow scheduling steps. The following formula is used to calculate the difference between the load balancing capabilities of two consecutive business flow scheduling steps:

[0077]

[0078] in, represents the maximum time slot resource utilization rate in step t, Ξ represents the ideal time slot resource utilization rate, represents the maximum time slot resource utilization in step t-1, and θ represents the weight coefficient.

[0079] During implementation, the resource utilization of each time slot is obtained based on the ratio of the total size of the business flow allocated to each time slot to the capacity of the time slot, and the maximum resource utilization is taken as

[0080] Specifically, the optimization constraints include: burst flow receiving time slot length constraint, time slot capacity constraint, and end-to-end bounded low delay, jitter and zero packet loss constraints for each real-time service flow transmitted in the network.

[0081] Specifically, the burst flow receiving time slot length constraint is: the product of the burst flow queue offset value and the time slot length is greater than or equal to the sum of the time slot length, the burst flow transmission delay on the switch interface, the burst flow transmission delay on the link and the clock deviation.

[0082] It should be noted that the queue offset value of the burst flow is predetermined. The transmission delay of the burst flow on the switch interface and the transmission delay of the burst flow on the link can be measured.

[0083] The time slot capacity constraint requires that the total number of data packets transmitted in each time slot does not exceed the maximum capacity of the time slot.

[0084] Specifically, the time slot capacity constraint is expressed as:

[0085]

[0086] in, Indicates the number of real-time business flows within the scheduling period. Indicates the total number of data packets sent by the i-th real-time service flow in a scheduling cycle, size i Indicates the data packet size of the i-th real-time service flow, Represents the indicator function. If the mth data packet of the i-th service flow is successfully scheduled in the ξth time slot, Otherwise C ξ Indicates the maximum capacity of the ξth time slot excluding the reserved time slot resources.

[0087] The end-to-end bounded low latency, jitter, and zero packet loss constraints for each real-time service flow transmitted in the network are expressed as:

[0088]

[0089] in, It represents the queue offset value allocated to the i-th real-time service flow on the n-th switch, T represents the time slot length, and N represents the number of switches. represents the end-to-end delay requirement of the i-th real-time service flow, It represents the delay required for the jth data packet of the i-th real-time service flow to be sent from the source to the destination. represents the jitter requirement of the i-th real-time service flow, Indicates the total number of data packets sent by the i-th real-time service flow in a scheduling cycle, represents the indicator function, and λ represents the number of time slots in the scheduling period.

[0090] A deep reinforcement learning network model is constructed, and the deep reinforcement learning network model is trained based on the real-time business flow information of the scheduling period, the optimization goal, and the optimization constraints to obtain a business flow scheduling model for the delay-sensitive network.

[0091] Specifically, the deep reinforcement learning model includes a prediction network and a target network. The prediction network and the target network have the same structure and both adopt a residual neural network structure.

[0092] When implemented, the prediction network and the target network have the same structure, but different parameters.

[0093] In order to enhance the ability to extract network status features and speed up the reasoning of the mapping relationship between real-time business flows and different time slots, the prediction network and the target network adopt a residual neural network structure.

[0094] The residual neural network includes multiple residual blocks connected in sequence. The residual blocks include dimension-upgraded residual blocks and basic residual blocks. The dimension-upgraded residual blocks and basic residual blocks are arranged alternately. During implementation, the first residual block of the residual neural network is the dimension-upgraded residual block.

[0095] During implementation, the first two layers based on the dimensionality-raising residual block used a 3*3 structured convolution kernel with padding of 1. The sliding step of the convolution kernel of the first layer network was 2, the sliding step of the convolution kernel of the second layer network was 1, and the third layer network used a 1*1 convolution kernel with default padding of 0 and a sliding step of 2.

[0096] The basic residual block uses a two-layer network with a convolution kernel size of 3*3, a padding of 1, and a sliding step of 1.

[0097] The residual neural network structure is adopted to effectively avoid the gradient explosion and gradient disappearance problems during the training process and accelerate the convergence of the algorithm.

[0098] Specifically, the deep reinforcement learning model is trained in the following way:

[0099] S31. For each round of training, the first real-time service flow is used as the current real-time service flow;

[0100] S32. Construct a current state and input it into the prediction network, predict the Q values ​​corresponding to the current real-time service flow when it is allocated at different time slots and offset values, and determine a current action based on the prediction results; the current state includes the offset value of each queue of each switch in each time slot, current network state information, and flow information of the current real-time service flow; the current action includes the time slot and offset value corresponding to the current real-time service flow;

[0101] S33. Determine whether the optimization constraint condition is satisfied based on the time slot and offset value corresponding to the current real-time service flow. If not, scheduling of the current real-time service flow fails. If satisfied, scheduling of the current real-time service flow succeeds. Generate experience samples based on the current state and current action and store them in an experience sample library. The experience sample library is used to update parameters of the prediction network and the target network.

[0102] S34. If the current real-time service flow scheduling fails, the time slot resource utilization is updated to the initial resource utilization, the current round of training ends, and step S35 is executed; if the current real-time service flow scheduling succeeds, the next real-time service flow is used as the current real-time service flow, and step S32 is returned; if there is no next real-time service flow, step S35 is executed;

[0103] S35. If the number of training rounds reaches the preset number, the training is terminated to obtain a service flow scheduling model for the delay-sensitive network. Otherwise, the next round of training is performed.

[0104] For example, in the first round of training, for the t-th step training, the offset value of each queue of each switch in each time slot, the current network state information and the flow information of the t-th real-time business flow are used as the state of the t-th step. The input prediction network predicts the Q value when the t-th real-time business flow is assigned to different time slots and queues. The ε-greedy strategy is used to select the action of the t-th step, that is, the time slot and queue assigned to the t-th real-time business flow.

[0105] Determine whether the predicted network allocation result can successfully schedule the t-th real-time business flow, that is, determine whether the optimization constraint conditions are met. If not, the scheduling of the t-th real-time business flow fails. If satisfied, the scheduling of the t-th real-time business flow succeeds.

[0106] If the tth real-time service flow is successfully scheduled, the remaining information of the t+1th real-time service flow, the offset values ​​of each queue on each switch within each time slot, and the current network state information are used as the current state for step t+1, i.e., the state of training for step t+1. This information is then input into the prediction network for training for step t+1. It should be noted that the current time slot resource utilization of each queue at this point is the time slot resource utilization of each queue after the tth real-time service flow is assigned. If all real-time service flows have been assigned, the current training round ends and the next round of training begins.

[0107] If the scheduling of the t-th real-time service flow fails, the time slot resource utilization of each time slot is updated to the initial resource utilization, the current round of training is ended, and the next round of training is carried out.

[0108] It should be noted that before the next round of training, the allocation results of the previous round are cleared and the allocation is restarted from the first real-time service flow. The initial resource utilization is the time slot resource utilization of each queue after the time slot resources are reserved for the burst flow in each time slot, but before the real-time service flow is allocated.

[0109] When the training rounds reach the preset number, the training is stopped to obtain the trained deep reinforcement learning model, that is, the business flow scheduling model. The state is input into the prediction network of the business flow scheduling model to obtain the time slots and queues allocated to the real-time business flow.

[0110] In order to enable the deep reinforcement learning model to allocate reasonable resources to real-time business flows and enable real-time business flows to be successfully scheduled, experience samples are generated during the training process and stored in the experience sample library to facilitate updating the prediction network and target network and parameters based on the experience sample library.

[0111] Specifically, based on the current state and current action, experience samples are generated and stored in the experience sample library, including:

[0112] Calculate the reward value based on the time slot and offset value corresponding to the current real-time business flow;

[0113] The current state, current action, reward value, next state and scheduling result of the current real-time business flow are stored as experience samples in an experience sample library.

[0114] The empirical sample is expressed as (s t ,a t ,r t ,,s t+1 ,d t ), s t Indicates the state of step t, a t represents the action of step t, r t represents the reward of step t, d tIndicates whether the real-time business flow of step t is successfully scheduled. If it is successfully scheduled, it is 1; otherwise, it is 0. t+1 Indicates the state at step t+1.

[0115] Update the parameters of the prediction network and target network based on the experience sample library, including:

[0116] After generating experience samples based on the state and current action and storing them in the experience sample library, randomly extract an experience sample from the experience sample, input the current state of the extracted experience sample into the experience network to obtain the predicted Q value, and input the next state into the target network to obtain the target Q value;

[0117] Calculate the loss based on the predicted Q value, target Q value, reward in the experience sample and whether it is scheduled, and update the parameters of the prediction network based on the calculated loss;

[0118] If the current number of training steps meets the target network update conditions, the parameters of the target network are updated to the parameters of the prediction network.

[0119] Specifically, the loss is calculated using the following formula:

[0120] L=E[(y i -Q fc ) 2 ] (10)

[0121]

[0122] Among them, Q fc Indicates the predicted Q value, Q T represents the target Q value, r i represents the reward in the i-th experience sample, d i Indicates whether the service flow in the i-th experience sample is successfully scheduled, γ represents a parameter, E[·] represents the expectation, and L represents the loss. During implementation, the target network update condition can be, for example, updating the target network every 100 training steps.

[0123] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0124] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A training method for a service flow scheduling model in a delay-sensitive network, characterized in that: The following steps are involved: Time slot resources are reserved for burst flows in each time slot. The state space and action space of reinforcement learning are constructed based on the multi-queue cyclic forwarding mechanism according to the time slot information after the time slot resources are reserved. Determine the optimization goal and optimization constraints of business flow scheduling; the optimization goal is to maximize the number of business flow scheduling and maximize resource load balancing; Constructing a deep reinforcement learning network model, and training the deep reinforcement learning network model based on real-time traffic flow information of the scheduling period, the optimization goal, and optimization constraints to obtain a traffic flow scheduling model for a delay-sensitive network; The deep reinforcement learning network model includes a prediction network and a target network; the prediction network and the target network have the same structure, both using a residual neural network structure; The residual neural network includes a plurality of residual blocks connected in sequence; The residual blocks include dimension-upgraded residual blocks and basic residual blocks, and the dimension-upgraded residual blocks and the basic residual blocks are alternately arranged; The state space and action space of reinforcement learning are constructed based on the multi-queue cyclic forwarding mechanism according to the time slot information after reserving time slot resources, including: The state space includes the current network state information after reserving time slot resources, real-time service flow information, and the offset value of each queue in each time slot; the offset value of each queue of each switch in each time slot is determined based on a multi-queue cyclic forwarding mechanism; The action space includes time slots and offset values ​​that can be allocated to real-time service flows; The deep reinforcement learning model is trained in the following way: S31. For each round of training, the first real-time service flow is used as the current real-time service flow; S32. Construct a current state and input it into the prediction network, predict the Q values ​​corresponding to the current real-time service flow when it is allocated at different time slots and offset values, and determine a current action based on the prediction results; the current state includes the offset value of each queue of each switch in each time slot, current network state information, and flow information of the current real-time service flow; the current action includes the time slot and offset value corresponding to the current real-time service flow; S33. Determine whether the optimization constraint condition is satisfied based on the time slot and offset value corresponding to the current real-time service flow. If not, scheduling of the current real-time service flow fails. If satisfied, scheduling of the current real-time service flow succeeds. Generate experience samples based on the current state and current action and store them in an experience sample library. The experience sample library is used to update parameters of the prediction network and the target network. S34. If the current real-time service flow scheduling fails, the time slot resource utilization is updated to the initial resource utilization, the current round of training ends, and step S35 is executed; if the current real-time service flow scheduling succeeds, the next real-time service flow is used as the current real-time service flow, and step S32 is returned; if there is no next real-time service flow, step S35 is executed; S35. If the number of training rounds reaches the preset number, the training is terminated to obtain a service flow scheduling model for the delay-sensitive network. Otherwise, the next round of training is performed. Generate experience samples based on the current state and current action and store them in the experience sample library, including: Calculate the reward value based on the time slot and offset value corresponding to the current real-time business flow; The current state, current action, reward value, next state and scheduling result of the current real-time business flow are stored as experience samples in the experience sample library; Update the parameters of the prediction network and target network based on the experience sample library, including: After generating experience samples based on the state and current action and storing them in the experience sample library, randomly extract an experience sample from the experience sample, input the current state of the extracted experience sample into the experience network to obtain the predicted Q value, and input the next state into the target network to obtain the target Q value; Calculate the loss based on the predicted Q value, target Q value, reward in the experience sample and whether it is scheduled, and update the parameters of the prediction network based on the calculated loss; If the current number of training steps meets the target network update conditions, the parameters of the target network are updated to the parameters of the prediction network; The optimization constraints include: burst flow receiving time slot length constraint, time slot capacity constraint, and end-to-end bounded low delay, jitter and zero packet loss constraints for each real-time service flow transmitted in the network; The burst flow receiving time slot length constraint is: the product of the queue offset value of the burst flow and the time slot length is greater than or equal to the sum of the time slot length, the transmission delay of the burst flow on the switch interface, the transmission delay of the burst flow on the link and the clock deviation.

2. The training method for a service flow scheduling model for a delay-sensitive network according to claim 1, characterized in that: The time slot capacity constraint is expressed as: in, Indicates the number of real-time business flows within the scheduling period. Indicates the total number of data packets sent by the i-th real-time service flow in a scheduling cycle, size i Indicates the data packet size of the i-th real-time service flow, Represents the indicator function. If the mth data packet of the i-th service flow is successfully scheduled in the ξth time slot, Otherwise C ξ Indicates the maximum capacity of the ξth time slot excluding the reserved time slot resources.

3. The training method for a service flow scheduling model for a delay-sensitive network according to claim 1, characterized in that: The end-to-end bounded low latency, jitter, and zero packet loss constraints for each real-time service flow transmitted in the network are expressed as: in, It represents the queue offset value allocated to the i-th real-time service flow on the n-th switch, T represents the time slot length, and N represents the number of switches. represents the end-to-end delay requirement of the i-th real-time service flow, It represents the delay required for the jth data packet of the i-th real-time service flow to be sent from the source to the destination. represents the jitter requirement of the i-th real-time service flow, Indicates the total number of data packets sent by the i-th real-time service flow in a scheduling cycle, represents the indicator function, and λ represents the number of time slots in the scheduling period.

4. The method for training a service flow scheduling model for a delay-sensitive network according to claim 1, wherein: The offset value of each queue of each switch in each time slot is determined based on multi-queue round-robin forwarding, including: Determine a sending queue of each switch in each time slot according to the time slot sequence number and set an offset value of the sending queue; The other queues are determined as receiving queues, and the offset values ​​of the other queues are calculated based on the distances from the other queues to the sending queue and the offset value of the sending queue.

5. The method for training a service flow scheduling model for a delay-sensitive network according to claim 4, wherein: Use the following formula to calculate the offset value of other queues: Among them, ψ base Indicates the offset value of the sending queue, ψ j represents the offset value of the jth queue, K represents the total number of queues, and ξ represents the time slot number.

Images