Time-sensitive network traffic scheduling method and electronic equipment

Abstract

The invention discloses a time-sensitive network flow scheduling method and electronic equipment, and the method comprises the steps: obtaining time-sensitive flow information and network topology information, and constructing a state matrix with a compression coefficient of theta; designing a reward function, and training the deep reinforcement learning model by using an action shielding method, assisted by a class A * backtracking method and combined with time-sensitive network traffic scheduling constraint and routing and adopting a competition depth Q network algorithm; and deploying the trained model to centralized user configuration, and scheduling and solving the input time-sensitive flow. According to the method, the convergence speed is higher, the scheduling quality is improved, the solving time is shortened, and the scheduling requirement of the time-sensitive flow in a complex scene is met. The method can be widely applied to the time-sensitive network technology in the industrial Internet of Things.

Description

Technical Field

[0001] The present invention relates to a time-sensitive network technology in an industrial Internet of Things, and in particular to a time-sensitive network traffic scheduling method and electronic equipment. Background Art

[0002] With the rapid development of applications such as industrial Internet, autonomous driving, telemedicine, and high-definition video transmission, unprecedented high requirements have been placed on the real-time, reliability, and precise time synchronization capabilities of network transmission. Although traditional Ethernet technology is flexible and widely used, it has obvious deficiencies in processing time-sensitive data. Therefore, time-sensitive network technology (TSN) came into being, aiming to provide deterministic, low-latency, and highly reliable network communication capabilities for critical applications through a series of standardized enhancements. Time-sensitive networking is a network technology based on the IEEE802.1 series of standards. It integrates multiple mechanisms such as time synchronization, traffic shaping, scheduling, and priority management to ensure that data packets can be accurately transmitted to the destination according to the predetermined schedule and path.

[0003] Traffic scheduling in time-sensitive networks is a key technology to ensure the quality of network data transmission, ensuring the real-time, deterministic and reliable network data transmission. Traffic scheduling refers to the process of orderly transmitting data packets in the network according to business needs and network resource conditions. In TSN, traffic scheduling controls the sending time and order of data packets to meet the requirements of time-sensitive applications for low latency, low jitter and high reliability. Therefore, the performance of the traffic scheduling strategy greatly affects the actual effect of TSN. Therefore, in time-sensitive networks, how to effectively schedule traffic is an important and challenging issue.

[0004] Among the existing scheduling algorithms, there are two main types of methods: using solvers to solve and using heuristic algorithms to solve. Among them, there are roughly three methods for using solvers, namely: integer linear programming (ILP), satisfiability modulo theory (SMT), and optimization modulo theory (OMT). There are also algorithms such as taboo search, genetic algorithm, ant colony optimization algorithm, simulated annealing algorithm, etc. for using heuristic algorithms to solve.

[0005] The principle of using solvers for traffic scheduling is mainly based on modeling and solving traffic scheduling problems. Solvers optimize routing and scheduling solutions by building mathematical models to meet the latency and jitter requirements of traffic. Solvers can also explore effective scheduling solutions and solve constraint problems by defining flow variables and constraints. However, the disadvantage of the solver algorithm is that it formulates the joint scheduling and routing problem as an NP-hard problem with high computational complexity. As the scale of the problem increases, the time spent on solving it will increase exponentially, or even become unsolvable.

[0006] The principle of using heuristic algorithms for traffic scheduling is to gradually approach the optimal solution through iteration. In each iteration, the algorithm selects a strategy to update the state of the solution based on the current state and the constraints of the problem. During the iteration process, the algorithm continuously evaluates the quality of the current solution and adjusts the search strategy based on the evaluation results. If the current solution meets the stopping criteria, the algorithm terminates and outputs the current solution as the final solution. The disadvantage of heuristic algorithms is that they reduce the solution space, making some flows that could have been scheduled unschedulable. Heuristic algorithms also have local optimal traps and cannot reach the global optimal. In addition, the convergence speed of some heuristic algorithms is relatively slow. Summary of the invention

[0007] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the purpose of the present invention is to provide a time-sensitive network traffic scheduling method, electronic device and medium based on deep reinforcement learning. The present invention has a faster convergence speed, improves the scheduling quality, and reduces the solution time to meet the scheduling requirements of time-sensitive flows in complex scenarios.

[0008] The first technical solution adopted by the present invention is:

[0009] A time-sensitive network traffic scheduling method comprises the following steps:

[0010] Obtain time-sensitive flow information and network topology information within the system and construct a state matrix with a compression factor of θ;

[0011] The reward function is designed based on the scheduled delay and heuristic function. The action masking method is supplemented by the A*-like backtracking method, combined with the time-sensitive network traffic scheduling constraints and routing, and the competitive deep Q-network algorithm is used to train the deep reinforcement learning model.

[0012] The trained model is deployed to a centralized user configuration to schedule and solve the input time-sensitive streams.

[0013] Furthermore, the time-sensitive flow information is defined as a five-tuple <Src i , Dst i , Size i , Li , T i >, where Src i 、Dst i Size i , L i 、T i They represent the source node, destination node, data size sent per cycle, maximum end-to-end delay, and cycle of flow i respectively;

[0014] The network topology is modeled as a directed graph G =<V,E> , where V and E represent the set of switches and terminals and the set of physical links respectively, and each physical link E i The weight is set as the inverse of the link speed.

[0015] Furthermore, the expression of the state matrix is:

[0016] s=[Load|Steps]

[0017] In the formula, Load is a value of size N. l *(H / θ) matrix, N l is the number of links in the network topology, H is the super period; Steps is a size of N l *N f The matrix, N f is the number of flows; the compression factor θ is used to control the size of the state matrix.

[0018] Furthermore, in the action masking method, when the action selected by the deep reinforcement learning model is an inactive action, the action is masked so that it can no longer be selected, and a new action is selected until the selected action is an active action or all actions are inactive.

[0019] Furthermore, in the A*-like backtracking method, the reward function is designed to be a function related to the scheduled delay and the heuristic function. When there are no feasible actions in the environment, the result with the largest reward function is extracted from all stored results, and the state matrix of the extracted result is used as the current state vector, and the action is selected according to the current state vector.

[0020] Furthermore, the reward function is expressed as:

[0021]

[0022] Where g is the time spent on the current scheduling step flow transmission, h is the estimated time required to complete the scheduling, and H is the super period.

[0023] Furthermore, the expression for estimating the time h required to complete the scheduling is:

[0024]

[0025] In the formula, f i For the flow that has not reached the end point, SL i is the shortest path length from the current node to the end point of flow i, f j ,f k There are two flows with the same node a in the historical scheduling.

[0026] Furthermore, the training steps of the deep reinforcement learning model include:

[0027] A1: The interactive environment collects time-sensitive flow information and network topology, and constructs the initialization state matrix based on the compression coefficient θ;

[0028] A2: Input the state matrix into a deep reinforcement learning framework network using two competitive deep Q networks of the same structure, and eliminate invalid actions through action masking;

[0029] A3: Determine whether there are still feasible actions: if yes, go to step A4; if no, go to step A5;

[0030] A4: Select the action with the largest Q value and execute it, obtain the new state through the interactive environment, and store the new state and reward value;

[0031] A5: Extract the state matrix with the highest reward value from the historical scheduling of this training as the new state; go to step A2;

[0032] A6: Determine whether all flows have been scheduled or all solution spaces have been explored: if so, end this training; if not, go to step A2.

[0033] Furthermore, the action vector a is:

[0034] a=(i,j)

[0035] Among them, i is the sequence number of the selected scheduling flow, and j is the sequence number of the selected scheduling link.

[0036] Furthermore, in the joint time-sensitive network traffic scheduling constraints and routing, when the action determines the scheduled flow and the scheduled link, the system will determine the earliest schedulable time that meets the scheduling constraints as the start time of scheduling based on the frame constraints, link constraints, flow transmission constraints, delay constraints, and frame isolation constraints.

[0037] Furthermore, in the competitive deep Q network algorithm, the calculation of the action value Q is decomposed into two parts: the state value and the advantage function through a deep Q network with a branching structure. The input data is first represented by a public network, and the data output from the public network enters the state value function network to output the state value on the one hand, and enters the advantage function network to output the advantage value corresponding to each action on the other hand.

[0038] The second technical solution adopted by the present invention is:

[0039] An electronic device comprises a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a time-sensitive network traffic scheduling method as described above.

[0040] The third technical solution adopted by the present invention is:

[0041] A computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, wherein the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a time-sensitive network traffic scheduling method as described above.

[0042] The fourth technical solution adopted by the present invention is:

[0043] A computer program product or a computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned time-sensitive network traffic scheduling method.

[0044] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0045] (1) The present invention is aimed at the design of the state vector matrix of deep reinforcement learning based on the characteristics of time-sensitive network transmission problems, and introduces a hyperparameter θ to control the size of the state vector matrix. In similar scheduling methods in the existing field of time-sensitive networks, complex network topology will cause the state vector matrix to be too large, slowing down the model training speed, and even making it impossible to perform model training because the required video memory is too large, exceeding the video memory of the server computing graphics card. By increasing the hyperparameter θ, the size of the state vector matrix can be compressed and the training speed can be accelerated. However, too large θ will make the environmental state information abstract, which is not conducive to model learning. Therefore, choosing a suitable θ can balance the model training speed and model learning effect.

[0046] (2) The present invention targets deep reinforcement learning of the characteristics of time-sensitive network transmission problems, and selects DuelingDQN as the network for deep reinforcement learning. In the traffic scheduling scenario, different actions have very different effects on the state, and some actions may even make the next state have no possible actions. Compared with the existing time-sensitive network traffic scheduling algorithms of the same type, Dueling DQN takes into account the differences in the effects of different actions on the value function under the same state, thereby improving the convergence speed.

[0047] (3) When the model is training, if the model selects an infeasible action, the action will be shielded so that it cannot be selected again, and then a new action will be selected until the selected action is feasible. Compared with the existing time-sensitive network similar traffic scheduling algorithms, action shielding can prevent the model from performing too many invalid actions and too many negative rewards, thereby improving learning efficiency.

[0048] (4) When the present invention encounters a state vector with no action vector during model training, it will select the state vector with the largest reward function result from the stored results as the new state vector. Compared with the existing time-sensitive network similar traffic scheduling algorithms, this selection method avoids directly proceeding to the next training without performing any operation, and also avoids randomly selecting a new state, which can speed up the training process and improve training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 This is a schematic diagram of a single training process of a model based on a competitive deep Q network of the present invention;

[0051] Figure 2 It is a flow chart of a time-sensitive network traffic scheduling method based on deep reinforcement learning of the present invention;

[0052] Figure 3 It is a schematic diagram of the algorithm based on competitive deep Q network of the present invention;

[0053] Figure 4 Schematic diagram of the competitive deep Q network structure in the present invention;

[0054] Figure 5 is a schematic diagram of the state matrix of the present invention;

[0055] Figure 6 It is a schematic diagram of the centralized management mechanism of the present invention. DETAILED DESCRIPTION

[0056] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0057] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0058] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0059] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0060] Example 1

[0061] like Figure 1 FIG. 1 is a schematic diagram of a single training process of a competitive deep Q network model in this embodiment. The training of the model includes the following steps:

[0062] Step 101: The interactive environment collects time-sensitive flow information and network topology, and constructs an initialization state matrix in combination with the compression coefficient θ.

[0063] Step 102: Input the state matrix into a deep reinforcement learning framework network using two competitive deep Q networks with the same structure, and eliminate invalid actions through action masking.

[0064] Step 103, determine whether there are any feasible actions: if so, go to step 104; if not, go to step 105.

[0065] Step 104: Select the action with the largest Q value and execute it, obtain a new state through the interactive environment, and store the new state and reward value.

[0066] Step 105: Extract the state matrix with the highest reward value from the historical scheduling of this training as the new state. Go to step 102.

[0067] Step 106: Determine whether all flows have been scheduled or all solution spaces have been explored: if yes, end this training; if not, go to step 102.

[0068] like Figure 2 As shown, this embodiment provides a time-sensitive network traffic scheduling method based on deep reinforcement learning, including the following steps:

[0069] Step 201, first set the parameters of the competitive deep Q network algorithm, the parameters include batch size, experience replay pool cache size, minimum number of replays, maximum number of training times, learning rate, target network update frequency.

[0070] Step 202: Initialize the training environment, including resetting the state matrix and reward value at the beginning of each training.

[0071] Step 203, obtain the current state. At the beginning of training, the current state is the initialization state. The subsequent state is the state obtained after the previous state performs an action and performs a state transition.

[0072] Step 204: Determine whether the number of caches in the experience replay pool is greater than the set minimum number of replays. If not, set the exploration rate to 0.4; if so, set the exploration rate according to the following formula:

[0073]

[0074] Among them, epoch is the number of training times.

[0075] Step 205: According to the current state, action masking is performed on the action space, and it is determined whether there are still feasible action vectors. If not, the state with the largest reward value stored in the historical schedule is taken out, and the process goes to step 204; if it exists, a random number ranging from 0 to 1 is generated, and it is determined whether the random number is greater than the exploration rate. If not, an action vector is randomly selected from the feasible action vectors; if so, an action vector with the largest q value is selected from the feasible action vectors. This can prevent the model from falling into a local optimum during training.

[0076] Step 206: Execute the selected action. The system obtains a new state vector according to the executed action and calculates the reward value according to the reward function.

[0077] Step 207: Store the current state vector matrix and the corresponding reward value in the history schedule. Put the current state vector, the selected action vector, the reward value, and the new state vector into the experience replay pool.

[0078] Step 208: Determine whether the number of the experience replay pool is greater than the minimum number of replays. If so, train the deep reinforcement learning network.

[0079] like Figure 3 The figure shows a schematic diagram of the competitive deep Q network algorithm in this embodiment. In this algorithm, the agent is composed of two competitive deep Q networks of the same structure. This embodiment uses the experience replay mechanism to store historical experience in the training process and stabilize the learning process. When training is required, batch_size training experience samples are extracted from the experience replay pool as a small batch to train the estimation network and the target network, where the target network adopts a delayed soft update method. After a certain number of iterations, the parameters of the target network will be updated from the estimation network. Batch_size is a set hyperparameter, which is the size of a single training batch.

[0080] like Figure 4 The figure shows the structure diagram of the competitive deep Q network, which includes an input layer, a hidden layer, a fully connected layer, a state output layer, and an action value output layer. In a time-sensitive network, different actions in the same state have very different effects on the value function. In some states, the action value function Q is highly correlated with the selected action, while in some states, the action value function Q is not closely related to the selected action. Therefore, the competitive deep Q network decomposes the calculation of the action value Q into two parts: the state value and the advantage function. This is achieved through a deep Q network with a branching structure. The input data is first represented by a public network; the data output from the public network, on the one hand, enters the state value function network to output the state value; on the other hand, enters the advantage function network to output the advantage value corresponding to each action.

[0081] Step 209: Determine whether all flows have completed the scheduling from the starting point to the end point. If not, go to step 203; if yes, determine whether the number of training times is an integer multiple of the target network update frequency. If yes, use the parameters of the estimated network to update the parameters of the target network; if not, determine whether the maximum number of training times has been reached. If not, go to step 202; if yes, end the program.

[0082] In this embodiment, the basic parameters of the algorithm can be appropriately adjusted according to the scale of the topology, computing resources, and solution time requirements to obtain the most ideal solution.

[0083] like Figure 5 As shown, this embodiment encodes the state matrix of deep reinforcement learning according to the characteristics of time-sensitive networks. State matrix s = [Load|Steps]. Each row represents each link in the time-sensitive network topology. For the Load matrix, each column represents a time period in a super cycle, which depends on the setting of parameter θ. For example, when parameter θ is set to 5, Where H is the super period, then t0 represents time period 0 to 4, t1 represents time period 5 to 9, and so on. When an action vector (f, l) is selected, the system determines the start time st and end time ed of the schedule based on the time-sensitive network traffic scheduling constraints. For the Steps matrix, each column represents each flow. The state transition equation is:

[0084]

[0085] α j =θ*j

[0086] β j =θ*(j+1)-1

[0087]

[0088] s` l,n+f+1 =step

[0089] Among them, step is the scheduling step number of this scheduling.

[0090] like Figure 6 As shown, this embodiment adopts a centralized management framework. Centralized user configuration is one of the key components in the time-sensitive network architecture. It is responsible for providing time-sensitive flow registration services to key applications, processing user needs for the network, collecting flow period, length, sending and receiving nodes, maximum delay and other information, and registering new devices and exchanging information with the centralized network configuration through the user network interface. The time-sensitive flow information is collected, consisting of a five-tuple <Src i , Dst i , Size i , L i , T i >, which represent the starting point of the time-sensitive flow, the end point of the time-sensitive flow, the amount of data sent per cycle, the maximum end-to-end delay, and the cycle respectively.

[0091] Centralized network configuration is responsible for implementing device monitoring management, network topology discovery, traffic monitoring and tuning, and calculating and configuring gate lists in the same time-sensitive network domain. When dynamic scheduling is performed in a large-scale network model, its core scheduling solution calculation method is required to have low time complexity to meet strict real-time requirements. Collect network information v ab Represents the link propagation rate.

[0092] Example 2

[0093] An embodiment of the present invention further provides an electronic device, the electronic device comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the following Figure 2 A time-sensitive network traffic scheduling method based on deep reinforcement learning is shown.

[0094] It is understood that the memory may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data created according to the use of the server, etc.

[0095] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor can integrate one or a combination of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes the operating system and application programs; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but implemented separately through a chip.

[0096] Since the electronic device is an electronic device corresponding to a time-sensitive network traffic scheduling method based on deep reinforcement learning in an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0097] Example 3

[0098] The embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the following Figure 2 A time-sensitive network traffic scheduling method based on deep reinforcement learning is shown.

[0099] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0100] Since the storage medium is a storage medium corresponding to a time-sensitive network traffic scheduling method based on deep reinforcement learning in an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0101] Example 4

[0102] In some possible implementations, various aspects of the method of the embodiment of the present invention may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a computer device, the program code is used to cause the computer device to execute the steps of a time-sensitive network traffic scheduling method based on deep reinforcement learning according to various exemplary embodiments of the present application described above in this specification. Among them, the executable computer program code or "code" for executing each embodiment can be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, structured query language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0103] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0104] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0105] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made based on the essence of the content of the present invention should be included in the protection scope of the present invention.

Claims

1. A time-sensitive network traffic scheduling method, characterized in that: The following steps are involved: Obtain time-sensitive flow information and network topology information, and construct a state matrix with a compression factor of θ; Design reward functions, use action masking methods, supplemented by A*-like backtracking methods, combine time-sensitive network traffic scheduling constraints and routing, and use competitive deep Q-network algorithms to train deep reinforcement learning models; The trained model is deployed to a centralized user configuration to schedule and solve the input time-sensitive streams.

2. A time-sensitive network traffic scheduling method according to claim 1, characterized in that: The time-sensitive flow information is defined as a five-tuple <Src i , Dst i , Size i , L i , T i >, where Src i 、Dst i Size i、 L i , T i They represent the source node, destination node, data size sent per cycle, maximum end-to-end delay, and cycle of flow i respectively; The network topology is modeled as a directed graph G =<V,E> , where V and E represent the set of switches and terminals and the set of physical links respectively, and each physical link E i The weight is set as the inverse of the link speed.

3. A time-sensitive network traffic scheduling method according to claim 1, characterized in that: The expression of the state matrix is: s=[Load|Steps] In the formula, Load is a value of size N. l *(H / θ) matrix, N l is the number of links in the network topology, H is the super period; Steps is a size of N l *N f The matrix, N f is the number of flows; the compression factor θ is used to control the size of the state matrix.

4. A time-sensitive network traffic scheduling method according to claim 1, characterized in that: In the action masking method, when the action selected by the deep reinforcement learning model is an inactive action, the action is masked so that it cannot be selected again, and a new action is selected until the selected action is an active action or all actions are inactive.

5. A time-sensitive network traffic scheduling method according to claim 1, characterized in that: In the A*-like backtracking method, the reward function is designed to be a function related to the scheduled delay and the heuristic function. When there are no feasible actions in the environment, the result with the largest reward function is extracted from all stored results, and the state matrix of the extracted result is used as the current state vector, and the action is selected according to the current state vector.

6. A time-sensitive network traffic scheduling method according to claim 1, characterized in that: The expression of the reward function is: Where g is the time spent on the current scheduling step flow transmission, h is the estimated time required to complete the scheduling, and H is the super period.

7. A time-sensitive network traffic scheduling method according to claim 1, characterized in that: The training steps of the deep reinforcement learning model include: A1: Collect time-sensitive flow information and network topology, and build the initialization state matrix based on the compression coefficient θ; A2: Input the state matrix into a deep reinforcement learning framework network using two competitive deep Q networks of the same structure, and eliminate invalid actions through action masking; A3: Determine whether there are still feasible actions: if yes, go to step A4; if no, go to step A5; A4: Select the action with the largest Q value and execute it, obtain the new state through the interactive environment, and store the new state and reward value; A5: Extract the state matrix with the highest reward value from the historical schedule of this training as the new state; go to step A2； A6: Determine whether all flows have been scheduled or all solution spaces have been explored: if so, end this training; if not, go to step A2.

8. A time-sensitive network traffic scheduling method according to claim 1, characterized in that: In the joint time-sensitive network traffic scheduling constraints and routing, when the action determines the scheduled flow and the scheduled link, the system will determine the earliest schedulable time that meets the scheduling constraints as the start time of scheduling based on the frame constraints, link constraints, flow transmission constraints, delay constraints, and frame isolation constraints.

9. A time-sensitive network traffic scheduling method according to claim 1, characterized in that: In the competitive deep Q network algorithm, the calculation of the action value Q is decomposed into two parts: the state value and the advantage function through a deep Q network with a branching structure. The input data is first represented by a public network, and the data output from the public network enters the state value function network to output the state value on the one hand, and enters the advantage function network to output the advantage value corresponding to each action on the other hand.

10. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method described in any one of claims 1 to 9.

Images