Multi-layer message optimization scheduling method applied to TSN networks

By constructing a CQF (Category QF) packet injection selection model in the TSN network, and combining deep reinforcement learning and incremental scheduling, the packet injection timing is optimized, solving the problems of long solution time and random injection selection in the TSN network, and realizing efficient multi-level network packet transmission.

CN123123154APending Publication Date: 2026-11-17MINZU UNIVERSITY OF CHINA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611309258.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2025-10-24
Filing Date
2026-08-27
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

The existing CQF scheduling mechanism in TSN networks has a long solution time and random packet injection selection, resulting in low scheduling efficiency and failing to effectively optimize the transmission of multi-layer network data.

Method used

A multi-layer message optimization scheduling method is adopted, which combines deep reinforcement learning and incremental scheduling. By constructing a CQF-packet injection selection model, the actor-commentator algorithm is used to optimize the timing of packet injection, and the network state is represented by a time-queue matrix to achieve continuous action optimization.

Benefits of technology

It improves the packet scheduling efficiency under the CQF mechanism of TSN network, reduces the computational resource overhead, realizes fine-grained scheduling of continuous injection time and optimization of cross-layer network data, and enhances the flexibility of network resource allocation and packet transmission.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123154A_ABST
    Figure CN123123154A_ABST
Patent Text Reader

Abstract

This invention discloses a multi-layer message optimization scheduling method for TSN networks, belonging to the CQF scheduling technology field of TSN networks. The method incorporates the message segmentation and mapping relationship between network layer messages and data link layer packets into the scheduling process. It constructs a CQF-packet injection selection model: using incremental scheduling combined with deep reinforcement learning, representing the network transmission state with a time-queue matrix, treating the ratio of packet injection time to its injection period as a continuous action, and using an actor-commentator algorithm to train and obtain the injection time scheme for each packet. This method eliminates the need for resolving complex constraints when the network state changes, enabling incremental scheduling of packets and messages. It achieves fine-grained scheduling with continuous injection time and cross-layer scheduling for multi-layer network data, significantly improving the efficiency of scheduling strategy acquisition and reducing computational resource overhead.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to a multi-layer message optimization scheduling method for TSN networks, belonging to the field of CQF scheduling technology for TSN networks. Background Technology

[0002] Time-Sensitive Networking (TSN) is a series of standard protocols that enhance the data link layer on top of standard Ethernet and virtual bridged LANs, and it is also a special type of local area network. Among the various transmission mechanisms in TSN networks, Cyclic Queuing and Forwarding (CQF) ensures the transmission of time-triggered (TT) messages by allocating different injection times to data packets and cyclically controlling the opening and closing of queues. Allocating injection times to different data packets to ensure that more grouped data packets or packets containing a specific message reach the target node, while simultaneously reducing queue resource occupancy, is the core issue of current CQF scheduling.

[0003] In the TSN network standard framework conforming to the IEEE 802.1Q standard, the data transmission of the CQF mechanism is similar to "creeping," sending one hop forward in each cycle. The latency range of a packet passing through a network switch is deterministic and independent of the network topology. Current CQF scheduling methods mostly employ constraint solving and reinforcement learning. Constraint solving methods combine network topology and traffic configuration, converting constraints into mathematical logical expressions of Satisfiability Modulo Theories (SMT) for planning and solving. The results often require a long-term, "intensive" solving process by the corresponding solver. Reinforcement learning algorithms define learning variables such as states, actions, and feedback to train an optimized packet injection scheme for CQF transmission in the TSN network. This scheduling technique uses uncertain gradients and discrete action spaces, resulting in a relatively random training process and poor response when there are too many packet injection options. Summary of the Invention

[0004] To address the drawbacks of long solution time and random packet injection selection in the CQF scheduling mechanism of TSN networks, this invention proposes a multi-layer message optimization scheduling method under the CQF scheduling mechanism of TSN networks. This method improves upon the CQF mechanism conforming to the IEEE 802.1Q protocol, and is termed the CQF-Packet Injection Selection Model. It transforms the complex constraints between packets and messages into a time slot range within which packets can be successfully scheduled. Its core features include: incorporating the message segmentation and mapping relationship between network layer messages and data link layer packets into the scheduling process; employing incremental scheduling, i.e., scheduling and injecting one packet before scheduling the next; and utilizing deep reinforcement learning, specifically the actor-commentator algorithm, to learn the network environment and the characteristics of the messages to be scheduled. A time-queue matrix represents the network transmission state, and the ratio of packet injection time to injection period is used as a continuous action, thereby quickly and efficiently obtaining the injection time scheme for each packet. This method is independent of the TSN network topology and can perform incremental scheduling when the network state changes without resolving complex constraints. The method of this invention can guarantee the feasibility of the solution, while reducing the randomness in machine learning and improving the solution efficiency of the CQF mechanism scheduling scheme.

[0005] Furthermore, to effectively model the state information, constraints, and optimization objectives in the improved CQF scheduling process, a time-constraint-aware tuple is proposed. This tuple consists of three elements: the packet injection time and transmission timing state, the constraints existing during transmission (such as queue resources, end-to-end latency, etc.), and the task-related optimization objectives (such as maximizing schedulable size, minimizing latency, load balancing, etc.). On the other hand, in the TD3 framework based on deep reinforcement learning (DRL), this tuple corresponds to the environment's state, the executable scheduling actions, and the resulting reward feedback, constituting the basic unit for the interaction between machine learning and the CQF mechanism. The CQF-packet injection selection model constructed through this structure can achieve adaptive optimization of long-term performance in a dynamic and constrained CQF scheduling environment, and can complete the scheduling of TT messages and the feedback on whether the transmission was successful through this time-constraint tuple.

[0006] The technical solution adopted in this invention is a multi-layer message optimization scheduling method applied to TSN networks, which includes the following steps:

[0007] Step 1: Initialize the TSN network, collect data packets transmitted in the network and initialize their attributes, initialize network environment parameters and the state of the CQF (Cellular QF) packet injection selection model, and sort the data packets to be scheduled.

[0008] Step 2, packet injection selection: Input the packet information into the packet injection selection model network to obtain the injection scheme for each packet;

[0009] Step 3: Data packet injection time allocation. The ratio of the injection time of the current data packet to be scheduled to its injection period is used as a continuous action. The injection time of the data packet is obtained by combining deep reinforcement learning.

[0010] Step 4, Environment Change and Feedback: Based on the data packet transmission path and action, construct a time-queue matrix, verify the legality of the action, and provide feedback on the benefits.

[0011] Step 5: Schedule the data packets under the CQF mechanism and deliver them to the target terminal; calculate the scheduling results of the data packets and messages to obtain the final scheduling scheme.

[0012] The method incorporates the message segmentation and mapping relationship between network layer messages and data link layer data packets into the scheduling process, adopts incremental scheduling combined with deep reinforcement learning, and uses a time-queue matrix to represent the network transmission status.

[0013] The method for constructing a CQF (Category QF) packet injection selection model in this invention includes: setting up six neural networks: an actual action network, a first actual evaluation network, a second actual evaluation network, a target action network, a first target evaluation network, and a second target evaluation network; wherein,

[0014] The actual action network is used to output basic actions based on the current state, the actual evaluation network is used to evaluate the benefits of the current state and actions, and the target action network and target evaluation network are used to calculate the target Q value during training.

[0015] An experience pool is set up to store the experience generated during the reinforcement learning phase. The experience is represented in the form of a quadruple, including the original state, action, cumulative reward, and post-action state.

[0016] Furthermore, the CQF-packet injection selection model adopts the actor-commentator algorithm in deep reinforcement learning to build a two-layer network architecture: the actor network is responsible for obtaining the injection time slot of the data packet, the commentator network is responsible for evaluating the transmission success rate of the injected data packet, and an experience pool is set up to store training experience.

[0017] The present invention constructs a CQF (Category QF) packet injection selection model, which includes:

[0018] Step A: Build a two-layer network architecture;

[0019] Step B is constructed, which involves allocating the time for packet injection. Step B1 involves obtaining the basic actions; Step B2 involves obtaining the noisy injection time; and Step B3 involves obtaining the noiseless injection time.

[0020] Step C is constructed, which involves environmental changes and feedback. Step C1 involves obtaining the queue usage status of the current data packet; Step C2 involves calculating the new state of the current data packet after executing the current action; Step C3 involves providing feedback on the action execution benefits; and Step C4 involves determining the scheduling result for this round.

[0021] Step D is constructed, which involves experience correction and storage. Specifically, step D1 sets the final reward; step D2 iteratively determines the cumulative reward for each step; and step D3 stores the experience.

[0022] Step E is constructed to reinforce learning and updating; wherein, step E1 is to select the experience group to be learned; and step E2 is to calculate the expected Q value.

[0023] Step E3: Calculate the actual Q value; Step E4: Calculate the loss and update the actual evaluation network; Step E5: Determine whether to perform a delayed update.

[0024] The technical advantages of the present invention compared to existing technologies are as follows:

[0025] (1) The method of the present invention can improve the efficiency of obtaining packet scheduling strategy under the CQF mechanism of TSN network and reduce the computational resource overhead. Compared with traditional constraint solving algorithms, it does not need to re-solve complex constraints when the network state changes, and can perform incremental scheduling of packets / messages.

[0026] (2) The method of this invention realizes fine-grained scheduling of continuous injection time under the CQF mechanism of TSN network. Traditional reinforcement learning algorithms usually treat injection time as discrete time intervals, while the method of this invention treats injection time as continuous moments. This change enables the scheduling strategy to be continuously optimized according to the network state, thereby improving the flexibility and fine-grainedness of network resource allocation and packet transmission scheduling.

[0027] (3) The method of the present invention can realize cross-layer scheduling for multi-layer network data, and combine the message segmentation and data packet mapping relationship between network layer messages and data link layer data packets, thereby optimizing the technical problem that the traditional CQF scheduling algorithm mainly faces single-layer data. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the application of the method of the present invention to multi-layer message optimization scheduling in TSN networks.

[0029] Figure 2 This is a flowchart of the CQF (Category QF) packet injection selection model constructed according to the present invention.

[0030] Figure 3 This is a diagram illustrating the segmentation of data packets for excessively large messages in a TSN network.

[0031] Figure 4 This is a schematic diagram of the TSN network topology and data packet transmission path as exemplified by the present invention.

[0032] Figure 5 This is a schematic diagram of ping-pong queue transmission using the CQF mechanism in a TSN network.

[0033] Figure 6 This is a classification diagram of the data packet injection selection method of the present invention.

[0034] Figure 7 This is a comparison chart of the packet scheduling success rates of different algorithms in the embodiments.

[0035] Figure 8 This is a comparison chart of message scheduling success rates for different algorithms in the embodiments.

[0036] Figure 9 This is a comparison chart of the time results of different algorithms solving the CQF-packet injection selection model in the embodiments. Detailed Implementation

[0037] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. The examples of the parameters listed are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

[0038] Part 1: Constructing a CQF mechanism for scheduling packet injection selection in a TSN network.

[0039] While time-aware shaping mechanisms can achieve microsecond-level scheduling for high-priority traffic flows on a hop-by-hop and packet-by-packet basis, they suffer from several drawbacks. Firstly, gating list configuration requires configuration across all terminal sites and bridge devices involved in the end-to-end process, and coordination between gating lists is essential to construct an isolated protection channel for high-priority traffic transmission. Therefore, the complexity of this configuration increases dramatically with the scale of the TSN network and the number of traffic flows. Secondly, the limited capacity of the gating lists on bridge devices prevents unlimited gating entries, leading to situations where a feasible scheduling solution to guarantee end-to-end transmission of high-priority traffic flows may not exist in complex network and service environments. Therefore, this invention improves upon the conventional scheduling method of the Circular Queuing Forwarding (CQF) mechanism conforming to the IEEE 802.1Q protocol, resulting in the CQF-Packet Injection Selection Model. The CQF-Packet Injection Selection Model transforms the complex constraints between packets and messages into a range of time slots within which packets can be successfully scheduled. This CQF-Packet Injection Selection Model under the CQF shaping mechanism is independent of the TSN network topology. The CQF (Category QF) packet injection selection model of this invention is beneficial for achieving efficient and refined data scheduling under the CQF mechanism of TSN networks, thereby solving the technical problems of existing scheduling algorithms in terms of efficiency, granularity, and multi-level network data coordination during the packet injection process.

[0040] To facilitate the construction of the CQF (Category QF) packet injection selection model, we will first illustrate the information related to the CQF mechanism in the required TSN network with examples.

[0041] TT messages in TSN network

[0042] Different terminal devices in a TSN network exchange network information by transmitting messages. In this invention, the scheduled TT message in the TSN network is denoted as... Multiple scheduled TT messages form a scheduling message set, denoted as . ,and .in, This indicates the first scheduled TT message. This indicates the second TT message to be scheduled. Indicates the first A scheduled TT message, This indicates the last TT message to be scheduled.

[0043] For ease of explanation, Also known as any TT message. Subscript This indicates the identifier number of the TT message.

[0044] In this invention, any TT message It includes the message transmission period P, the message transmission deadline D, and the message size S. The TT message is represented in triplet form. Similarly, we can conclude that... The triplet TT message is . The triplet TT message is . The triplet TT message is .

[0045] Data packets in a TSN network

[0046] In a TSN network, messages are encapsulated into data packets. For messages that are too large, the terminal device will split them into several data packets to comply with the TSN network's Maximum Transmission Unit (MTU) limit. In this invention, a data packet is denoted as... The data packet set is denoted as... ,and .in, This indicates the first data packet. This indicates the second data packet. Indicates the first Data packets, This indicates the last data packet.

[0047] For ease of explanation, Also known as any data packet. Subscript The identifier number of the data packet.

[0048] Any data packet The attributes include the transmission path Messages carried Message transmission cycle Message transmission deadline and data packet size That is, data packet attributes are denoted as ,and The elements P and D inherit from the TT message. Transmission path This is a TT message. A set of ordered nodes that are traversed via transmission.

[0049] In this invention, see Figure 3 As shown, a TT message is divided into one or more smaller data segments and stored in data packets for transmission. Any TT message... At least one data packet was allocated. Inside. Any data packet. It can contain at most the data of a TT message.

[0050] To implement CQF scheduling of TT messages in a TSN network, examples are provided to illustrate how a TT message might be assigned to one or more data packets.

[0051] For example, TT messages They were assigned to 8 different data packets, namely , , , , , , and For example, TT messages Assigned to data packets In, that is For example, TT messages It was assigned to two data packets, namely and For example, TT messages It was allocated into 5 data packets, that is... , , , and .

[0052] Representation of transmission path

[0053] A TSN topology network consists of connections between multiple nodes. These nodes primarily include switches and terminals, denoted as follows: and Data packets are generated and scheduled at the terminal, originating from the source terminal, passing through several switches, and arriving at the destination terminal.

[0054] In this invention, the unidirectional transmission between two nodes is referred to as a logical link, denoted as . Multiple logical links form a transmission path, denoted as . The number of switches that data passes through during transmission is called the hop count, denoted as . See also Figure 4 For ease of explanation, each logical link is listed as an example.

[0055] For example, by the terminal To the switch The logical link for transmission is denoted as ,and And by the switch To the terminal The logical link for transmission is denoted as ,and .

[0056] Similarly, it can be concluded that, from the terminal To the switch The logical link for transmission is denoted as ,and And by the switch To the terminal The logical link for transmission is denoted as ,and .

[0057] Similarly, it can be concluded that, from the terminal To the switch The logical link for transmission is denoted as ,and And by the switch To the terminal The logical link for transmission is denoted as ,and .

[0058] Similarly, it can be concluded that, from the terminal To the switch The logical link for transmission is denoted as ,and And by the switch To the terminal The logical link for transmission is denoted as ,and .

[0059] Similarly, it can be concluded that, from the terminal To the switch The logical link for transmission is denoted as ,and And by the switch To the terminal The logical link for transmission is denoted as ,and .

[0060] Then by terminal To the terminal Transmission path Including logical links Logical Link Logical Link , recorded as The number of hops in the data along this path It is 2.

[0061] Representation of time units

[0062] In this invention, the basic unit for periodic division of time is called a time slot, denoted as . This basic unit can be changed for different message configurations, but it should ensure that the transmission period P of all TT messages can be changed. Divisible, i.e. .

[0063] In this invention, the shortest non-repeating time of all periodic scheduling tasks in the scheduling system is called the supercycle, denoted as . This time unit should be equal to the least common multiple of the transmission period P of all TT messages, i.e. .

[0064] In this invention, any time point is denoted as t. For ease of explanation, the identifier of the time slot in which the current time t is located is denoted as n, i.e., t and n satisfy... .

[0065] Ping-Pong Queue Mechanism of CQF Scheduling

[0066] In this invention, any one switch or terminal In any logical link connected to it Each of the above has an independent port, and each port has 8 queues, denoted as . These are used to transmit data. Among these eight queues, and The two queues are dedicated transmission queues for TT messages. The opening and closing states of the control gates for the two queues are always opposite, i.e. When open closure, When open Close, see Figure 5 As shown. The control gates for the two queues are in a single time slot. The ping-pong queue cyclically changes its open and closed states. A high-precision time mechanism ensures the permanent correctness of the cyclical queuing and forwarding of the ping-pong queue.

[0067] Effective length of queue

[0068] In this invention, the queue for transmitting TT messages is denoted as... The set of queues for all ports is denoted as . ,and .in, This indicates the queue for the first transmission of a TT message. This indicates the second queue for transmitting TT messages. Indicates the first A queue for transmitting TT messages. This indicates the queue that last transmitted the TT message.

[0069] For ease of explanation, Also known as any queue for transmitting TT messages. (Subscript) This indicates the identifier of the queue transmitting TT messages.

[0070] In this invention, the maximum amount of data that a TSN network can transmit per unit time is called the network bandwidth, denoted as . Bandwidth, often simply called bandwidth, determines a network's ability to transmit data packets. The maximum amount of data a queue can hold is called the effective length of the queue. CQF's ping-pong queuing mechanism simultaneously satisfies the full forwarding principle, meaning the queue operates within a single time slot. Data received and queued for forwarding must be in the next time slot. Data packets exceeding the storage limit were sent completely within the specified timeframe. Packet loss will be performed. Therefore, it can be concluded that the amount of data the queue can hold is less than or equal to one time slot. The amount of data that can be transmitted within, i.e. Therefore, the effective length of the queue satisfy .

[0071] Queue Usage and Time – Queue Matrix

[0072] In this invention, arbitrary time slots are... any queue The maximum bandwidth utilization rate is called the occupancy rate of the queue in that time slot, denoted as . Its value range is [0,1]. A two-dimensional matrix is ​​introduced. ,in This represents the current time slot number n, with an upper limit of 1. , representing the observation time, i.e., the current moment. The time slot ; This represents the current queue number k, with an upper limit of K, and represents the observation queue, i.e., the current queue. This matrix is ​​called the time-queue matrix, denoted as . And the element in the nth row and kth column of the matrix is .

[0073] In this invention, incremental scheduling is used to schedule a data packet... After scheduling and injection are completed, proceed to the next data packet. Scheduling is performed. A time-queue matrix is ​​used to represent the usage of all queues within one supercycle after injecting several data packets; this is called the queue usage of injected data packets, denoted as [formula missing]. ,and Before The time-queue matrix after a data packet is scheduled and injected into the TSN network is called the time before injection. The queue usage of each data packet is denoted as . .

[0074] For example This indicates that no packets were scheduled and injected into the TSN network, and The matrix satisfies that all elements are 0, i.e. Similarly, we can conclude that... This indicates that the first data packet has been scheduled and injected into the TSN network. Similarly, we can deduce that... This indicates that the first two data packets have been scheduled and injected into the TSN network. Similarly, we can conclude that... Indicates the preceding One packet has been scheduled and injected into the TSN network. Similarly, it can be concluded that... This indicates that the last data packet has been scheduled and injected into the TSN network.

[0075] Using a time-queue matrix representation, only scheduling and injection of the first... The time-queue matrix after the first data packet is called the injection-only-then-first-packet matrix. The queue usage of each data packet is denoted as . Furthermore, the queue usage varies depending on the injection time, denoted as... ,in This is the ratio of injection time to injection cycle.

[0076] For example Indicates only the previous Each packet is scheduled and injected into the TSN network. Similarly, it can be concluded that... This indicates that only the first packet is scheduled and injected into the TSN network. Similarly, we can deduce that... This indicates that only the first two packets are scheduled and injected into the TSN network. Similarly, it can be concluded that... This indicates that only the last packet was scheduled and injected into the TSN network.

[0077] Set variables for CQF—Packet Injection Selection Model

[0078] In this invention, a deep reinforcement learning algorithm combined with incremental scheduling is mainly used to learn the network environment and the characteristics of the messages to be scheduled, so as to achieve fast and high-quality scheduling of data packets. Its three main variables are: state s, action, and so on. 、Revenue r.

[0079] The current transmission state of the TSN network is called the state. This refers to the time-queue matrix consisting of currently scheduled data packets. For A data packet scheduling system exists. There are several states. For ease of explanation, let's denote any state as . , indicating the preceding The time-queue matrix after each data packet has been scheduled and injected into the TSN network. State transitions. Represented as the first After a packet is injected into the TSN network, the usage of all queues changes within a supercycle.

[0080] The ratio of the injection time of the currently scheduled data packet to its injection period is called the action. It is a continuous variable. The injection time is always less than or equal to the injection period, therefore the action... It takes values ​​in the range [0,1]. The injection period of the currently scheduled data packet is called the action space. For a scheduling system with J data packets, there are J actions. For ease of explanation, any action is denoted as... , which represents the ratio of the injection time of the j-th data packet to the injection period.

[0081] The feedback value obtained at time t in the current transmission state of the TSN network for the data packets to be scheduled is called the gain r. For each scheduling step, there exists a gain value r, that is, for a scheduling system with J data packets, there are J gains. For ease of explanation, any gain is denoted as . , indicating the state The j-th data packet is processed according to the action. Network environment feedback obtained by injecting into the TSN network.

[0082] In this invention, based primarily on deep reinforcement learning algorithms, two sets of networks were designed. Each set of networks has three different types of networks with different functions, namely, the actual action network. Actual evaluation network and Target Action Network Target evaluation network and There are a total of 6 neural networks. Among them, Enter the current status To obtain actual actions; and Input the current state and the current action to obtain the benefit of the action in that state; Input the target state to obtain the target action; and Given the target state and the optimal action, obtain the expected reward of the action in that state. and The network obtains a value in the range [0,1], representing the proportional position of the selected action in the action space. Further processing is required to obtain the final action. .

[0083] For ease of explanation, use and They represent and network, and They represent and Network parameters. Use and They represent and network, and They represent and The network parameters; similarly, we can derive the following using... and They represent and network, and They represent and Network parameters. and The network input is state. ,use and They represent and Network output; , , and The network input is state. and actions ,use , , and They represent , , and Network output.

[0084] In this invention, for ease of explanation, a notation system for DRL parameters is established. The learning rate is represented as... The noise figure is expressed as The delayed update frequency is expressed as The soft update coefficient is expressed as The attenuation coefficient is expressed as .

[0085] CQF—Experience Pool and Selection of Experience in Packet Injection

[0086] This invention primarily utilizes deep reinforcement learning algorithms. The data that the neural network needs to learn when updating its parameters is called experience; one experience... The original state should be included. ,action ,income Post-action state , represented as a quadruple In the reinforcement learning phase, experience is generated; in the deep learning phase, the neural network learns the experience generated. Therefore, an experience pool is set up to store the experience generated in the reinforcement learning phase. The experience pool has a storage limit. When the storage limit is exceeded, the earliest stored experience is discarded according to the time it was generated.

[0087] In the deep learning phase, to ensure learning efficiency and effectiveness, a batch of experiences should be selected for learning each time. The experience pool is sized according to the batch size of the selected experiences. Random selection experience Form an experience group, denoted as ,and .in, This refers to the experience group. The first lesson learned is... This refers to the experience group. The second lesson learned, This refers to the experience group. The first in One experience, This refers to the experience group. The last lesson in the book, namely the first... One piece of experience. For ease of explanation, Also known as any experience. Subscript An identifier representing experience.

[0088] To address the drawbacks of long solution time and random packet injection selection in the CQF scheduling mechanism of TSN networks, this invention designs a CQF scheduling mechanism compliant with the IEEE 802.1Q protocol. The construction of the CQF-packet injection selection model includes the following steps:

[0089] Step A: Build a two-layer network architecture;

[0090] To improve the efficiency of the CQF-packet injection selection model in obtaining CQF scheduling schemes for TSN networks and the stability of scheduling performance, the actor-commentator algorithm from deep reinforcement learning is adopted.

[0091] In the CQF (Clustered Quadruple Decision Function) packet injection selection model, each round requires collecting a complete trajectory based on a policy and calculating the reward for that trajectory. The Actor-Critic Algorithm is a reinforcement learning method that combines policy gradient and temporal difference learning. The "Actor" refers to the policy function of the CQF packet injection selection model, i.e., learning an injection selection model to complete as many packet transmissions as possible. The "Critic" refers to the value function of the CQF packet injection selection model, estimating the expected reward of the current scheduling policy, i.e., evaluating the actor's performance. Thanks to the value function, the Actor-Critic Algorithm can update parameters step-by-step, without waiting for the end of the round. Both the policy function and the value function in the Actor-Critic Algorithm are functions to be learned and need to be learned simultaneously during training.

[0092] (1) In the CQF-packet injection selection model, there are two network layers: the actor network is responsible for obtaining the injection time slot of the data packet; the commentator network is responsible for evaluating the transmission success rate of the injected data packet.

[0093] (2) An experience pool is set in the CQF-packet injection selection model. When the experience pool reaches its storage limit, the earliest stored experience is discarded first, i.e., first-in-first-out. Set the experience pool storage limit, batch size when selecting experience, and other experience pool parameters.

[0094] In this invention, the main design element is the actual action network. Actual evaluation network and Target Action Network Target evaluation network and There are a total of 6 networks. Among them, the actual action network... Primarily used for selecting the timing of packet injection, and for actual network evaluation. and Target Action Network Target evaluation network and It is primarily used for training and parameter updating of deep reinforcement learning neural networks. Among them, and A four-layer network is used (1 input layer, 2 hidden layers, 1 output layer; the input layer consists of states). The dimensions are fixed; the number of neurons in the hidden layer is 256, and the number of neurons in the output layer is 1. , , and A four-layer network is used (1 input layer, 2 hidden layers, 1 output layer; the number of neurons in the input layer is determined by the state). Dimensions and Actions With the dimensions determined, the number of hidden layer neurons is 256, and the number of output layer neurons is 1. Initially, the parameters of the network neurons can be randomly set, and the network optimizer is Adam.

[0095] In the CQF-packet injection selection model of this invention, the learning rate is set according to the learning effect. Noise figure Delayed update frequency Soft update coefficient attenuation coefficient Initialize and clear the experience pool, and set the experience pool storage limit. Experience in selecting batch size .

[0096] Step B involves allocating time for packet injection.

[0097] Step B1: Obtain basic movements;

[0098] Towards Current status of network input , obtain output And this is called the basic movement.

[0099] For example, the Data packets The state before scheduling is The basic actions obtained are Similarly, the first data packet... The state before scheduling is The basic actions obtained are Similarly, the second data packet... The state before scheduling is The basic actions obtained are Similarly, the first... Data packets The state before scheduling is The basic actions obtained are .

[0100] During training, to ensure the exploratory nature of the DRL algorithm within the TD3 framework of Deep Reinforcement Learning (DRL), noise needs to be added to the basic actions, but not when obtaining the final scheduling scheme. Therefore, during training, continue with step B2; otherwise, proceed to step B3.

[0101] Step B2, obtain the noisy injection time;

[0102] Action noise follows a normal distribution. In practical applications, it is randomly selected from [-1, 1] according to the normal distribution probability and multiplied by the noise coefficient. As noise .

[0103] data packet basic movements With noise Summing, pruning within the range [0,1], and obtaining the action of the data packet. Then multiply by the current data packet. Action space That is, to obtain the noisy injection time of the data packet.

[0104] For example, the Data packets basic movements The value is 0.05, the noise is -0.15, the post-cutting action is 0.00, and the final action is... The value is 0.00. The action space is 16, and the noisy injection time is 0.00. Similarly, the first data packet... basic movements The value is 0.10, the noise is 0.15, the post-cutting motion is 0.25, and the final motion... The value is 0.25. The action space is 4, and the noisy injection time is 1.00. Similarly, the second data packet... basic movements The value is 0.85, the noise is 0.25, the post-cutting motion is 1.00, and the final motion... The value is 1.00. The action space is 2, and the noisy injection time is 2.00. Similarly, the... Data packets basic movements The value is 0.58, the noise is -0.05, the post-cutting motion is 0.53, and the final motion... The value is 0.53. The action space is 8, and the noisy injection time is 4.24.

[0105] Step B3: Obtain the noise-free injection timing;

[0106] data packet basic movements As the action of this data packet Multiply by the current data packet Action space That is, to obtain the noise-free injection time of the data packet.

[0107] For example, the Data packets basic movements The final action is 0.05. The value is 0.05. The action space is 16, and the injection time without noise is 0.80. Similarly, the first data packet... basic movements The final action is 0.10. The value is 0.10. The action space is 4, and the noise-free injection time is 0.40. Similarly, the second data packet... basic movements The final action is 0.85. The value is 0.85. The action space is 2, and the injection time without noise is 1.70. Similarly, the... Data packets basic movements The final action is 0.58. The value is 0.58. The action space is 8, and the injection time without noise is 4.64.

[0108] In step B, the basic action is obtained by inputting the current state into the actor network; the basic action is then multiplied by the action space of the data packet to be scheduled to obtain the injection time of the data packet.

[0109] Step C: Environmental Changes and Feedback;

[0110] Based on the transmission path of the current data packet to be scheduled and the injection time, construct a time-queue matrix that only injects the current data packet; sum the queue usage of the injected data packets with the time-queue matrix that only injects the current data packet to obtain the expected new time-queue matrix; verify the legality of the injection time based on the expected new time-queue matrix and the hop count of the current data packet, and obtain the single-step benefit based on the legality verification result.

[0111] Step C1: Obtain the queue usage status for injecting the current data packet;

[0112] Get the current data packet transmission path According to the transmission path and actions Construct a time-queue matrix and use it as the queue usage for injecting the current data packet.

[0113] For example, the Data packets The final action is The queue usage for injecting only the current data packet is as follows. Similarly, the first data packet... The final action is The queue usage for injecting only the current data packet is as follows. Similarly, the second data packet... The final action is The queue usage for injecting only the current data packet is as follows. Similarly, the first... Data packets The final action is The queue usage for injecting only the current data packet is as follows. .

[0114] Step C2: Calculate the new state of the current data packet after performing the current action;

[0115] Step (1) Check the queue usage of the injected data packets. According to the current action Inject only the queue usage of the current data packet Sum the results to obtain the queue usage of the expected new packets to be injected into the current data packet. .

[0116] For example, the Data packets The final action is Only inject the queue usage information for the current data packet. The queue usage of packets already injected before this packet was: The expected queue usage for newly injected current packets is .

[0117] Similarly, the first data packet The final action is Only inject the queue usage information for the current data packet. The queue usage of packets already injected before this packet was: The expected queue usage for newly injected current packets is .

[0118] Similarly, the second data packet The final action is Only inject the queue usage information for the current data packet. The queue usage of packets already injected before this packet was: The expected queue usage for newly injected current packets is .

[0119] Similarly, the first Data packets The final action is Only inject the queue usage information for the current data packet. The queue usage of packets already injected before this packet was: The expected queue usage for newly injected current packets is .

[0120] Step (2), based on the expected queue usage of the newly injected current data packets. and number of jumps This verifies whether the action performed by the data packet is legal. There are two main criteria for determining the legality of the action: one is whether the queue overflows, i.e., whether the data overflows. First, does any element in the matrix exist that is greater than 1? Second, is the deadline for the data packet satisfied? .

[0121] For example, the Data packets The queue usage of packets already injected before this packet was: The expected queue usage for newly injected current packets is The number of jumps is ,satisfy The matrix element is less than 1, which satisfies If the data packet executes the current action, then it is legal and has been injected before. The queue usage of each data packet is as follows: .

[0122] Similarly, the first data packet The queue usage of packets already injected before this packet was: The expected queue usage for newly injected current packets is The number of jumps is ,satisfy The matrix element is less than 1, which satisfies If the data packet executes the current action, then it is valid. The usage status of the queue that was injected with the previous data packet is as follows: .

[0123] Similarly, the second data packet The queue usage of packets already injected before this packet was: The expected queue usage for newly injected current packets is The number of jumps is ,satisfy The matrix element is less than 1, which does not satisfy the condition. If the data packet cannot execute the current action, then the queue usage of the first two injected data packets is as follows: .

[0124] Similarly, the first Data packets The queue usage of packets already injected before this packet was: The expected queue usage for newly injected current packets is The number of jumps is Not satisfied The matrix element is less than 1, which satisfies If the data packet cannot execute the current action, it is invalid and has already been injected. The queue usage of each data packet is as follows: .

[0125] Step C3: Provide feedback on the benefits of the action execution;

[0126] The TSN network environment returns data packets in this action. The following benefits If the validity of the current action of the data packet is verified, then the data packet scheduling and injection are successful, and the benefit is realized. The value is 1; if the verification of the legality of the current action of the data packet fails, the scheduling and injection of the data packet will fail, and the benefit will be lost. It is 0.

[0127] For example, the Data packets Execute action If it's legal, then you'll benefit. The value is 1. Similarly, the first data packet... Execute action If it's legal, then you'll benefit. The value is 1. Similarly, the second data packet... Execute action Illegal, therefore profitable It is 0. Similarly, the first... Data packets Execute action Illegal, therefore profitable It is 0.

[0128] Based on the number of packets currently scheduled Then determine whether to continue scheduling and transmitting data packets. If... If so, repeat steps three and four; Then proceed to step five.

[0129] Step C4: Determine the scheduling result for this round;

[0130] Step (1), based on the data packet Execute the current action The validity verification results determine the number of successfully scheduled data packets. .

[0131] For example, the Data packets Execute action If the data packet is valid, then the packet scheduling and injection were successful. Add 1. Similarly, for the first data packet... Execute action If the data packet is valid, then the packet scheduling and injection were successful. Add 1. Similarly, for the second data packet... Execute action If invalid, the packet scheduling and injection will fail. Unchanged. Similarly, the... Data packets Execute action If invalid, the packet scheduling and injection will fail. constant.

[0132] Step (2), according to the message The number of successfully scheduled messages is determined by verifying the validity of all allocated data packets. .

[0133] For example, TT messages They were assigned to 8 different data packets, namely , , , , , , and If all data packets pass the validity verification and the scheduling and injection are successful, then... Add 1.

[0134] TT News Assigned to data packets In, that is If all data packets pass the validity verification and the scheduling and injection are successful, then... Add 1.

[0135] TT News It was assigned to two data packets, namely and Among them, data packets The data packet passed the legality verification, scheduling, and injection. If the validity verification fails, scheduling and injection will fail. constant.

[0136] TT News It was allocated into 5 data packets, that is... , , , and Among them, data packets and The data packet passed the legality verification, scheduling, and injection. , and If the validity verification fails, scheduling and injection will fail. constant.

[0137] Step (3), based on the injected... Queue usage for each data packet Determined in a supercycle The maximum average queue bandwidth utilization is the highest of the average values ​​of any queue bandwidth utilization over time. .right By averaging over time, we obtain the set of average queue bandwidth utilization rates. Select the maximum value from them as .

[0138] For example China represents the first A queue for transmitting TT messages The amount is ,but .

[0139] Similarly The middle line represents the queue for transmitting the first TT message. The amount is ,but .

[0140] Similarly The middle line represents the queue for transmitting the second TT message. The amount is ,but .

[0141] For example China represents the first A queue for transmitting TT messages The amount is ,but .

[0142] Then the maximum average queue bandwidth utilization in this round of scheduling for .

[0143] Step D: Experience correction and storage;

[0144] Step D1: Set the final revenue;

[0145] The scheduling objective is to comprehensively maximize the transmission success rate and minimize the maximum average queue bandwidth utilization, thus ultimately maximizing the benefit. Set as message Transmission success rate, data packets Transmission success rate, one supercycle The product of the maximum value of the average bandwidth utilization of any queue over time.

[0146] For example, when the first Data packets After injection, the number of successfully transmitted data packets was: The number of messages successfully transmitted is The maximum average queue bandwidth utilization rate is So, the final benefit for .

[0147] Step D2: Iteratively determine the cumulative reward for each step;

[0148] In this invention, the reward that is directly fed back from the environment and can only reflect the quality of the current action is called the single-step reward. The accumulated gains that reflect the quality of overall actions are called cumulative gains. According to the iterative formula of cumulative returns Calculate the actual cumulative profit for each step. Among them, satisfy .

[0149] For example, after all data packets have been scheduled... Data packets In state Select action Cumulative gains obtained for Similarly, after all data packets have been scheduled, the first... Data packets In state Select action Cumulative gains for Similarly, the second data packet is sent after all data packets have been scheduled. In state Select action Cumulative gains for Similarly, the first data packet after all data packets have been scheduled. In state Select action Cumulative gains for .

[0150] Step D3: Store experience;

[0151] The adjusted cumulative returns and the original state ,action Post-action state Forming a quadruple , and expressed as experience This experience Stored in the experience pool.

[0152] For example, in this round of scheduling, the first Data packets The experience generated (i.e., the first) (One experience) Similarly, in this round of scheduling... Data packets The experience generated (i.e., the first) (One experience) Similarly, in this round of scheduling, the second data packet... The experience generated (i.e., the second piece of experience) is Similarly, in this round of scheduling, the first data packet... The experience generated (i.e., the first piece of experience) is .

[0153] In step D, the final reward is set according to the scheduling results of all data packets, and the cumulative reward for each step is determined iteratively. The original state, action, cumulative reward and post-action state are combined into experience and stored in the experience pool.

[0154] Step E involves reinforcement learning and updating.

[0155] In this invention, firstly, a set of experience groups to be learned is selected. Based on the set batch size for experience selection, a set of experience groups is randomly selected from the experience pool, and any experience is parsed into a four-tuple form. Then, the expected Q-value is calculated. Based on the state after the action, the expected action is calculated using the target action network. Based on the state after the action and the expected action, two target evaluation networks calculate their respective output values, and the smaller value is selected as the target Q-value. Then, the expected Q-value is calculated based on the reward and the target Q-value. The actual Q-value is calculated. Based on the original state and action, two actual evaluation networks calculate their respective output values. Then, the loss is calculated and the actual evaluation networks are updated. The loss is calculated for both the expected Q-value and the two actual Q-values. Optimizer gradient zeroing, loss backpropagation, and optimizer stepping steps are performed on the two actual evaluation networks respectively. Based on the current training epoch and the delayed update frequency, it is determined whether a delayed update is needed. Finally, determine whether to perform a delayed update: Based on the original state, calculate the basic actions using the actual action network; based on the original state and the basic actions, calculate the output values ​​using the actual evaluation network; average the output values ​​and take the inverse as the loss for the actual action network; perform optimizer gradient zeroing, loss backpropagation, and optimizer stepping on the actual action network; based on the soft update coefficients, perform soft update operations on the parameters of the actual action network and the parameters of the two target evaluation networks. Specifically, this includes five aspects:

[0156] Step E1: Select the experience group to be learned;

[0157] Batch size selected based on established experience. A group of experience points is randomly selected from the experience pool. and any experience Parsing into the form of quadruplets Where s represents the initial state, Let r represent the action and r represent the benefit. This indicates the state after an action has been performed.

[0158] Step E2: Calculate the expected Q value;

[0159] Step (1), proceed to the post-action state ,use Calculate the expected action The process is roughly the same as step three. The calculation formula is: To obtain the desired action .

[0160] For example, the experience Post-action state The selected noise is Then the expected action satisfy Similarly, the first piece of experience... Post-action state The selected noise is Then the expected action satisfy Similarly, the second piece of experience... Post-action state The selected noise is Then the expected action satisfy Similarly, the first... experience Post-action state The selected noise is Then the expected action satisfy .

[0161] Step (2), proceed to the post-action state and expected action ,use and The network calculates its respective output value. and and compare and Choose the smaller value as the target Q value ,and .

[0162] For example, the experience Post-action state The selected noise is Then the expected action satisfy .pass and After network calculation If the value is smaller, then Similarly, the first piece of experience... Post-action state The selected noise is Then the expected action satisfy .pass and After network calculation If the value is smaller, then Similarly, the second piece of experience... Post-action state The selected noise is Then the expected action satisfy .pass and After network calculation If the value is smaller, then Similarly, the first... experience Post-action state The selected noise is Then the expected action satisfy .pass and After network calculation If the value is smaller, then .

[0163] Step (3), based on the revenue And the target Q value, according to the formula Calculate the expected Q value .

[0164] For example, the experience Post-action state The selected noise is Then the expected action satisfy .pass and After network calculation If the value is smaller, then The profit is Then the expected Q value satisfy Similarly, the first piece of experience... Post-action state The selected noise is Then the expected action satisfy .pass and After network calculation If the value is smaller, then The profit is Then the expected Q value satisfy Similarly, the second piece of experience... Post-action state The selected noise is Then the expected action satisfy .pass and After network calculation If the value is smaller, then The profit is Then the expected Q value satisfy Similarly, the first... experience Post-action state The selected noise is Then the expected action satisfy .pass and After network calculation If the value is smaller, then The profit is Then the expected Q value satisfy .

[0165] Step E3: Calculate the actual Q value;

[0166] Based on the initial state s and the action ,use and The network calculates its respective output value. and .

[0167] For example, the experience original state Selected action Then through and The output values ​​after networking are respectively and Similarly, the first piece of experience... original state Selected action Then through and The output values ​​after networking are respectively and Similarly, the second piece of experience... original state Selected action Then through and The output values ​​after networking are respectively and Similarly, the first... experience original state Selected action Then through and The output values ​​after networking are respectively and .

[0168] Step E4: Calculate the loss and update the actual evaluation network;

[0169] Step (1), obtain the expected Q value With two actual Q values and Calculate the loss separately and obtain and .

[0170] Step (2), for and The network performs optimizer gradient zeroing, loss backpropagation, and optimizer step steps respectively.

[0171] Based on the current training round and delayed update frequency Determine whether a delayed update is needed. If a delayed update is required, proceed to step E5; otherwise, skip this step.

[0172] Step E5: Determine whether to perform a delayed update;

[0173] Step (1), based on the original state ,use Basic network computing actions Based on the original state and basic movements ,use Network calculation output value .

[0174] For example, the experience original state The basic action is ,pass The output value after networking is Similarly, the first piece of experience... original state The basic action is ,pass The output value after networking is Similarly, the second piece of experience... original state The basic action is ,pass The output value after networking is Similarly, the first... experience original state The basic action is ,pass The output value after networking is .

[0175] Step (2), for use Output value of network computation Take the average and then take the opposite as... Network loss .

[0176] Step (3), for The network performs optimizer gradient zeroing, loss backpropagation, and optimizer step steps.

[0177] Step (4), based on the soft update coefficient ,right Network parameters , Network parameters and Network parameters Perform a soft update operation, i.e. , and .

[0178] In step E, an experience group is randomly selected from the experience pool, the expected Q value and the actual Q value are calculated, the loss is calculated and the actual evaluation network is updated; it is determined whether to perform delayed update based on the current training round and the delayed update frequency. If necessary, the actual action network is updated and the parameters of the actual action network and the target evaluation network are softly updated.

[0179] Part Two: Application of CQF—Packet Injection Selection Model in TSN Networks

[0180] See Figure 1 , Figure 2 As shown, the CQF-packet injection selection model trained in the first part is applied to the TSN network for multi-layer message optimization scheduling. This scheduling method includes the following steps:

[0181] Step 1: Initialize the TSN network;

[0182] In this invention, before performing multi-layer message optimization scheduling based on the CQF-Packet Injection Selection Model, it is necessary to collect data packets and initialize their attributes, initialize TSN network environment parameters and CQF-Packet Injection Selection Model status, and sort the data packets to be scheduled.

[0183] Step 101, initialize data packet attributes;

[0184] Collect all data packets transmitted in the TSN network and for any data packet Initialize the attributes to obtain the data packet – the attributes are denoted as ,and .in, Indicates data packet The transmission path Indicates data packet TT message carried in , Indicates data packet TT message carried in The injection cycle, Indicates data packet TT message carried in The transmission deadline Indicates data packet Size.

[0185] Similarly, we can deduce that the data packet The attribute is Data packets The attribute is Data packets The attribute is .

[0186] Step 102: Initialize TSN network environment parameters;

[0187] Based on any data packet—attributes Initialize TSN network environment parameters. Set network bandwidth. And according to the data packet injection cycle Set the timeout period of the scheduling system. .

[0188] Calculate time slots Maximum value and effective queue length The maximum value. For any data packet. Injection period P, time slot All meet Therefore, the maximum value of the time slot satisfy ,and Effective queue length satisfy Therefore, the maximum effective length of the queue satisfy .

[0189] Input time slot and queue effective length and the calculated maximum value and The comparison is performed, and if the conditions are met, the parameter setting is complete.

[0190] Step 103, Initialize CQF—Packet Injection Selection Model State;

[0191] State of DRL Initialization is performed. In this invention, the current transmission state of the TSN network is referred to as state s, and is represented using a time-queue matrix. Initially, no data packets have been scheduled and injected into the TSN network, therefore the system's time-queue matrix at this time is... ,and That is, initially, state s is a A matrix in which all elements are 0.

[0192] Step 104: Sort the data packets to be scheduled;

[0193] To meet the schedulable message count Maximizing and switch queue bandwidth utilization To minimize this, the order of messages to be scheduled needs to be reordered before scheduling begins. In this invention, the reordering method is based on the product of the injection period of the message to be scheduled and the sum of the sizes of all the data packets it contains. Determine the scheduling order.

[0194] Step 2, Selecting data packets for injection;

[0195] In this invention, data packet information is sent to the data packet injection selection model network to obtain a data packet injection scheme, which represents the time when each data packet enters the TSN.

[0196] See Figure 6 As shown, when no scheduling is performed, the data packets In the same time slot Internal network injection may exceed queue capacity, leading to packet loss and network congestion. In contrast, a strategy-based network injection allocates different entry times for packets into the TSN network, for example... exist Injection within time slots, exist Injection within time slots. This will significantly reduce potential packet congestion, lower the probability of packet loss, and improve network transmission efficiency.

[0197] Step 3, Data Packet Injection Time Allocation;

[0198] In this invention, the ratio of the injection time to the previous injection period is used as a continuous action in the data packet injection time allocation, thereby obtaining the data packet injection time in deep reinforcement learning.

[0199] Towards The network takes the current state s as input and obtains the output. And this is called the basic movement.

[0200] Based on the basic motion, motion noise needs to be added to create a realistic motion. The motion noise follows a normal distribution. In practical applications, it is randomly selected from [-1, 1] according to the normal distribution probability and multiplied by a noise coefficient. As noise .

[0201] data packet basic movements As the action of this data packet Multiply by the current data packet Action space That is, to obtain the noise-free injection time of the data packet.

[0202] Step 4, Environmental Changes and Feedback;

[0203] In this invention, the legality of actions is verified and the benefits are fed back by constructing a time-queue matrix.

[0204] Get the current data packet transmission path According to the transmission path and actions Construct a time-queue matrix and use it as the queue usage for injecting the current data packet.

[0205] Queue usage of injected data packets According to the current action Inject only the queue usage of the current data packet Sum the results to obtain the queue usage of the expected new packets to be injected into the current data packet. .

[0206] Step 5: Schedule the data packets to reach the target terminal completely under the CQF mechanism;

[0207] In this invention, the data packets and messages are scheduled and arrive at the target terminal under the CQF mechanism, and the scheduling results are calculated to obtain the final scheduling scheme.

[0208] According to the allocated injection time, the data packets enter the network sequentially and are transmitted according to the CQF mechanism of the TSN network to reach the target terminal.

[0209] Step 501: Calculate the data packet and message scheduling results;

[0210] Based on the expected queue usage of newly injected current data packets and number of jumps This verifies whether the data packet can be successfully scheduled at the current injection time. There are two main criteria: one is whether the queue overflows, i.e., whether... First, does any element in the matrix exist that is greater than 1? Second, is the deadline for the data packet satisfied? .

[0211] Based on the TSN network environment, determine whether the data packet was successfully scheduled. If the data packet is successfully injected into the TSN network and reaches the target node, it is recorded as successfully scheduled, and the reward r is 1; otherwise, the reward r is 0.

[0212] Calculate the number of successfully scheduled data packets based on the packet scheduling success rate. If message If all allocated data packets are successfully scheduled, the message is considered successfully scheduled; calculate the number of successfully scheduled messages. .

[0213] Step 502: Obtain the final scheduling scheme;

[0214] Under the current CQF-packet injection selection model, the injection time (i.e., action) of each packet is selected according to the method in step three to form the final scheduling scheme.

[0215] Example 1

[0216] This embodiment is based on the PyCharm platform and utilizes... Figure 4 Based on the network topology shown and the TT message and data packet configuration information shown in Table 1, network scheduling design is performed according to the fast traffic scheduling method proposed in this invention. The network bandwidth is set to 1000. The period is set to 250. The queue length is set to 3125. .

[0217] Table 1 Data Packets and TT Messages

[0218]

[0219] After 10 rounds of training, a satisfactory scheduling result was obtained. The scheduling success rate was 100%, and the maximum average occupancy rate was 0.1655. The optimal injection time for each data packet, based on the CQF-packet injection selection model output, is shown in Table 2.

[0220] Table 2 Optimal Data Packet Injection Time

[0221]

[0222] Example 2

[0223] This embodiment is also based on the PyCharm platform. Building upon Embodiment 1, it configures more complex TT messages and data packet information, and compares the scheduling method proposed in this invention with other baseline scheduling methods in terms of scheduling success rate and efficiency in obtaining the CQF (Category QF) packet scheduling policy. The network bandwidth is set to 1000. T is set to 250 The queue length is set to 3125. .

[0224] In the simulation, Satisfiability Modulo Theories (SMT), Joint Routing and Scheduling Algorithm (JRSA), Double Deep Q-learning Network-based Resource Allocation (DDTA), and Deep Reinforcement Learning-assisted Scheduling Algorithm for CQF (DeepCQF) were selected as baseline algorithms for comparison. SMT is a traditional constraint-solving algorithm, while JRSA, DDTA, and DeepCQF are all deep reinforcement learning algorithms. The proposed method (TAMS) was compared with the four baseline methods on 10 different experimental examples to observe the scheduling performance.

[0225] like Figure 7 The simulated packet scheduling success rates are as follows: In small-scale scenarios (c1-c3), both TAMS and SMT can schedule all packets, while DDTA and DeepCQF have average scheduling success rates of 82% and 91%, respectively. In medium-scale scenarios (c4-c6), TAMS achieves a scheduling success rate exceeding 90%, improving upon SMT, JRSA, DDTA, and DeepCQF by 11%, 50%, 28%, and 17%, respectively. In large-scale scenarios (c7-c10), TAMS reaches 72%, improving upon DDTA and DeepCQF by 48% and 17%, respectively, and is 2.3 times that of JRSA. Overall, the packet scheduling success rates of TAMS, DeepCQF, SMT, DDTA, and JRSA decrease sequentially, with TAMS leading by 18%, 44%, 45%, and 122%, respectively.

[0226] like Figure 8The simulated message scheduling success rate results are generally consistent with those of packet scheduling success rate. In medium-scale scenarios of c4-c6, TAMS outperforms SMT, JRSA, DDTA, and DeepCQF by 21%, 57%, 32%, and 27%, respectively. Overall, the performance of TAMS, DeepCQF, DDTA, SMT, and JRSA decreases in that order, with TAMS leading by 26%, 51%, 88%, and 177%, respectively.

[0227] like Figure 8 The simulation results for solving the CQF-packet injection selection model show that SMT has the longest solution time, exceeding 30 minutes in medium-scale scenarios (c4-c6) and approaching 3 hours in the CEV scenario. JRSA has the shortest solution time, requiring only 11 minutes even in the CEV scenario. TAMS requires only 17 seconds, 4 minutes, and 15 minutes in small, medium, and large-scale scenarios, respectively, ranking second only to JRSA and outperforming DeepCQF and DDTA. The speed at which each DRL method obtains the optimal injection slot scheme is approximately four times that of SMT.

Claims

1. A multi-layer message optimization scheduling method applied to TSN networks, characterized in that, Includes the following steps: Step 1: Initialize the TSN network; Collect all data packets to be scheduled in the TSN network and initialize the attribute information of each data packet; The attribute information includes the transmission path, the TT message carried, the injection period, the transmission deadline, and the data packet size; initialize the network bandwidth, timeout, time slot, and effective queue length; initialize the state of deep reinforcement learning, which is the time-queue matrix composed of the data packets that have been scheduled; determine the scheduling order of data packets according to the product of the injection period of the message to be scheduled and the sum of the sizes of all data packets it contains; And the state of the CQF—Packet Injection Selection Model; Step 2, Selecting data packets for injection; The information of the currently scheduled data packet is sent to the pre-built CQF-data packet injection selection model. The strategy is based on the actor-commentator architecture of deep reinforcement learning to obtain the injection scheme of the data packet. Step 3, Data Packet Injection Time Allocation; Input the current state into the actual action TSN network to obtain the basic action; During training, noise that satisfies a normal distribution is added to the basic action, and the result is cropped to the range of [0,1] as the final action; the ratio of the injection time of the current data packet to be scheduled to its injection period is taken as the continuous action, and the injection time of the data packet is obtained by combining deep reinforcement learning; Step 4, Environmental Changes and Feedback; Based on the transmission path of the current data packet and the injection time, construct a time-queue matrix that only injects the current data packet; sum the queue usage of the injected data packets with the time-queue matrix that only injects the current data packet to obtain the expected new time-queue matrix; verify the legality of the injection time based on the expected new time-queue matrix and the number of hops of the current data packet, and obtain the single-step profit based on the legality verification result. Step 5: Schedule the data packet under the CQF mechanism and deliver it to the target terminal; According to the allocated injection time, the data packets enter the network sequentially and are transmitted according to the CQF mechanism to reach the target terminal; Count the number of successfully scheduled data packets and messages, and obtain the final scheduling scheme; The method incorporates the message segmentation and mapping relationship between network layer messages and data link layer data packets into the scheduling process, adopts incremental scheduling combined with deep reinforcement learning, and uses a time-queue matrix to represent the network transmission status.

2. The multi-layer message optimization scheduling method applied to TSN networks according to claim 1, characterized in that, The CQF (Category QF) packet injection selection model adopts the actor-commentator algorithm in deep reinforcement learning and builds a two-layer network architecture: the actor network is responsible for obtaining the injection time slot of the data packet, and the commentator network is responsible for evaluating the transmission success rate of the injected data packet and setting up an experience pool to store training experience.

3. The multi-layer message optimization scheduling method applied to TSN networks according to claim 2, characterized in that, The method for constructing the CQF-packet injection selection model includes: setting up six neural networks: an actual action network, a first actual evaluation network, a second actual evaluation network, a target action network, a first target evaluation network, and a second target evaluation network; wherein, The actual action network is used to output basic actions based on the current state, the actual evaluation network is used to evaluate the benefits of the current state and actions, and the target action network and target evaluation network are used to calculate the target Q value during training. An experience pool is set up to store the experience generated during the reinforcement learning phase. The experience is represented in the form of a quadruple, including the original state, action, cumulative reward, and post-action state.

4. The multi-layer message optimization scheduling method applied to TSN networks according to claim 1, characterized in that, In step three, the state is the time-queue matrix composed of the data packets that have been scheduled; the action is a continuous variable that takes values ​​in the range [0,1], and its action space is the injection period of the current data packet.

5. The multi-layer message optimization scheduling method applied to TSN networks according to claim 1, characterized in that, The determination of the legality of the action in step four includes: first, determining whether there is a queue overflow situation in the time-queue matrix where the element is greater than 1; and second, determining whether the data packet meets its transmission deadline.

6. A method for constructing a CQF-packet injection selection model based on CQF scheduling conforming to the IEEE 802.1Q protocol, characterized in that... The steps include: Step A involves building a two-layer network architecture. An actor network and a commentator network are set up. The actor network is used to obtain the injection time slots of data packets, and the commentator network is used to evaluate the transmission success rate of the injected data packets. An experience pool is set up to store experience. When the experience pool reaches its storage limit, the previously stored experience is discarded first. Step B involves allocating time for packet injection. Input the current state into the actor network to obtain the basic action; multiply the basic action by the action space of the current data packet to obtain the injection time of the data packet; Step C: Environmental Changes and Feedback; Based on the transmission path of the current data packet to be scheduled and the injection time, construct a time-queue matrix that only injects the current data packet; sum the queue usage of the injected data packets with the time-queue matrix that only injects the current data packet to obtain the expected new time-queue matrix; verify the legality of the injection time based on the expected new time-queue matrix and the hop count of the current data packet, and obtain the single-step benefit based on the legality verification result; Step D: Experience correction and storage; The final reward is set based on the scheduling results of all data packets, and the cumulative reward for each step is determined iteratively; the original state, action, cumulative reward, and post-action state are combined into experience and stored in the experience pool. Step E involves reinforcement learning and updating. Randomly select an experience group from the experience pool, calculate the expected Q value and the actual Q value, calculate the loss and update the actual evaluation network; determine whether to perform delayed update based on the current training round and the delayed update frequency, and if necessary, update the actual action network and perform soft update on the parameters of the actual action network and the target evaluation network.

7. The CQF-packet injection selection model construction method according to claim 6, characterized in that, The construction step B specifically includes: Step B1, Obtain the basic action: Input the current state into the actual action network, obtain the output and use it as the basic action; Step B2, obtaining the noisy injection time: During the training process, noise is randomly selected according to the normal distribution and multiplied by the noise coefficient to obtain the action noise. The basic action is summed with the action noise and clipped within a preset range to obtain the action. Then, it is multiplied by the action space of the current data packet to obtain the noisy injection time of the data packet. Step B3, Obtain the noise-free injection time: When obtaining the final scheduling scheme, the basic action is directly used as the action of the data packet, and then multiplied by the action space of the current data packet to obtain the noise-free injection time of the data packet.

8. The CQF-packet injection selection model construction method according to claim 6, characterized in that, The construction step C specifically includes: Step C1: Obtain the transmission path of the current data packet, construct a time-queue matrix based on the transmission path and action, and use it as the queue usage for injecting the current data packet. Step C2: Sum the queue usage of the injected data packets with the queue usage of the data packets injected only according to the current action to obtain the expected queue usage of the newly injected data packets; and verify whether the data packet is legal to perform the current action based on the expected queue usage of the newly injected data packets and the number of hops. The criteria for determining the legality of the action include: whether there is any element greater than 1 in the expected new time-queue matrix, and whether the data packet meets the deadline requirement. Step C3, feedback on action execution benefit: If the data packet passes the legality verification of the current action, the data packet scheduling and injection are successful, and the benefit is 1; otherwise, the data packet scheduling and injection fail, and the benefit is 0. Step C4: Determine the number of successfully scheduled data packets based on the validity verification result of the data packet performing the current action, determine the number of successfully scheduled messages based on the validity verification result of all data packets allocated to the message, and determine the maximum value of the average bandwidth utilization of any queue over time within a supercycle based on the queue usage of the injected data packets.

9. The method according to claim 1 or 6, characterized in that, The final benefit of the method is set as the product of the message transmission success rate, the data packet transmission success rate, and the maximum value of the average queue bandwidth utilization over time within a supercycle.

10. The method according to claim 1 or 6, characterized in that, In the CQF mechanism, each port has eight queues, two of which are dedicated transmission queues for time-triggered messages. The control gates of these two queues are always in opposite states and switch cyclically with a time slot as the cycle.