Graph network and hierarchical decision deep reinforcement learning network traffic scheduling method

Abstract

The present application relates to the field of computer network, traffic scheduling, for proposing a kind of time sensitive network traffic scheduling algorithm, can realize TSN traffic routing time slot joint optimization scheduling, improve the maximum schedulable quantity of traffic and the real-time performance of model, and can be generalized to different scale topologies.Therefore, the time sensitive network traffic scheduling algorithm based on graph network and deep reinforcement learning, first, network topology features, traffic features and time slot occupation state are modeled to obtain state feature vector, input to the graph attention neural network with virtual node for feature extraction, obtain global feature vector and node feature vector input to the strategy network based on pointer network;Finally, the selected node feature is input to the decoding MLP network to select time slot block, finally obtain accurate time slot selection to complete the scheduling of this traffic.The present application is mainly applied to computer network traffic scheduling occasions.

Description

Technical Field

[0001] This invention belongs to the field of computer networks and traffic scheduling, specifically involving traffic scheduling methods for graph networks and hierarchical decision-making deep reinforcement learning networks. Background Technology

[0002] With the increasing demand for time-critical applications such as the Industrial Internet and autonomous driving, higher requirements are being placed on the determinism, reliability, and scalability of today's network transmissions. Traditional networks cannot meet these requirements, while Time-Sensitive Networking (TSN) has become an irreplaceable core technology in these fields due to its commitment to data transmission reliability and determinism. However, achieving deterministic data transmission in TSN relies on sophisticated scheduling tables.

[0003] To obtain the scheduling table for a network, traditional methods mainly employ solver-based approaches and manually designed heuristics. Solver-based methods can provide accurate optimal solutions, but their computation time is typically very long, increasing exponentially with the problem size, making them unsuitable for the scale and dynamic scenarios of modern networks. Manually designed heuristics, on the other hand, quickly obtain a feasible solution through heuristic rules, without traversing the entire solution space, thus significantly reducing the time required to generate the scheduling table. However, this method relies on specialized domain knowledge and lacks strong scalability.

[0004] In recent years, deep reinforcement learning methods have shown potential in the field of TSN traffic scheduling, training an agent through interaction with the environment to continuously learn how to obtain optimal solutions. However, despite significant progress, several challenges remain. First, due to the enormous computational complexity and resource costs, most existing deep reinforcement learning methods assume that traffic routing is pre-configured and do not consider joint optimization of routing and scheduling. Second, the trained models are only applicable to fixed topologies and cannot be applied to topologies of other sizes or changing network scenarios, resulting in poor scalability. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention aims to propose a time-sensitive network traffic scheduling algorithm based on deep reinforcement learning. This method can achieve joint optimization scheduling of TSN traffic routing slots, improve the maximum schedulable traffic volume and the real-time performance of the model, and can be generalized to topologies of different scales. To this end, the technical solution adopted by this invention is a time-sensitive network traffic scheduling algorithm based on graph networks and deep reinforcement learning. First, network topology features, traffic features, and slot occupancy status are modeled to obtain a state feature vector. The slot portion features in the state feature vector are extracted using a multilayer perceptron coding (MLP) network and then concatenated with other features in the state feature vector except for the slot portion features. The resulting concatenated state feature vector is input into a graph attention neural network (GAT) with virtual nodes connecting all nodes for feature extraction, yielding a global feature vector and node feature vectors. These global and node feature vectors are then input into a pointer-based network (Ptrn). In the policy network of et, the input of the policy network consists of two parts: a background vector and an input vector. The background vector is represented by the global feature vector and the initial parameter vector in the first hop node selection of the traffic. In subsequent node selection, it is represented by the global feature vector, the first hop node feature vector, and the previous hop node feature vector. All node feature vectors are masked and used as the input vector of the policy network. The policy network outputs the selection of the next hop node for the current traffic. Finally, the selected node features are input into the decoding MLP network to select the time slot block. Within the time slot block, the time slot allocation method based on periodic-urgent perception is used to obtain the accurate time slot selection to complete the scheduling of this traffic.

[0006] The specific implementation steps are as follows:

[0007] Step 1: Identify the network topology features in the original problem. Flow characteristics and time slot occupancy status Modeling the state features of reinforcement learning on a link-by-link basis, the final state feature vector is obtained. in For the first in the network The characteristics of an edge, where an edge is a physical link in the network;

[0008] Step 2: Construct the routing decision model: The routing decision model includes a feature extraction network based on GAT and a policy network based on Ptrnet. First, a virtual node V connecting all nodes is created and initialized with the average of all state feature vectors as the global feature. The feature extraction network extracts the input state features. and global features Feature extraction is performed to obtain updated node features. With global features ,in For the first Features of the strip edge Features updated by the layered graph attention neural network GAT, in the selection of the node for the first hop of the flow; global feature vector. and initial parameter vector As the background vector input to the Ptrnet-based policy network, where For a learnable weight matrix, The vector after passing through the mask matrix serves as the input vector to the PtrNet-based policy network. In the PtrNet policy network's two-layer single-head attention mechanism, the first layer is used to update the background vector, and the second layer outputs the next-hop node for the current traffic. In subsequent node selection, the node features selected in the previous step will be used... Features of source nodes splicing and combining The superposition serves as the background vector for the current step;

[0009] Step 3: Construct a time-slot hierarchical decision model: The model includes high-level decision-making in the time-slot hierarchical MLP network and manually designed low-level decision-making. The high-level decision-making is based on the node features output in Step 2. Input time slot hierarchical MLP network for time slot block The selection of time slot blocks is based on a time slot allocation method using a periodic-emergency sensing approach. The time slots within the time slot are scored, and the time slot with the highest score is selected. As the forwarding time of the next hop in the traffic;

[0010] Step 4: During the training phase, as new traffic continuously arrives, the end of this round of sampling is marked when the agent cannot successfully schedule the current traffic. The agent is the part covered and learned by the entire neural network, including the routing decision model and the time slot hierarchical decision model. The collected sequence samples are stored in the experience replay buffer and learned using the policy gradient algorithm REINFORCE. Important sample mining technology assists in updating. The routing decision model and the time slot hierarchical decision model are updated by maximizing the expectation of the reward function. During the testing phase, the network topology and traffic demand are input into the trained routing decision model and the time slot hierarchical decision model, which then output an accurate scheduling table.

[0011] In step 1, the network topology features in the original problem are... Flow characteristics and time slot occupancy status Modeling the state features of reinforcement learning on a link-by-link basis is defined as the sum of the features of all edges. The network topology model consists of a directed graph. It means that, among them, A set of nodes, representing a switch in the network. For the number of nodes, Let be the set of directed edges connecting two nodes, representing a unidirectional transmission link in the network. The number of edges, if in node and If there is a bidirectional physical link between them, then... The left side of the brackets represents the source node of the edge, and the right side represents the destination node of the edge.

[0012] Traffic transmitted in a Time-Sensitive Network (TSN) is time-triggered (TT) traffic, and each TT traffic is represented by a 5-tuple. ,in, Represents traffic The node that issued the signal, Represents traffic The destination node Represents traffic The size of the transmitted frame. Represents traffic The maximum acceptable end-to-end delay, Represents traffic The sending period represents the set of traffic in the network as... , The total number of traffic; since TT traffic has different periods, the greatest common multiple of all TT traffic periods is defined as the global period. Once a scheduling table for a global cycle is obtained, it can be repeated in any global cycle without recalculation.

[0013] Scheduling a traffic flow involves selecting the next-hop forwarding link for each flow one by one. and forwarding time on the link To reduce computational complexity, the entire global cycle is divided into several time slots of equal size. The size of the time slots is adjusted according to different application requirements. This ensures that each time slot can carry a frame with a frame length of the Maximum Transmission Unit (MTU) while avoiding significant bandwidth waste. Time slot occupancy is defined as... ,in and For any link and time slot, the value is 1 when a time slot on the link has already been allocated traffic, and 0 otherwise. That is, a time slot can only be occupied by one traffic flow. When a period is Traffic In the link On When a time slot is forwarded, the time slot occupancy status is updated:

[0014]

[0015] flow The final scheduling result is expressed as follows:

[0016] ={( ),( ),…,( )}

[0017] in This represents an edge originating from the traffic source node. This represents an edge pointing to the destination node of the traffic;

[0018] In addition, the following scheduling constraints are fundamental conditions for ensuring the correctness of scheduling results:

[0019] Single-cycle forwarding constraint

[0020] Each piece of traffic has a different cycle, and data frames are continuously forwarded according to the cycle. In order to avoid affecting the forwarding of subsequent frames, a piece of traffic must be forwarded within its own cycle time range:

[0021] in, Indicates that it will be on the link Traffic forwarded on . Indicates the actual forwarding time of the traffic;

[0022] End-to-end delay constraints

[0023] End-to-end latency refers to the total time it takes for traffic to travel from the source node to the destination node. The actual end-to-end latency of each traffic item cannot exceed its maximum acceptable end-to-end latency.

[0024] Scheduling timing constraints

[0025] Traffic travels from the source node to the destination node through multiple links, and these links have a time sequence relationship, meaning that the forwarding time of a later link should be later than that of an earlier link.

[0026]

[0027]

[0028]

[0029]

[0030] in, Indicates flow rate In the link The transmission delay experienced;

[0031] The original problem is modeled as a state feature of reinforcement learning, where the features of each edge include the following parts: (1) the distance of the current edge from the current flow destination node. distance , is the shortest distance between the current edge and the destination node and the source node. With the target node (2) The ratio of the shortest distances between the two sides; (3) The occupancy rate of the current slot. , which is the time slot occupied by the current edge. The sum of the sums and the ratio of the supercycle, where the supercycle is the least common multiple of all flow cycles. (3) Whether the current edge is an adjacent edge of the current edge or the edge of the source node and has not been visited; (4) Whether the current edge has not been fully occupied in its time slot and will not form a loop; (5) The urgency of the traffic. , representing the remaining time of the traffic and the maximum end-to-end delay. The ratio; (6) the degree of the current side. , is the ratio of the number of edges connected to the current edge to the maximum degree; (7) Time slot resource characteristics Each 1ms contains 64 time slots, forming a time slot block. The time slot occupancy rate is calculated for each time slot block and used as input to the MLP network. The time period is 32ms. The MLP network contains four fully connected layers with sizes of 32, 32, 16, and 8. The final output features are used as time slot resource features. (8) Flow cycle , which represents the current traffic cycle.

[0032] In step 2, a virtual node V connecting all nodes is first created and initialized as the average of all state feature vectors as the global feature. The feature extraction network extracts the input state features. and global features Feature extraction is performed to obtain updated node features. With global features In the selection of the first hop node for traffic, The initial parameter vector is used as the background vector for the Ptrnet-based policy network. After passing through the mask matrix, the vector is used as the input vector to the policy network. In the attention mechanism of the policy network, the first layer is used to update the background vector, and the second layer outputs the next-hop node for the current traffic. In subsequent node selection, the node features selected in the previous step will be used... Features of source nodes splicing and combining The sum is used as the background vector for the current step; specifically:

[0033] Using state feature vectors Input to a GAT-based feature extraction network, where, The dimension of the feature vector is represented by the GAT network, which consists of several graph attention layers. In each layer, a node is defined as the feature vector corresponding to a physical link in the network, and an edge is defined as the adjacency relationship of a physical link in the actual network. In addition, a virtual node V is added to connect all nodes in the graph, whose features are... Initialized as the average of the sum of the feature vectors of all other nodes, and updated jointly with other nodes, the graph attention layer of the GAT network uses an attention mechanism. Obtain the attention coefficients of the two nodes:

[0034]

[0035] in This represents the attention coefficients of nodes i and j. This represents a learnable weight matrix. Each node calculates attention coefficients with all its neighboring nodes. For subsequent calculations, these attention coefficients are normalized.

[0036] in This represents the normalized attention coefficient. Represents the normalization function. The set of first-order neighbors of node i, and the attention mechanism. The weight vector generated by a single-layer feedforward neural network is parameterized and then subjected to a nonlinear transformation using a nonlinear activation function.

[0037]

[0038] in, Represents a non-linear activation function. This represents the concatenation operation of two vectors. Finally, node i performs a weighted sum of the transformed features of all its neighboring nodes based on the obtained attention coefficients, and then performs a nonlinear transformation through a nonlinear activation function to obtain the updated features.

[0039]

[0040] in The GAT network has two layers, representing a non-linear activation function. During the update process, a multi-head attention mechanism is also used, where the two heads independently and in parallel compute the updated features, which are then concatenated in the first layer and averaged in the second and final layer.

[0041] The updated node features are obtained from the feature extraction network. With global features The input is fed into a PtrNet-based policy network. The input to the policy network consists of two parts: the background vector. The input vector and background vector are represented as follows:

[0042]

[0043] in, For learnable parameter vectors, The weight matrix is ​​a learnable matrix. and These are the node feature vectors output at time step t=1 and the previous time step, respectively. Assume the current edge is a node. Then the input vector is the node For all nodes that are considered valid actions, the input vector is obtained by applying all node features through a mask matrix. :

[0044]

[0045]

[0046] That is, among them For the mask matrix, Represents the mask vector, when the node With nodes Adjacent nodes The value is 1 if the node has not been visited, otherwise it is 0. The policy network consists of two attention mechanisms, where the first layer updates the background vector and the second layer outputs the node selection. In the first attention mechanism, the background vector... Used to calculate query vector Input vector Used to calculate key-value vectors Sum value vector ,use and After calculating the attention score, it is normalized using the Softmax function. Finally, the input vector is weighted and summed based on the attention score, and then fed into a fully connected layer to obtain the updated background vector.

[0047]

[0048]

[0049] in The dimension of the parameter vector. For learnable weight matrices, Let be the attention coefficient between the feature vector and the background vector of node i. The input is fed into the second-layer attention mechanism, where an attention score is calculated and then transformed into a probability distribution using a softmax function.

[0050]

[0051] Finally, based on the probability distribution To select the next hop node for the traffic.

[0052] In step 3, the node features output in step 2 are... Input MLP network for time slot blocks The selection process involves the following steps: after selecting the next-hop node for the traffic, the features of the selected node are input into the MLP network, and the MLP scores each time slot block. Finally, the probability distribution is normalized using the softmax function. And according to the probability distribution Output time slot block :

[0053]

[0054] Each time slot is scored using a time slot allocation method based on periodic-emergency awareness, and the time slot with the highest score is selected. The forwarding time of the next hop in the traffic flow:

[0055] First, traffic with shorter cycles requires more time slots than traffic with longer cycles, therefore, for time slots... Since subsequent time slots may be occupied and can only carry traffic for a larger period, the time slots will be allocated to the period. The load-bearing capacity is expressed as:

[0056]

[0057] in This represents the period in the set of traffic periods. Indicates the time slot on node i Whether it is occupied; if occupied, the value is 1; otherwise, the value is 0. Move the node... Upper Time Slot The periodic bearing capacity is expressed as:

[0058] in This represents a set of traffic cycles. A higher-score time slot can carry more cycles, while a lower-score time slot can only carry fewer larger time slots. Therefore, lower-score time slots are prioritized for scheduling to preserve cycle carrying capacity and thus allocate more traffic. Note that there may be time slots with similar scores, and the maximum end-to-end delay constraint of the traffic should also be considered. Therefore, the urgency of the traffic should be considered. As a second indicator, to eliminate the impact of the difference in the value range of the two sub-indicators, the periodic carrying capacity is... After normalizing the fractions and The weighted sum is the total score:

[0059]

[0060] in, and These are the weighting coefficients. This represents the total number of periods in the periodic set.

[0061] In step 4, during the training phase, as new traffic continuously arrives, the end of the current sampling round is marked when the agent cannot successfully schedule the current traffic. The collected sequence samples are stored in the experience replay buffer and learned using the policy gradient algorithm REINFORCE. Important sample mining technology assists in updating, and the routing decision model and time slot hierarchical decision model are updated by maximizing the expectation of the reward function. During the testing phase, the network topology and traffic demand are input into the pre-trained routing decision model and time slot hierarchical decision model, which then output an accurate scheduling table. The specific process is as follows:

[0062] The agent's goal is to schedule as much traffic as possible while maintaining latency constraints, so the reward function is set as follows:

[0063]

[0064]

[0065]

[0066] in This indicates whether the current traffic has been successfully scheduled. Represents the latency of the current traffic. This represents the load balancing level of the network, and its value is the difference between the time slot occupancy rate of the selected node and the average time slot occupancy rate of all nodes. , and These are the weighting coefficients;

[0067] The policy gradient algorithm REINFORCE calculates the reward value. The expectation relative to the model parameters The model is updated using gradients. The end of the current sampling round is marked when the agent fails to schedule traffic. The collected sequence samples of state, action, and reward are stored in an experience replay buffer. The agent's goal is to update the model parameters by learning from these samples to find an optimal policy. To maximize rewards Expectations:

[0068] in As the decay factor, the update formula for the enhancement algorithm REINFORCE is derived from this:

[0069]

[0070] in It's a baseline, used to reduce the variance of gradient estimation during model updates, making the training process more stable. The learning rate; in addition, important sample mining techniques are used to improve the training efficiency of the agent, and an auxiliary buffer is created. The sample with the lowest advantage in each training iteration is added to the buffer pool. At the same time These samples contain more information and are more valuable than other samples. They are crucial for the subsequent learning of routing decision models and time-slot hierarchical decision models. and The samples in the training data are uniformly sampled, so that these samples are repeatedly learned, thereby improving training efficiency.

[0071] Because it involves hierarchical actions, the probability function of the actions consists of joint probabilities. If the action at time step t If the traffic does not reach its destination node, the intelligent system updates the network status. Select the next action; otherwise, complete the scheduling of this traffic and obtain the request for the next new traffic, starting the scheduling from the source node.

[0072] The features and beneficial effects of this invention are:

[0073] (1) The method of the present invention realizes a time-sensitive network traffic scheduling framework based on deep reinforcement learning, including a routing decision model and a time slot hierarchical decision model, realizing joint optimization of routing and time slot scheduling, and improving training efficiency through important sample mining technology.

[0074] (2) A feature extraction network based on GAT is proposed, which can capture the importance of nodes by learning weights and perceive the global state of the network, learn more effectively from the network topology, improve the generalization ability of the model, and enable the trained model to be applied to topologies of different scales. The policy network based on Ptrnet adds action masking, which transforms the traditional node classification problem into a continuous node classification problem, reduces the computational complexity of the model, and improves the training efficiency and scheduling performance of the model.

[0075] (3) A time slot hierarchical decision method is proposed. In view of the huge time cost of traversing all nodes to calculate scores, the MLP network is used to select time slot blocks first, and then the precise time slot is selected within the time slot blocks by the time slot allocation method based on periodic-emergency perception. This improves the real-time performance of the model while ensuring scheduling quality. Attached Figure Description

[0076] Figure 1 : Schematic diagram of the model framework. Detailed Implementation

[0077] This invention belongs to the field of computer networks and designs an intelligent traffic scheduling method for Time-Sensitive Networks (TSNs), aiming to improve network resource utilization efficiency, real-time performance, and scalability under high reliability and deterministic requirements. It is applicable to time-critical application scenarios such as autonomous driving, telemedicine, intelligent manufacturing, and industrial Ethernet. Specifically, it involves a deep reinforcement learning-based traffic scheduling method for time-sensitive networks (TSNs) based on graph attention network feature extraction and hierarchical decision-making.

[0078] The technical solution adopted in this invention is a time-sensitive network traffic scheduling algorithm based on graph networks and deep reinforcement learning. First, network topology features, traffic features, and time slot occupancy status are modeled to obtain a state feature vector. The time slot features in this state feature vector are extracted using a multilayer perceptron coding (MLP) network and then concatenated with other features in the state feature vector except for the time slot features. The resulting concatenated state feature vector is then input into a graph attention neural network (GAT) with virtual nodes connecting all nodes for feature extraction, yielding a global feature vector and node feature vectors. These global and node feature vectors are then input into a pointer-based network (Ptrne). In the policy network of t, the input of the policy network consists of two parts: a background vector and an input vector. The background vector is represented by the global feature vector and the initial parameter vector in the first hop node selection of the traffic, and by the global feature vector, the first hop node feature vector, and the previous hop node feature vector in subsequent node selections. All node feature vectors are masked and then used as the input vector of the policy network. The policy network outputs the selection of the next hop node for the current traffic. Finally, the selected node features are input into the decoding MLP network to select time slot blocks. Within the time slot block, the time slot allocation method based on periodic-urgent perception is used to obtain the accurate time slot selection to complete the scheduling of this traffic.

[0079] The specific implementation steps are as follows:

[0080] Step 1: Identify the network topology features in the original problem. Flow characteristics and time slot occupancy status Modeling the state features of reinforcement learning on a link-by-link basis, the final state feature vector is obtained. in For the first in the network The characteristics of an edge, where an edge is a physical link in the network;

[0081] Step 2: Construct the routing decision model: The routing decision model includes a feature extraction network based on GAT and a policy network based on Ptrnet. First, a virtual node V connecting all nodes is created and initialized with the average of all state feature vectors as the global feature. The feature extraction network extracts the input state features. and global features Feature extraction is performed to obtain updated node features. With global features ,in For the first Features of the strip edge Features updated by the layered graph attention neural network GAT, in the selection of the node for the first hop of the flow; global feature vector. and initial parameter vector As the background vector input to the Ptrnet-based policy network, where For a learnable weight matrix, The vector after passing through the mask matrix serves as the input vector to the PtrNet-based policy network. In the PtrNet policy network's two-layer single-head attention mechanism, the first layer is used to update the background vector, and the second layer outputs the next-hop node for the current traffic. In subsequent node selection, the node features selected in the previous step will be used... Features of source nodes splicing and combining The superposition serves as the background vector for the current step;

[0082] Step 3: Construct a time-slot hierarchical decision model: The model includes high-level decision-making in the time-slot hierarchical MLP network and manually designed low-level decision-making. The high-level decision-making is based on the node features output in Step 2. Input time slot hierarchical MLP network for time slot block The selection of time slot blocks is based on a time slot allocation method using a periodic-emergency sensing approach. The time slots within the time slot are scored, and the time slot with the highest score is selected. As the forwarding time of the next hop in the traffic;

[0083] Step 4: During the training phase, as new traffic continuously arrives, the end of this round of sampling is marked when the agent cannot successfully schedule the current traffic. The agent is the part covered and learned by the entire neural network, including the routing decision model and the time slot hierarchical decision model. The collected sequence samples are stored in the experience replay buffer and learned using the policy gradient algorithm REINFORCE. Important sample mining technology assists in updating. The routing decision model and the time slot hierarchical decision model are updated by maximizing the expectation of the reward function. During the testing phase, the network topology and traffic demand are input into the trained routing decision model and the time slot hierarchical decision model, which then output an accurate scheduling table.

[0084] In step 1, the network topology features in the original problem are... Flow characteristics and time slot occupancy status Modeling the state features of reinforcement learning on a link-by-link basis is defined as the sum of the features of all edges. The network topology model consists of a directed graph. It means that, among them, A set of nodes, representing a switch in the network. For the number of nodes, Let be the set of directed edges connecting two nodes, representing a unidirectional transmission link in the network. The number of edges, if in node and If there is a bidirectional physical link between them, then... The left side of the brackets represents the source node of the edge, and the right side represents the destination node of the edge.

[0085] Traffic transmitted in a Time-Sensitive Network (TSN) is time-triggered (TT) traffic, and each TT traffic is represented by a 5-tuple. ,in, Represents traffic The node that issued the signal, Represents traffic The destination node Represents traffic The size of the transmitted frame. Represents traffic The maximum acceptable end-to-end delay, Represents traffic The sending period represents the set of traffic in the network as... , The total number of traffic; since TT traffic has different periods, the greatest common multiple of all TT traffic periods is defined as the global period. Once a scheduling table for a global cycle is obtained, it can be repeated in any global cycle without recalculation.

[0086] Scheduling a traffic flow involves selecting the next-hop forwarding link for each flow one by one. and forwarding time on the link To reduce computational complexity, the entire global cycle is divided into several time slots of equal size. The size of the time slots is adjusted according to different application requirements. This ensures that each time slot can carry a frame with a frame length of the Maximum Transmission Unit (MTU) while avoiding significant bandwidth waste. Time slot occupancy is defined as... ,in and For any link and time slot, the value is 1 when a time slot on the link has already been allocated traffic, and 0 otherwise. That is, a time slot can only be occupied by one traffic flow. When a period is Traffic In the link On When a time slot is forwarded, the time slot occupancy status is updated:

[0087]

[0088] flow The final scheduling result is expressed as follows:

[0089] ={( ),( ),…,( )}

[0090] in This represents an edge originating from the traffic source node. This represents an edge pointing to the destination node of the traffic;

[0091] In addition, the following scheduling constraints are fundamental conditions for ensuring the correctness of scheduling results:

[0092] Single-cycle forwarding constraint

[0093] Each piece of traffic has a different cycle, and data frames are continuously forwarded according to the cycle. In order to avoid affecting the forwarding of subsequent frames, a piece of traffic must be forwarded within its own cycle time range:

[0094] in, Indicates that it will be on the link Traffic forwarded on . Indicates the actual forwarding time of the traffic;

[0095] End-to-end delay constraints

[0096] End-to-end latency refers to the total time it takes for traffic to travel from the source node to the destination node. The actual end-to-end latency of each traffic item cannot exceed its maximum acceptable end-to-end latency.

[0097] Scheduling timing constraints

[0098] Traffic travels from the source node to the destination node through multiple links, and these links have a time sequence relationship, meaning that the forwarding time of a later link should be later than that of an earlier link.

[0099]

[0100]

[0101]

[0102]

[0103] in, Indicates flow rate In the link The transmission delay experienced;

[0104] The original problem is modeled as a state feature of reinforcement learning, where the features of each edge include the following parts: (1) the distance of the current edge from the current flow destination node. distance , is the shortest distance between the current edge and the destination node and the source node. With the target node (2) The ratio of the shortest distances between the two sides; (3) The occupancy rate of the current slot. , which is the time slot occupied by the current edge. The sum of the sums and the ratio of the supercycle, where the supercycle is the least common multiple of all flow cycles. (3) Whether the current edge is an adjacent edge of the current edge or the edge of the source node and has not been visited; (4) Whether the current edge has not been fully occupied in its time slot and will not form a loop; (5) The urgency of the traffic. , representing the remaining time of the traffic and the maximum end-to-end delay. The ratio; (6) the degree of the current side. , is the ratio of the number of edges connected to the current edge to the maximum degree; (7) Time slot resource characteristics Each 1ms contains 64 time slots, forming a time slot block. The time slot occupancy rate is calculated for each time slot block and used as input to the MLP network. The time period is 32ms. The MLP network contains four fully connected layers with sizes of 32, 32, 16, and 8. The final output features are used as time slot resource features. (8) Flow cycle , which represents the current traffic cycle.

[0105] In step 2, a virtual node V connecting all nodes is first created and initialized as the average of all state feature vectors as the global feature. The feature extraction network extracts the input state features. and global features Feature extraction is performed to obtain updated node features. With global features In the selection of the first hop node for traffic, The initial parameter vector is used as the background vector for the Ptrnet-based policy network. After passing through the mask matrix, the vector is used as the input vector to the policy network. In the attention mechanism of the policy network, the first layer is used to update the background vector, and the second layer outputs the next-hop node for the current traffic. In subsequent node selection, the node features selected in the previous step will be used... Features of source nodes splicing and combining The sum is used as the background vector for the current step; specifically:

[0106] Using state feature vectors Input to a GAT-based feature extraction network, where, The dimension of the feature vector is represented by the GAT network, which consists of several graph attention layers. In each layer, a node is defined as the feature vector corresponding to a physical link in the network, and an edge is defined as the adjacency relationship of a physical link in the actual network. In addition, a virtual node V is added to connect all nodes in the graph, whose features are... Initialized as the average of the sum of the feature vectors of all other nodes, and updated jointly with other nodes, the graph attention layer of the GAT network uses an attention mechanism. Obtain the attention coefficients of the two nodes:

[0107]

[0108] in This represents the attention coefficients of nodes i and j. This represents a learnable weight matrix. Each node calculates attention coefficients with all its neighboring nodes. For subsequent calculations, these attention coefficients are normalized.

[0109] in This represents the normalized attention coefficient. Represents the normalization function. The set of first-order neighbors of node i, and the attention mechanism. The weight vector generated by a single-layer feedforward neural network is parameterized and then subjected to a nonlinear transformation using a nonlinear activation function.

[0110]

[0111] in, Represents a non-linear activation function. This represents the concatenation operation of two vectors. Finally, node i performs a weighted sum of the transformed features of all its neighboring nodes based on the obtained attention coefficients, and then performs a nonlinear transformation through a nonlinear activation function to obtain the updated features.

[0112]

[0113] in The GAT network has two layers, representing a non-linear activation function. During the update process, a multi-head attention mechanism is also used, where the two heads independently and in parallel compute the updated features, which are then concatenated in the first layer and averaged in the second and final layer.

[0114] The updated node features are obtained from the feature extraction network. With global features The input is fed into a PtrNet-based policy network. The input to the policy network consists of two parts: the background vector. The input vector and background vector are represented as follows:

[0115]

[0116] in, For learnable parameter vectors, The weight matrix is ​​a learnable matrix. and These are the node feature vectors output at time step t=1 and the previous time step, respectively. Assume the current edge is a node. Then the input vector is the node For all nodes that are considered valid actions, the input vector is obtained by applying all node features through a mask matrix. :

[0117]

[0118]

[0119] That is, among them For the mask matrix, Represents the mask vector, when the node With nodes Adjacent nodes The value is 1 if the node has not been visited, otherwise it is 0. The policy network consists of two attention mechanisms, where the first layer updates the background vector and the second layer outputs the node selection. In the first attention mechanism, the background vector... Used to calculate query vector Input vector Used to calculate key-value vectors Sum value vector ,use and After calculating the attention score, it is normalized using the Softmax function. Finally, the input vector is weighted and summed based on the attention score, and then fed into a fully connected layer to obtain the updated background vector.

[0120]

[0121]

[0122] in The dimension of the parameter vector. For learnable weight matrices, Let be the attention coefficient between the feature vector and the background vector of node i. The input is fed into the second-layer attention mechanism, where an attention score is calculated and then transformed into a probability distribution using a softmax function.

[0123]

[0124] Finally, based on the probability distribution To select the next hop node for the traffic.

[0125] In step 3, the node features output in step 2 are... Input MLP network for time slot blocks The selection process involves the following steps: after selecting the next-hop node for the traffic, the features of the selected node are input into the MLP network, and the MLP scores each time slot block. Finally, the probability distribution is normalized using the softmax function. And according to the probability distribution Output time slot block :

[0126]

[0127] Each time slot is scored using a time slot allocation method based on periodic-emergency awareness, and the time slot with the highest score is selected. The forwarding time of the next hop in the traffic flow:

[0128] First, traffic with shorter cycles requires more time slots than traffic with longer cycles, therefore, for time slots... Since subsequent time slots may be occupied and can only carry traffic for a larger period, the time slots will be allocated to the period. The load-bearing capacity is expressed as:

[0129]

[0130] in This represents the period in the set of traffic periods. Indicates the time slot on node i Whether it is occupied; if occupied, the value is 1; otherwise, the value is 0. Move the node... Upper Time Slot The periodic bearing capacity is expressed as:

[0131] in This represents a set of traffic cycles. A higher-score time slot can carry more cycles, while a lower-score time slot can only carry fewer larger time slots. Therefore, lower-score time slots are prioritized for scheduling to preserve cycle carrying capacity and thus allocate more traffic. Note that there may be time slots with similar scores, and the maximum end-to-end delay constraint of the traffic should also be considered. Therefore, the urgency of the traffic should be considered. As a second indicator, to eliminate the impact of the difference in the value range of the two sub-indicators, the periodic carrying capacity is... After normalizing the fractions and The weighted sum is the total score:

[0132]

[0133] in, and These are the weighting coefficients. This represents the total number of periods in the periodic set.

[0134] In step 4, during the training phase, as new traffic continuously arrives, the end of the current sampling round is marked when the agent cannot successfully schedule the current traffic. The collected sequence samples are stored in the experience replay buffer and learned using the policy gradient algorithm REINFORCE. Important sample mining technology assists in updating, and the routing decision model and time slot hierarchical decision model are updated by maximizing the expectation of the reward function. During the testing phase, the network topology and traffic demand are input into the pre-trained routing decision model and time slot hierarchical decision model, which then output an accurate scheduling table. The specific process is as follows:

[0135] The agent's goal is to schedule as much traffic as possible while maintaining latency constraints, so the reward function is set as follows:

[0136]

[0137]

[0138]

[0139] in This indicates whether the current traffic has been successfully scheduled. Represents the latency of the current traffic. This represents the load balancing level of the network, and its value is the difference between the time slot occupancy rate of the selected node and the average time slot occupancy rate of all nodes. , and These are the weighting coefficients;

[0140] The policy gradient algorithm REINFORCE calculates the reward value. The expectation relative to the model parameters The model is updated using gradients. The end of the current sampling round is marked when the agent fails to schedule traffic. The collected sequence samples of state, action, and reward are stored in an experience replay buffer. The agent's goal is to update the model parameters by learning from these samples to find an optimal policy. To maximize rewards Expectations:

[0141] in As the decay factor, the update formula for the enhancement algorithm REINFORCE is derived from this:

[0142]

[0143] in It's a baseline, used to reduce the variance of gradient estimation during model updates, making the training process more stable. The learning rate; in addition, important sample mining techniques are used to improve the training efficiency of the agent, and an auxiliary buffer is created. The sample with the lowest advantage in each training iteration is added to the buffer pool. At the same time These samples contain more information and are more valuable than other samples. They are crucial for the subsequent learning of routing decision models and time-slot hierarchical decision models. and The samples in the training data are uniformly sampled, so that these samples are repeatedly learned, thereby improving training efficiency.

[0144] Because it involves hierarchical actions, the probability function of the actions consists of joint probabilities. If the action at time step t If the traffic does not reach its destination node, the intelligent system updates the network status. Select the next action; otherwise, complete the scheduling of this traffic and obtain the request for the next new traffic, starting the scheduling from the source node.

[0145] The present invention will be further described in detail below with reference to the accompanying drawings and specific examples.

[0146] Step 1: Identify the network topology features in the original problem. Flow characteristics and time slot occupancy status Modeling state features for reinforcement learning on a link-by-link basis .

[0147] Step 2, Routing Decision Model: The model includes a feature extraction network based on GAT and a policy network based on Ptrnet. First, a virtual node V connecting all nodes is created, initialized with the average of all state feature vectors as the global feature. The feature extraction network extracts the input state features. and global features Feature extraction is performed to obtain updated node features. With global features In the selection of the first hop node for traffic, The initial parameter vector is used as the background vector for the Ptrnet-based policy network. The masked matrix is ​​then used as input to the policy network. Through an attention mechanism, the first layer updates the background vector, and the second layer outputs the next-hop node for the current traffic. In subsequent node selection, the features of the nodes selected in the previous step will be used... Features of source nodes splicing and combining The sum is used as the background vector for the current step.

[0148] Step 3, Time-Slot Hierarchical Decision Model: The model includes high-level decision-making using a Multilayer Perceptron (MLP) network and manually designed low-level decision-making. The node features output from Step 2 are then used... Input MLP network for time slot blocks The selection of time slots, and then the allocation of time slots based on the periodic-emergency sensing method, are performed on the time slot blocks. The time slots within the time slot are scored, and the time slot with the highest score is selected. As the forwarding time of the next hop in traffic.

[0149] Step 4: During the training phase, as new traffic continuously arrives, the sampling cycle ends when the agent cannot successfully schedule the current traffic. The collected sequence samples are stored in the experience replay buffer, and the REINFORCE algorithm is used for learning. Important sample mining techniques are used to assist in updates, and the model is updated by maximizing the expected value of the reward function. During the testing phase, the network topology and traffic requirements are input into the trained model, and the model outputs an accurate scheduling table.

[0150] In step 1, the network topology features in the original problem are... Flow characteristics and time slot occupancy status Modeling the state features of reinforcement learning on a link-by-link basis is defined as the sum of the features of all edges. The problem model corresponds to the network in reality. For example, switches, computers and mobile phones are nodes, and the connecting links are edges. Similarly, graph neural networks also have nodes and edges. Each node has its own characteristics, and the edges connect the nodes. Therefore, the nodes and edges here correspond to graph networks. The adjacency relationship between the edges in the network and the physical links in the network corresponds to the real network. The edges in the network are connected to the physical links in the network. The network topology features, traffic features and time slot occupancy status are modeled to obtain the state feature vector. Specifically, the original problem is modeled as the state features of reinforcement learning. The features of each edge include the following parts: (1) the distance of the current edge from the current traffic destination node. , is the ratio of the shortest distance between the current edge and the destination node to the shortest distance between the source node and the destination node. (2) Time slot occupancy rate of the current edge. , which is the ratio of the time slot occupied by the current edge to the timeout, where the timeout is the least common multiple of all flow cycles. (3) Whether the current edge is an adjacent edge of the current edge or the edge of the source node and has not been visited. (4) Whether the current edge's time slot is not fully occupied and will not form a loop. (5) The urgency of the traffic. , is the ratio of the remaining time of the flow to the maximum end-to-end delay. (6) Degree of the current edge. , which is the ratio of the number of edges connected to the current edge to the maximum degree. (7) Time slot resource characteristics Each 1ms contains 64 time slots, forming a time slot block. The time slot occupancy rate is calculated for each time slot block and used as input to the MLP network. Assuming a timeout of 32ms, the MLP network contains four fully connected layers with sizes of 32, 32, 16, and 8. The final output features serve as the time slot resource features. (8) Flow cycle , which represents the current traffic cycle.

[0151] In order to enable the agent to learn better, the features in the original model are not used as input. Instead, a certain design is made, which is the final state features. That is, the original problem is modeled as the state features of reinforcement learning.

[0152] The network topology model consists of a directed graph. It means that, among them, A set of nodes, representing a switch in the network. For the number of nodes, Let be the set of directed edges connecting two nodes, representing a unidirectional transmission link in the network. The number of edges, if in node and If there is a bidirectional physical link between them, then... The left side of the brackets represents the source node of the edge, and the right side represents the destination node of the edge.

[0153] Traffic transmitted in a Time-Sensitive Network (TSN) is time-triggered (TT) traffic, and each TT traffic is represented by a 5-tuple. ,in, Represents traffic The node that issued the signal, Represents traffic The destination node Represents traffic The size of the transmitted frame. Represents traffic The maximum acceptable end-to-end delay, Represents traffic The sending period. The set of traffic in the network is represented as... , The total number of traffic; since TT traffic has different periods, the greatest common multiple of all TT traffic periods is defined as the global period. Once a global scheduling table is obtained, it can be repeated in any global cycle without recalculation.

[0154] Scheduling a traffic flow essentially involves selecting the next-hop forwarding link for each flow one by one. and forwarding time on the link To reduce computational complexity, the entire global cycle is divided into several time slots of equal size. The size of the time slots is adjusted according to different application requirements. This paper sets the size of the time slots to [value missing]. Soon The time slots are divided into 64 equal segments, ensuring that each segment can carry a frame with a maximum transmission unit (MTU) length while avoiding significant bandwidth waste. The time slot occupancy is defined as follows: ,in and For any link and time slot, the value is 1 when a time slot on the link has already been allocated traffic, and 0 otherwise. That is, a time slot can only be occupied by one traffic flow. When a period is Traffic In the link On When a time slot is forwarded, the time slot occupancy status is updated:

[0155]

[0156] flow The final scheduling result is expressed as follows:

[0157] ={( ),( ),…,( )}

[0158] in This represents an edge originating from the traffic source node. This represents an edge pointing to the destination node of the traffic.

[0159] In step 2, a virtual node V connecting all nodes is first created and initialized as the average of all state feature vectors as the global feature. The feature extraction network extracts the input state features. and global features Feature extraction is performed to obtain updated node features. With global features In the selection of the first hop node for traffic, The initial parameter vector is used as the background vector for the Ptrnet-based policy network. The masked matrix is ​​then used as input to the policy network. Through an attention mechanism, the first layer updates the background vector, and the second layer outputs the next-hop node for the current traffic. That is, the edge pointing to the destination node of the traffic. In subsequent node selection, the features of the nodes selected in the previous step will be used... Features of source nodes splicing and combining The sum is used as the background vector for the current step. In reinforcement learning, t=0 and t=n are both time steps.

[0160] Using state feature vectors Input to a GAT-based feature extraction network, where, This represents the dimension of the feature vector. The GAT network consists of several graph attention layers. In each layer, a node is defined as the feature vector corresponding to a physical link in the network, and an edge is defined as the adjacency relationship of a physical link in the actual network. In addition, a virtual node V connecting all nodes in the graph is added, whose features... It is initialized as the average of the sum of the feature vectors of all other nodes and updated together with other nodes. The graph attention layer uses an attention mechanism. Obtain the attention coefficients of the two nodes:

[0161]

[0162] in This represents the attention coefficients of nodes i and j. This represents the learnable weight matrix. Each node calculates attention coefficients with all its neighboring nodes. For subsequent calculations, these attention coefficients are normalized.

[0163] in This represents the normalized attention coefficient. Represents the normalization function. Let represent the set of first-order neighbors of node i. Attention mechanism. The weight vector can be parameterized by a single-layer feedforward neural network and transformed nonlinearly using a nonlinear activation function.

[0164]

[0165] in, Represents a non-linear activation function. This represents the concatenation operation of two vectors. Finally, node i performs a weighted sum of the transformed features of all its neighboring nodes based on the obtained attention coefficients, and then performs a non-linear transformation through a non-linear activation function to obtain its updated features:

[0166]

[0167] in This represents a non-linear activation function. The GAT network has two layers and employs a multi-head attention mechanism during the update process. The two heads independently and in parallel compute the updated features, which are then concatenated in the first layer and averaged in the second and final layer.

[0168] The updated node features are obtained from the feature extraction network. With global features The input is fed into a PtrNet-based policy network. The input to the policy network consists of two parts: the background vector and the background vector. And the input vector. The background vector is represented as:

[0169]

[0170] in, For learnable parameter vectors, The weight matrix is ​​a learnable matrix. and These are the node feature vectors output at time step t=1 and the previous time step, respectively. Assume the current edge is a node. Then the input vector is the node For all nodes that can be used as valid actions, obtain the input vector by applying the features of all nodes through a mask matrix. :

[0171]

[0172]

[0173] That is, among them For the mask matrix, Represents the mask vector, when the node With nodes Adjacent nodes The value is 1 if the node has not been visited; otherwise, the value is 0. The policy network consists of two attention mechanisms: the first layer updates the background vector, and the second layer outputs the node selection. In the first attention mechanism, the background vector... Used to calculate query vector Input vector Used to calculate key-value vectors Sum value vector ,use and After calculating the attention score, it is normalized using the Softmax function. Finally, the input vector is weighted and summed based on the attention score, and then fed into a fully connected layer to obtain the updated background vector.

[0174]

[0175]

[0176] in The dimension of the parameter vector. For learnable weight matrices, Let be the attention coefficient between the feature vector and the background vector of node i. The input is fed into the second-layer attention mechanism, where an attention score is calculated and then transformed into a probability distribution using a softmax function. :

[0177]

[0178] Finally, based on the probability distribution To select the next hop node for the traffic.

[0179] In step 3, the node features output in step 2 are... Input MLP network for time slot blocks The selection of time slots, and then the allocation of time slots based on the periodic-emergency sensing method, are performed on the time slot blocks. The time slots within the time slot are scored, and the time slot with the highest score is selected. Forwarding time as the next hop in traffic .

[0180] Simply using the occupancy status of all time slots as input to the policy network results in an unacceptably large action space with unacceptable computational costs. Hierarchical decision-making decomposes the original single action into coarse-grained high-level decisions and precise low-level decisions, thereby reducing computational complexity and alleviating the huge action space. In the high-level decision-making part, all time slots are equally divided into multiple time slot blocks, and an MLP network is used to select time slots from these blocks, rather than all time slots, thus limiting the selection of subsequent time slots to a finite range. In the low-level decision-making part, the problem scale is smaller than the high-level problem, the need for intelligent selection is low, and it would increase unnecessary computational complexity and model training difficulty. Therefore, a rule-based algorithm, namely a periodic-emergency perception-based time slot allocation method, is designed to select precise time slots, improving decision accuracy while ensuring real-time performance.

[0181] After selecting the next-hop node for the traffic, the features of the selected node are input into the MLP network, and the MLP scores each time slot block. Finally, the probability distribution is normalized using the softmax function. And based on its output time slot block :

[0182]

[0183] The selection of time slot blocks, as a high-level decision in time slot selection, limits the selection of subsequent time slots to a finite range. This eliminates the need for manually designed rule algorithms to calculate all time slots, significantly improving the model's real-time performance. Within each selected time slot block, we score each time slot using a time slot allocation method based on periodic-emergency awareness.

[0184] First, traffic with shorter cycles requires more time slots than traffic with longer cycles, therefore, for time slots... Because subsequent time slots may be occupied, they can only carry traffic for a larger period. Therefore, the time slots are assigned to the period... The load-bearing capacity is expressed as:

[0185]

[0186] in This represents the period in the set of traffic periods. Indicates the time slot on node i Whether it is occupied; if occupied, the value is 1; otherwise, the value is 0. Move the node... Upper Time Slot The periodic bearing capacity is expressed as:

[0187] in This represents the set of traffic cycles. High-score time slots can carry more cycles, while low-score time slots can only carry fewer larger time slots. Therefore, this method prioritizes scheduling low-score time slots to maintain cycle carrying capacity and thus schedule more traffic. Note that there may be time slots with similar scores, and the maximum end-to-end delay constraint of the traffic should also be considered. Therefore, we prioritize the urgency of the traffic. As a second indicator, to eliminate the impact of the difference in the value range of the two sub-indicators, the periodic carrying capacity is... After normalizing the fractions and The weighted sum is the total score:

[0188]

[0189] in, and These are the weighting coefficients. This represents the total number of periods in the periodic set.

[0190] In step 4, during the training phase, as new traffic continuously arrives, the end of the current sampling round is marked when the agent cannot successfully schedule the current traffic. The collected sequence samples are stored in the experience replay buffer, and the REINFORCE algorithm is used for learning. Important sample mining techniques assist in updating the model, which is then updated by maximizing the expected value of the reward function. During the testing phase, the network topology and traffic requirements are input into the trained model, which outputs an accurate scheduling table. The specific process is as follows:

[0191] The agent's goal is to schedule as much traffic as possible while maintaining latency constraints, so the reward function is set as follows:

[0192]

[0193]

[0194]

[0195] in This indicates whether the current traffic has been successfully scheduled. Represents the latency of the current traffic. This represents the load balancing level of the network, and its value is the difference between the time slot occupancy rate of the selected node and the average time slot occupancy rate of all nodes. , and These are the weighting coefficients.

[0196] The REINFORCE algorithm calculates the reward value. The expectation relative to the model parameters The gradient is used to update the model. The end of the current sampling round is marked when the agent fails to schedule traffic. The collected sequence samples of state, action, and reward are stored in the experience replay buffer. The agent's goal is to update the model parameters by learning from these samples to find an optimal policy. To maximize rewards Expectations:

[0197] in As the decay factor, the update formula for the REINFORCE algorithm is derived from this:

[0198]

[0199] in It's a baseline, used to reduce the variance of gradient estimation during model updates, making the training process more stable. The learning rate is used. In addition, important sample mining techniques are employed to improve the training efficiency of the agent. An auxiliary buffer is created. The sample with the lowest advantage in each training iteration is added to the buffer pool. At the same time These samples contain more information and are more valuable than other samples. In subsequent model learning, [they will be used to...]. and The samples in the training data are uniformly sampled, so that these samples are repeatedly learned, thereby improving training efficiency.

[0200] Because it involves hierarchical actions, the probability function of the actions consists of joint probabilities. If the action at time step t If the traffic does not reach its destination node, the intelligent system updates the network status. Select the next action; otherwise, complete the scheduling of this traffic and obtain the request for the next new traffic, starting the scheduling from the source node.

[0201] The complete training process of this invention is as follows:

[0202] a. Data Loading: Input network topology features and a set of traffic containing attributes and time slot occupancy status .

[0203] b. Parameter Initialization: Initialize network parameters Experience replay buffer and auxiliary buffer Transforming the input data into the initial state of the reinforcement learning environment. This marks the beginning of the first iteration of the intelligent agent.

[0204] c. Feature Extraction: Create a virtual node V and initialize features. The feature extraction network based on GAT extracts the state features of the input. and global features Feature extraction is performed to obtain updated node features. With global features .

[0205] d. Action selection: In the selection of the first hop node for the traffic, The initial parameter vector is used as the background vector for the Ptrnet-based policy network. After masking matrix Then used as input vector Output the next hop node for the current traffic. In subsequent node selection, the features of the nodes selected in the previous step will be used... Features of source nodes splicing and combining The sum is used as the background vector for the current step. Output node features. Input MLP network for time slot blocks The selection of time slots, and then the allocation of time slots based on the periodic-emergency sensing method, are performed on the time slot blocks. The time slots within the time slot are scored, and the time slot with the highest score is selected. As the forwarding time of the next hop in traffic.

[0206] e. Parameter Update: When the agent fails to schedule the current traffic, it marks the end of this round of sampling, and the collected samples are stored in the experience replay buffer. and auxiliary buffer In this model, the REINFORCE algorithm is used to learn from the samples, and important sample mining techniques are used to assist in updating the model parameters to maximize the reward. The expected outcome. After updating, repeat the above process for the next round of training, iterating until the maximum number of training rounds is reached. Save the training model.

[0207] The testing process uses a specified set of topologies and a randomly generated set of traffic as input to the model. The model schedules traffic one by one and calculates latency, bandwidth utilization, the number of scheduled traffic flows, and the time taken. The test terminates when traffic scheduling fails, and calculates the maximum number of schedulable flows, bandwidth utilization, total scheduling time, and average latency to measure model performance.

[0208] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A time-sensitive network traffic scheduling algorithm based on graph networks and deep reinforcement learning, characterized in that, firstly, the network topology features, traffic features, and time slot occupancy status are modeled to obtain a state feature vector, wherein, The time slot features in the state feature vector are extracted by a multilayer perceptron coding (MLP) network and then concatenated with other features in the state feature vector except for the time slot features. The resulting concatenated state feature vector is input into a graph attention neural network (GAT) with virtual nodes connecting all nodes for feature extraction, yielding a global feature vector and node feature vectors. These global and node feature vectors are then input into a policy network based on a pointer network (Ptrnet). The policy network's input consists of two parts: a background vector and an input vector. The background vector is represented by the global feature vector and the initial parameter vector in the first hop node selection of the traffic flow, and by the global feature vector, the first hop node feature vector, and the previous hop node feature vector in subsequent node selections. All node feature vectors are masked and used as the input vector to the policy network, which outputs the next hop node selection for the current traffic flow. Finally, the selected node features are input into a decoding (MLP) network for time slot block selection. Within the time slot block, a time slot allocation method based on periodic-urgent awareness is used to obtain the precise time slot selection to complete the scheduling of this traffic flow.

2. The time-sensitive network traffic scheduling algorithm based on graph networks and deep reinforcement learning as described in claim 1, characterized in that, The specific implementation steps are as follows: Step 1: Identify the network topology features in the original problem. Flow characteristics and time slot occupancy status Modeling the state features of reinforcement learning on a link-by-link basis, the final state feature vector is obtained. in For the first in the network The characteristics of an edge, where an edge is a physical link in the network; Step 2: Construct the routing decision model: The routing decision model includes a feature extraction network based on GAT and a policy network based on Ptrnet. First, a virtual node V connecting all nodes is created and initialized with the average of all state feature vectors as the global feature. The feature extraction network extracts the input state features. and global features Feature extraction is performed to obtain updated node features. With global features ,in For the first Features of the strip edge Features updated by the layered graph attention neural network GAT, in the selection of the node for the first hop of the flow; global feature vector. and initial parameter vector As the background vector input to the Ptrnet-based policy network, where For a learnable weight matrix, The vector after passing through the mask matrix serves as the input vector to the PtrNet-based policy network. In the PtrNet policy network's two-layer single-head attention mechanism, the first layer is used to update the background vector, and the second layer outputs the next-hop node for the current traffic. In subsequent node selection, the node features selected in the previous step will be used... Features of source nodes splicing and combining The superposition serves as the background vector for the current step; Step 3: Construct a time-slot hierarchical decision model: The model includes high-level decision-making in the time-slot hierarchical MLP network and manually designed low-level decision-making. The high-level decision-making is based on the node features output in Step 2. Input time slot hierarchical MLP network for time slot block The selection of time slot blocks is based on a time slot allocation method using a periodic-emergency sensing approach. The time slots within the time slot are scored, and the time slot with the highest score is selected. As the forwarding time of the next hop in the traffic; Step 4: During the training phase, as new traffic continuously arrives, the end of this round of sampling is marked when the agent cannot successfully schedule the current traffic. The agent is the part covered and learned by the entire neural network, including the routing decision model and the time slot hierarchical decision model. The collected sequence samples are stored in the experience replay buffer and learned using the policy gradient algorithm REINFORCE. Important sample mining technology assists in updating. The routing decision model and the time slot hierarchical decision model are updated by maximizing the expectation of the reward function. During the testing phase, the network topology and traffic demand are input into the trained routing decision model and the time slot hierarchical decision model, which then output an accurate scheduling table.

3. The time-sensitive network traffic scheduling algorithm based on graph networks and deep reinforcement learning as described in claim 1, characterized in that, In step 1, the network topology features in the original problem are... Flow characteristics and time slot occupancy status Modeling the state features of reinforcement learning on a link-by-link basis is defined as the sum of the features of all edges. The network topology model consists of a directed graph. It means that, among them, A set of nodes, representing a switch in the network. For the number of nodes, Let be the set of directed edges connecting two nodes, representing a unidirectional transmission link in the network. The number of edges, if in node and If there is a bidirectional physical link between them, then... The left side of the brackets represents the source node of the edge, and the right side represents the destination node of the edge. Traffic transmitted in a Time-Sensitive Network (TSN) is time-triggered (TT) traffic, and each TT traffic is represented by a 5-tuple. ,in, Represents traffic The node that sends out, Represents traffic The destination node Represents traffic The size of the transmitted frame. Represents traffic The maximum acceptable end-to-end delay, Represents traffic The sending period represents the set of traffic in the network as... , The total number of traffic; since TT traffic has different periods, the greatest common multiple of all TT traffic periods is defined as the global period. Once a global scheduling table is obtained, it can be repeated in any global cycle without recalculation. Scheduling a traffic flow involves selecting the next-hop forwarding link for each flow one by one. and forwarding time on the link To reduce computational complexity, the entire global cycle is divided into several time slots of equal size. The size of the time slots is adjusted according to different application requirements. This ensures that each time slot can carry a frame with a frame length of the Maximum Transmission Unit (MTU) while avoiding significant bandwidth waste. Time slot occupancy is defined as... ,in and For any link and time slot, the value is 1 when a time slot on the link has already been allocated traffic, and 0 otherwise. That is, a time slot can only be occupied by one traffic flow. When a period is Traffic In the link On When a time slot is forwarded, the time slot occupancy status is updated: flow The final scheduling result is expressed as follows: ={( ),( ),…,( )} in This represents an edge originating from the traffic source node. This represents an edge pointing to the destination node of the traffic; In addition, the following scheduling constraints are fundamental conditions for ensuring the correctness of scheduling results: Single-cycle forwarding constraint Each piece of traffic has a different cycle, and data frames are continuously forwarded according to the cycle. In order to avoid affecting the forwarding of subsequent frames, a piece of traffic must be forwarded within its own cycle time range: in, Indicates that it will be on the link Traffic forwarded on , Indicates the actual forwarding time of the traffic; End-to-end delay constraints End-to-end latency refers to the total time it takes for traffic to travel from the source node to the destination node. The actual end-to-end latency of each traffic item cannot exceed its maximum acceptable end-to-end latency. Scheduling timing constraints Traffic travels from the source node to the destination node through multiple links, and these links have a time sequence relationship, meaning that the forwarding time of a later link should be later than that of an earlier link. in, Indicates flow rate In the link The transmission delay experienced; The original problem is modeled as a state feature of reinforcement learning, where the features of each edge include the following parts: (1) the distance of the current edge from the current flow destination node. distance , is the shortest distance between the current edge and the destination node and the source node. With the target node (2) The ratio of the shortest distances between the two sides; (3) The time slot occupancy rate of the current side. , which is the time slot occupied by the current edge. The sum of the sums and the ratio of the supercycle, where the supercycle is the least common multiple of all flow cycles. (3) Whether the current edge is an adjacent edge of the current edge or the edge of the source node and has not been visited; (4) Whether the current edge has not been fully occupied in its time slot and will not form a loop; (5) The urgency of the traffic. , representing the remaining time of the traffic and the maximum end-to-end delay. The ratio; (6) the degree of the current side. , is the ratio of the number of edges connected to the current edge to the maximum degree; (7) Time slot resource characteristics Each 1ms contains 64 time slots, forming a time slot block. The time slot occupancy rate is calculated for each time slot block and used as input to the MLP network. The time period is 32ms. The MLP network contains four fully connected layers with sizes of 32, 32, 16, and 8. The final output features are used as time slot resource features. (8) Flow cycle , which represents the current traffic cycle.

4. The time-sensitive network traffic scheduling algorithm based on graph networks and deep reinforcement learning as described in claim 3, characterized in that, In step 2, a virtual node V connecting all nodes is first created and initialized as the average of all state feature vectors as the global feature. The feature extraction network extracts the input state features. and global features Feature extraction is performed to obtain updated node features. With global features In the selection of the first hop node for traffic, The initial parameter vector is used as the background vector for the Ptrnet-based policy network. After passing through the mask matrix, the vector is used as the input vector to the policy network. In the attention mechanism of the policy network, the first layer is used to update the background vector, and the second layer outputs the next-hop node for the current traffic. In subsequent node selection, the node features selected in the previous step will be used... Features of source nodes splicing and combining The sum is used as the background vector for the current step; specifically: Using state feature vectors Input to a GAT-based feature extraction network, where, The dimension of the feature vector is represented by the GAT network, which consists of several graph attention layers. In each layer, a node is defined as the feature vector corresponding to a physical link in the network, and an edge is defined as the adjacency relationship of a physical link in the actual network. In addition, a virtual node V is added to connect all nodes in the graph, whose features are... Initialized as the average of the sum of the feature vectors of all other nodes, and updated jointly with other nodes, the graph attention layer of the GAT network uses an attention mechanism. Obtain the attention coefficients of the two nodes: in This represents the attention coefficients of nodes i and j. This represents a learnable weight matrix. Each node calculates attention coefficients with all its neighboring nodes. For subsequent calculations, these attention coefficients are normalized. in This represents the normalized attention coefficient. Represents the normalization function. The set of first-order neighbors of node i, and the attention mechanism. The weight vector generated by a single-layer feedforward neural network is parameterized and then subjected to a nonlinear transformation using a nonlinear activation function. in, Represents a non-linear activation function. This represents the concatenation operation of two vectors. Finally, node i performs a weighted sum of the transformed features of all its neighboring nodes based on the obtained attention coefficients, and then performs a nonlinear transformation through a nonlinear activation function to obtain the updated features. in The GAT network has two layers, representing a non-linear activation function. During the update process, a multi-head attention mechanism is also used, where the two heads independently and in parallel compute the updated features, which are then concatenated in the first layer and averaged in the second and final layer. The updated node features are obtained from the feature extraction network. With global features The input is fed into a PtrNet-based policy network. The input to the policy network consists of two parts: the background vector. The input vector and background vector are represented as follows: in, For learnable parameter vectors, The weight matrix is ​​a learnable matrix. and These are the node feature vectors output at time step t=1 and the previous time step, respectively. Assume the current edge is a node. Then the input vector is the node For all nodes that are considered valid actions, the input vector is obtained by applying all node features through a mask matrix. : That is, among them For the mask matrix, Represents the mask vector, when the node With nodes Adjacent nodes The value is 1 if the node has not been visited, otherwise it is 0. The policy network consists of two attention mechanisms, where the first layer updates the background vector and the second layer outputs the node selection. In the first attention mechanism, the background vector... Used to calculate query vector Input vector Used to calculate key-value vectors Sum value vector ,use and After calculating the attention score, it is normalized using the Softmax function. Finally, the input vector is weighted and summed based on the attention score, and then fed into a fully connected layer to obtain the updated background vector. in The dimension of the parameter vector. For learnable weight matrices, Let the attention coefficients between the feature vector and the background vector of node i be... The input is fed into the second-layer attention mechanism, where an attention score is calculated and then transformed into a probability distribution using a softmax function. Finally, based on the probability distribution To select the next hop node for the traffic.

5. The time-sensitive network traffic scheduling algorithm based on graph networks and deep reinforcement learning as described in claim 4, characterized in that, In step 3, the node features output in step 2 are... Input MLP network for time slot blocks The selection process involves the following steps: after selecting the next-hop node for the traffic, the features of the selected node are input into the MLP network, and the MLP scores each time slot block. Finally, the probability distribution is normalized using the softmax function. And according to the probability distribution Output time slot block : Each time slot is scored using a time slot allocation method based on periodic-emergency awareness, and the time slot with the highest score is selected. The forwarding time of the next hop in the traffic flow: First, traffic with shorter cycles requires more time slots than traffic with longer cycles, therefore, for time slots... Since subsequent time slots may be occupied and can only carry traffic for a larger period, the time slots will be allocated to the period. The load-bearing capacity is expressed as: in This represents the period in the set of traffic periods. Indicates the time slot on node i Is the node occupied? If occupied, the value is 1; otherwise, the value is 0. Upper Time Slot The periodic bearing capacity is expressed as: in This represents a set of traffic cycles. A higher-score time slot can carry more cycles, while a lower-score time slot can only carry fewer larger time slots. Therefore, lower-score time slots are prioritized for scheduling to preserve cycle carrying capacity and thus allocate more traffic. Note that there may be time slots with similar scores, and the maximum end-to-end delay constraint of the traffic should also be considered. Therefore, the urgency of the traffic should be considered. As a second indicator, to eliminate the impact of the difference in the value range of the two sub-indicators, the periodic carrying capacity is... After normalizing the fractions and The weighted sum is the total score: in, and These are the weighting coefficients. This represents the total number of periods in the periodic set.

6. The time-sensitive network traffic scheduling algorithm based on graph networks and deep reinforcement learning as described in claim 4, characterized in that, In step 4, during the training phase, as new traffic continuously arrives, the end of the current sampling round is marked when the agent cannot successfully schedule the current traffic. The collected sequence samples are stored in the experience replay buffer and learned using the policy gradient algorithm REINFORCE. Important sample mining technology assists in updating, and the routing decision model and time slot hierarchical decision model are updated by maximizing the expectation of the reward function. During the testing phase, the network topology and traffic demand are input into the pre-trained routing decision model and time slot hierarchical decision model, which then output an accurate scheduling table. The specific process is as follows: The agent's goal is to schedule as much traffic as possible while maintaining latency constraints, so the reward function is set as follows: in This indicates whether the current traffic has been successfully scheduled. Represents the latency of the current traffic. This represents the load balancing level of the network, and its value is the difference between the time slot occupancy rate of the selected node and the average time slot occupancy rate of all nodes. , and These are the weighting coefficients; The policy gradient algorithm REINFORCE calculates the reward value. The expectation relative to the model parameters The model is updated using gradients. The end of the current sampling round is marked when the agent fails to schedule traffic. The collected sequence samples of state, action, and reward are stored in an experience replay buffer. The agent's goal is to update the model parameters by learning from these samples to find an optimal policy. To maximize rewards Expectations: in As the decay factor, the update formula for the enhancement algorithm REINFORCE is derived from this: in It's a baseline, used to reduce the variance of gradient estimation during model updates, making the training process more stable. The learning rate; in addition, important sample mining techniques are used to improve the training efficiency of the agent, and an auxiliary buffer is created. The sample with the lowest advantage in each training iteration is added to the buffer pool. At the same time These samples contain more information and are more valuable than other samples. They are crucial for the subsequent learning of routing decision models and time-slot hierarchical decision models. and The samples in the training data are uniformly sampled, so that these samples are repeatedly learned, thereby improving training efficiency. Because it involves hierarchical actions, the probability function of the actions consists of joint probabilities. If the action at time step t If the traffic does not reach its destination node, the intelligent system updates the network status. Proceed to the next action selection; otherwise, complete the scheduling of this traffic and obtain the request for the next new traffic, starting the scheduling from the source node.

Images