Route scheduling method based on multi-cycle CSQF mechanism and gdrl

Abstract

The present invention belongs to the field of route scheduling. Disclosed is a route scheduling method based on a multi-cycle CSQF mechanism and GDRL. The method comprises the following steps: S1, performing initialization to set a cycle index detection mechanism, a queue mapping and a queue mapping constraint of multi-cycle CSQF; S2, constructing a DFRLLS model; S3, on the basis of a GDRL network, optimizing the DFRLLS model; S4, offline training a learning policy of the GDRL model; and S5, making a decision online on the basis of the trained GDRL model. The present invention uses the routing scheduling method based on the multi-cycle CSQF mechanism and GDRL uses a GCN network to extract topology information among networks, such that more flow scheduling can be achieved compared with methods merely using reinforcement learning; the performance is also stable even under a complex network topology; and compared with a CSQF mechanism, the multi-cycle CSQF mechanism can reduce a delay from the source to the destination of a flow.

Description

Routing scheduling method based on multi-cycle CSQF mechanism and GDRL Technical Field

[0001] This invention relates to the field of routing scheduling technology, and in particular to a routing scheduling method based on the multi-cycle CSQF mechanism and GDRL. Background Technology

[0002] Cyclic Queuing and Forwarding (CQF) was proposed as a creep shaper, alternately opening and closing two queues on a port. It divides time into equal-length periods T, where packets sent by a previous node in period C must be received by subsequent nodes within the same period, and then sent out in period C+1. While CQF provides good control over the latency of each hop (up to two periods), this mechanism is not very scalable, is only suitable for small networks, and requires complete synchronization between nodes.

[0003] To improve flexibility and scalability, the Periodic Queuing Forwarding (CSQF) mechanism was designed as an emerging standard draft by the IETF DN working group, as an evolution of the CQF mechanism. The CSQF mechanism proposes using more queues to delay packets and specifying corresponding periods for packet transmission. Internally, in a router supporting the CSQF mechanism, each output port will be equipped with N queues, and within these N queues... ( The queue is reserved for time-critical traffic, while the remaining non-critical traffic is used for other traffic. The queues are used for best-effort (BE) traffic. These N queues transmit packets in a round-robin fashion, meaning that in each cycle, only one queue is active, used to send packets to the physical link, while the other (N - 1) inactive queues are closed and packets are queued for future transmission. It's important to note that the number of packets enqueued in each inactive queue is related to the buffer size of each queue; improper enqueuing can lead to packet loss. The ND time-sensitive queues are dedicated to time-critical flows through resource reservation. Assigning packets to specific queues effectively determines their transmission cycle, and packets can be delayed by a maximum of (N - 1) cycles.

[0004] Existing research on Specified Periodic Queuing Forwarding (CSQF) based on deterministic networks mostly focuses on routing and scheduling under single-link rates, where all nodes have the same period length. In practical industrial scenarios, multi-link rates are also quite common. In networks composed of multiple link rates, in order to be compatible with low-speed links, the period of high-speed links needs to be set to be the same as that of low-speed links, resulting in waste of high-speed link resources and extremely high latency from the start to the end of the flow.

[0005] Meanwhile, existing methods generally use deep reinforcement learning to solve routing problems in deterministic networks (DNs), but this approach cannot fully utilize the topological information between networks for fusion feature extraction (mainly because topological information is irregular graph structure information, while the fully connected neural networks used in deep reinforcement learning are for Euclidean data). Summary of the Invention

[0006] To address the aforementioned issues, this invention provides a routing scheduling method based on a multi-cycle CSQF mechanism and GDRL. It uses a GCN network to extract topology information between networks, achieving more flow scheduling compared to methods that only use reinforcement learning. Furthermore, it maintains stable performance even in complex network topologies. The multi-cycle CSQF mechanism can also reduce the start and end point delays of flows.

[0007] To achieve the above objectives, this invention provides a routing scheduling method based on a multi-cycle CSQF mechanism and GDRL, comprising the following steps:

[0008] S1. Initialize and set the periodic index detection mechanism, queue mapping, and queue mapping constraints of the multi-period CSQF;

[0009] S2. Construct the DFRLLS model;

[0010] S3. Optimize the DFRLLS model based on GDRL;

[0011] S4. Learning strategies for offline training of GDRL models;

[0012] S5. Make decisions online based on the trained GDRL model.

[0013] The present invention has the following beneficial effects:

[0014] 1. Using GCN networks to extract topology information between networks can achieve more flow scheduling compared to methods that only use reinforcement learning, and the performance remains stable even in complex network topologies;

[0015] 2. Using a multi-cycle CSQF mechanism can achieve lower flow start-to-end latency compared to the same-cycle CSQF in multi-link rate networks.

[0016] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0017] Figure 1 is a framework diagram of the GDRL described in the routing scheduling method based on the multi-cycle CSQF mechanism and GDRL of the present invention.

[0018] Figure 2 shows four queue mapping diagrams of the multi-cycle CSQF based routing scheduling method based on multi-cycle CSQF mechanism and GDRL of the present invention; wherein Figure 2(a) is the mapping from high-speed port to high-speed port; Figure 2(b) is the mapping from high-speed port to low-speed port; Figure 2(c) is the mapping from low-speed port to low-speed port; and Figure 2(d) is the mapping from low-speed port to high-speed port.

[0019] Figure 3 shows the link transmission diagram of a CSQF network operating at the same cycle;

[0020] Figure 4 is a comparison of the number of deterministic flow scheduling operations described in the simulation experiment of this invention;

[0021] Figure 5 shows the comparison results of the start-to-end delay of the flow under three topologies in the simulation experiment of the present invention; wherein Figure 5(a) is a random topology; Figure 5(b) is a stepped topology; and Figure 5(c) is an AFDX topology. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages disclosed in the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the embodiments of the present invention and are not intended to limit the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments in this application without creative effort are within the scope of protection of this application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout.

[0023] It should be noted that the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0024] Similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0025] To fully utilize the topology information between networks, this invention introduces graph neural networks. The graph neural network extracts the topology information between networks and fuses it into deep reinforcement learning for route allocation. Furthermore, to address the problem of low utilization of high-speed link resources in multi-link-rate networks, this patent proposes a multi-cycle CSQF mechanism. By setting a cycle adapted to the link rate, it reduces the origin-to-end delay of the flow and extends it to general scenarios. Based on this, scheduling constraints for multi-cycle CSQF are proposed.

[0026] The specific discussion is as follows:

[0027] As shown in Figure 1, the routing scheduling method based on the multi-cycle CSQF mechanism and GDRL includes the following steps:

[0028] S1. Initialize and set the periodic index detection mechanism, queue mapping, and queue mapping constraints of the multi-period CSQF;

[0029] In the periodic index detection mechanism described in step S1, the detection period length is set to... , To detect short periods, and simultaneously detect short periods and long cycle and use short cycles As the detection granularity, the output periodic index is used. ;

[0030] The output cycle index will be used in the queue mapping described in step S1. Mapped to the exit queue In the middle, and when the long cycle With short cycle ratio When = 2, queue mapping is divided into the following four scenarios as shown in Figure 2:

[0031] High-speed port to high-speed port mapping: based on the output cycle index of the stream received by the node. The offset of the flow at the current node is The stream enters the sending queue, and the mapping relationship is represented as follows:

[0032]

[0033] Mapping from high-speed port to low-speed port: the i-th The flow arrives periodically, because periodic detection uses short periods. To determine the granularity of detection, a transformation is required in the long-cycle queue, and the mapping relationship is expressed as follows:

[0034]

[0035] Mapping from low-speed port to low-speed port: Since the detection granularity is... Therefore, it is the same as the mapping from high-speed port to low-speed port, and the mapping relationship is expressed as:

[0036]

[0037] Mapping from low-speed port to high-speed port: This is the same as mapping from high-speed port to high-speed port, and the mapping relationship is expressed as follows:

[0038]

[0039] In the queue mapping constraint described in step S1, the mapping rules are divided into two types based on the period size of the source port and the destination port: high-speed port to low-speed port mapping and low-speed port to high-speed port mapping. This represents the ratio of the source port cycle time to the destination port cycle time.

[0040] When a flow is forwarded from a high-speed port to a low-speed port, i.e. from a short cycle to a long cycle, including cases with equal cycles, the mapping rules are as follows:

[0041]

[0042] In the formula, It is the ratio of the period of the destination port to the shortest period; This is the number of queues at the destination port; ;

[0043] When a flow is forwarded from a low-speed port to a high-speed port, i.e. from a long-period to a short-period direction, including cases with equal periods, the mapping rule is as follows:

[0044] ;

[0045] In the formula, This is the ratio of the source port's period to the shortest period. Number of destination port queues; .

[0046] In step S1, the multi-cycle CSQF is extended, and the extension condition is:

[0047] If the switch supports Different CSQF cycles Its shortest period Longest period Simultaneously satisfying the ratio between any two periods It is an integer;

[0048] For routers with multi-cycle CSQF that meet the expansion conditions, the shortest cycle is used. To detect particle size, the detection cycle is... The output of data packets arriving within the detection period Scope Inside.

[0049] In this embodiment, as shown in Figure 3, in a CSQF network operating with the same period, to ensure compatibility with low-speed links, the period is set to T=500us. At this time, the delay from the start to the end of the flow is 8×500us=4000us. Conversely, if CSQF with different periods is running on GE and FE, , Therefore, the delay of multi-cycle CSQF is only 2×500 + 4×250 + 2×500 = 3000µs. Thus, the multi-cycle CSQF scheme can better utilize the speed of high-speed links in multi-link rate networks. Furthermore, matching the link rate with an appropriate CSQF cycle—that is, using short cycles on high-speed links and long cycles on low-speed links—is a reasonable multi-cycle strategy. This can reduce the start-to-end delay of deterministic flows.

[0050] S2. Construct the DFRLLS model (Deterministic Routing and Low-Latency Scheduling Model).

[0051] Step S2 specifically includes the following steps:

[0052] S21. Constructing a network flow model:

[0053] Abstract the network topology into a directed graph, and use... Represents the network topology, where It represents the set of DN routing nodes. This indicates a connection between two network nodes. A set of edges, and nodes The full-duplex physical link between them consists of two directed edges, which are represented as follows: ∈ as well as Each link The attributes are represented by quadruples:

[0054]

[0055] In the formula, Link speed; The period of the multi-cycle CSQF on the link; Effective time slot capacity; This refers to the link's latency, which includes propagation latency, transmission latency, and processing latency.

[0056] S22. Construct the DFRLLS model:

[0057] S221. For a flow to be scheduled The controller determines the only feasible scheduling link. The link sequence is Among them, the link From the source node Beginning, Link End at the destination node ,and and They are adjacent links, and the following constraints apply:

[0058]

[0059]

[0060]

[0061] In the formula, Indicates link The starting point; Indicates the starting point of the flow; Indicates link The end point; Indicates the destination node; Indicates link The end point; Indicates link The starting point;

[0062] S222. Define an integer variable. Represents a stream All data packets on each link The offset;

[0063] Based on the different cycle lengths of different links, assuming the flow The first data packet in Reaching the source node This indicates that it is in the period From the source node Issued, and in the cycle + To reach the next node, use This indicates that the flow will be... The period is represented by an integer sequence ( , , , ),in This indicates that the first data packet is in the corresponding node. The arriving circular index, then the arrival period of the remaining groups is passed through... Calculated;

[0064] S223, from the source node to the destination node flow scheduling Represented as {( , ),( , ),...,( , )};

[0065] S224. Use the following conditions to determine convection Is it effective?

[0066] Flow start-to-end delay constraint: Flow The start-to-end delay of all message flows must not exceed the maximum start-to-end delay limit of the flow. :

[0067]

[0068] Periodic capacity constraint: If the flow The grouping is determined in the cycle Link Transmission at that location is represented as The queue capacity for a given period is shared among the scheduled flows, therefore, during a period... Any edge inside The business load is all within its queue capacity limit:

[0069]

[0070] In the formula, Represents all streams The set that constitutes; Indicates link The capacity in one period t;

[0071] S225. The flow scheduling problem can be expressed as follows: Given a network topology and DN flows, find an efficient schedule for all flows such that all TT flows are scheduled, and define it as follows:

[0072]

[0073] In the formula, Indicates the total number of scheduled flows; This represents the stream processing function, which is activated when stream scheduling is successful. =1, otherwise 0.

[0074] In step S225, a DN flow is defined as periodic unicast traffic from the source node to the destination node, and the set of DN flows is represented as... And the DN flow Defined as a tuple ( , , , , ),in, and Representing flow The source node and the destination node; Represents a stream The cycle, that is, each of the source nodes Data packets sent periodically; Represents a stream Size; This represents the time delay from the start point to the end point of the maximum flow.

[0075] Because the flow has different periods Define an overall scheduling period, called the supercycle, in which all network behaviors are identical, and the supercycles of all flows are considered. Calculated as the least common multiple of all flow cycles.

[0076] S3. Optimize the DFRLLS model based on GDRL (Graph Deep Reinforcement Learning);

[0077] In step S3, reinforcement learning modeling is performed on the DFRLLS model, and an MDP (Markov Decision Process) model is constructed. The MDP model includes a state space. Action space and reward space ;

[0078] Among them, the state space The design is as follows: System status is set. This represents all link information across the entire network, allowing observation and use of system status via a DRL proxy. For a stream Generate a schedule Then, the system state is represented by extracting network features from the following three aspects. Topology information, flow information, and network periodic load information;

[0079] In this embodiment It can be divided into And each It consists of the following seven parts:

[0080] 1) If the edge Edge or source node of the previous action The edges are adjacent;

[0081] 2) This edge to the destination node The distance;

[0082] 3) Does this edge form a loop with the edge of the previous action? The scheduling of the TT process should avoid network link loops, which waste network resources and lead to increased latency.

[0083] 4) The congestion level of the available offset period refers to the current edge The congestion status of the available queues of the source node; in CSQF scheduling, the available offset period of a node depends on its number of queues, and the congestion level of the available offset period is the average load of the available queues;

[0084] 5) The congestion level on this side is the ratio of all available cycles to the total number of cycles within a supercycle.

[0085] 6) The difference between the currently selected period and the flow delay boundary;

[0086] 7) Cyclic load value (in percentage).

[0087] Design another one reachability matrix ,in It represents the number of edges in the entire network topology; the reachability matrix represents the links. With Link Does there exist a path between them that contains one node?

[0088] The calculations are recalculated before each action decision is made, and Count only once. This represents information about each edge. It is related to network topology.

[0089] Action Space The design is as follows: GDRL divides the path of a TT stream into a set of adjacent edges. When using DRL to solve scheduling problems, the action space is large due to the numerous routing choices on network nodes and the large number of frame transmission times. GDRL reduces the operation space by dividing the path of a TT stream into a set of adjacent edges. Each action determines only one edge, rather than the entire route. In other words, the scheduling of TT streams... {( , ),( , ),...,( , Scheduling Composed of a series of sub-actions Composition, a sub-action Defined tuple ( , ), Indicates the edge that the flow needs to pass through. Indicates that the flow reaches the link The time of the source node, Represents a stream On the side Offset in the source node + It is from The time when the source node sends the data; in this embodiment, an effective flow scheduling The constraints in the sub-actions, including the formulas for the constraints, the time delay constraints, and the periodic capacity constraints, must be satisfied.

[0090] Reward Space When making a decision to act At that time, reward space The reward space is calculated using the Environment from the GDRL model. Train the DRL model and GCN network of the GDRL model, and adjust their parameters. Update; and determine the flow using the following formula. Was it successfully scheduled?

[0091] ;

[0092] Furthermore, the scheduling of each stream consists of a series of sub-actions. =( , The reward space of the entire sub-action is composed of... It consists of two parts: the first part is the flow. In the link The degree value corresponding to the offset is determined by the offset allocation method. This method assigns a degree value to the offset period of a node; a higher degree value indicates a lower load within that offset period. The reward for the sub-action is also given by this degree value. The reward for the first part of the sub-action is... as follows:

[0093]

[0094] In the formula, Represents a stream In the link Offset in source node The availability of the cycle;

[0095] The second part concerns whether the sub-action is a flow. The reward for a successful flow scheduling is only effective when it is a component of the complete scheduling process; that is, the reward for a successful flow scheduling is only given when the sub-action... This only takes effect when it is the last edge to be effectively scheduled, and is the last action in selecting a valid route. Afterwards, check if the flow was successfully scheduled, and then update the sub-actions by adding additional rewards to the second part in a decaying manner. , , ..., The reward for the second sub-action as follows:

[0096]

[0097] In the formula, In this embodiment, the attenuation factor is represented. Set it to 0.5.

[0098] Step S3 specifically includes the following steps:

[0099] S31. Set the optimization objective as maximizing the number of scheduled flows given a certain number of flows, while obtaining the optimal DFRLLS scheduling strategy:

[0100]

[0101] In the formula, This represents maximizing the long-term reward of the flows mapped into the network; Representation Strategy Remodeling rewards; Discount factor Discount factor represents the ratio of the current reward to future rewards. It is an adjustable parameter similar to a decay factor, which may vary in different task scenarios, but its value ranges between (0, 1); in this embodiment, it is set to 0.5;

[0102] S32. Using a GDRL-based network model for edge selection, the optimization problem of the DFRLLS model is transformed into an optimization problem of maximizing expected future discounted revenue:

[0103] The spatial dependencies of the network topology are extracted using GCN (Graph Convolutional Network), and the extracted features are fused with link features. The main network is then used to extract features and output the Q-value.

[0104] Step S32 specifically includes the following steps:

[0105] S321. Using a three-layer GCN network as a feature extractor, extract the spatial correlation of the network topology:

[0106] Environment will link state and reachability matrix The input is the main network, which uses the GCN network to process each link by aggregating the features of its neighboring links. The aggregated features are then combined with the original link state. The input function approximator generates the next hop for each stream and assigns an offset to this hop.

[0107] In step S321, the offset is selected based on the number of receiving node queues:

[0108] In MCCSQF, the nodes have different periods, the number of queues used for TT stream scheduling is different, and the range of offset selection is also different. Therefore, this embodiment uses a selection method based on the number of receiving node queues in the offset selection. Specifically, an appropriate offset... Will make the edge It can handle more TT streams. When a stream arrives at the corresponding port of a node, if the queue load is not considered and only fast forwarding is guaranteed, it is easy to be mapped to the next queue to be sent, causing the corresponding period of that queue to be overloaded and unable to schedule more streams. The offset allocation method fully considers the load of the receiving queue. It selects a low-load queue for scheduling. The main idea is to take the difference between the load of each receiving queue and the average load of all receiving queues, and consider the stream within a super-period. Average load sent.

[0109] The load of each receive queue is subtracted from the average load of all receive queues, and the flow is considered within one supercycle. The average load sent, and Defined as:

[0110]

[0111]

[0112]

[0113] In the formula, Represents a stream Reaching the node The time slot; Represents a stream The cycle; Represents a stream Supercycle; Indicates link At any moment The availability of time slots (remaining capacity); Indicates port The number of queues; Indicates the range of values ​​for the offset; Indicates the current flow The number of times a period can be sent within a supercycle; Indicates in the link Mid-offset The load under;

[0114] in, A higher value indicates a lower time slot load corresponding to the current offset, so the maximum value is taken. corresponding offset For flow exist Node offset.

[0115] S322, In performing the action Afterwards, the Environment provided feedback and rewards. Observation, feedback, and rewards and the next state of Environment The target network generates an estimated Q-value:

[0116] network status The features extracted by GCN are input into the main neural network. The network makes an action decision.

[0117]

[0118] In performing the action Afterwards, feedback and rewards were received. and will Status updated to Then through This action can be obtained. The value, its specific calculation expression is (where ):

[0119] =

[0120] S323. Calculate the loss using the mean squared error (MSE);

[0121]

[0122] Update parameters using gradient descent ;

[0123] S4. Learning strategies for offline training of GDRL models;

[0124] In step S4, the Q-network trained on the GDRL model based on TD error is replayed using a dual-channel experience replay mechanism, and historical experience during the training process is stored using the experience replay mechanism:

[0125] First, TT streams are randomly generated. The GDRL model calculates the timetable for each generated TT stream. When the scheduling of a stream is completed, a reward is provided for all actions taken during the scheduling process.

[0126] S5. Make decisions online based on the trained GDRL model.

[0127] In this embodiment, the GDRL algorithm does not always generate feasible actions (i.e., actions that can be converted into valid schedules). Therefore, to generate feasible action decisions, action control is used to omit obviously invalid edges. In this embodiment, two types of edges are considered invalid: 1) non-adjacent edges: the selected edge should be immediately adjacent to the last edge; 2) edge loops that form an edge cycle waste network resources and cause significant latency. GDRL outputs the probability of selecting each edge. Then, action control marks invalid edges. Finally, the edge with the highest probability is selected from the valid edges. Through the above action control techniques, errors caused by lack of training can be compensated for, making the agent more likely to take effective actions.

[0128] Simulation Experiment

[0129] Experimental environment: Based on a machine with a 13th Gen Intel(R) Core(TM) i7-13700KF @ 3.40 GHz, and the software environment is based on Python 3.7 and PyTorch 1.12.0.

[0130] In this experiment, the proposed multi-cycle CSQF mechanism and GDRL algorithm are evaluated. The main focus is on the impact of the multi-cycle CSQF mechanism on the start-to-end delay of flows under different network topologies.

[0131] To verify the applicability of the proposed multi-cycle CSQF mechanism, a comparative experiment on the start-to-end delay of the multi-cycle CSQF mechanism flow was conducted using three network topologies: a simplified AFDX (avionics full-duplex switched Ethernet) network used on the Airbus A380, a ladder network topology used on the train communication network, and a randomly generated network topology.

[0132] Random topology: Besides random topology, the other two topologies have specific structures. Therefore, a random algorithm is used to generate random topologies. The number of nodes in each topology is randomly selected between (10, 20) and is uniformly distributed. Any two nodes... and The probability of an edge connecting any two nodes is 0.35. In other words, for any two nodes... and Generate a random number between (0, 1). If the value is greater than 0.35, add a connection. and The edge.

[0133] Besides the network topology, the TT stream is also another input to the scheduler. Different randomly generated streams are used in both the training and evaluation phases, and the TT stream... By tuple ( , , , , Definition: For each random flow, a source node is randomly selected from all nodes in the network. and destination node Frame length It is an integer randomly selected from (64, 1500). Period (In milliseconds) Selected from the set {8, 16, 32}. Maximum latency. It is randomly selected between 8 and 24 milliseconds, with a period set to 500ms.

[0134] Three agents were used: one for a simplified AFDX topology, one for a stepped topology of varying sizes, and a last one for a random topology. Each agent was trained to convergence using the aforementioned relevant topologies and randomly generated flows. The three agents were then evaluated using the same topologies as during training, but with different flows.

[0135] To illustrate the universality of the multi-cycle CSQF mechanism in different node topologies, 40% of the nodes were randomly selected as high-speed nodes with different cycle lengths than the other nodes, thus forming a multi-cycle CSQF. For example, when R=2, since the global cycle is set to 500ms, the cycle of the high-speed nodes is 250ms. Furthermore, in the experiment, a super-cycle scheduling was considered, with a super-cycle length of... For convenience, the link scheduling delay is set to the period length of one node, that is, the link delay for high-speed port nodes is 250ms, and the link delay for low-speed port nodes is 500ms.

[0136] In addition, to verify the scheduling effectiveness of the GDRL algorithm described in this invention, the multi-branch reinforcement learning algorithm in "Deep Reinforcement Learning-Based Deterministic Routing and Scheduling for Mixed-Criticism Flows"

[0013] by Hao Yu, Tarik Taleb, Senior Member, IEEE, and Jiawei Zhang was modified into a single-branch algorithm for route selection (named DRL). At the same time, the low-load offset selection method was used to select the offset, and then the algorithm was compared with the GDRL algorithm proposed in this invention under the same environment.

[0137] Comparison of deterministic flow scheduling quantities:

[0138] To eliminate the randomness of the experimental results, 10 network topologies were randomly generated under the corresponding network conditions, with 1000 TT flows generated in each topology. The trained GDRL agent and DRL agent were used for testing. The experimental results are shown in Figure 4. It can be seen that in the case of random topologies, the incremental scheduling of flows by the two algorithms is basically the same. However, in special topologies such as staircase topologies and AFDX topologies, the GCN-based DRL incremental scheduling algorithm is significantly better than the DRL scheduling algorithm, achieving deterministic flow scheduling.

[0139] Comparison of delays from the start to the end of the stream:

[0140] To verify the impact of multi-cycle CSQF scheduling on the start-to-end latency of a flow while reducing scheduling randomness, experiments were conducted using multiple sets of topologies with varying numbers of nodes for each topology. Specifically, 15, 20, and 25 nodes were initialized in the random topology; 10, 12, and 14 nodes were initialized in the ladder topology; and 8 and 11 nodes were initialized in the AFDX topology. Experiments were performed using a trained agent in both the same-cycle CSQF and multi-cycle CSQF models. Each topology was tested 10 times with the corresponding number of nodes. The experimental results are shown in Figure 5. It can be seen that, under any topology, multi-cycle CSQF can significantly reduce the start-to-end latency of a flow.

[0141] Therefore, this invention adopts the above-mentioned routing scheduling method based on multi-period CSQF mechanism and GDRL, and uses GCN network to extract topology information between networks. Compared with the method that only uses reinforcement learning, it can achieve more flow scheduling, and its performance is equally stable under complex network topologies. Multi-period CSQF can reduce the flow start-to-end delay compared to CSQF.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A routing scheduling method based on multi-cycle CSQF mechanism and GDRL, characterized in that: Includes the following steps: S1. Initialize and set the periodic index detection mechanism, queue mapping, and queue mapping constraints of the multi-period CSQF; S2. Construct the DFRLLS model; S3. Optimize the DFRLLS model based on GDRL network; S4. Learning strategies for offline training of GDRL models; S5. Make decisions online based on the trained GDRL model.

2. The routing scheduling method based on multi-cycle CSQF mechanism and GDRL according to claim 1, characterized in that: In the periodic index detection mechanism described in step S1, the detection period length is set to... , To detect short periods, and simultaneously detect short periods. and long cycle and use short cycles As the detection granularity, the output periodic index is... ; The output cycle index will be used in the queue mapping described in step S1. Mapped to the exit queue In the middle, and when the long cycle With short cycle ratio When = 2, queue mapping is divided into the following four scenarios: High-speed port to high-speed port mapping: based on the output cycle index of the stream received by the node. The offset of the flow at the current node is The stream enters the sending queue, and the mapping relationship is represented as follows: ； Mapping from high-speed port to low-speed port: the i-th The flow arrives periodically, because periodic detection uses short periods. To determine the granularity of detection, a transformation is required in the long-cycle queue, and the mapping relationship is expressed as follows: ； Mapping from low-speed port to low-speed port: Since the detection granularity is... Therefore, it is the same as the mapping from high-speed port to low-speed port, and the mapping relationship is expressed as: ； Mapping from low-speed port to high-speed port: This is the same as mapping from high-speed port to high-speed port, and the mapping relationship is expressed as follows: ； In the queue mapping constraint described in step S1, the mapping rules are divided into two types based on the period size of the source port and the destination port: high-speed port to low-speed port mapping and low-speed port to high-speed port mapping. This represents the ratio of the source port cycle time to the destination port cycle time. When a flow is forwarded from a high-speed port to a low-speed port, i.e. from a short cycle to a long cycle, including cases with equal cycles, the mapping rules are as follows: ； In the formula, It is the ratio of the period of the destination port to the shortest period; This is the number of queues at the destination port; ； When a flow is forwarded from a low-speed port to a high-speed port, i.e. from a long-period to a short-period direction, including cases with equal periods, the mapping rule is as follows: ； In the formula, The ratio of the source port's period to the shortest period. Number of destination port queues; 。 3. The routing scheduling method based on multi-cycle CSQF mechanism and GDRL according to claim 2, characterized in that: In step S1, the multi-cycle CSQF is extended, and the extension condition is: If the switch supports Different CSQF cycles Its shortest period Longest period Simultaneously satisfying the ratio between any two periods It is an integer; For routers with multi-cycle CSQF that meet the expansion conditions, the shortest cycle is used. To detect particle size, the detection cycle is 3. The output of data packets arriving within the detection period Scope Inside.

4. The routing scheduling method based on multi-cycle CSQF mechanism and GDRL according to claim 1, characterized in that: Step S2 specifically includes the following steps: S21. Constructing a network flow model: Abstract the network topology into a directed graph, and use... and Represents the network topology, where It represents the set of DN routing nodes. This indicates a connection between two network nodes. of A set of edges, and nodes and nodes The full-duplex physical link between them consists of two directed edges, which are represented as follows: ∈ as well as , each link The attributes are represented by quadruples: ； In the formula, Link speed; The period of the multi-cycle CSQF on the link; Effective time slot capacity; This refers to the link's latency, which includes propagation latency, transmission latency, and processing latency. S22. Construct the DFRLLS model: S221. For a flow to be scheduled The controller determines the only feasible scheduling link. The link sequence is Among them, the link From the source node Beginning, Link End at the destination node ,and and They are adjacent links, and the following constraints apply: ； ； ； In the formula, Indicates link The starting point; Indicates the starting point of the flow; Indicates link The end point; Indicates the destination node; Indicates link The end point; Indicates link The starting point; S222. Define an integer variable. Represents a stream All data packets on each link The offset; Based on the different cycle lengths of different links, assuming the flow The first data packet in Reaching the source node This indicates that it is in the period From the source node Issued, and in the cycle + To reach the next node, use This indicates that the flow will be... The period is represented by an integer sequence ( , , , ),in This indicates that the first data packet is in the corresponding node. The arriving circular index, then the arrival period of the remaining groups is passed through... Calculated; S223, from the source node to the destination node Target flow scheduling Represented as {( , ),( , ),...,( , )}； S224. Use the following conditions to determine convection Is it effective? Flow start-to-end delay constraint: Flow The start-to-end delay of all message flows must not exceed the maximum start-to-end delay limit of the flow. ： ； Periodic capacity constraint: If the flow The grouping is determined in the cycle Link Transmission at that location is represented as The queue capacity for a given period is shared among the scheduled flows, therefore, during a period... Any edge inside The business load is all within its queue capacity limit: ； In the formula, Represents all streams The set that constitutes; Indicates link The capacity in one period t; S225. The flow scheduling problem can be expressed as follows: Given a network topology and DN flows, find an efficient schedule for all flows such that all TT flows are scheduled, and define it as follows: ； In the formula, Indicates the total number of scheduled flows; This represents the stream processing function, which is activated when stream scheduling is successful. =1, otherwise 0.

5. The routing scheduling method based on multi-cycle CSQF mechanism and GDRL according to claim 4, characterized in that: In step S3, reinforcement learning modeling is performed on the DFRLLS model, and an MDP model is constructed. The MDP model includes a state space. Action space and reward space ; Among them, the state space The design is as follows: System status is set. This represents all link information across the entire network, allowing observation and use of system status via a DRL proxy. For a stream Generate a schedule Then, the system state is represented by extracting network features from the following three aspects. Topology information, flow information, and network periodic load information; Design another one reachability matrix ,in It represents the number of edges in the entire network topology; the reachability matrix represents the links. With Link Does there exist a path between them that contains one node? Action Space The design is as follows: GDRL divides the path of a TT flow into a set of adjacent edges, and schedules the TT flow. {( , ),( , ),...,( , Scheduling Composed of a series of sub-actions Composition, a sub-action Defined tuple ( , )， Indicates the edge that the flow needs to pass through. Indicates that the flow reaches the link The time of the source node, Indicates flow in link The offset in + From The time it was sent; Reward Space When making a decision to act At that time, reward space The reward space is calculated using the Environment from the GDRL model. Train the DRL model and GCN network of the GDRL model, and adjust their parameters. Update; and determine the flow using the following formula. Was it successfully scheduled? ； Furthermore, the scheduling of each stream consists of a series of sub-actions. =( , The reward space of the entire sub-action is composed of... It consists of two parts: the first part is the flow. In the link The degree value corresponding to the offset is determined by the offset allocation method. This method assigns a degree value to the offset period of a node; a higher degree value indicates a lower load within that offset period. The reward for the sub-action is also given by this degree value. The reward for the first part of the sub-action is... as follows: ； In the formula, Represents a stream In the link Offset in source node The availability of the cycle; The second part concerns whether the sub-action is a flow. The reward for a successful flow scheduling is only effective when it is a component of the complete scheduling process; that is, the reward for a successful flow scheduling is only given when the sub-action... This only takes effect when it is the last edge to be effectively scheduled, and is the last action in selecting a valid route. Afterwards, check if the flow was successfully scheduled, and then update the sub-actions by adding additional rewards to the second part in a decaying manner. ， ，…， The reward for the second sub-action as follows: ； In the formula, This represents the decay factor, which updates the reward of the sub-action by increasing the additional reward of the second part in a decaying manner.

6. The routing scheduling method based on multi-cycle CSQF mechanism and GDRL according to claim 5, characterized in that: Step S3 specifically includes the following steps: S31. Set the optimization objective as maximizing the number of scheduled flows given a certain number of flows, while obtaining the optimal DFRLLS scheduling strategy: ； In the formula, This represents maximizing the long-term reward of the flows mapped into the network; Representation strategy Remodeling rewards; discount factor This indicates the ratio of the current reward to future rewards; S32. Using the GCN-based GDRL algorithm for edge selection, the optimization problem of the DFRLLS model is transformed into an optimization problem of maximizing the expected future discounted revenue: The spatial dependencies of the network topology are extracted using GCN, and the extracted features are fused with link features. The main network is then used to extract features and output the Q-value.

7. The routing scheduling method based on multi-cycle CSQF mechanism and GDRL according to claim 6, characterized in that: Step S32 specifically includes the following steps: S321. Using a three-layer GCN network as a feature extractor, extract the spatial correlation of the network topology: Environment will link state and reachability matrix The input is the main network, which uses the GCN network to process each link by aggregating the features of its neighboring links. The aggregated features are then combined with the original link state. The input function approximator generates the next hop for each stream and assigns an offset to this hop. S322, In performing the action Afterwards, the Environment provided feedback and rewards. Observation, feedback, and rewards and the next state of Environment The target network generates an estimated Q-value: network status The features extracted by GCN are input into the main neural network. The network makes an action decision. ； ； In performing the action Afterwards, feedback and rewards were received. and will Status updated to Then through This action can be obtained. The value, its specific calculation expression is (where ): = ； The loss is calculated using the mean squared error, and the specific formula is as follows: ； Update parameters using gradient descent based on mean square error ； 8. The routing scheduling method based on multi-cycle CSQF mechanism and GDRL according to claim 7, characterized in that: In step S321, the offset is selected based on the number of receiving node queues: The load of each receive queue is subtracted from the average load of all receive queues, and the flow is considered within one supercycle. The average load sent, and Defined as: ； ； ； In the formula, Represents a stream Reaching the node The time slot; Represents a stream The cycle; Represents a stream Supercycle; Indicates link At any moment The availability of time slots (remaining capacity); Indicates port The number of queues; Indicates the range of values ​​for the offset; Indicates the current flow The number of times a period can be sent within a supercycle; Indicates in the link Mid-offset The load under; in, A higher value indicates a lower time slot load corresponding to the current offset, so the maximum value is taken. corresponding offset For flow exist Node offset.

9. The routing scheduling method based on multi-cycle CSQF mechanism and GDRL according to claim 1, characterized in that: In step S4, the Q-network trained on the GDRL model based on TD error is replayed using a dual-channel experience replay mechanism, and historical experience during the training process is stored using the experience replay mechanism: First, TT streams are randomly generated. The GDRL model calculates the timetable for each generated TT stream. When the scheduling of a stream is completed, a reward is provided for all actions taken during the scheduling process.

10. The routing scheduling method based on multi-cycle CSQF mechanism and GDRL according to claim 4, characterized in that: In step S225, a DN flow is defined as periodic unicast traffic from the source node to the destination node, and the set of DN flows is represented as... And the DN flow Defined as a tuple ( , , , , ),in, and Representing flow The source node and the destination node; Represents a stream The period, that is, each of the source nodes Data packets sent periodically; Represents a stream Size; This represents the time delay from the start point to the end point of the maximum flow. Because the flow has different periods Define an overall scheduling period, called the supercycle, in which all network behaviors are identical, and the supercycles of all flows are considered. Calculated as the least common multiple of all flow cycles.

Images