Time-sensitive networking routing and scheduling method with link awareness and redundancy weighting

Abstract

The application discloses a link-aware and redundancy-weighted time-sensitive network routing scheduling method, which comprises the following steps: firstly, a low-redundancy path set is generated based on a dynamic link weight penalty mechanism; the link weight is exponentially increased by using the number of times of using the selected path and the number of times of attempting the rejected path, so as to guide subsequent paths to avoid hot links; then, a reinforcement learning state containing flow characteristics, candidate path characteristics and global link state is constructed, and a PPO intelligent agent is used to adaptively select path combinations; finally, time-aware shapers are used to complete time slot allocation and gate list generation on-line and incrementally. The reward function comprehensively considers the scheduling success rate, reliability, time delay, load balancing and resource occupation, and an entropy coefficient annealing mechanism is introduced to balance exploration and convergence. The application can realize intelligent incremental scheduling for reliability, time delay constraint and load balancing in the scene of random arrival of time-sensitive flows, and effectively improves the scheduling success rate and link load balancing performance.

Description

Technical Field

[0001] This invention relates to the fields of time-sensitive networking, industrial internet communication, deterministic network scheduling, and intelligent route optimization, specifically to a link-aware and redundancy-weighted time-sensitive network routing scheduling method. Background Technology

[0002] Time-Sensitive Networking (TSN) is a deterministic communication network designed for scenarios such as industrial control, vehicular networks, smart manufacturing, and automation systems. Its core objective is to provide low-latency, low-jitter, and highly reliable data transmission capabilities for mission-critical flows within an Ethernet environment. In the centralized configuration model of TSN, the user / network configuration interface is typically handled jointly by a Centralized User Configuration Entity (CUC) and a Centralized Network Configuration Entity (CNC). The CUC is responsible for collecting communication requirements from terminal devices or applications, such as the source node, destination node, period, frame size, deadline, and reliability requirements of time-sensitive flows. The CNC, based on the global network topology, link status, and traffic requirements, uniformly calculates the routing path, scheduling time slots, and gating control list configuration for time-sensitive flows. For time-sensitive streams with strict periodicity and deadline constraints, the CNC not only needs to allocate transmission paths that meet end-to-end delay requirements, but also needs to properly configure the gating control list on the switch port so that data frames can be transmitted within the predetermined time slot.

[0003] In existing time-sensitive network scheduling methods, common schemes are typically based on static routing, shortest path routing, or heuristic path selection strategies. These methods achieve good scheduling results when network load is low, but as the number of time-sensitive flows increases, some critical links are easily reused repeatedly, leading to uneven link resource utilization and the formation of hotspot links or bottleneck links. The emergence of hotspot links reduces the continuous idle time slots available for subsequent flows, thereby reducing the success rate of online incremental scheduling. On the other hand, to improve the transmission reliability of time-sensitive flows, redundant multipath transmission is often adopted, that is, selecting multiple paths for the same time-sensitive flow to replicate transmission, ensuring that at least one path's successful transmission meets the service reliability requirements. However, if there is a high degree of link overlap among candidate redundant paths, a common link failure will simultaneously affect multiple paths, weakening the reliability gain brought by multipath redundancy. Simultaneously, multipath transmission consumes more link and time slot resources; without an effective path combination selection mechanism, this may lead to excessive network resource consumption, further exacerbating load imbalance.

[0004] Furthermore, time-sensitive flows in real-world networks do not arrive simultaneously but randomly. Therefore, scheduling algorithms need online incremental decision-making capabilities, meaning they must select appropriate path combinations and allocate time slots for currently arriving flows without altering the time slot allocation results of already scheduled flows. Traditional heuristic algorithms struggle to make adaptive decisions based on the current network state, candidate path quality, and flow service requirements, and find it difficult to simultaneously consider reliability, latency constraints, scheduling success rate, and global load balancing.

[0005] Given the above situation, there is an urgent need for an intelligent optimization method for online incremental scheduling of time-sensitive networks. Under the condition that time-sensitive flows arrive randomly, it can dynamically generate low-overlap redundancy paths, adaptively select path combinations based on network link status, and complete incremental time slot allocation by combining a time-aware shaper. This can improve the link load balancing degree and scheduling success rate in high-load scenarios while meeting reliability and latency constraints. Summary of the Invention

[0006] The present invention proposes a link-aware and redundancy-weighted time-sensitive network routing scheduling method, which can at least solve one of the technical problems in the background art. It can be applied to scenarios such as online redundant path selection, time slot resource allocation, and gating control list generation for time-sensitive flows in time-sensitive networks based on time-aware shapers.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: A link-aware and redundancy-weighted time-sensitive network routing scheduling method, comprising the following steps performed by a computer device: Step 1: Construct the time-sensitive network topology and link time slot resource model. Represent the time-sensitive network as a directed graph structure, where nodes represent terminal hosts or switches, and directed links represent transmission links in the network. Using the time-period as the scheduling unit, divide the transmission resources of each link into several discrete time slots, and maintain the time slot occupancy status of each link within the current time-period.

[0008] Step 2: Obtain information on currently arriving time-sensitive flows. For each currently arriving time-sensitive flow, obtain its source node, destination node, period, packet size, end-to-end latency constraints, and reliability requirements, and use these as inputs for subsequent path generation and path combination decisions.

[0009] Step 3: Generate a reliability-oriented set of low-overlap redundancy paths. First, calculate the shortest path from the current time-sensitive flow source node to the destination node and use it as the initial member path. Then, dynamically adjust the link weights based on the number of times links in the selected paths have been used and the number of times links in the rejected candidate paths have been attempted. This ensures that links that have been used or repeatedly failed have a higher cost in subsequent path searches, thus guiding the path search process to avoid selected paths and reducing link overlap within the candidate redundant path set. This module dynamically adjusts link weights to ensure that links used by selected paths have a higher cost in subsequent searches, and combines path length, reliability, and overlap constraints to filter candidate paths.

[0010] Step 4: Construct the reinforcement learning state. The reinforcement learning state consists of current time-sensitive flow features, candidate redundant path features, and global link state features. Among them, flow features include source node, destination node, period, packet size, latency requirements, and reliability requirements; candidate path features include the set of links traversed by the candidate path or one-hot encoding of the links; global link state features include link utilization, normalized value of the maximum consecutive idle time slot, and link fragmentation index.

[0011] Step 5: Construct the path combination action space. Let the number of candidate redundant paths in the current flow be... Then the agent's action is defined as starting from... A non-empty subset of candidate paths is selected as the transport path combination for the current stream; therefore, the action space size is... .

[0012] Step 6: Utilize the PPO agent for path combination decisions. Input the current state into the Actor network. The Actor network outputs the probability distribution of each discrete path combination action, and samples or selects path combination actions based on this probability distribution. The selected action corresponds to a non-empty subset of candidate paths, which serves as the transmission path combination for the current time-sensitive stream.

[0013] Step 7: Perform online incremental time slot allocation. Based on the selected path combination, the scheduling environment allocates transmission time slots to each link of the selected path for the current flow, without altering the time slot allocation results of the already scheduled flows. If feasible time slots that satisfy link idle constraints, transmission order constraints, and end-to-end delay constraints can be found, the current flow scheduling is successful, and the time slot occupancy status of the relevant links is updated, generating the corresponding gating control list; if the above constraints cannot be satisfied, the current flow scheduling fails.

[0014] Step 8: Calculate the immediate reward. Calculate the immediate reward based on the current flow scheduling result, path combination reliability, end-to-end latency, link load balancing changes, and resource usage. The reward includes a scheduling success reward, a reliability reward, a latency reward, a load balancing reward, and a resource usage penalty. Specifically, the scheduling success reward encourages the model to prioritize path combinations capable of completing the scheduling; the reliability reward guides the agent to select path combinations that meet or exceed the flow reliability requirements; the latency reward encourages the model to select path combinations that meet end-to-end latency constraints and have lower transmission latency; the load balancing reward encourages a more even distribution of global link utilization after scheduling; and the resource usage penalty suppresses the agent from selecting too many redundant paths without constraints. The immediate reward is designed to simultaneously reflect factors such as scheduling feasibility, reliability constraints, latency constraints, load balancing, and resource usage.

[0015] Step 9: Store the current state obtained in Step 4, the path combination action obtained in Step 6, the reward value obtained in Step 8, the next state after executing the combination action, the termination flag, and the probability of the old strategy action into the experience pool.

[0016] Step 10: Calculate the dominance function and target value, and update network parameters. The Critic network is used to estimate state values, and the generalized dominance estimation method is employed to calculate the dominance function for each sample. The target value is then calculated using the dominance function and the value estimate. Subsequently, an importance sampling ratio is constructed based on the current policy probability and the old policy probability. The Actor network parameters are updated using the PPO pruning target function, and the Critic network parameters are updated using the value function loss. Simultaneously, a policy entropy regularization term is added to the Actor loss function, and an entropy coefficient annealing mechanism is used to maintain strong exploration capabilities in the early stages of training and improve policy stability in the later stages.

[0017] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0018] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0019] As can be seen from the above technical solution, the link-aware and redundancy-weighted time-sensitive network routing scheduling method of the present invention is used to solve the problems existing in the current online scheduling method for time-sensitive networks, such as high overlap of candidate paths, uneven distribution of link load, easy formation of hot links or link bottlenecks, difficulty in controlling multi-path resource consumption, and decreased scheduling success rate under high load scenarios. The present invention generates a set of low-overlap redundancy paths by introducing a dynamic link weight penalty mechanism, and uses a PPO agent to select transmission path combinations based on current flow characteristics, candidate path characteristics, and global link status. Then, the time-aware shaper online incremental scheduling module completes the time slot allocation and gating list generation, forming a closed-loop intelligent scheduling process for randomly arriving time-sensitive flows. The overall process is a closed-loop optimization process of "redundant path generation - intelligent path combination decision - path time slot allocation".

[0020] In summary, this invention discloses a joint optimization method for routing and scheduling based on link awareness and redundancy weighting in time-sensitive networks (TSNs), applicable to routing and scheduling scenarios in TSNs based on time-aware shapers. This method addresses the issues of random arrival of time-sensitive flows, flow-by-flow access, and dynamic changes in network link resources by constructing a closed-loop scheduling framework of "low-overlap redundancy path generation—PPO intelligent path combination decision—online time slot allocation for paths." First, based on the source node, destination node, latency requirements, and reliability requirements of the current arriving flow, a set of low-overlap candidate redundant paths is generated using a dynamic link weight penalty mechanism to reduce the proportion of shared links among multiple paths and improve the reliability of path combinations. Then, a reinforcement learning state containing flow characteristics, candidate path characteristics, and global link state characteristics is constructed, and the PPO agent selects transmission path combinations from a non-empty subset of the candidate path set. Finally, the scheduling module performs online incremental time slot allocation for the selected path combinations and calculates immediate rewards based on scheduling feasibility, end-to-end latency, path combination reliability, link load balancing, and resource occupancy, which are used to update the PPO policy network and value network. During training, generalized advantage estimation, PPO pruning objective function, and policy entropy coefficient annealing mechanism are employed to improve policy update stability and prevent premature convergence of action selection. Compared with existing scheduling methods based on shortest paths or fixed redundant paths, this invention can effectively distribute link load, reduce the probability of hot links or link bottlenecks, and improve the success rate of online incremental scheduling in high-load scenarios while satisfying the reliability and latency constraints of time-sensitive flows.

[0021] Compared with the prior art, the present invention has the following beneficial effects: It offers higher performance. This invention enables the system to adaptively select transmission path combinations based on the current network state through low-overlap redundancy path generation, PPO path combination decision-making, and collaborative optimization of transmission time slot allocation. Experimental results show that, compared with traditional routing and scheduling methods, this invention can improve scheduling success rate and path combination reliability, and effectively improve link load balancing performance.

[0022] This invention is highly innovative. It introduces maximum continuous idle time slots and fragmentation metrics on top of link utilization, improving the model's ability to perceive link schedulability. Simultaneously, it employs a low-overlap redundancy path generation algorithm to reduce link overlap between candidate paths, and comprehensively considers reliability, latency, load balancing, and resource consumption in the reward function, achieving joint optimization of routing and scheduling for the online incremental scheduling process of time-sensitive traffic in time-sensitive networks.

[0023] This invention has excellent applicability. It is suitable for online scheduling scenarios in time-sensitive networks where time-sensitive flows arrive randomly and network states change dynamically. It can select path combinations and allocate time slots based on different flow quantities, reliability requirements, latency constraints, and network load conditions. This method does not rely on offline planning with a fixed flow set, and has good engineering application value and scalability. Attached Figure Description

[0024] Figure 1 This is a diagram illustrating the overall architecture of this method; Figure 2 Generate an algorithm flowchart for this method path; Figure 3 This is a diagram illustrating the information interaction framework of this method. Figure 4 Flowchart of sampling and experience storage in this method; Figure 5 Flowchart of model advantage function calculation and network parameter update for this method; Figure 6 This is the training reward change curve of an embodiment of the present invention; Figure 7 This is a comparison of link load imbalance using different algorithms in embodiments of the present invention; Figure 8 This is a comparison of the scheduling success rates of different algorithms in embodiments of the present invention; Figure 9 This is a comparison of the reliability of selected actions using different algorithms in embodiments of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0026] like Figure 1 As shown, the link-aware and redundancy-weighted time-sensitive network routing scheduling method described in this embodiment executes the following steps via a computer device: S1: Construct a time-sensitive network topology and link time slot resource model, represent the time-sensitive network as a directed graph structure, use the supercycle as the scheduling unit, divide the transmission resources of each link into several discrete time slots, and maintain the time slot occupancy status of each link in the current supercycle. S2: Obtain information about the currently randomly arriving time-sensitive streams, including source node, destination node, period, packet size, end-to-end latency constraints, and reliability requirements; S3: Generate a set of low-overlap redundancy paths for reliability. First, calculate the shortest path from the current time-sensitive flow source node to the destination node as the initial member paths. Then, dynamically adjust the link weights based on the number of times the links in the selected paths are used and the number of times the links in the rejected candidate paths are tried. This makes links that have been used or repeatedly failed have a higher cost in subsequent path searches, thereby guiding the path search process to avoid the selected paths, reducing the link overlap within the candidate redundant path set, and combining path length, reliability, and overlap constraints to filter candidate paths. S4: Construct a reinforcement learning state, which consists of current time-sensitive flow features, candidate redundant path features, and global link state features. The flow features include source node, destination node, period, data packet size, latency requirements, and reliability requirements. The candidate path features include the set of links traversed by the candidate path or one-hot encoding of the links. The global link state features include link utilization, normalized value of maximum continuous idle time slot, and link fragmentation index. S5: Construct the path combination action space, assuming the number of candidate redundant paths in the current flow is . Then the agent's action is defined as starting from... Select a non-empty subset of candidate paths as the transport path combination for the current stream. The action space size is [size missing]. ; S6: Use PPO agents to make path combination decisions. Input the current state into the Actor network. The Actor network outputs the probability distribution of each discrete path combination action. Based on the probability distribution, sample or select path combination actions. The selected action corresponds to a non-empty candidate path subset as the transmission path combination of the current time-sensitive stream. S7: Perform online incremental time slot allocation. Based on the selected path combination, the scheduling environment allocates transmission time slots to the current flow on each link of the selected path without changing the time slot allocation result of the already scheduled flow. If a feasible time slot that satisfies the link idle constraint, transmission order constraint, and end-to-end delay constraint can be found, the current flow scheduling is successful, and the time slot occupancy status of the relevant links is updated, generating the corresponding gating control list; if the above constraints cannot be satisfied, the current flow scheduling fails. S8: Calculate the immediate reward based on the current flow scheduling result, path combination reliability, end-to-end latency, link load balancing changes, and resource usage. The reward includes scheduling success reward, reliability reward, latency reward, load balancing reward, and resource usage penalty. The immediate reward is designed to simultaneously reflect scheduling feasibility, reliability constraints, latency constraints, load balancing, and resource usage factors. S9: Store the current state obtained in step S4, the path combination action obtained in step S6, the reward value obtained in step S8, the next state after executing the combination action, the termination flag, and the probability of the old strategy action into the experience pool. S10: Calculate the advantage function and target value and update the network parameters. Estimate the state value using the Critic network and calculate the advantage function for each sample using the generalized advantage estimation method. Calculate the target value using the advantage function and value estimation. Then, construct the importance sampling ratio based on the current policy probability and the old policy probability. Update the Actor network parameters using the PPO pruning target function and update the Critic network parameters using the value function loss. At the same time, add a policy entropy regularization term to the Actor loss function and use an entropy coefficient annealing mechanism to maintain strong exploration ability in the early stage of training and improve policy stability in the later stage of training.

[0027] The following are detailed explanations: The first step is to abstract the Time-Sensitive Network (TSN) into a directed graph. ,in Represents a set of nodes. This represents a set of directed links. Nodes in a network include terminal hosts and switches; directed links... Represents a node Output ports and nodes The input port is connected. The scheduling process is performed in units of time intervals. Considering that flow requests arrive randomly during online incremental scheduling, to avoid frequent changes in time intervals due to new flows, a set of service periods supported by the system is pre-defined, and the least common multiple of the periods in this set is used as the scheduling time interval. Let a time interval be divided into... Each discrete time slot, for each link Maintain a length of The time slot occupancy vector for the link. In the The decision-making time step, i.e., the first step The next scheduling decision, if the flow In the If each time slot occupies resources, then define a binary variable: (1) This occupancy vector can be used to determine the current available time slot resources of a link, providing a basis for online incremental scheduling of subsequent flows. Based on this, the calculation of each link... Link utilization Normalized value of maximum continuous idle time slot length and degree of fragmentation Among them, "link utilization" reflects the current link resource occupancy level, "maximum continuous idle time slot length" reflects whether the link still has a relatively long continuous schedulable window, and "fragmentation index" reflects whether idle time slots are dispersed into multiple short segments. The link utilization calculation formula is: (2) in, Indicates the number of flows in the scheduled flow set. Assume a link The idle time slots within the current supercycle are divided into several consecutive idle segments: (3) in, Indicates the first The length of a consecutive free segment This indicates the number of idle time slots. The total number of idle time slots is: (4) The formula for calculating the maximum continuous idle time slot length is: (5) The normalized value is: (6) in, Definition of the first The proportion of each idle segment to the total idle time slots is: (7) The degree of link fragmentation is: (8) in, The larger the index, the more dispersed the idle time slots of the link are, and the fewer the continuous schedulable windows are; the smaller the index, the more concentrated the idle time slots are, which is more conducive to the time slot allocation of the first arriving flow.

[0028] The second step, during online incremental scheduling, involves time-sensitive flows entering the scheduling system one by one according to a random arrival pattern. For the currently arriving flow... Obtain its source node , destination node ,cycle Data packet size End-to-end delay threshold and reliability requirements Therefore, the flow passes through tuples express.

[0029] The third step involves calculating a set of redundant paths for the current time-sensitive flow using a low-overlap redundancy path generation algorithm based on dynamic weights. First, the shortest path from the source node to the destination node is calculated based on the current network topology, and this path is added to the candidate redundant path set. Then, to generate more low-overlap candidate paths, the algorithm maintains two link count counters: one is a link selection counter. One is used to record the number of times a link is used by the selected path; the other is a link attempt counter. This is used to record the number of times a link appears in rejected candidate paths. Before each search for a new candidate path, the link weight is dynamically updated based on the number of link selections and link attempts. (9) in: :link The number of times the selected path has been used; :link The number of times it appears in rejected candidate paths; Selected path penalty factor; Try the penalty factor.

[0030] If a link has been heavily used by selected paths or appears multiple times among candidate paths that do not meet the constraints, the search cost for that link is increased, causing subsequent path search processes to tend to avoid it. This reduces the overlap of links between different candidate redundant paths. For each newly generated candidate path, determine whether it meets the following conditions: First, the length of the candidate path cannot exceed the maximum number of hops constraint; Second, the maximum overlap between candidate paths and selected paths cannot exceed the current overlap threshold. Third, after candidate paths are added to the set, the reliability of the resulting path combinations should meet the reliability requirements of the current flow.

[0031] If a candidate path meets the above conditions, it is added to the redundant path set, and the number of times it has traversed a link is updated. If it does not meet the conditions, the candidate path is rejected, and the number of times it has attempted to traverse a link is updated. When a sufficient number of candidate paths cannot be obtained under the current overlap threshold, the overlap threshold is gradually relaxed to improve the success rate of path generation.

[0032] For a single path, its reliability is determined by the product of the probabilities of successful transmission across all links on the path. If the links... Packet loss rate Then the path The reliability can be expressed as: (10) in, For path The set of links on a path combination. For a multipath combination, its reliability represents the probability that at least one member path successfully transmits data. Simply assuming that each path is completely independent can easily lead to overestimating reliability when shared links exist. Therefore, this invention considers the link overlap relationship between paths when calculating the reliability of a multipath combination and uses the inclusion-exclusion principle for correction, thereby more accurately evaluating the reliability of the path combination. Let the path combination be... The formula for calculating the reliability of path combinations is: (11) Path overlap is used to measure the proportion of shared links between two or more paths. The calculation formula is: (12) in, Representing a path The number of links on the path. The higher the path overlap, the more links are shared between different member paths; the lower the path overlap, the better the link independence between redundant paths, which is beneficial to improving the reliability of multi-path transmission and distributing link load.

[0033] The fourth step, reinforcement learning state can be represented as (13) in, Represents the first step in the interaction process Each time step This represents the currently arriving time-sensitive stream. Indicates the current time-sensitive flow characteristics. This represents the path characteristics in the set of candidate redundant paths for the current flow. For global link state characteristics, This represents the matrix vectorization operation. For the state space dimension.

[0034] (14) in, and These represent the current time-sensitive streams. The source node and the destination node, This indicates a one-hot encoding operation for nodes. , , and These represent the normalized flow period, packet size, end-to-end latency threshold, and reliability requirement, respectively. For flow feature dimensions.

[0035] Let the current flow The generated set of candidate redundant paths is as follows: (15) The candidate path feature matrix is ​​then represented as: (16) in, Let be the number of candidate paths, for the The candidate paths are represented by their one-hot edges as follows: (17) (18) Representing a path The set of links traversed, This indicates the number of directed links in the network.

[0036] (19) in, For link In the decision-making step State characteristics: (20) Through the above state design, the PPO agent can simultaneously perceive current business needs, candidate path quality, and global network resource distribution, thereby providing a basis for decision-making on path combination selection.

[0037] Fifth step, assume the current stream is generated Candidate redundant paths, denoted as An agent's action is defined as selecting a non-empty subset of paths from the candidate path set as the transport path combination for the current stream. Therefore, the action space, which is the set of all non-empty subsets of the candidate path set, can be represented as: (twenty one) Its number of actions is To facilitate the neural network output and environment execution, each action is represented by an integer number, and the action number is mapped to a length of [value missing]. binary vector: (twenty two) (twenty three) Therefore, action The corresponding path combination can be represented as: (twenty four) Step 6: The PPO agent includes an Actor network. and Critic Network The Actor network outputs the probability distribution of combined actions for each path based on the current state; the Critic network evaluates the value of the current state. During the interaction, the state is... Inputting the Actor network yields the action probability distribution. Then, sampling actions are performed based on this probability distribution. and the action This is mapped to a specific path combination. This path combination is then submitted to the scheduling environment for path slot allocation. Since the action probability distribution is dynamically generated based on the current state, the PPO agent can adaptively select different path combinations based on the service requirements of different time-sensitive flows, the candidate path set, and the network resource status.

[0038] Step 7: After receiving the path combination action, the scheduling environment prioritizes scheduling the longer path. Overlapping links in subsequent paths can reuse existing time slot allocations. Time slot allocation is performed hop-by-hop for each member path in the path combination. The time slot allocation process must satisfy the following constraints: First, there is a link idle constraint, which means that the time slot occupied by the current flow cannot conflict with the time slot occupied by the scheduled flow on the same link.

[0039] Second, transmission order constraints, that is, the transmission times of the same data frame on each hop link on the path should meet the sequential order.

[0040] Third, end-to-end latency constraints, meaning the total transmission time from the source node to the destination node cannot exceed the latency threshold of the stream.

[0041] If the selected path combination can complete the time slot allocation that satisfies the constraints, the current flow scheduling is considered successful, and the time slot occupancy vector of the relevant links is updated, while a corresponding gating control list is generated. If no feasible time slots can be found, the current flow scheduling is considered a failure, and a failure penalty is returned.

[0042] Step 8: To guide the PPO agent to simultaneously optimize scheduling success rate, reliability, latency, and load balancing, this invention designs a comprehensive real-time reward function. This reward function includes the following parts: (25) in: Successful scheduling reward A positive reward is given when the current flow completes the feasible time slot allocation and meets the delay constraint.

[0043] Reliability Bonus : (26) in, To ensure the reliability of the path combination corresponding to the action; For flow Reliability requirements; A reward is given when the reliability of a path combination meets or exceeds the reliability requirements; otherwise, a penalty is imposed.

[0044] Delay Reward : (27) in, The maximum end-to-end delay of the path after the current flow is scheduled; For flow The end-to-end latency threshold, the latency reward is only calculated when scheduling is successful, that is... .

[0045] Load balancing rewards : (28) in, The standard deviation of the utilization of all links before scheduling. This represents the standard deviation of the utilization of all links after scheduling. This standard deviation measures the degree to which the utilization of each link deviates from the mean, thus assessing the overall balance. The formula for calculating the standard deviation is: (29) in, For link utilization rate At the same time The average utilization of all links in the network.

[0046] Compare the changes in the standard deviation of global link utilization before and after the current flow scheduling. If the link utilization distribution is more balanced after scheduling, a positive reward is given; if the link utilization distribution is more uneven after scheduling, the reward is reduced or a penalty is imposed.

[0047] Resource usage penalty : (30) in, For the post-scheduling link Link utilization For scheduling the link Link utilization. This penalty is used to prevent the agent from excessively selecting multiple paths, resulting in unnecessary resource consumption.

[0048] Through the above reward function, the agent can, while satisfying reliability and latency constraints, tend to choose path combinations that have reasonable resource consumption and are conducive to global load balancing.

[0049] Step 9: During PPO training, first fix the parameters of the old policy. For each scheduling moment, the agent outputs the action probability distribution according to the old policy and samples the actions. After the environment performs the action, a reward is returned. Next state and termination mark The samples generated at each interaction step are stored as follows: Among them, the probability of the old strategy action. It must be saved during sampling and cannot be recalculated after network parameters are updated to ensure the correctness of the PPO importance sampling ratio.

[0050] Step 10: After sampling, the state value is calculated using the Critic network, and the advantage function for each sample is calculated using the generalized advantage estimation method. Specifically, this is first calculated based on the immediate reward. Current state value estimation Next state value estimation and termination mark Calculate the timing difference error: (31) Then along Step sampling trajectory Calculate the advantage estimate for each step recursively from back to front: (32) Right now (33) in, This is called the smoothing parameter. This is called the discount factor. This represents the total number of sampling steps. If a step is the episode termination state... If the value is not propagated further, then subsequent values ​​will not be propagated. To improve training stability, the advantage function in the same sampling batch is standardized so that its mean is close to 0 and its standard deviation is close to 1, thereby reducing the impact of reward scale differences on network updates. In a sampling batch, we can obtain... One advantage estimate The standardization steps are as follows: (1) Calculate the mean : (34) (2) Calculate the standard deviation : (35) (3) Standardization updates: (36) in, It is a very small constant to prevent the denominator from being 0.

[0051] Calculate the target value using the advantage estimate and the state value estimated by the old value: (37) Step 12: After obtaining the advantage function and target value, randomly sample mini-batch samples from the experience pool for multiple rounds of parameter updates. For the Actor network, firstly, recalculate the probability of the sampled actions based on the current policy. and the probability of actions in the old strategy Construct importance sampling ratio: (38) Then, the policy update magnitude is limited using the PPO pruning objective function, which is: (39) in, This is a hyperparameter used to limit the maximum allowable proportion of changes between old and new policies, preventing excessive policy changes from occurring in a single update. Simultaneously, a policy entropy regularization term is added to the Actor loss function. Policy entropy measures the dispersion of action probability distributions; a higher entropy value indicates the agent engages in more exploration, while a lower entropy value indicates the policy tends towards certainty. The calculation formula is: (40) By maximizing policy entropy, we can prevent the policy from prematurely converging to a single path combination in the early stages of training. The Actor loss function after incorporating policy entropy can be written as: (41) To balance exploration during the initial training phase and stable convergence during the later training phase, this invention employs an entropy coefficient annealing mechanism. The annealing formula is as follows: (42) in, This represents the initial entropy coefficient, used to control the exploration intensity in the early stages of training. This represents the minimum entropy coefficient, used to prevent the entropy regularization term from completely disappearing. Indicates the current training round. This represents the total number of annealing steps, i.e., the total number of training rounds required for the entropy coefficient to decay from its initial value to its minimum. A larger entropy coefficient is used in the early stages of training to encourage the agent to explore different path combinations; as the number of training rounds increases, the entropy coefficient is gradually reduced, allowing the policy to smoothly transition from the exploration phase to the exploitation phase. For the Critic network, parameters are updated by minimizing the mean squared error between the state value estimate and the target reward. (43) This enables value networks to more accurately assess long-term scheduling benefits under different network states.

[0052] During the training phase, a predetermined number of time-sensitive flows are continuously scheduled as one episode. Once all time-sensitive flows in an episode have been scheduled, the network environment and link resource states are reset, and the next round of training begins. After training, the Actor network is retained as the online path combination decision model. In practical applications, when a new time-sensitive flow arrives, the system first generates a set of candidate redundant paths, then constructs the current state and inputs it into the trained Actor network. The Actor network outputs the probability values ​​of each behavior in the state's behavior space, and the behavior with the highest probability is used as the path combination selection result. Finally, the scheduling module completes the time slot allocation and gating list generation. Through this process, this invention enables intelligent incremental scheduling oriented towards reliability, latency constraints, and load balancing in online scenarios where time-sensitive flows arrive randomly.

[0053] The following are the simulation results of the embodiments of the present invention: 1. Algorithm training convergence Figure 6 The graph shows the training reward curve for the invented algorithm. The horizontal axis represents the number of training rounds, and the vertical axis represents the cumulative reward. The light blue curve represents the original reward for each round, and the orange curve represents the moving average reward with a window size of 50. As can be seen from the graph, the reward increases rapidly in the early stages of training, and then gradually stabilizes, indicating that the model policy converges continuously during training, and the scheduling decision-making ability is improved.

[0054] 2. Algorithm Performance Comparison Figures 7-9The performance comparison of three algorithms—RB-PPO (the algorithm of this invention), DQN, and RMRSA—under different test flow numbers is presented. The horizontal axis represents the number of flows in the test scenario, and the vertical axis represents the link load imbalance value, average scheduling success rate, and average selected action reliability, respectively.

[0055] As shown in the figure, with the increase in the number of test flows, the link load imbalance values ​​of the three algorithms generally increase, while the scheduling success rate generally decreases, indicating that increased network load increases scheduling difficulty. Compared with DQN and RMRSA, RB-PPO has a lower link load imbalance value and maintains a higher scheduling success rate in most load scenarios. At the same time, the reliability of the selected actions in RB-PPO remains at a high level, indicating that this algorithm can better balance reliability constraints and resource utilization during path selection.

[0056] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0057] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0058] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the link-aware and redundancy-weighted time-sensitive network routing scheduling methods described in the above embodiments.

[0059] It is understood that the systems, devices, and storage media provided in the embodiments of the present invention correspond to the methods provided in the embodiments of the present invention, and the explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.

[0060] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0061] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0062] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0063] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A link-aware and redundancy-weighted time-sensitive network routing scheduling method, characterized in that, Includes the following steps, S1. Construct a directed graph topology model for time-sensitive networks, discretize link transmission resources into time slots using time-periods as the scheduling unit, and maintain the occupancy status of link time slots; S2. Obtain the source node, destination node, period, data packet size, end-to-end latency constraints, and reliability requirements of the current time-sensitive stream; S3. Based on the dynamic link weight adjustment mechanism, a set of redundant path candidates with low overlap is generated. The link weight is adaptively updated according to the number of times it is used in the selected path and the number of attempts in the rejected path, so as to guide the path search to avoid the selected path and reduce the link overlap. S4. Construct a reinforcement learning state space, which consists of flow features, candidate path features, and global link state features; S5. Construct a path combination action space, wherein the action is defined as selecting a non-empty subset from the candidate path set as the transmission path combination of the current stream; S6. Use the PPO agent to make path combination decisions. Input the current state into the Actor network, output the probability distribution of each path combination action, and sample or select actions. S7. Perform online incremental time slot allocation. Based on the selected path combination, without changing the time slot allocation of the scheduled flow, allocate transmission time slots that meet the constraints on each path link for the current flow, and update the link time slot occupancy status. S8. Calculate the real-time reward based on the scheduling results, path combination reliability, end-to-end latency, changes in link load balancing, and resource usage. S9. Store the current state, path combination action, immediate reward, next state, termination flag and old strategy action probability into the experience pool. S10. Based on the experience pool samples, the state value is estimated using the Critic network, the advantage function is calculated using generalized advantage estimation, the Actor network parameters are updated by pruning the objective function through PPO, and the Critic network parameters are updated by the value function loss. At the same time, an entropy regularization term is added to the Actor loss and an entropy coefficient annealing mechanism is adopted.

2. The link-aware and redundancy-weighted time-sensitive network routing scheduling method according to claim 1, characterized in that: In S3, adaptive updates use a dynamic update formula for link weights: in: :link The number of times the selected path has been used; :link The number of times it appears in rejected candidate paths; Selected path penalty factor ; Try the punishment factor. ; The initial weight of the link is At this point, the algorithm finds the path with the fewest hops and dynamically updates the link weights based on the search results. The selected path penalty is determined by the number of times the link has been used by the selected path, and the attempt penalty is determined by the number of times the link appears in the rejected candidate paths. The penalty increases exponentially to guide subsequent candidate paths to avoid links that have been used frequently or have failed to be tried multiple times.

3. The link-aware and redundancy-weighted time-sensitive network routing scheduling method according to claim 2, characterized in that: In step 3, when screening candidate paths, it is simultaneously determined whether the candidate path meets the maximum hop count constraint, path overlap constraint, and path reliability constraint. If the candidate path meets the above constraints, it is added to the candidate redundant path set. If it does not meet the constraints, the number of attempts of the links traversed by the candidate path is updated. When a sufficient number of candidate redundant paths cannot be obtained under the current overlap threshold, the overlap threshold is gradually relaxed to improve the success rate of path generation.

4. The link-aware and redundancy-weighted time-sensitive network routing scheduling method according to claim 3, characterized in that: In step 3, single-path reliability is calculated based on the probability that all links on the path successfully transmit, using the following formula: in, For path The set of links on; For link The packet loss rate; The reliability of multipath combination is calculated based on the probability of successful transmission of at least one member path. When calculating the reliability of multipath combination, the link overlap relationship between different member paths is considered, and the principle of inclusion-exclusion is used to correct the reliability overestimation caused by shared links. Let the path combination of the flow be The formula for calculating the reliability of multi-path combinations of a flow is: 。 5. The link-aware and redundancy-weighted time-sensitive network routing scheduling method according to claim 1, characterized in that: In step 4, the reinforcement learning state is represented as follows: in, The first one represents the interaction with the environment. Each time step This represents the currently arriving time-sensitive stream. Indicates flow characteristics, This represents the path feature matrix in the set of candidate redundant paths for the current flow. Represents the global link state matrix. This represents the matrix vectorization operation. For state space dimension; in, and These represent the current time-sensitive streams. The source node and the destination node, This indicates a one-hot encoding operation for nodes. , , and These represent the normalized flow period, packet size, end-to-end latency threshold, and reliability requirement, respectively. For flow feature dimensions; in, Let be the number of candidate paths, for the The candidate paths are represented by their one-hot edges as follows: in, Representing a path The set of links traversed, This indicates the number of directed links in the network; link At time step The characteristics are: in, Indicates link At any moment Link utilization; Indicates link At any moment The normalized value of the maximum consecutive idle time slots; Indicates link At any moment Fragmented metrics.

6. The link-aware and redundancy-weighted time-sensitive network routing scheduling method according to claim 5, characterized in that: The formula for calculating the link utilization rate is as follows: in, Indicates the number of flows in the scheduled flow set. Assume link The idle time slots within the current supercycle are divided into several consecutive idle segments: in, Indicates the first The length of a consecutive free segment This indicates the number of idle segments. The total number of idle time slots is: The maximum number of consecutive idle time slots is: The normalized value of the maximum consecutive idle time slot number is: in, ; Define the first The proportion of each idle segment to the total idle time slots is: The link fragmentation metric is: in, The larger the index, the more dispersed the idle time slots of the link are and the fewer the continuous schedulable windows; the smaller the index, the more concentrated the idle time slots are, which is more conducive to the time slot allocation of the first-arriving flow.

7. The link-aware and redundancy-weighted time-sensitive network routing scheduling method according to claim 1, characterized in that: In step 8, the immediate reward consists of scheduling success reward, reliability reward, latency reward, load balancing reward, and resource occupancy penalty, namely: Among them, the scheduling success reward is used to characterize whether the current time-sensitive flow has successfully completed the time slot allocation, the reliability reward is used to characterize the degree to which the reliability of the selected path combination meets the reliability requirements of the flow, the latency reward is used to characterize the degree to which the end-to-end latency of the current flow meets the latency threshold, the load balancing reward is used to characterize the change in the standard deviation of global link utilization before and after scheduling, and the resource occupation penalty is used to limit the link resource consumption caused by the agent excessively selecting multiple redundant paths.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it causes the processor to perform the steps of the method as described in any one of claims 1 to 7.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, it causes the processor to perform the steps of the method as described in any one of claims 1 to 7.

Images