Method for multi-objective routing of traffic flows in a cross-domain deterministic data center network

Abstract

The application discloses a kind of service flow multi-objective routing methods for cross-domain deterministic data center network.The system receives service flow request by deterministic service controller and signs service level agreement, constructs global weighted graph by comprehensively each autonomous domain's topology and resource information.Controller establishes the multi-objective optimization model of simultaneously minimizing end-to-end delay, maximizing scheduling success rate and improving bandwidth utilization, and solves by using the deep reinforcement learning framework of fusion graph attention network and Transform, generates optimal routing and time slot resource arrangement table that meet SLA constraint, realizes the deterministic forwarding and intelligent dynamic scheduling of service flow under cross-domain deterministic data center environment.The application has intelligent self-adaptation, efficiency and robustness, can realize low latency, high reliability and high resource utilization in complex cross-domain scene, significantly improves the service quality, scalability and running stability of deterministic data center interconnection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of service flow routing in cross-domain deterministic data center networks, and in particular to a multi-service flow routing method based on a graph attention network and a Transformer deep reinforcement learning G-TDRL designed for a cross-domain deterministic network environment. BACKGROUND

[0002] In recent years, with the rapid development of cloud computing, artificial intelligence, and large-scale distributed storage, the demand for data center interconnection (DCI) is increasing. By transmitting data between different data centers through high-speed links, enterprises can achieve cross-regional resource sharing, load balancing, and collaborative computing. However, with the continuous growth of data size and business types, if only relying on traditional IP networks or static links, the network will face problems such as increased end-to-end latency, bandwidth congestion, and decreased reliability, making it difficult to meet the urgent needs of emerging businesses such as real-time video analysis, remote medical diagnosis, smart grid scheduling, and cross-center AI model training for ultra-low latency and high reliability. Especially in the context of complex cross-domain interconnection, the non-deterministic characteristics of the existing IP network architecture further amplify the transmission latency fluctuations and service instability, making it difficult to achieve predictable and controllable communication performance.

[0003] To make up for the shortcomings of traditional networks in low latency and high reliability, deterministic networking (DetNet) has emerged. DetNet achieves controllability of latency, jitter, bandwidth, and packet loss rate through end-to-end resource reservation and precise scheduling, and has become an important technology to support multiple types of critical task-based businesses. In scenarios such as industrial Internet, vehicle networking, and remote medical care that are highly sensitive to latency and reliability, DetNet effectively alleviates the problems of uncontrollable latency, bandwidth congestion, and insufficient reliability under traditional network architecture through mechanisms such as path reservation, fine-grained scheduling, and PREOF (Packet Replication, Elimination, and Ordering Function). In cross-data center interconnection (DCI), DetNet further achieves deterministic guarantee of cross-regional service flow through joint control of the service layer and the transport layer, significantly improving link utilization and fault tolerance, and thus becoming one of the key technologies to support high-performance DCI.

[0004] With the evolution of business requirements, DetNet research gradually extends from single domain to cross-domain, and deeply integrates with key technologies such as Time-Sensitive Networking (TSN) and Segment Routing (SR) to achieve deterministic transmission in a wide range. However, cross-domain DetNet faces many new technical challenges, including cross-domain clock synchronization and scheduling consistency, routing optimization and resource allocation in complex topologies, inter-domain coordination and control interface standardization, etc. In the future, it still needs to further focus on multi-tenant isolation, heterogeneous network coordination, cross-data center large-scale deployment, and energy efficiency optimization. Overall, DetNet is moving from the laboratory to commercial deployment, and is expected to play a key role in cross-data center interconnection and other key infrastructure, promoting the next generation of information infrastructure and industry digital transformation.

[0005] Although the IETF DetNet working group has clearly defined the overall architecture of deterministic networks in RFC 8655, and defined key technologies such as data plane mechanisms (MPLS and IP-based forwarding), service layer functions, and PREOF in subsequent standards, its standard system still mainly focuses on data forwarding and service guarantee framework, and has not yet given a unified specification for end-to-end traffic routing and scheduling methods. Existing implementations mostly rely on extensions of RSVP-TE or Segment Routing for resource reservation, or by centralized controllers (such as PCE / SDN) to calculate paths. However, in the cross-domain scenario facing large-scale multi-service flows, such methods are difficult to simultaneously meet the multi-objective requirements of low latency, high reliability, and high resource utilization, becoming a key bottleneck hindering the commercialization and landing of DetNet.

[0006] On this basis, the existing routing and scheduling methods face more prominent challenges in the cross-domain scenario. Most current technologies use heuristic methods and deep reinforcement learning (DRL, Deep Reinforcement Learning) methods to solve them, but as the scenario shifts from single domain to cross domain and the complexity of business requirements increases, their limitations become increasingly apparent:

[0007] 1. Traditional static or heuristic methods lack adaptability and can usually only achieve better results under specific topological structures and business conditions. They are often slow to respond to dynamically arriving service requests and link state fluctuations in cross-domain networks, making it difficult to consistently guarantee end-to-end deterministic performance. More critically, heuristic algorithms rely on artificial rules and lack universality, making it difficult to generalize to cross-domain scenarios of different scales and business patterns.

[0008] 2. Although exact optimization methods (such as integer programming and linear programming) can theoretically find the global optimal solution, their computational complexity increases exponentially with the network size. They cannot meet the real-time scheduling requirements in large-scale cross-domain environments, which limits their application in actual deployment.

[0009] 3. Deep reinforcement learning (DRL)-based methods have shown potential in dynamic decision-making, enabling a shift from static open-loop to dynamic closed-loop. However, most existing studies have failed to adequately model multi-domain collaborative models from single-domain to cross-domain scenarios, and only optimize for single objectives (such as minimum latency or maximum reliability), lacking a holistic consideration of resource utilization, scheduling efficiency, and latency. Furthermore, DRL methods still suffer from insufficient applicability when facing concurrent multi-service flows, cross-domain coordination, heterogeneous domain control differences, and strict SLA constraints. The training process often requires a large number of samples and computational resources, making it difficult to directly transfer to real-world cross-domain environments.

[0010] In summary, DetNet still has significant room for improvement in cross-domain service flow routing and scheduling, which provides new research opportunities for developing multi-objective optimization methods that can simultaneously achieve low latency, high scheduling efficiency, and high resource utilization. Summary of the Invention

[0011] The purpose of this invention is to provide a multi-target routing method for service flows in cross-domain deterministic data center networks.

[0012] The technical solution to achieve the purpose of this invention is: a multi-target routing method for service flows in cross-domain deterministic data center networks, comprising the following steps:

[0013] Step 1: The user service flow initiates a service access request to the deterministic service controller in the scheduling plane; the deterministic service controller signs a Service Level Agreement (SLA) with the service flow according to the user service requirements to clarify the reliability, end-to-end latency, bandwidth and resource constraints of the service flow; at the same time, the source host of the user service flow waits for the deterministic service controller to issue the routing configuration result and the corresponding time slot resource orchestration table for deterministic forwarding after the service is established.

[0014] Step 2: Each Autonomous Domain Controller (ADC) acquires network information for its own domain. This network information includes: node geographic location information, link propagation delay, node port time slot resource information, and link status information. Each ADC aggregates the network information to the Deterministic Service Controller (DSC), which integrates the data to form a global network topology view. The cross-domain network is then abstracted into a weighted directed graph, where the edge weights reflect link propagation delay, bandwidth utilization, and reliability metrics.

[0015] Step 3: The deterministic service controller combines user service demand information with the topology information of the cross-domain deterministic data center network to establish a multi-objective optimization model that comprehensively considers latency, bandwidth and scheduling success rate. This model aims to minimize end-to-end latency of service flows, improve the scheduling success rate of time-triggered (TT) service flows and improve bandwidth utilization, thus transforming the routing and scheduling problem of the cross-domain deterministic network into a multi-objective optimization problem that satisfies the service level agreement constraints.

[0016] Step 4: The deterministic service controller solves the multi-service flow routing problem based on service flow information, cross-domain deterministic data center network topology information, and a multi-objective optimization model. During the solution process, a fusion graph attention network (GAT) and a Transformer-based deep reinforcement learning algorithm framework (G-TDRL) are used to model the complex cross-domain network state and perform dynamic decision-making. The reinforcement learning policy network automatically generates a set of optimal routing paths for multiple service flows that satisfy SLA constraints and the corresponding time slot resource orchestration table according to the state-action-reward mechanism.

[0017] Step 5: The deterministic service controller distributes the time slot resource orchestration table corresponding to the calculated optimal route path set to each autonomous system controller (AS / A). Each AS / A performs scheduling and forwarding in its local data plane based on the optimal route configuration and time slot resource orchestration table allocated by the deterministic service controller, thereby achieving end-to-end deterministic transmission of service flows in a cross-domain environment. Through path reservation, time slot synchronization, and distributed scheduling coordination mechanisms, the delay and reliability requirements of the service flows are ensured to meet the SLA constraints.

[0018] Furthermore, in step 1, the user service flow includes a time-sensitive flow (TT) and a best-effort flow (BE); wherein, the time-sensitive flow... By six-tuple Indicates; among which The source node representing the business flow. Indicates the destination node of the business flow. The periodicity attribute represents the time-sensitive flow and is used to describe its time window constraints in periodic scheduling. This represents the upper bound of the latency of the business flow. This indicates the bandwidth requirements of the service flow. Indicates the priority of the business flow.

[0019] Furthermore, in step 2, the cross-domain deterministic data center network environment is abstracted as a weighted directed graph. ,in, Indicates by A set of nodes consisting of 100 nodes Represents a set of links, each link Represents a node and Communication links between; links By binary It means that, among them This indicates the transmission delay of the link. This indicates the bandwidth resources included in the link.

[0020] Furthermore, the multi-objective optimization model established in step 3 includes the following objectives: minimizing the service latency of time-sensitive flow (TT) and best-effort flow (BE), maximizing the scheduling success rate of time-sensitive flow, and improving the utilization rate of bandwidth resources; the objective function of the optimization model is specifically defined as follows:

[0021] 3.1 Objective function 1 is to minimize the average delay of the TT and BE streams.

[0022] The weighted average latency of the TT and BE streams is the overall average latency of the user's service flow. The delay calculation method is as follows:

[0023]

[0024] in, As a variable, it represents the business flow. From the host node The offset time slot when accessing its adjacent TSN domain. and These represent the business flow respectively. Queuing delays in intra-domain switches or routers and offset time slots during cross-domain transmissions;

[0025] Objective function 1 is:

[0026]

[0027] in, Represents a set of time-sensitive streams. Represents a set of streams that do their best. This indicates the number of TT streams, while Indicates the number of BE streams;

[0028] 3.2 Objective function 2 is to maximize the scheduling success rate of TT stream.

[0029] The number of successfully scheduled TT streams is defined as follows:

[0030]

[0031] Objective function 2 is:

[0032]

[0033] in, It is a 0-1 variable; when its value is 1, it represents a time-sensitive flow. By the deadline The target node was successfully reached within the time limit; a value of 0 indicates that the target node was not reached within the time limit.

[0034] 3.3 Objective function 3 is to maximize resource utilization.

[0035]

[0036] This formula is defined as a comparison between the number of time slot resources occupied by all successfully scheduled flows and the number of time slot resources consumed in the network environment.

[0037] Furthermore, the average latency of the service flow in step 3.1 needs to be normalized, expressed as:

[0038] The normalized definition of the average latency of a business flow is:

[0039]

[0040] In the formula, This indicates the number of time-sensitive flows in a cross-domain deterministic data center network. This represents the number of best-effort flows in a cross-domain deterministic data center network. This represents the upper limit of the single-stream delay baseline for TT streams. This indicates the upper limit of the single-stream reference for the BE stream.

[0041] Furthermore, the number of successfully scheduled time-sensitive flows in step 3.2 needs to be normalized, specifically expressed as follows:

[0042] The normalized definition of the number of successful schedulings for time-sensitive streams is:

[0043]

[0044] Furthermore, steps 3.1 and 3.2 need to consider the constraints of multiple services to implement a multi-objective service flow routing optimization model, the constraints of which are as follows:

[0045] 1): Time slot offset limitation

[0046]

[0047]

[0048] In the formula, This is the number of queues on the switch / router. This constraint ensures that the time slot offset of each TT flow within a certain domain of a deterministic network does not exceed the queue rotation period of that domain; where, Indicates from the first The edge router of the domain to the first The time slot offset of the edge routers of each domain must be less than the queue round-robin period of the next domain;

[0049] 2): Access time slot restrictions

[0050]

[0051] In the formula, Represents a stream The period, in milliseconds (ms). Indicates the first The size of the time slot in each domain;

[0052] 3): Maximum latency limit for services

[0053] Business Flow The total latency from the source node to the destination node satisfies the following constraint:

[0054]

[0055] The total latency includes: the access time slot offset generated when the service flow enters the system; the queuing time slot offset generated during the intra-domain or cross-domain transmission of the service flow; and the propagation latency generated when the service flow is transmitted on the link. These constraints ensure that the service flow... The end-to-end delay does not exceed its preset deadline. ;

[0056] 4) Limitations of time slot resources

[0057] In a cross-domain deterministic data center network environment, resource constraints consist of the following two parts:

[0058] ①

[0059] ②

[0060] Constraint ① limits the total time slot queue resources occupied by a service flow on a node path to not exceeding the available resource capacity of the selected path link within any given system time. Constraint ② is used to define the total time slot resource capacity of the path links selected by the service flow at any given time, where, Indicates the size of a single time slot. This indicates the number of queues in this field.

[0061] Furthermore, in step 4, the G-TDRL deep reinforcement learning framework based on GAT and Transformer is used to obtain path routing selection for multiple service flows, specifically as follows:

[0062] 4.1 The encoder first maps the topology information of the cross-domain deterministic network environment, including network device locations, link latency, link status, and service flow information, to a high-dimensional space through a linear layer; then it maps node features and edge features to a higher-dimensional space through a connection layer. and In the dimensional embedding space, the initial node and edge information representation is obtained; the specific formula is as follows:

[0063]

[0064]

[0065] in, Represents the 0th level node Embedded, Indicates a Layer 0 link Embedded, This indicates the embedding of business flow information at layer 0. , , and , , These are learnable parameters; through the above operations, the initial node and edge information is embedded into a high-dimensional space and provides input for subsequent deep learning models.

[0066] After transformation, node and edge features are embedded into a high-dimensional representation of business flow information, specifically as follows:

[0067]

[0068] in, Indicates the 0th layer The embedded business flow information includes the source node, destination node, periodicity attribute, bandwidth requirement, maximum latency limit and priority information of the flow;

[0069] Deterministic cross-domain data center network environment information includes time slot resource status characteristics; first, this raw data undergoes a linear transformation through the input layer to obtain the initial time slot resource status; the status of nodes and links is mapped to a space of the same dimension as network topology and service information through linear transformation;

[0070] The obtained initial embedding information of the network topology , The final embedding information is obtained by processing L identical residual E-GAT modules;

[0071] Node information in a cross-domain deterministic data center network environment is

[0072]

[0073] in, For the first The mean of node features output by layer graph convolution. For the first The feature vector of each node The total number of nodes;

[0074] Business information in a cross-domain deterministic data center network environment is

[0075] )

[0076] in, For the first Mean of business flow characteristics of the layer For the first Feature vectors of each business flow For the number of business flows;

[0077] The time slot resource information for cross-domain deterministic data center networks is

[0078]

[0079] in, For the first Mean value of time slot resource characteristics of the layer For the first Feature vector of each time slot resource The number of time slots;

[0080] Finally, all the embedded information is connected through a linear layer to obtain global information.

[0081]

[0082] The mean() function represents the mean operation, and || represents the feature concatenation operation. This is the weight matrix for the second fully connected layer. The bias term; the final fused result For cross-domain deterministic data center networks in the first Global state information of the layer;

[0083] 4.2 The decoding process models the cross-domain deterministic data center network routing problem as a Markov decision process (MDP). In this process, the cross-domain network topology, global link information, the time slot resource status of each domain, and the service requirements of the business flow are regarded as a set of states, which are used to characterize the environmental characteristics of the system at each moment. The action set corresponds to selecting the next-hop node for the current business flow from the candidate node set, and then gradually constructing the complete routing path of the business flow. The state transition function is jointly determined by the currently selected node and the available resources of the deterministic cross-domain data center network, while the reward function comprehensively considers end-to-end latency, bandwidth constraints, reliability, and scheduling success rate indicators to reflect the impact of routing decisions on overall performance.

[0084] 4.3 The decoder uses contextual information from the encoder, takes the current business flow as the query vector, and candidate nodes as key and value vectors, and combines them with masking rules to generate the probability distribution of the business flow selecting different candidate nodes; the agent selects the next-hop node at each time step according to the deep reinforcement learning DRL policy until a complete routing path that satisfies the SLA constraint is generated.

[0085] 4.4 After obtaining a set of multiple candidate routing paths, the deterministic service controller uses a reinforcement learning method based on the improved baseline REINFORCE algorithm to train and optimize the policy network, so as to evaluate and update the policy parameters and obtain the final path selection result that can optimally match the cross-domain network state and service flow requirements. The reinforcement learning process includes policy initialization, sample collection, baseline update, policy optimization, parameter update and policy output, which are used to realize dynamic adaptive routing and time slot scheduling of cross-domain deterministic data center networks.

[0086] Furthermore, the reinforcement learning process of the reinforcement learning method based on the improved baseline REINFORCE algorithm includes the following steps:

[0087] 1. Initialization steps: Establish the policy network and the baseline policy network, and initialize the parameters respectively. and Set the learning rate, decay coefficient, discount factor, and other hyperparameters; perform pre-training by interacting with the environment, collecting initial sample data of state, action, and reward; and 1. Parameter sets of the policy network and the baseline policy network; 2. Sample collection steps: In each training iteration, based on the interaction between the current policy network and the environment, multiple complete state-action-reward sequences are generated, the cumulative reward value of each sequence is calculated, and it is used as the input of training samples;

[0088] 3. Baseline Update Steps: A baseline policy network is used to replace the traditional critic network to estimate the reference reward for the current state; the parameters of the baseline policy network from the previous cycle are frozen during each training round. This serves as a stable reference.

[0089] 4. Policy optimization steps: Calculate the policy advantage value based on the outputs of the current policy network and the baseline policy network; use the policy advantage value to optimize the policy network parameters. Perform gradient direction updates; when the policy performance metrics are better than the baseline reference values, copy the current policy network parameters to the baseline policy network to implement a dual-actor structure to improve training stability.

[0090] 5. Parameter update steps: The Adam optimization algorithm is used to update the parameters of the policy network and the baseline policy network; the learning rate is dynamically decayed during training to prevent gradient oscillation and overfitting; after each K training rounds, the model performance is evaluated for reliability, and the learning rate and update frequency are adjusted according to the evaluation results, where K is the preset number of training rounds.

[0091] 6. Strategy Output Steps: When the model's average return or performance metric reaches a preset threshold, training is stopped; the final policy network parameters are used as the optimal policy output and applied to the dynamic routing and time slot scheduling process of cross-domain deterministic data center networks.

[0092] Compared with existing technologies, the present invention has the following significant advantages: (1) The present invention integrates the structural advantages of Graph Attention Network (GAT) and Transformer in cross-domain deterministic network scenarios, and can simultaneously capture network topology features and cross-domain latency correlation, thereby maintaining high adaptability and stability even when dynamic service flows arrive and link states fluctuate. (2) Compared with traditional schemes that rely on integer programming or heuristic methods, the present invention's dynamic decision-making mechanism based on deep reinforcement learning significantly reduces computational complexity and can meet the real-time routing requirements in large-scale cross-domain environments. (3) At the same time, the present invention effectively guarantees the Service Level Agreement (SLA) requirements of service flows during transmission by introducing access time slot limits, end-to-end latency limits, and resource capacity constraints, avoiding waste caused by excessive resource redundancy and improving the utilization efficiency of link and node resources.

[0093] In summary, this invention demonstrates stronger real-time performance, reliability, and scalability in cross-domain deterministic network routing, and can support mission-critical services with extremely high requirements for low latency and high reliability, such as data center interconnection, vehicle networking, and telemedicine. Attached Figure Description

[0094] Figure 1This is a schematic diagram of the overall architecture for service flow routing and scheduling in cross-domain deterministic data center networks according to the present invention.

[0095] Figure 2 This is a flowchart illustrating a multi-service flow routing method that satisfies a Service Level Agreement (SLA) in a cross-domain deterministic data center network environment, according to the present invention.

[0096] Figure 3 This is a schematic diagram illustrating the process of solving multi-service flow routing and scheduling using deep reinforcement learning G-TDRL based on GAT and Transformer proposed in this invention. Detailed Implementation

[0097] The present invention will now be further described with reference to the accompanying drawings and embodiments.

[0098] To address the aforementioned issues, this invention introduces a deterministic service controller to achieve dynamic routing and scheduling of cross-domain service flows. Specifically, the topology information and resource status of the cross-domain network are fed back to the deterministic service controller in real time. The deterministic service controller generates routing configurations and time slot orchestration tables based on service requirements and SLA constraints, and then distributes them to the data plane for execution. Simultaneously, the deterministic service controller can continuously process dynamically arriving new service requests, realizing a closed-loop routing and scheduling mechanism in the cross-domain deterministic network. Based on this, the proposed technical solution mainly includes the following steps:

[0099] Step 1: The user service flow initiates a service access request to the deterministic service controller in the scheduling plane; the deterministic service controller signs a Service Level Agreement (SLA) with the service flow according to the user service requirements to clarify the reliability, end-to-end latency, bandwidth and resource constraints of the service flow; at the same time, the source host of the user service flow waits for the deterministic service controller to issue the routing configuration result and the corresponding time slot resource orchestration table for deterministic forwarding after the service is established.

[0100] Step 2: Each Autonomous Domain Controller (ADC) acquires network information for its own domain. This network information includes: the geographical location information of nodes, the propagation delay of links, the time slot resource information of node ports, and the link status information. Each ADC aggregates this network information to the Deterministic Service Controller (DSC), which integrates it to form a global network topology view. The ADC then abstracts the cross-domain network into a weighted directed graph, where the edge weights reflect the propagation delay, bandwidth utilization, and reliability indicators of the links.

[0101] Step 3: The deterministic service controller combines user service demand information with the topology information of the cross-domain deterministic network to establish a multi-objective optimization mathematical model that comprehensively considers factors such as latency, bandwidth, and scheduling success rate. The model aims to minimize end-to-end latency of service flows, improve the scheduling success rate of time-triggered service flows (TT flows), and improve bandwidth utilization, thus transforming the routing and scheduling problem of the cross-domain deterministic network into a multi-objective optimization problem that satisfies the service level agreement constraints.

[0102] Step 4: The deterministic service controller solves the multi-service flow routes based on service flow information, cross-domain network topology information, and a multi-objective optimization model. During the solution process, a fusion graph attention network (GAT) and a graph-transformer deep reinforcement learning (G-TDRL) algorithm framework are used to model the complex cross-domain network state and perform dynamic decision-making. The reinforcement learning policy network automatically generates a set of multiple candidate routing paths that satisfy SLA constraints and a corresponding time slot resource orchestration table according to the state-action-reward mechanism.

[0103] Step 5: The deterministic service controller distributes the calculated optimal routing configuration and the corresponding time slot resource orchestration table to each autonomous system controller (AS / A). Each AS / A performs scheduling and forwarding in its local data plane based on the routing configuration and time slot resource orchestration table, thereby achieving end-to-end deterministic transmission of service flows in a cross-domain environment. Through path reservation, time slot synchronization, and distributed scheduling coordination mechanisms, the delay and reliability requirements of the service flows are ensured to meet the SLA constraints.

[0104] Due to the highly complex and multidimensional nature of cross-domain deterministic data center networks and service information, traditional algorithms struggle to efficiently process high-dimensional complex data and fail to characterize the nonlinear relationships within it. To address this, the proposed method employs a Graph Attention Network (GAT) to extract structural features of the network topology, utilizes a Transformer to encode service requirements and time slot resource status, and maps multi-source information to a high-dimensional space in the encoder module. Finally, a decoder generates routing decisions for cross-domain multi-service flows. This process effectively models complex nonlinear relationships, enabling efficient processing of high-dimensional data and ensuring that the selected path meets Service Level Agreement (SLA) requirements under low latency and high reliability conditions.

[0105] The following is in conjunction with the appendix Figure 2 Appendix Figure 3 The present invention will be further described in conjunction with the embodiments.

[0106] 1. User service flows issue access requests to access the corresponding edge domain (TSN domain), and sign a Service Level Agreement (SLA) with the deterministic service controller in the scheduling plane. Meanwhile, in the data plane, the source host of the flow awaits the routing configuration and time slot orchestration results from the deterministic service controller.

[0107] The Service Level Agreement (SLA) signed between the user and the deterministic service controller primarily includes service flow latency and resource requirements. The deterministic service controller must provide services to the user in accordance with the SLA. The service flow consists linearly of time-sensitive flows and best-effort flows. The Each business flow is defined by a tuple. ,in The source node representing the business flow. Indicates the destination node of the business flow. This indicates the period of a time-sensitive flow (not applicable to best-effort flow). This represents the upper bound of the latency of the business flow. This indicates the bandwidth requirements of the service flow. Indicates the priority of the business flow.

[0108] 2. The deterministic service controller obtains network topology information of the cross-domain deterministic network and abstracts it into an undirected network graph.

[0109] The deterministic cross-domain data center network environment is modeled as a weighted directed graph using network modeling methods. , express A set consisting of nodes It is a set of links in a network environment, where Represents a node and Links between; links Composed of binary pairs ,in , Representing links respectively Transmission latency and bandwidth resources.

[0110] 3. By combining business information and edge computing network topology information, establish a multi-objective optimization model with routing constraints for multiple service flows.

[0111] The service flow consists of TT flow and BE flow. The weighted average latency of TT flow and BE flow is the average latency of the service flow. The delay is:

[0112]

[0113] In the formula, It is a variable that represents the business flow. From the host node Access the offset time slot of its adjacent TSN domain. and These represent the business flow respectively. Within the domain, traffic flows are transmitted at offset time slots (i.e., queuing delays) on switches / routers. Offset time slots for cross-domain transmission.

[0114] 1) Objective function 1 is to minimize the average delay of the TT and BE streams.

[0115]

[0116] In the formula, Represents a set of time-sensitive streams. Represents a set of streams that do their best. This indicates the number of TT streams, while Indicates the number of BE streams.

[0117] 2) Objective function 2 is to maximize the scheduling success rate of TT stream.

[0118] Objective function 2 is:

[0119]

[0120] In the formula, It is a 0-1 variable; if it is 1, it means that the TT stream is due at the specified deadline. If the value is 0, it means the target node has been reached. If the value is 0, it means the node has not been reached.

[0121] 3) Objective function 3 is to maximize resource utilization.

[0122] Objective function 3 is to maximize resource utilization.

[0123]

[0124] The formula expresses the comparison between the number of time slot resources occupied by all successfully scheduled flows and the number of time slot resources consumed in the network environment.

[0125] The average latency of the business flow in Objective 1 needs to be normalized, and is expressed as:

[0126] The normalized definition of the average latency of a business flow is:

[0127]

[0128] In the formula, This indicates the number of time-sensitive flows in a cross-domain deterministic data center network. This represents the number of best-effort flows in a cross-domain deterministic data center network. This represents the upper limit of the single-stream delay baseline for TT streams. This indicates the upper limit of the single-stream reference for the BE stream.

[0129] The number of successfully scheduled time-sensitive streams in Objective 2 needs to be normalized, specifically expressed as follows:

[0130] The normalized definition of the number of successful schedulings for time-sensitive streams is:

[0131]

[0132] Constraints:

[0133] 1): Time slot offset limitation

[0134]

[0135]

[0136] In the formula, This is the number of queues on the switch / router. This constraint ensures that the time slot offset of each TT flow within a domain of a deterministic network does not exceed the queue round-robin cycle of that domain. Indicates from the first The edge router of the domain to the first The time slot offset of the edge routers in each domain must be less than the queue rotation period of the next domain.

[0137] 2): Access time slot restrictions

[0138]

[0139] In the formula, Represents a stream The period (unit: ms). Indicates the first The time slot size of each domain. This constraint ensures the flow The transmission time offset in the access domain does not exceed its period, thus ensuring that the transmission of each time-sensitive service flow does not exceed the time window of the next sub-flow and avoids mutual interference.

[0140] 3): Maximum latency limit for services

[0141] Business Flow The total latency from the source node to the destination node satisfies the following constraint:

[0142]

[0143] The total latency includes: the access time slot offset generated when the service flow enters the system; the queuing time slot offset generated during the intra-domain or cross-domain transmission of the service flow; and the propagation latency generated when the service flow is transmitted on the link. These constraints ensure that the service flow... The end-to-end delay does not exceed its preset deadline. This ensures the deterministic transmission performance of the system.

[0144] 4) Limitations of time slot resources

[0145] In this implementation method, resource constraints consist of the following two parts:

[0146] ①

[0147] ②

[0148] Constraint ① limits the total time slot queue resources occupied by the flow on the node path to not exceeding the available resource capacity of the link within any given system time period. Constraint ② is used to define the time slot resource capacity of the link at any given time, wherein, Indicates the size of a single time slot. This indicates the number of queues in this field.

[0149] The above constraints ensure that the time slot resources occupied by the service flow on network nodes and links do not exceed the available resources of the system within the corresponding time period, thereby ensuring the schedulability of the deterministic network and the rationality of resource allocation.

[0150] 4. Employing G-TDRL, a deep reinforcement learning approach based on GAT and Transformer, for multi-service flow routing in cross-domain deterministic data center networks.

[0151] 1) Encoding of cross-domain deterministic network topology information, service flow information, and time slot resource information

[0152] Cross-domain deterministic network environment information includes node information and link information. Node information includes the geographical location of network transmission devices, the number of port queues, and the size of time slots. Link information includes link propagation delay and bandwidth resources. Service flow information includes the type of service flow, the service scheduling rate and latency requirements specified by the Service Level Agreement (SLA), packet loss rate, and required bandwidth resources.

[0153] The encoder embeds deterministic cross-domain network environment information, i.e., network topology, into a high-order space. First, it needs to map node features and edge features through a fully connected layer to... and The embedding space of the dimension is used to obtain the initial embedding information, and then the business flow information and time slot resource information are also mapped to it:

[0154]

[0155]

[0156] To facilitate further processing in subsequent models; among which, Represents the 0th level node Embedded, Indicates a Layer 0 link Embedded, and It is a learnable weight matrix;

[0157] The obtained initial embedding information and Input encoder, via The final embedding information is obtained by processing the same residual E-GAT modules, with the first one being the first. Taking layers as an example, attention coefficient Represents a node For nodes The importance of which

[0158]

[0159] in, Indicates transpose. This indicates a splicing operation. and This represents the learnable vector and weights. This represents the LeakyReLu activation function. In the... In the first layer, the feature vector of each node is updated through the attention mechanism described above, therefore the first... The output of the layer is specifically as follows

[0160]

[0161] It is a function computed by layer L, and can be represented as

[0162]

[0163] here, This represents the learnable weight matrix. Finally, the graph embeddings obtained from the final layer are used to compute the overall graph embedding, specifically...

[0164]

[0165] Furthermore, after the node features and edge features are transformed, the business flow information and time slot resource information are embedded into a high-dimensional vector representation. Taking the processing of business flow information as an example, the processing of time slot resource information is consistent with it, and the specific form is as follows:

[0166]

[0167] in, Indicates the 0th layer The embedded business flow information includes information such as the source node, destination node, periodicity attribute, bandwidth requirement, maximum latency limit, and priority of the flow.

[0168] Business flow information passes through The same layer is processed to obtain the final embedded information. Each layer includes a multi-head attention layer, residual connections, a normalization layer, and a feedforward network layer. We will now describe this processing flow in detail:

[0169] The input information is first processed by a multi-head attention layer, where it is divided into multiple sub-parts, each called a "head". Each attention head has an independent weight matrix, enabling it to focus on different features in the input information from different perspectives, thereby improving the model's ability to model complex dependencies. Specifically, for different embedded information, the model generates corresponding query, key, and value vectors through linear transformations to calculate the correlation between the embedded information.

[0170]

[0171]

[0172]

[0173] In the formula, , , These represent the query, key, and value weight matrices for the i-th head at level 0, respectively. , , These represent the query, key, and value vector of the i-th head at level 0, respectively.

[0174] Subsequently, the similarity between the query vector and the key vector is calculated using a dot product; a larger dot product value indicates a stronger correlation. Then, the Softmax function is used to normalize these similarity scores into a probability distribution, thereby obtaining the attention weights. This is used to indicate the key vectors that each query vector should focus on:

[0175]

[0176] Subsequently, the attention weights are weighted and summed with their corresponding value vectors to obtain the output of a single attention head. Finally, the outputs of all attention heads are concatenated and mapped to the target space through a linear transformation to generate the final output representation.

[0177]

[0178]

[0179] In the formula, Represents the business flow of the i-th head at layer 0. Attention output, Represents Layer 0 business flow The final output of the attention layer.

[0180] The output of the attention layer The input is fed into the residual connection and the normalization layer. The output of the normalization layer is then fed into the feedforward network layer for nonlinear transformation.

[0181] 2) Service Flow Routing Decoding

[0182] At each time step The intelligent system can adjust its response based on the current environmental conditions. The environment state of the previous action The agent performs corresponding actions and continuously maximizes the cumulative reward function, i.e., maximizing the business flow routing benefit, through repeated training. Therefore, the decoder combines the encoder's output to decode business flow routes. Then, based on the probability vector of the business flow routing node selected by the decoder, the agent selects a routing node for each business flow according to the DRL model to achieve business flow routing selection.

[0183] Current environmental status This mainly includes network topology information, service flow information, and time slot resource information, indicating the current environmental status. The environment state of the previous action Connect Subsequently, calculations are performed through the multi-head attention layer (MHA). The context information, the query vector is the embedding of the agent, and the key vector and value vector are defined as the embeddings of the edge nodes:

[0184] The decoder consists of two layers: the first layer computes the context vector using a multi-head attention mechanism, and the second layer generates the probability distribution on the candidate nodes. During the decoding process, at each time step... The decoder is based on the embeddings of all nodes and the previous outputs. Select a node .

[0185] At time step The context vector of the decoder It is embedded in the network topology Previously selected node Embedded, first selected node The embedding of the input stream information corresponding to the current step. The remaining time slot resources ,

[0186]

[0187] This represents a matrix of learnable weights. The first layer of the decoder takes the context vector. As input, and using a multi-head attention mechanism to generate a new context vector. .

[0188] In summary, based on node embedding and context vector definition 3D key vector Value vector and query vector

[0189]

[0190]

[0191]

[0192] Then, the attention coefficients are calculated using an attention mechanism on the query vector and key vector.

[0193]

[0194] Not all nodes can be selected at the moment, so a mask rule is set to exclude nodes that cannot be routed. The mask rule for determining whether a node j can be routed is defined as follows:

[0195]

[0196] At time step t, nodes that violate the constraints are masked by setting their attention coefficients to -∞, thus ensuring that the node... Will not be affected by business flow Choose. Then, use the softmax activation function on... Normalize.

[0197] The normalized attention coefficient calculated for the h-th head is expressed as: The vectors generated by each head are concatenated and then passed through a fully connected layer to obtain the final context vector.

[0198]

[0199] The input to the second decoder layer uses single attention head processing. Attention coefficient The calculation uses the following formula

[0200]

[0201] Finally, a probability vector for selecting nodes by the business flow is generated by combining the masking rules and the compatibility vector between nodes and business flows. The routing node for the business flow is then selected from this probability vector.

[0202]

[0203] The reward function is defined as:

[0204]

[0205] 3) Model Training

[0206] In this step, the improved baseline REINFORCE algorithm is used to optimize the neural network parameters. The algorithm is trained and optimized to maximize the expected cumulative reward. It improves training stability by introducing a baseline network to replace the critic network in the traditional actor-critic algorithm, forming a dual-actor structure. First, the loss function is defined as:

[0207]

[0208] parameter By using a rollout baseline The REINFORCE algorithm utilizes gradient descent for optimization, and its gradient form is as follows:

[0209]

[0210] In this method, the function originally performed by the critic network is replaced by the baseline policy network, which can be viewed as a "two-actor structure". This replacement strategy freezes the baseline policy network during each training cycle. The parameters are similar to the practice of freezing the target Q-network in DQN.

[0211] At the end of each training cycle, a greedy decoding algorithm is used to compare the results of the current policy network with those of the baseline policy network. The baseline network parameters are only updated if the new policy is statistically significantly better than the baseline policy. Significance testing is performed using paired t-tests, with a significance level set to [value missing]. And it was conducted on 10,000 evaluation instances.

[0212] Furthermore, code-level optimization methods are incorporated during training, including learning rate annealing of the Adam optimizer and normalization of the reward function, to further improve training performance. This invention refers to this improved method as the improved baseline REINFORCE algorithm (Rollout algorithm).

[0213] In summary, the multi-service flow routing scheme of this invention, applicable to cross-domain deterministic network environments, effectively reduces end-to-end latency, improves link reliability, and increases bandwidth resource utilization by rationally selecting primary and backup paths while ensuring Service Level Agreements (SLAs). The implementation of this technology can be widely applied to business scenarios with extremely stringent latency and reliability requirements, such as cross-data center interconnection, industrial internet, and vehicle-to-everything (V2X) networks. It effectively alleviates the challenges posed by resource scarcity and dynamic fluctuations in cross-domain environments, promotes the practical implementation of deterministic networks in mission-critical services, and has significant social and economic value.

Claims

1. A multi-target routing method for service flows in cross-domain deterministic data center networks, characterized in that, Includes the following steps: Step 1: The user service flow initiates a service access request to the deterministic service controller in the scheduling plane; the deterministic service controller signs a Service Level Agreement (SLA) with the service flow according to the user service requirements to clarify the reliability, end-to-end latency, bandwidth and resource constraints of the service flow; at the same time, the source host of the user service flow waits for the deterministic service controller to issue the routing configuration result and the corresponding time slot resource orchestration table for deterministic forwarding after the service is established. Step 2: Each autonomous domain controller obtains the network information of its own domain. The network information includes: the geographical location information of the node, the propagation delay of the link, the time slot resource information of the node port, and the link status information. Each autonomous system controller (AS / RS) aggregates network information to the deterministic service controller (DSC), which then integrates the information to form a global network topology view. The cross-domain network is abstracted as a weighted directed graph, where the weights of the edges reflect the propagation delay, bandwidth utilization, and reliability metrics of the links. Step 3: The deterministic service controller combines user service demand information with the topology information of the cross-domain deterministic data center network to establish a multi-objective optimization model that comprehensively considers latency, bandwidth and scheduling success rate. This model aims to minimize end-to-end latency of service flows, improve the scheduling success rate of time-triggered (TT) service flows and improve bandwidth utilization, thus transforming the routing and scheduling problem of the cross-domain deterministic network into a multi-objective optimization problem that satisfies the service level agreement constraints. Step 4: The deterministic service controller solves the multi-service flow routing problem based on service flow information, cross-domain deterministic data center network topology information, and a multi-objective optimization model. During the solution process, a fusion graph attention network (GAT) and a Transformer-based deep reinforcement learning algorithm framework (G-TDRL) are used to model the complex cross-domain network state and perform dynamic decision-making. The reinforcement learning policy network automatically generates a set of optimal routing paths for multiple service flows that satisfy SLA constraints and the corresponding time slot resource orchestration table according to the state-action-reward mechanism. Step 5: The deterministic service controller distributes the time slot resource orchestration table corresponding to the calculated optimal route path set to each autonomous system controller (AS / A). Each AS / A performs scheduling and forwarding in its local data plane based on the optimal route configuration and time slot resource orchestration table allocated by the deterministic service controller, thereby achieving end-to-end deterministic transmission of service flows in a cross-domain environment. Through path reservation, time slot synchronization, and distributed scheduling coordination mechanisms, the delay and reliability requirements of the service flows are ensured to meet the SLA constraints.

2. The multi-target routing method for service flows in cross-domain deterministic data center networks according to claim 1, characterized in that: In step 1, the user service flow includes a time-sensitive flow (TT) and a best-effort flow (BE); wherein, the time-sensitive flow... By six-tuple Indicates; among which The source node representing the business flow. Indicates the destination node of the business flow. The periodicity attribute represents the time-sensitive flow and is used to describe its time window constraints in periodic scheduling. This represents the upper bound of the latency of the business flow. This indicates the bandwidth requirements of the service flow. Indicates the priority of the business flow.

3. The multi-target routing method for service flows in cross-domain deterministic data center networks according to claim 2, characterized in that: In step 2, the cross-domain deterministic data center network environment is abstracted as a weighted directed graph. ,in, Indicates by A set of nodes consisting of 100 nodes Represents a set of links, each link Represents a node and Communication links between; links By binary It means that among them This indicates the transmission delay of the link. This indicates the bandwidth resources included in the link.

4. The multi-target routing method for service flows in cross-domain deterministic data center networks according to claim 3, characterized in that: The multi-objective optimization model established in step 3 includes the following objectives: minimizing the service latency of Time-Sensitive Stream (TT) and Best-effort Stream (BE), maximizing the scheduling success rate of Time-Sensitive Streams, and improving the utilization rate of bandwidth resources; the specific definition of the objective function of the optimization model is as follows: 3.1 Objective function 1 is to minimize the average delay of the TT and BE streams. The weighted average latency of the TT and BE streams is the overall average latency of the user's service flow. The delay calculation method is as follows: ， in, As a variable, it represents the business flow. From the host node The offset time slot when accessing its adjacent TSN domain. and These represent the business flow respectively. Queuing delays in intra-domain switches or routers and offset time slots during cross-domain transmissions; Objective function 1 is: ， in, Represents a set of time-sensitive streams. Represents a set of streams that do their best. This indicates the number of TT streams, while Indicates the number of BE streams; 3.2 Objective function 2 is to maximize the scheduling success rate of TT stream. The number of successfully scheduled TT streams is defined as follows: ， Objective function 2 is: ， in, It is a 0-1 variable; when its value is 1, it represents a time-sensitive flow. By the deadline The target node was successfully reached within the time limit; a value of 0 indicates that the target node was not reached within the time limit. 3.3 Objective function 3 is to maximize resource utilization. ， This formula is defined as a comparison between the number of time slot resources occupied by all successfully scheduled flows and the number of time slot resources consumed in the network environment.

5. The multi-target routing method for service flows in cross-domain deterministic data center networks according to claim 4, characterized in that: The average latency of the service flow in step 3.1 needs to be normalized, and is expressed as: The normalized definition of the average latency of a business flow is: ， In the formula, This indicates the number of time-sensitive flows in a cross-domain deterministic data center network. This represents the number of best-effort flows in a cross-domain deterministic data center network. This represents the upper limit of the single-stream delay baseline for TT streams. This indicates the upper limit of the single-stream reference for the BE stream.

6. The multi-target routing method for service flows in cross-domain deterministic data center networks according to claim 4, characterized in that: The number of successfully scheduled time-sensitive flows in step 3.2 needs to be normalized, specifically expressed as follows: The normalized definition of the number of successful schedulings for time-sensitive streams is: 。 7. The multi-target routing method for service flows in cross-domain deterministic data center networks according to claim 4, characterized in that: Steps 3.1 and 3.2 require considering the constraints of multiple services to implement a multi-objective service flow routing optimization model, with the following constraints: 1): Time slot offset limitation ， ， In the formula, This is the number of queues on the switch / router. This constraint ensures that the time slot offset of each TT flow within a certain domain of a deterministic network does not exceed the queue rotation period of that domain; where, Indicates from the first The edge router of the domain to the first The time slot offset of the edge routers of each domain must be less than the queue round-robin period of the next domain; 2): Access time slot restrictions ， In the formula, Represents a stream The period, in milliseconds (ms). Indicates the first The size of the time slot in each domain; 3): Maximum latency limit for services Business Flow The total latency from the source node to the destination node satisfies the following constraint: ， The total latency includes: the access time slot offset generated when the service flow enters the system; the queuing time slot offset generated during the intra-domain or cross-domain transmission of the service flow; and the propagation latency generated when the service flow is transmitted on the link. These constraints ensure that the service flow... The end-to-end delay does not exceed its preset deadline. ; 4) Limitations of time slot resources In a cross-domain deterministic data center network environment, resource constraints consist of the following two parts: ① ， ② ， Constraint ① limits the total time slot queue resources occupied by a service flow on a node path to not exceeding the available resource capacity of the selected path link within any given system time. Constraint ② is used to define the total time slot resource capacity of the path links selected by the service flow at any given time, where, Indicates the size of a single time slot. This indicates the number of queues in this field.

8. The multi-target routing method for service flows in cross-domain deterministic data center networks according to claim 3, characterized in that: Step 4 employs the G-TDRL deep reinforcement learning framework based on GAT and Transformer to obtain path routing selection for multiple service flows, specifically as follows: 4.1 The encoder first maps the topology information of the cross-domain deterministic network environment, including network device locations, link latency, link status, and service flow information, to a high-dimensional space through a linear layer; then it maps node features and edge features to a higher-dimensional space through a connection layer. and In the dimensional embedding space, the initial node and edge information representation is obtained; the specific formula is as follows: ， ， in, Represents the 0th level node Embedded, Indicates a Layer 0 link Embedded, This indicates the embedding of business flow information at layer 0. , , and , , These are learnable parameters; through the above operations, the initial node and edge information is embedded into a high-dimensional space and provides input for subsequent deep learning models. After transformation, node and edge features are embedded into a high-dimensional representation of business flow information, specifically as follows: ， in, Indicates the 0th layer The embedded business flow information includes the source node, destination node, periodicity attribute, bandwidth requirement, maximum latency limit and priority information of the flow; Deterministic cross-domain data center network environment information includes time slot resource status characteristics; first, this raw data undergoes a linear transformation through the input layer to obtain the initial time slot resource status; the status of nodes and links is mapped to a space of the same dimension as network topology and service information through linear transformation; The obtained initial embedding information of the network topology , The final embedding information is obtained by processing L identical residual E-GAT modules; Node information in a cross-domain deterministic data center network environment is ， in, For the first The mean of node features output by layer graph convolution. For the first The feature vector of each node The total number of nodes; Business information in a cross-domain deterministic data center network environment is )， in, For the first Mean of business flow characteristics of the layer For the first Feature vectors of each business flow For the number of business flows; The time slot resource information for cross-domain deterministic data center networks is ， in, For the first Mean value of time slot resource characteristics of the layer For the first Feature vector of each time slot resource The number of time slots; Finally, all the embedded information is connected through a linear layer to obtain global information. , ， The mean() function represents the mean operation, and || represents the feature concatenation operation. This is the weight matrix for the second fully connected layer. The bias term; the final fused result For cross-domain deterministic data center networks in the first Global state information of the layer; 4.2 The decoding process models the cross-domain deterministic data center network routing problem as a Markov decision process (MDP). In this process, the cross-domain network topology, global link information, the time slot resource status of each domain, and the service requirements of the business flow are regarded as a set of states, which are used to characterize the environmental characteristics of the system at each moment. The action set corresponds to selecting the next-hop node for the current business flow from the candidate node set, and then gradually constructing the complete routing path of the business flow. The state transition function is jointly determined by the currently selected node and the available resources of the deterministic cross-domain data center network, while the reward function comprehensively considers end-to-end latency, bandwidth constraints, reliability, and scheduling success rate indicators to reflect the impact of routing decisions on overall performance. 4.3 The decoder uses contextual information from the encoder, takes the current business flow as the query vector, and candidate nodes as key and value vectors, and combines them with masking rules to generate the probability distribution of the business flow selecting different candidate nodes; the agent selects the next-hop node at each time step according to the deep reinforcement learning DRL policy until a complete routing path that satisfies the SLA constraint is generated. 4.4 After obtaining a set of multiple candidate routing paths, the deterministic service controller uses a reinforcement learning method based on the improved baseline REINFORCE algorithm to train and optimize the policy network, so as to evaluate and update the policy parameters and obtain the final path selection result that can optimally match the cross-domain network state and service flow requirements. The reinforcement learning process includes policy initialization, sample collection, baseline update, policy optimization, parameter update and policy output, which are used to realize dynamic adaptive routing and time slot scheduling of cross-domain deterministic data center networks.

9. The multi-target routing method for service flows in cross-domain deterministic data center networks according to claim 8, characterized in that: The reinforcement learning process of the reinforcement learning method based on the improved baseline REINFORCE algorithm includes the following steps:

1. Initialization steps: Establish the policy network and the baseline policy network, and initialize the parameters respectively. and Set the learning rate, decay coefficient, discount factor, and other hyperparameters; perform pre-training by interacting with the environment, collecting initial sample data of state, action, and reward; and 1. Parameter sets of the policy network and the baseline policy network; 2. Sample collection steps: In each training iteration, based on the interaction between the current policy network and the environment, multiple complete state-action-reward sequences are generated, the cumulative reward value of each sequence is calculated, and it is used as the input of training samples; 3. Baseline Update Steps: A baseline policy network is used to replace the traditional critic network to estimate the reference reward for the current state; the parameters of the baseline policy network from the previous cycle are frozen during each training round. This serves as a stable reference.

4. Policy optimization steps: Calculate the policy advantage value based on the outputs of the current policy network and the baseline policy network; use the policy advantage value to optimize the policy network parameters. Perform gradient direction updates; when the policy performance metrics are better than the baseline reference values, copy the current policy network parameters to the baseline policy network to implement a dual-actor structure to improve training stability.

5. Parameter update steps: The Adam optimization algorithm is used to update the parameters of the policy network and the baseline policy network; the learning rate is dynamically decayed during training to prevent gradient oscillation and overfitting; after each K training rounds, the model performance is evaluated for reliability, and the learning rate and update frequency are adjusted according to the evaluation results, where K is the preset number of training rounds.

6. Strategy Output Steps: When the model's average return or performance metric reaches a preset threshold, training is stopped; the final policy network parameters are used as the optimal policy output and applied to the dynamic routing and time slot scheduling process of cross-domain deterministic data center networks.

Images