A method and apparatus for deterministic network inter-domain mixed flow routing and scheduling

Abstract

The application discloses a kind of certainty network cross-domain hybrid flow routing and scheduling method and device, belong to network routing and scheduling technical field.The method includes: using directed graph to build cross-domain deterministic system network model;Modeling service flow and end-to-end delay, construct the joint routing and scheduling problem of cross-domain hybrid flow in cross-domain deterministic system network model, and formulate the coordination mechanism of cross-domain hybrid flow in cross-domain deterministic system network model;The joint routing and scheduling problem of cross-domain hybrid flow following coordination mechanism is modeled as multi-agent Markov decision process;Based on graph convolution network and multi-agent collaborative scheduling distributed reinforcement learning algorithm, the multi-agent Markov decision process is solved, and the optimal routing and scheduling strategy of cross-domain hybrid flow is obtained.The application can better meet the actual demand of high reliability, low delay and strict jitter constraint in industrial control scene.

Description

Technical Field

[0001] This invention relates to the field of network routing and scheduling technology, and in particular to a method and apparatus for deterministic cross-domain hybrid flow routing and scheduling in networks. Background Technology

[0002] With the development of industrial automation and intelligent manufacturing, time-sensitive services such as industrial control and process monitoring place higher demands on communication networks. These networks require not only highly reliable data transmission but also strict constraints on end-to-end latency and jitter to ensure deterministic operation of the control system. To address this, Time-Sensitive Networking (TSN) provides deterministic transmission capabilities at the link layer through mechanisms such as time synchronization, periodic scheduling, bandwidth reservation, and traffic shaping. Deterministic Networking (DetNet) further extends deterministic guarantees to the network layer, achieving end-to-end latency, packet loss, and jitter control across multi-hop paths. In multi-subdomain industrial networks, different subdomains typically employ independent control planes or scheduling mechanisms, leading to differences in time slot allocation, time-lapse length, and queue management methods. For hybrid services in cross-domain scenarios that simultaneously include Time-Triggered Streaming (TT), Audio / Video Bridging (AVB), and Best-Effect Streaming (BE), existing technologies primarily rely on end-to-end resource reservation based on engineering rules, centralized or hierarchical path calculation, and scheduling strategies based on heuristics or single-domain learning. These methods can maintain certain deterministic performance in single-domain or small-scale networks, but in large-scale, multi-domain, high-concurrency DetNet environments, they are prone to resource fragmentation and compromise end-to-end deterministic guarantees.

[0003] In recent years, multi-agent reinforcement learning and hierarchical reinforcement learning have been introduced into the field of network routing and scheduling to solve distributed collaborative decision-making problems, thus improving the system's adaptability to some extent. However, when these learning methods are directly applied to cross-domain hybrid stream joint routing and scheduling in DetNet, problems such as insufficient end-to-end performance guarantees and weak engineering feasibility still exist.

[0004] Therefore, the existing technologies still have the following shortcomings: (1) Low end-to-end scheduling success rate of cross-domain hybrid streams: In the multi-domain DetNet scenario, there are many hybrid streams and complex mapping relationships. Routing selection and queue scheduling are coupled with each other, making it difficult to ensure the stable arrival of cross-domain hybrid streams; (2) Insufficient collaborative efficiency under local observation conditions: Nodes can usually only obtain local or neighboring state information, making it difficult to accurately assess the impact of their own decisions on end-to-end performance, resulting in limited multi-agent collaborative effects; (3) Cross-domain heterogeneity and high latency cause environmental non-stationarity: The differences in scheduling mechanisms of different subdomains and the large propagation and queuing delay of cross-domain links make the learning environment exhibit obvious non-stationarity characteristics, affecting the stability and generalization ability of the strategy; (4) Lack of verifiable service guarantee mechanism: Existing learning-based methods generally lack formal constraints on TT slot continuity, AVB minimum bandwidth and BE bounded delay, which poses a risk of service level violation and makes it difficult to meet the strict requirements of determinism for industrial applications. Summary of the Invention

[0005] This invention addresses the challenges of cross-domain hybrid flow routing and scheduling in deterministic networks by designing a method and apparatus for such routing and scheduling. The method models the joint routing and scheduling problem as a partially observable Markov decision process, utilizes graph convolutional networks to extract topological and neighborhood state features, and combines multi-agent reinforcement learning for centralized training and distributed execution, thereby solving the problems of multi-agent credit allocation and topology awareness.

[0006] To achieve the above objectives, the present invention employs the following technical solution:

[0007] This invention provides a method for deterministic cross-domain hybrid flow routing and scheduling in a network, comprising:

[0008] A network model for cross-domain deterministic systems is constructed using directed graphs;

[0009] Modeling of business flows and end-to-end latency, constructing a joint routing and scheduling problem for cross-domain hybrid flows in a cross-domain deterministic system network model, and formulating a coordination mechanism for the cross-domain hybrid flows in the cross-domain deterministic system network model;

[0010] The joint routing and scheduling problem of cross-domain hybrid flows following the aforementioned cooperative mechanism is modeled as a multi-agent Markov decision process;

[0011] A distributed reinforcement learning algorithm based on graph convolutional networks and multi-agent cooperative scheduling is used to solve the multi-agent Markov decision process, thereby obtaining the optimal routing and scheduling strategy for cross-domain hybrid flows.

[0012] Preferably, the business flow modeling is as follows: , in, Represents a stream This includes time-triggered streams, audio / video bridging streams, and best-effort streams. Represents a stream The source node, Represents a stream destination node Represents a stream Maximum end-to-end transmission delay constraint Represents a stream bandwidth requirements Represents a stream Stream type Represents a stream bag size, Represents a stream Cross-domain identifiers, Represents a stream Minimum bandwidth requirement Represents a stream Priority weights;

[0013] The end-to-end delay model is as follows:

[0014] ,

[0015] in End-to-end cumulative delay For flow The scheduling path Indicates the node number on the scheduling path. For link Available bandwidth capacity for service flows within the current time slot, link Indicates from node To the node Directed communication channel, For link The delay in transmission, For flow At the node The waiting time in the queue.

[0016] Preferably, the joint routing and scheduling problem of cross-domain hybrid flows in the cross-domain deterministic system network model is expressed as:

[0017] ,

[0018] in For decision variables, if flow If all scheduling constraints are met, then ,otherwise ; This represents the total number of service flows successfully scheduled within the current scheduling period.

[0019] The scheduling constraints include:

[0020] Delay constraints: ,

[0021] in, For flow The delay constraint activation flag is set for time-triggered streams and audio / video bridging streams. For best-effort flow, set ;

[0022] Link capacity constraints: ,

[0023] in, Whether to flow From node Send to node Scheduled, with a value of 0 or 1. For a set of links in a cross-domain deterministic system network model;

[0024] Queue capacity constraints: ,

[0025] in, For nodes In the time slot Above, the first Queue length for different types of business flows. The maximum buffer capacity supported by the node scheduling port. This represents the set of nodes in a cross-domain deterministic system network model. For time-triggered streams, For audio and video bridging streams, To do one's best to flow.

[0026] Preferably, the cooperative mechanism of the cross-domain hybrid flow in the cross-domain deterministic system network model includes:

[0027] Set the time-triggered stream as the highest priority, the audio / video bridging stream as the second highest priority, and the best-effort stream as the lowest priority.

[0028] For time-triggered streams, determine whether all nodes on the end-to-end path have available slots within the required continuous time interval. If any node exists... Not satisfied , For nodes In the time slot The set of slots used for time-triggered flow scheduling returns a verification failure signal and triggers a rollback strategy; the set of slots is obtained based on the slot availability function.

[0029] For audio and video bridging streams, each node maintains an integer token indicator. When a node receives an audio and video bridging stream scheduling request, it needs to determine whether the request is satisfied. If the conditions are met, scheduling is allowed, where For audio and video bridging streams in time slots The integer token value, The minimum token threshold required to allow audio / video bridging stream scheduling;

[0030] For best-effort streams, transmission is only permitted when both the time-triggered stream and the audio / video bridging stream queues are idle or have unoccupied non-reserved slots. When a high-priority stream request arises and needs to occupy a slot already occupied by a best-effort stream, the current transmission of the best-effort stream is forcibly interrupted, and the slot is immediately released to the high-priority stream.

[0031] Preferably, the slot availability function is expressed as:

[0032] ,

[0033] in For slot availability function, Represents a node Time index within supercycle The above has scheduling capabilities for a certain type of flow. The total number of slots within the supercycle is the least common multiple of the transmission cycles of all types of service flows.

[0034] Preferably, the joint routing and scheduling problem of cross-domain hybrid flows following the aforementioned cooperative mechanism is modeled as a multi-agent Markov decision process, including:

[0035] Treat each node as an intelligent agent.

[0036] agent state This includes node state and flow state, where node state is represented as: , For nodes state, For nodes The length of the stream queue. For nodes The current integer token value of the audio / video bridging stream. For nodes A boolean variable indicating whether the current time is within a time-triggered flow schedulable slot. For nodes Boundary node identifiers, For nodes The number of cross-domain streams cached in the middle. For nodes The remaining available bandwidth, For nodes Does the current scheduling period contain any audio or video bridging streams that have not met bandwidth guarantees?

[0037] The flow state is represented as: , In flow state, This represents the current node position of the flow. The remaining tolerable delay is for the flow;

[0038] The action of an agent is defined as: ,

[0039] in Indicates the first Actions of an agent, where an action of 0 represents an empty action, meaning the current candidate flow to be scheduled will not be scheduled; actions This indicates that the current candidate flow to be scheduled is to be forwarded to neighboring nodes. Receive queue , For nodes The neighborhood group, For a collection of stream queues, , A collection of intelligent agents;

[0040] The joint action of all nodes during the scheduling period is represented as: , Indicates the node in the scheduling cycle The action, This represents the combined action of all nodes during the scheduling period;

[0041] The agent's reward is represented as:

[0042] ,

[0043] in In order to take action Immediate rewards returned after the environment is restored. For cross-domain flow priority weights, if For cross-domain flows, then >0, if =0, then For intra-domain flow, For indicator functions, The delay penalty coefficient, This is the weight for timeout penalties.

[0044] Preferably, the step of using a distributed reinforcement learning algorithm based on graph convolutional networks and multi-agent cooperative scheduling to solve the multi-agent Markov decision process to obtain the optimal routing and scheduling strategy for cross-domain hybrid flows includes:

[0045] Graph convolutional networks combine network topology information with the system node states from the current scheduling period. The constructed node feature matrix is ​​jointly mapped to a structure-aware node embedding matrix. , For structure-aware vectors;

[0046] flow state The flow state feature vector is obtained by extracting and encoding locally observable static attributes. Structure-aware vectors generated by graph convolutional networks The local policy network of the agent is input together, and the scheduled actions are output.

[0047] In each scheduling cycle, the node Based on event priority, a current candidate flow to be scheduled is determined. The scheduling actions of the agents only apply to the candidate flow to be scheduled at the corresponding node. The scheduling actions of all agents are executed synchronously in each scheduling cycle, forming a joint action. This refers to the optimal routing and scheduling strategy for cross-domain mixed flows.

[0048] Preferably, the distributed reinforcement learning algorithm based on graph convolutional networks and multi-agent cooperative scheduling simultaneously employs online network parameter mapping during the training phase. and target network parameters Selecting an action is represented as:

[0049] ,

[0050] in, for Time by the first Each agent estimates the optimal action chosen based on the current online network. , For a collection of intelligent agents, For all possible actions Select the action that maximizes the Q value. For the first The action value function of an agent. It is the first An intelligent agent in Local observation at a given moment The target value for time-series difference is... for Instant rewards for each moment This is a termination flag, indicating whether the current interaction has reached the termination state. The degree of importance attached to future returns, Let be the joint value function of the target network. for The state of the agent at all times;

[0051] Simultaneously, the Huber loss function is used to optimize the error between the online network joint value function and the temporal difference objective value, thereby updating the network parameters of the online network joint value function, expressed as:

[0052] ,

[0053] in, Represents the loss function. Sampling weights for the importance of priority experience replay. For the joint value function of the online network, For Huber functions, This represents the expectation of the sample distribution in the experience replay buffer.

[0054] Preferably, during the training phase, every This training will use online network parameters Periodically synchronize to target network parameters .

[0055] The present invention also provides an apparatus for deterministic cross-domain hybrid flow routing and scheduling, used to implement the above-described deterministic cross-domain hybrid flow routing and scheduling method, the apparatus comprising:

[0056] The initial modeling module is used to construct a network model of a cross-domain deterministic system using directed graphs;

[0057] The problem construction module is used to model the business flow and end-to-end latency, construct the joint routing and scheduling problem of cross-domain hybrid flows in the cross-domain deterministic system network model, and formulate the coordination mechanism of the cross-domain hybrid flows in the cross-domain deterministic system network model.

[0058] The problem transformation module is used to model the joint routing and scheduling problem of cross-domain hybrid flows following the aforementioned cooperation mechanism as a multi-agent Markov decision process;

[0059] The optimization scheduling module is used to solve the multi-agent Markov decision process by employing a distributed reinforcement learning algorithm based on graph convolutional networks and multi-agent cooperative scheduling, thereby obtaining the optimal routing and scheduling strategy for cross-domain hybrid flows.

[0060] The advantages and beneficial effects of this invention are as follows:

[0061] This invention expands the solution approach for collaborative management of time-sensitive and non-time-sensitive services by defining the optimization object in detail and decomposing the end-to-end transmission delay precisely. It provides bounded end-to-end delay guarantee for low-priority services while ensuring that the performance of high-priority traffic is not compromised.

[0062] This invention implements a multi-domain heterogeneous collaborative scheduling mechanism and a multi-flow classification collaborative mechanism to meet the service requirements of TT, AVB, and BE traffic flows and to achieve differentiated allocation of resources between domains. By identifying cross-domain boundary nodes and monitoring cross-domain queues, it guides the Graph Convolutional Network (GCN) to perceive topological boundary characteristics, thereby resolving scheduling conflicts caused by periodic differences and achieving end-to-end deterministic service for cross-domain hybrid flows.

[0063] This invention presents a distributed reinforcement learning algorithm based on Graph Convolutional Networks (GCNs) and Multi-Agent Cooperative Scheduling (GACS). It models the joint scheduling problem as a Decentralized Partial Observable Markov Decision Process (Dec-POMDP), utilizes GCNs to extract topological and neighborhood state features, and combines multi-agent reinforcement learning (MARL) for centralized training and distributed execution, thereby solving the problems of multi-agent credit allocation and topology awareness. To improve training efficiency, a priority experience replay and asynchronous update mechanism is introduced to accelerate convergence and enhance stability, thus addressing the non-stationarity of the environment caused by limited cross-domain resources and improving the service guarantee for different priorities and the transfer and generalization capabilities of learning strategies.

[0064] This invention can overcome the shortcomings of traditional centralized and purely distributed methods in terms of scalability, collaborative optimization and formal service assurance, thereby better meeting the actual needs of industrial control scenarios for high reliability, low latency and strict jitter constraints. Attached Figure Description

[0065] Figure 1 This is a schematic diagram of the cross-domain hybrid flow routing and scheduling method provided in an embodiment of the present invention;

[0066] Figure 2 This is a schematic diagram of the adaptive topology-aware autonomous scheduling learning framework provided in an embodiment of the present invention;

[0067] Figure 3 This is a schematic diagram of the distributed reinforcement learning algorithm flow of graph convolutional networks and multi-agent cooperative scheduling (GACS) provided in an embodiment of the present invention. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0069] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0070] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0071] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0072] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0073] It should be emphasized here that the step markers mentioned below are not a limitation on the order of the steps, but should be understood as meaning that the steps can be executed in the order mentioned in the embodiments, or in a different order than in the embodiments, or several steps can be executed simultaneously.

[0074] See Figure 1 This invention provides a deterministic method for cross-domain hybrid flow routing and scheduling in networks, mainly including: constructing a system, flow, and latency model under cross-domain hybrid flow. This model includes modeling network and link attributes, describing each flow in the form of nine-tuples, and precisely defining transmission constraints and metrics using an end-to-end latency decomposition formula. A global multi-domain heterogeneous collaborative scheduling mechanism and a multi-flow classification collaborative mechanism are adopted. These mechanisms include abstracting each mechanism in different domains as a set of "available scheduling slots" exposed by nodes in future time, and periodic scheduling, rate limiting, and preemptive mechanisms. The above model problem is formulated as an observable Markov decision process, including a state space, action space, and reward space. Finally, a distributed reinforcement learning algorithm of graph convolutional network and multi-agent collaborative scheduling (GACS) is executed. Routing and scheduling combining centralized training and distributed execution are achieved through priority experience replay, dual networks, and asynchronous updates. Updates include reward mechanisms and action selection strategies, and routing includes neighbor node observation, path selection, and queue selection.

[0075] Based on the above inventive concept, the method for deterministic cross-domain hybrid flow routing and scheduling in a network provided by this invention is implemented as follows:

[0076] S1. Construct a cross-domain deterministic system network model.

[0077] The cross-domain deterministic system network model constructed in this invention consists of a directed graph. It means that among them This represents a set of nodes in the network, where each scheduling node corresponds to a device with local awareness and scheduling capabilities. Indicates the first 1 node This represents the total number of nodes in the network. This represents the set of physical links between nodes. Each link... Indicates from node To the node A directed communication channel with a fixed bandwidth capacity. With propagation delay In this system, network nodes act as autonomous agents, combining local observations and neighbor interactions to model the topology using graph neural networks and independently execute flow scheduling decisions. Centralized learning enables local policies to coordinate; during the training phase, the central trainer learns parameters using global state and joint reward, allowing individual agents to coordinate globally based on their local inputs during decentralized execution. Furthermore, the network is divided into several domains, each employing an independent scheduling mechanism. For each node... Add domain identifiers and boundary identifiers. Define domain type identifiers and scheduling mechanism identifiers for each scheduling node. Add a flag indicating whether it is a boundary node and a cross-domain queue length status item to the node status.

[0078] S2. Accurately model the service flow and end-to-end latency to construct a joint routing and scheduling problem for cross-domain hybrid flows.

[0079] In this invention, the business flow model is defined in the form of a nine-tuple, and the business flow modeling is as follows:

[0080] The elements within the parentheses represent streams. The source node, destination node, maximum end-to-end transmission delay constraint, flow bandwidth requirement, flow type, packet size, cross-domain identifier, minimum bandwidth requirement, and priority weight.

[0081] To differentiate the transmission characteristics and path differences of different service flows, and considering the differences in propagation delays within and outside the domain, as well as the delay differences caused by the congestion levels of different service flows at different domain nodes, the end-to-end cumulative delay is modeled as follows:

[0082] ,

[0083] in End-to-end cumulative delay For flow The scheduling path Indicates the node number on the scheduling path. For link Available bandwidth capacity for service flows within the current time slot, intra-domain link bandwidth Inter-domain link bandwidth , For link The delay in transmission, For link The average hop factor of the path in question. For link Logical or physical length, These are the weighting coefficients. For the node The waiting time in the queue.

[0084] The optimization objective is to guide the system to prioritize the scheduling success rate of high-priority services under resource-constrained conditions through a weighted scheduling value maximization strategy, thereby improving overall service efficiency and scheduling fairness. Simultaneously, it introduces latency and scheduling behavior constraint mechanisms to ensure deterministic guarantees for time-sensitive services and worst-case delay control for non-critical services. Defined as:

[0085] ,

[0086] in For decision variables, if flow If the following constraints on latency, link capacity, queue capacity, and time slot feasibility are met, then... ,otherwise By employing a weighted scheduling value maximization strategy, the system is guided to prioritize the scheduling success rate of high-priority services under limited resource conditions, thereby improving overall service efficiency and scheduling fairness. At the same time, by introducing constraints on latency and scheduling behavior, deterministic guarantees for time-sensitive services and worst-case delay control for non-critical services are ensured.

[0087] The delay constraints are as follows:

[0088] ,

[0089] in For flow Delay constraint activation flag, ;like Then the flow must satisfy ;like If the constraint is invalidated, then the constraint is automatically invalidated. For flow Maximum end-to-end transmission delay constraint. The delay constraint limits the maximum allowed end-to-end latency for Time-Triggered Streams (TT streams) and Audio / Video Bridged Streams (AVB streams). For real-time streams such as TT and AVB streams, setting... Configure the specific maximum allowable latency. For business flows without strict latency requirements, such as best-effort flows (BE flows), set... This allows the time delay limit to be automatically ignored without changing the form of the constraint.

[0090] The link capacity constraints are as follows:

[0091] ,

[0092] in Whether to flow From node Send to node Perform scheduling (0 or 1). For streaming bandwidth requirements, For link The available bandwidth capacity for service flows within the current time slot. Link capacity constraints ensure that the link load does not exceed the available bandwidth in any time slot.

[0093] The queue capacity constraints are as follows:

[0094] ,

[0095] in For nodes In the time slot Above, the first Queue length for different types of service flows (e.g., TT, AVB, BE). This specifies the maximum buffer capacity supported by the node scheduling port. Queue capacity constraints limit the queue length of each node to no more than the port buffer limit to avoid congestion and packet loss.

[0096] For time slot feasibility constraints, nodes in the path are required to have a common available time slot during scheduling; and for service quality constraints, such as maintaining minimum bandwidth or meeting token conditions for some services, these are given in step S3 when implementing the multi-domain heterogeneous and multi-flow classification coordination mechanism.

[0097] S3. Construct a collaborative mechanism for multi-domain heterogeneous and multi-stream classification.

[0098] To address the challenges of mixed flows (TT, AVB, and BE) in multi-domain heterogeneous industrial networks, including concurrent transmission with differentiated QoS requirements and cross-domain scheduling engineering constraints, a multi-domain heterogeneous and multi-flow classification and coordination mechanism is implemented. This mechanism ensures the time determinism of TT flows while providing bandwidth guarantees for AVB flows and best-effort service for BE flows. Furthermore, it unifies and abstracts different scheduling implementations within and between domains into algorithm-level accessible "available scheduling slots" information and several executable hard constraint interfaces, thereby providing executable constraint verification and rollback mechanisms for upper-layer learning and decision-making.

[0099] First, the flows are classified and queued. Based on the service flow type, quality of service constraint strength, and cross-domain attributes, TT flows are assigned the highest priority, AVB flows the second highest priority, and BE flows the lowest priority. For TT flows, a circular queue group is used to ensure deterministic transmission slots exist within a given transmission cycle. For AVB flows, multiple receive queues are set up on top of the circular queue group to increase reception fault tolerance. For BE flows, a non-periodic ordinary queue is used, and forwarding is only allowed when high-priority queues are idle or unoccupied time slots exist. When a flow arrives, the classification module at the control plane or node edge maps the flow to the corresponding queue group based on its attributes, records its priority, and sets the transmission cycle. Each cycle must ensure that all packets in the queue are transmitted within that cycle. A supercycle is set for different transmission cycles in the network, and its size is set to the least common multiple of all transmission cycles. The supercycle is used for time slot allocation, queue switching, and multi-domain alignment at the link layer. Each domain can use this supercycle concept for local time slot mapping during initialization or cycle synchronization, thereby minimizing cross-domain time slot fragmentation in engineering.

[0100] Building upon this, a multi-domain heterogeneous collaborative mechanism is proposed, which abstracts these mechanisms at the mechanism layer into a set of scheduling slots available to nodes in future timeframes. For each node... Define its slot availability function on the supercycle time axis. :

[0101] ,

[0102] in Represents a node Time index within supercycle The above has scheduling capabilities for a certain type of flow. This represents the total number of slots within the supercycle.

[0103] Specifically, for the multi-stream classification and coordination mechanism, for TT streams, before executing end-to-end path delivery or scheduling decisions, the algorithm needs to call the "slot availability check" interface provided by the mechanism to determine whether the consecutive time slots required by the stream during its transmission cycle are simultaneously available at all nodes on the path. If any node... Not satisfied , For nodes In the time slot The set of slots available for TT flow scheduling. This constraint means that there is at least one common set of available scheduling slots on the end-to-end path of the service flow, so that all nodes in the path can forward the time-triggered flow at the same time position, thereby satisfying its end-to-end time determinism requirement. Otherwise, it is judged as "slot discontinuous".

[0104] AVB streaming, on the other hand, introduces integer token state monitoring at each node. Maintain an integer token indicator; when a node receives an AVB stream scheduling request, it needs to determine whether the condition is met. , For AVB streams in time slots The integer token value, To allow the minimum token threshold for AVB stream scheduling, scheduling is guaranteed only if the integer token value of the AVB stream is not less than the minimum token threshold.

[0105] Simultaneously, periodic bandwidth monitoring and feedback are employed to compare the actual bandwidth usage. Minimum bandwidth requirements when enabled ,like The system records bandwidth insufficiency events and updates the local state vector to adjust subsequent scheduling priorities. For BE (Banded Stream) flows, a preemption rule is used; BE flows are only allowed to enter the transmission phase when both the TT (Transmission Time) and AVB (Available Stream) queues are idle or have unoccupied non-reserved slots. To prevent BE flow occupancy from affecting the timing determinism or bandwidth guarantee of TT / AVB, when a high-priority flow demands a slot already occupied by a BE flow, the current transmission of the BE flow is forcibly interrupted, and the slot is immediately released to the high-priority flow. Simultaneously, the preemption event is recorded so that the algorithm layer can make retransmission or compensation decisions.

[0106] The specific implementation of the multi-domain heterogeneous and multi-stream classification collaboration mechanism in this step includes the following steps:

[0107] Step 3.1: Set priority levels and queue group mapping rules based on business flow type, service quality constraint strength, and cross-domain attributes.

[0108] Step 3.2: Establish a queue group on each physical port. , , At the same time, multiple receive queues are established. To enhance resilience to unexpected events;

[0109] Step 3.3: Set the transmission bandwidth for each queue group to complete the transmission of all data in the queue within one transmission cycle;

[0110] Step 3.4: Calculate the timeout based on the transmission period of each queue and generate a local slot table. The slot table records the scheduling capability of each node for a certain type of flow in each time slot within the timeout.

[0111] Step 3.5: Initialize the integer token parameters of the AVB stream and send all the above configuration information to the node local state and the upper-layer control unit.

[0112] S4. The optimization objective of the joint routing and scheduling problem of cross-domain hybrid flows is modeled as a multi-agent Markov decision process.

[0113] To facilitate the application of multi-agent reinforcement learning in solving the joint routing and scheduling problem of cross-domain hybrid flows, the problem is formalized as a Decentralized Partial Observable Markov Decision Process (Dec-POMDP), represented by a quintuple. , For a collection of intelligent agents, For the global state space, For joint action space, Let be the state transition probability function. This is an immediate reward function. The state is observed in each scheduling cycle. Each node is considered an intelligent agent, and each node's intelligent agent bases its actions on its local observations. Independent selection of local actions All local actions are executed synchronously and by Transition to the next state Instant rewards for environmental feedback .

[0114] State: The state Divided into node states and flow states,

[0115] The node state is represented as follows: These represent the queue length of the business flow, the integer token value of the current AVB flow, and the node, respectively. The current status includes a boolean variable indicating whether the node is in a TT stream schedulable slot, the boundary node identifier, the number of cross-domain streams cached in the node, the remaining available bandwidth, and whether the node contains AVB streams that have not met bandwidth guarantees within the current scheduling period.

[0116] The flow state is represented as: These are the flow type, flow bandwidth requirement, current node location, remaining tolerable latency, cross-domain flag, and priority weight. The flow state is then... The flow state feature vector is obtained by extracting and encoding locally observable static attributes. Nodes generated by Graph Convolutional Networks (GCN) Structure-aware vectors The local policy networks of all agents are input together, and the output is the scheduled action. The local action value functions of all agents are aggregated through a hybrid network.

[0117] Action: Each agent in each scheduling cycle The system makes local decisions regarding its current candidate flow to be scheduled: determining whether to schedule the flow and, if permitted, selecting the next hop and target queue. Agent actions, centered on node autonomy and local forwarding, aim to improve structural generalization and distributed execution performance, avoiding the communication and computational bottlenecks of centralized control. Nodes The local action space is defined as:

[0118] ,

[0119] An action of 0 indicates an empty action, meaning the current candidate flow to be scheduled will not be scheduled; action This indicates that the current candidate flow to be scheduled is to be forwarded to neighboring nodes. Receive queue , For nodes The neighborhood group, This is a set of flow queues. The combined actions of all nodes occur during the scheduling cycle. The above is: , Indicates during the scheduling period upper node The action.

[0120] Reward: for taking action Then, the environment returns an immediate reward and proceeds to the next state. The immediate reward can be formalized as:

[0121] ,

[0122] in In order to take action Immediate rewards returned after the environment is restored. For cross-domain flow priority weights, if For cross-domain flows, then >0, if =0, then For intra-domain flow, For indicator functions, The delay penalty coefficient, This is the weight for timeout penalties.

[0123] S5. Execution Graph Convolutional Network and Multi-Agent Cooperative Scheduling (GACS) Distributed Reinforcement Learning Algorithm.

[0124] This invention constructs a learning framework for Adaptive Topology-Aware Autonomous Scheduling (ATLAS), such as... Figure 2 As shown, the graph convolutional network (GCN) is used to extract topological and local state features to achieve autonomous scheduling that combines graph convolutional encoder training with multi-agent value decomposition training. The control layer is trained using global state and joint reward learning parameters, enabling global coordination among individual agents based on their local inputs during decentralized execution. The underlying environment layer is responsible for interacting with the environment and processing data, receiving the current state. and perform actions Later transitioned to a new state This process is recorded as experience and stored in an experience replay buffer for use in randomly sampled mini-batch data during training. This data eventually flows into the GCN network, where it integrates information from different agents through a hybrid network. Finally, the loss function calculates the error between the predicted value and the actual reward to guide model optimization.

[0125] Based on this learning framework, a distributed reinforcement learning algorithm for GACS is proposed and implemented. First, GCN transfers network topology information... With the current scheduling cycle System node status The constructed node feature matrix The joint mapping is a structure-aware node embedding matrix. , For structure-aware vectors;

[0126] flow state The flow state feature vector is obtained by extracting and encoding locally observable static attributes. Nodes generated by Graph Convolutional Networks (GCN) Structure-aware vectors The local policy network of the agent is input together, and the output is the scheduled action. To ensure the executability of the action at the engineering level and avoid violating hard constraints such as TT / AVB, an executability mask is introduced between action selection and execution. The mask is calculated by the mechanism layer and referenced by the agent when selecting an action.

[0127] Local action In the implementation, it is mapped as a discrete index. , For discrete actions that are distinguishable within the scheduling cycle, index 0 is reserved for empty actions. A fixed maximum action dimension is preset for each agent during network training / inference. To avoid implementation complexity caused by different dimensions, actions that do not actually exist are filled in and masked during loss / selection. During training... Make action selections under greedy or other exploration strategies, and ensure that hidden actions are not selected.

[0128] In each scheduling cycle, the node Based on event priority, a current candidate flow to be scheduled is determined, and the agent's actions... This applies only to the candidate flow to be scheduled. The local actions of all agents are executed synchronously in each scheduling cycle, forming a joint action. .

[0129] During the training phase, the local action value function Aggregated into a joint value function by a hybrid network Based on this, the temporal difference objective and gradient are calculated for parameter updates.

[0130] During the execution phase, each agent relies on local observations. The mechanism greedily or nearly greedily selects actions based on local parameters, while simultaneously performing a final check on the executability of the actions and either actually executing them or rolling them back.

[0131] The algorithm uses an empirical replay buffer. Store state transition tuple Prioritized Experience Replay (PER) is employed to improve the reuse rate of key samples and enhance training efficiency. Simultaneously, online network parameter optimization is used. and target network parameters The target value calculation method using Double Q-learning is extended to the online network to select the next joint action. The target network is used to evaluate the selected action online, as shown in the following expression:

[0132] ,

[0133] in, for Time by the first Each agent estimates the optimal action chosen based on the current online network. , For a collection of intelligent agents, For all possible actions Choose the action that maximizes the Q value. For the first The action value function of an agent. It is the first An intelligent agent in Local observation at a given moment The target value for time-series difference is... for Instant rewards for each moment This is a termination flag, indicating whether the current interaction has reached the termination state. The degree of importance attached to future returns, For the joint value function of the target network, for The state of the intelligent agent at all times.

[0134] Simultaneously, the Huber loss function is used to optimize the error between the online network joint value function and the temporal difference objective value, thereby updating the network parameters of the online network joint value function:

[0135] ,

[0136] in Sampling weights for the importance of priority experience replay. For the joint value function of the online network, For Huber functions, For experience playback buffer pool The expected value of the sample distribution.

[0137] To ensure the stability of the target value, online network parameters It will periodically synchronize with the target network parameters. That is, every Execute once per training session or several episodes: .

[0138] The training process of the distributed reinforcement learning algorithm of GACS in this invention is as follows: Figure 3 As shown, this includes: first, initializing the training process by pre-determining training-related parameters, including the maximum time step for each training epoch. Synchronization update interval of the target network and initialize the global time step counter to Simultaneously, the network structure is initialized, including online network parameters. Target network parameters And build an experience replay buffer pool. Used to store interaction samples. The environment is then initialized by executing... Get the initial environment state During the interaction at each time step, the system first acquires current observation information from the environment, including node features and system operating status, and constructs the acquired information into graph-structured input data. Then, it uses a graph convolutional network (GCN) to encode the graph-structured data to obtain the embedding representation of each node, and calculates the local action value function of each agent based on this embedding representation. Based on this, a hybrid network is used to fuse and calculate the value functions of each local action to obtain the global joint action value function, and the action to be executed at the current time step is selected.

[0139] After the action is performed, the environment returns the state of the next time step and the corresponding reward, forming a state transition sample. The transferred sample is then stored in the experience replay buffer. Then, it checks whether the current time step has reached the preset maximum step size, i.e., whether the condition is met. If the target is not reached, update the counter. Then continue executing the environment interaction for the next time step. If the maximum step size is reached, further determine the experience replay buffer. Does the sampling condition meet? If the sampling condition meets, replay the data from the empirical buffer. A minibatch of samples is randomly selected from the data, and the target value is calculated based on the Double-Q mechanism. Subsequently, a loss function is constructed based on the target value, and the network gradient is calculated to update the online network parameters, while maintaining a preset synchronization interval. The target network parameters are synchronized and updated periodically, meaning the online network parameters are periodically synchronized to the target network. Finally, it is determined whether the current training epoch meets the requirements. ,in This indicates the maximum number of training rounds; if the condition is met, the next training round continues; otherwise, the entire training process ends.

[0140] Based on the above-described inventive concept, the present invention also provides an apparatus for deterministic cross-domain hybrid flow routing and scheduling, used to implement the above-described method for deterministic cross-domain hybrid flow routing and scheduling, the apparatus comprising:

[0141] The initial modeling module is used to construct a network model of a cross-domain deterministic system using directed graphs;

[0142] The problem construction module is used to model the business flow and end-to-end latency, construct the joint routing and scheduling problem of cross-domain hybrid flows in the cross-domain deterministic system network model, and formulate the coordination mechanism of the cross-domain hybrid flows in the cross-domain deterministic system network model.

[0143] The problem transformation module is used to model the joint routing and scheduling problem of cross-domain hybrid flows following the aforementioned cooperation mechanism as a multi-agent Markov decision process;

[0144] The optimization scheduling module is used to solve the multi-agent Markov decision process by employing a distributed reinforcement learning algorithm based on graph convolutional networks and multi-agent cooperative scheduling, thereby obtaining the optimal routing and scheduling strategy for cross-domain hybrid flows.

[0145] It is worth noting that this device embodiment corresponds to the above method embodiment. The implementation methods of the above method embodiments are all applicable to this device embodiment and can achieve the same or similar technical effects, so they will not be described in detail here.

[0146] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0147] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0148] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0149] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for cross-domain hybrid flow routing and scheduling in a deterministic network, characterized in that, include: A network model for cross-domain deterministic systems is constructed using directed graphs; Modeling of business flows and end-to-end latency, constructing a joint routing and scheduling problem for cross-domain hybrid flows in a cross-domain deterministic system network model, and formulating a coordination mechanism for the cross-domain hybrid flows in the cross-domain deterministic system network model; The joint routing and scheduling problem of cross-domain hybrid flows following the aforementioned cooperative mechanism is modeled as a multi-agent Markov decision process; A distributed reinforcement learning algorithm based on graph convolutional networks and multi-agent cooperative scheduling is used to solve the multi-agent Markov decision process, thereby obtaining the optimal routing and scheduling strategy for cross-domain hybrid flows.

2. The method for deterministic network cross-domain hybrid flow routing and scheduling according to claim 1, characterized in that, The business flow model is as follows: ， in, Represents a stream This includes time-triggered streams, audio / video bridging streams, and best-effort streams. Represents a stream The source node, Represents a stream destination node Represents a stream Maximum end-to-end transmission delay constraint Represents a stream bandwidth requirements Represents a stream Stream type Represents a stream bag size, Represents a stream Cross-domain identifiers, Represents a stream Minimum bandwidth requirement Represents a stream Priority weights; The end-to-end delay model is as follows: ， in End-to-end cumulative delay For flow The scheduling path Indicates the node number on the scheduling path. For link Available bandwidth capacity for service flows within the current time slot, link Indicates from node To the node Directed communication channel, For link The delay in transmission, For flow At the node The waiting time in the queue.

3. The method for deterministic network cross-domain hybrid flow routing and scheduling according to claim 2, characterized in that, The joint routing and scheduling problem of cross-domain hybrid flows in the cross-domain deterministic system network model is expressed as follows: ， in For decision variables, if flow If all scheduling constraints are met, then ,otherwise ; This represents the total number of service flows successfully scheduled within the current scheduling period. The scheduling constraints include: Delay constraints: , in, For flow The delay constraint activation flag is set for time-triggered streams and audio / video bridging streams. For best-effort flow, set ; Link capacity constraints: , in, Whether to flow From node Send to node Scheduled, with a value of 0 or 1. For a set of links in a cross-domain deterministic system network model; Queue capacity constraints: , in, For nodes In the time slot Above, the first Queue length for different types of business flows. The maximum buffer capacity supported by the node scheduling port. This represents the set of nodes in a cross-domain deterministic system network model. For time-triggered streams, For audio and video bridging streams, To do one's best to flow.

4. The method for deterministic network cross-domain hybrid flow routing and scheduling according to claim 3, characterized in that, The cooperative mechanism of the cross-domain hybrid flow in the cross-domain deterministic system network model includes: Set the time-triggered stream as the highest priority, the audio / video bridging stream as the second highest priority, and the best-effort stream as the lowest priority. For time-triggered streams, determine whether all nodes on the end-to-end path have available slots within the required continuous time interval. If any node exists... Not satisfied , For nodes In the time slot The set of slots used for time-triggered flow scheduling returns a verification failure signal and triggers a rollback strategy; the set of slots is obtained based on the slot availability function. For audio and video bridging streams, each node maintains an integer token indicator. When a node receives an audio and video bridging stream scheduling request, it needs to determine whether the request is satisfied. If the conditions are met, scheduling is allowed, where For audio and video bridging streams in time slots The integer token value, The minimum token threshold required to allow audio / video bridging stream scheduling; For best-effort streams, transmission is only permitted when both the time-triggered stream and the audio / video bridging stream queues are idle or have unoccupied non-reserved slots. When a high-priority stream request arises and needs to occupy a slot already occupied by a best-effort stream, the current transmission of the best-effort stream is forcibly interrupted, and the slot is immediately released to the high-priority stream.

5. A method for deterministic network cross-domain hybrid flow routing and scheduling according to claim 4, characterized in that, The slot availability function is expressed as: ， in For slot availability function, Represents a node Time index within supercycle The above has scheduling capabilities for a certain type of flow. The total number of slots within the supercycle is the least common multiple of the transmission cycles of all types of service flows.

6. The method for deterministic network cross-domain hybrid flow routing and scheduling according to claim 4, characterized in that, The joint routing and scheduling problem of cross-domain hybrid flows following the aforementioned cooperative mechanism is modeled as a multi-agent Markov decision process, including: Treat each node as an intelligent agent. agent state This includes node state and flow state, where node state is represented as: , For nodes state, For nodes The length of the stream queue. For nodes The current integer token value of the audio / video bridging stream. For nodes A boolean variable indicating whether the current time is within a time-triggered flow schedulable slot. For nodes Boundary node identifiers, For nodes The number of cross-domain streams cached in the middle. For nodes The remaining available bandwidth, For nodes Does the current scheduling period contain any audio or video bridging streams that have not met bandwidth guarantees? The flow state is represented as: , In flow state, This represents the current node position of the flow. The remaining tolerable delay is for the flow; The action of an agent is defined as: , in Indicates the first Actions of an agent, where an action of 0 represents an empty action, meaning the current candidate flow to be scheduled will not be scheduled; actions This indicates that the current candidate flow to be scheduled is to be forwarded to neighboring nodes. Receive queue , For nodes The neighborhood group, For a collection of stream queues, , A collection of intelligent agents; The joint action of all nodes during the scheduling period is represented as: , Indicates the node in the scheduling cycle The action, This represents the combined action of all nodes during the scheduling period; The agent's reward is represented as: ， in In order to take action Immediate rewards returned after the environment is restored. For cross-domain flow priority weights, if For cross-domain flows, then >0, if =0, then For intra-domain flow, For indicator functions, The delay penalty coefficient, This is the weight for timeout penalties.

7. A method for deterministic network cross-domain hybrid flow routing and scheduling according to claim 6, characterized in that, The method employs a distributed reinforcement learning algorithm based on graph convolutional networks and multi-agent cooperative scheduling to solve the multi-agent Markov decision process, thereby obtaining the optimal routing and scheduling strategy for cross-domain hybrid flows, including: Graph convolutional networks combine network topology information with the system node states from the current scheduling period. The constructed node feature matrix is ​​jointly mapped to a structure-aware node embedding matrix. , For structure-aware vectors; flow state The flow state feature vector is obtained by extracting and encoding locally observable static attributes. Structure-aware vectors generated by graph convolutional networks The local policy network of the agent is input together, and the scheduled actions are output. In each scheduling cycle, the node Based on event priority, a current candidate flow to be scheduled is determined. The scheduling actions of the agents only apply to the candidate flow to be scheduled at the corresponding node. The scheduling actions of all agents are executed synchronously in each scheduling cycle, forming a joint action. This refers to the optimal routing and scheduling strategy for cross-domain mixed flows.

8. A method for deterministic network cross-domain hybrid flow routing and scheduling according to claim 7, characterized in that, The distributed reinforcement learning algorithm based on graph convolutional networks and multi-agent cooperative scheduling employs online network parameter mapping during the training phase. and target network parameters Selecting an action is represented as: ， in, for Time by the first Each agent estimates the optimal action chosen based on the current online network. , For a collection of intelligent agents, For all possible actions Select the action that maximizes the Q value. For the first The action value function of an agent. It is the first An intelligent agent in Local observation at a given moment The target value for time-series difference is... for Instant rewards for each moment This is a termination flag, indicating whether the current interaction has reached the termination state. The degree of importance attached to future returns, For the joint value function of the target network, for The state of the agent at all times; Simultaneously, the Huber loss function is used to optimize the error between the online network joint value function and the temporal difference objective value, thereby updating the network parameters of the online network joint value function, expressed as: ， in, Represents the loss function. Sampling weights for the importance of priority experience replay. For the joint value function of the online network, For Huber functions, This represents the expectation of the sample distribution in the experience replay buffer.

9. A method for deterministic network cross-domain hybrid flow routing and scheduling according to claim 8, characterized in that, During the training phase, every This training will use online network parameters Periodically synchronize to target network parameters .

10. An apparatus for deterministic network cross-domain hybrid flow routing and scheduling, characterized in that, The apparatus for implementing the deterministic network cross-domain hybrid flow routing and scheduling method of claim 1, the apparatus comprising: The initial modeling module is used to construct a network model of a cross-domain deterministic system using directed graphs; The problem construction module is used to model the business flow and end-to-end latency, construct the joint routing and scheduling problem of cross-domain hybrid flows in the cross-domain deterministic system network model, and formulate the coordination mechanism of the cross-domain hybrid flows in the cross-domain deterministic system network model. The problem transformation module is used to model the joint routing and scheduling problem of cross-domain hybrid flows following the aforementioned cooperation mechanism as a multi-agent Markov decision process; The optimization scheduling module is used to solve the multi-agent Markov decision process by employing a distributed reinforcement learning algorithm based on graph convolutional networks and multi-agent cooperative scheduling, thereby obtaining the optimal routing and scheduling strategy for cross-domain hybrid flows.

Images