A time-sensitive network cross-domain traffic scheduling method, system, medium and device

Abstract

The present invention relates to the technical field of data transmission, and discloses a time-sensitive network cross-domain traffic scheduling method, system, medium and device, including: for the current time step, in the state, after selecting an action through the main network and the greedy strategy, calculate the reward after the action is executed and the state of the next time step; store the state, action, reward and state of the next time step of the current time step as a transfer to the replay memory, and when the number of transfers in the replay memory reaches a threshold, sample multiple transfers from the replay memory; for each sampled transfer, calculate the time difference target value of each branch through the target network and the Munchausen mechanism; calculate the predicted Q value of each branch through the main network; calculate the mean square error, and update the parameters of the main network based on the mean square error; assign the parameters of the main network to the target network every several time steps. The real-time performance of traffic scheduling and the reliability of routing selection are guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of data transmission technology, and in particular to a method, system, medium and device for cross-domain traffic scheduling in a time-sensitive network. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] The existing Time-Sensitive Networking (TSN) technology has been widely used in industrial automation, intelligent transportation, the Internet of Things and other fields, providing a guarantee for low-latency and high-reliability data transmission.

[0004] However, with the continuous expansion of network scale and the complexity of industrial automation systems, the problem of collaborative work and traffic scheduling between multiple TSN domains has become increasingly prominent. Especially in a multi-domain environment, the existing traffic scheduling and routing optimization algorithms are difficult to effectively deal with cross-domain time synchronization, traffic congestion and resource scheduling problems, resulting in increased end-to-end delay, packet loss and low network resource utilization. Summary of the invention

[0005] In order to solve the above problems, the present invention provides a method, system, medium and device for cross-domain traffic scheduling in a time-sensitive network. Through Markov decision process modeling, the cross-domain scheduling and routing problems are converted into decision problems in a multidimensional action space, and the main network, target network and Munchausen mechanism are combined to optimize the selection of time slots and routes, thereby ensuring the real-time performance of traffic scheduling and the reliability of route selection.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A first aspect of the present invention provides a method for scheduling cross-domain traffic in a time-sensitive network, comprising:

[0008] Initialize the state of the time-sensitive network;

[0009] For the current time step, in the state, after selecting an action through the main network and the greedy strategy, the reward after the action is executed and the state of the next time step are calculated;

[0010] The state, action, reward of the current time step and the state of the next time step are stored in the replay memory as a transition. When the number of transitions in the replay memory reaches a threshold, multiple transitions are sampled from the replay memory.

[0011] For each sampled transfer, the target time difference value of each branch is calculated through the target network and the Munchausen mechanism; the predicted Q value of each branch is calculated through the main network; based on the target time difference value and the predicted Q value of each branch, the mean square error is calculated, and the parameters of the main network are updated based on the mean square error;

[0012] At intervals of several time steps, the parameters of the main network are assigned to the target network;

[0013] Determine whether the end condition is met. If so, output the scheduling strategy and perform traffic scheduling; otherwise, enter the next time step and return to the main network and greedy strategy to select actions.

[0014] Furthermore, the time-sensitive network adopts a multi-domain time-sensitive network architecture, including a control layer, a network layer and a field layer. The control layer manages resource allocation and scheduling strategies of multiple domains through centralized network configuration and user configuration modules, and the network layer is used to transmit cross-domain time-sensitive traffic.

[0015] Furthermore, the time-sensitive network uses the IEEE 802.1AS protocol for time synchronization, each domain performs time synchronization through a clock host within its domain, and performs time synchronization error correction through a software-defined network controller.

[0016] Furthermore, the time-sensitive network uses a network-wide unified scheduling time slot based on circular queue forwarding, and realizes the reception and forwarding of data packets by dividing time into multiple time slots of fixed length.

[0017] Furthermore, the reward is calculated using a reward function, and the reward function is expressed as: ,in, Indicates whether the traffic scheduling is successful under the action of the current time step. and represents the weight parameter, Represents adjacent edges The average link utilization rate is is the time slot utilization of the adjacent edge.

[0018] Furthermore, the time difference target value is defined as: ,in, It's a reward. is the scaling factor, represents the Munchausen term obtained by the Munchausen mechanism, is the discount factor, is the action in action branch b estimated by the target network The Q value, ( s , a , r ,s' ) represents sampling transfer, s , a , r and s' Sample the current time step state, action, reward and next time step state in the transition respectively, A b Represents the action space of action branch b.

[0019] Further, the mean square error is defined as: , where B represents the number of action branches b, Indicates the time difference target value, is the action in action branch b estimated by the main network Q value.

[0020] A second aspect of the present invention provides a time-sensitive network cross-domain traffic scheduling system, comprising:

[0021] An initialization module, configured to: initialize the state of the time-sensitive network;

[0022] An action selection module is configured to: for the current time step, in the state, after selecting an action through the main network and the greedy strategy, calculate the reward after the action is executed and the state of the next time step;

[0023] A replay memory module is configured to: store the state, action, reward of the current time step and the state of the next time step as a transition in the replay memory, and when the number of transitions in the replay memory reaches a threshold, sample multiple transitions from the replay memory;

[0024] The main network update module is configured as follows: for each sampled transfer, the time difference target value of each branch is calculated through the target network and the Munchausen mechanism; the predicted Q value of each branch is calculated through the main network; based on the time difference target value and the predicted Q value of each branch, the mean square error is calculated, and the parameters of the main network are updated based on the mean square error;

[0025] A target network update module is configured to: assign the parameters of the main network to the target network at intervals of a certain number of time steps;

[0026] The scheduling module is configured to: determine whether the end condition is met, if so, output the scheduling strategy and perform traffic scheduling; otherwise, enter the next time step and return to the main network and greedy strategy for action selection.

[0027] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor. When the program is executed by the processor, the steps in a time-sensitive network cross-domain traffic scheduling method as described above are implemented.

[0028] The fourth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the program, the steps in the time-sensitive network cross-domain traffic scheduling method as described above are implemented.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The present invention provides a method for cross-domain traffic scheduling in a time-sensitive network. The method converts cross-domain scheduling and routing problems into decision problems in a multidimensional action space through Markov decision process modeling, and optimizes the selection of time slots and routes by combining the main network, the target network and the Munchausen mechanism, thereby ensuring the real-time performance of traffic scheduling and the reliability of routing selection.

[0031] The present invention provides a time-sensitive network cross-domain traffic scheduling method, which realizes time synchronization and deterministic traffic scheduling in a complex multi-domain environment through a multi-domain time-sensitive network architecture and its cross-domain scheduling mechanism.

[0032] The present invention provides a method for cross-domain traffic scheduling in a time-sensitive network, in which time synchronization is achieved between all TSN domains through the IEEE802.1AS protocol, and a unified scheduling time slot mechanism based on CQF is used to ensure the real-time performance and low latency of cross-domain data transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which constitute a part of the specification of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but do not constitute limitations of the present invention.

[0034] Figure 1 This is a flow chart of a method for scheduling cross-domain traffic in a time-sensitive network according to the first embodiment of the present invention;

[0035] Figure 2 This is a diagram of the MD-TSN architecture of the first embodiment of the present invention;

[0036] Figure 3 A schematic diagram of time synchronization error between adjacent TSN domains according to the first embodiment of the present invention;

[0037] Figure 4 This is a schematic diagram of a circular queuing forwarding mechanism according to the first embodiment of the present invention. DETAILED DESCRIPTION

[0038] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0039] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0040] In the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other. The present invention is further described below with reference to the accompanying drawings and embodiments.

[0041] Terminology explanation:

[0042] IEEE 802.1AS: A network protocol used primarily for time synchronization in real-time Ethernet networks. IEEE 802.1AS is part of the IEEE 802.1 family and is designed to ensure time consistency between devices in a network, thereby supporting high-precision time-sensitive applications such as audio and video streaming. IEEE 802.1AS is based on the IEEE 1588 Precision Time Protocol (PTP), which provides high-precision time synchronization capabilities, allowing network devices to synchronize time within microseconds.

[0043] Munchausen mechanism: The Munchausen mechanism introduces an innovative reward recalibration method in deep reinforcement learning, which aims to improve the learning efficiency of agents in sparse reward environments. This mechanism encourages agents to explore under-visited state spaces by considering "false rewards" in the Q-value update process, that is, re-evaluation based on historical states.

[0044] Q-value: A key concept in reinforcement learning that represents the expected reward of taking an action in a specific state. It is the core of Q-learning and many other reinforcement learning algorithms, used to evaluate the value of an agent at a given state and action combination.

[0045] Branching Dueling Q-Network (BDQN): An enhanced deep reinforcement learning algorithm that combines the advantages of branching structure and duel learning to improve the decision-making ability of intelligent agents in complex environments; BDQN introduces multiple branches, allowing the network to simultaneously learn the value and advantages of different actions, thereby more effectively capturing the potential value of state-action pairs.

[0046] Embodiment 1

[0047] The purpose of this first embodiment is to provide a method for cross-domain traffic scheduling in a time-sensitive network.

[0048] This embodiment provides a time-sensitive network cross-domain traffic scheduling method, provides a multi-domain time-sensitive network (MD-TSN) architecture and a unified scheduling time slot mechanism based on CQF (Cyclic queuing and forwarding), and proposes a cross-domain scheduling and routing optimization algorithm, aiming to solve the complexity of cross-domain communication in multi-domain time-sensitive networks and ensure low-latency and high-reliability data transmission.

[0049] This embodiment provides a method for scheduling cross-domain traffic in a time-sensitive network, including the following steps:

[0050] Step 1: Build the MD-TSN architecture.

[0051] In this embodiment, if Figure 2 As shown in the figure, the time-sensitive network adopts MD-TSN. The MD-TSN architecture is divided into the control layer, network layer and field layer, and the industrial network is divided into multiple TSN domains. The division of labor among the control layer, network layer and field layer ensures collaborative work in a multi-domain environment.

[0052] Among them, the control layer acts as the core authority for high-level coordination and decision-making in the industrial network, responsible for managing the scheduling strategy, routing strategy, resource allocation and network configuration of the entire network. This function is achieved through a centralized software-defined network (SDN) controller (such as ONOS), which supervises the entire multi-domain time-sensitive network architecture. Each domain adopts a fully centralized configuration model, covering the central network configuration (CNC) and the central user configuration (CUC). The CNC configures the network according to the capabilities of the TSN switch and the network topology to ensure the reliable and timely transmission of critical data between multiple domains. The CUC is responsible for collecting application requirements and sending them to the CNC, which then allocates resource reservations and scheduling parameters to the TSN switch, and the CUC forwards these instructions to the terminal device. This centralized control architecture supports complex routing and scheduling strategies, and is specially designed for cross-domain operations, which significantly improves the efficiency and reliability of data transmission in industrial environments.

[0053] Among them, the network layer guarantees the transmission of cross-domain time-sensitive traffic (inter-domain data). As an intermediary between the application layer and the field layer, the network layer ensures seamless data transmission across domain boundaries and consistency of network configuration. This layer implements deterministic traffic scheduling based on the configuration of the application layer and manages data flows to meet strict time and performance requirements. In addition, the network layer also integrates advanced traffic shaping mechanisms, quality of service (QoS) guarantees, and synchronization functions, which are necessary to maintain reliable communication in a time-sensitive network ecosystem. These mechanisms work together to ensure efficient and reliable data transmission in complex industrial environments.

[0054] The field layer is responsible for interacting with industrial control equipment and controlling industrial equipment. The field layer contains multiple PLCs (Programmable Logic Controllers), and each PLC is connected to at least one actuator and sensor.

[0055] like Figure 2 In the time-sensitive network shown in the figure, the network layer contains three TSN domains, namely the first TSN domain (TSN domain 1), the second TSN domain (TSN domain 2) and the third TSN domain (TSN domain 3). The first TSN domain contains a first CNC (CNC1), a first CUC (CUC1) and multiple TSN switches connected by links; the second TSN domain contains a second CNC (CNC2), a second CUC (CUC2) and multiple TSN switches connected by links; the third TSN domain contains a third CNC (CNC3), a third CUC (CUC3) and multiple TSN switches connected by links. Each TSN domain connects two PLCs through a link. PLCs in different TSN domains implement C2C (controller to controller) connection through TSN domains. PLCs and actuators in different TSN domains implement C2D (controller to device) connection through TSN domains. Each TSN domain is connected to the control layer through C2C.

[0056] The devices in each domain maintain precise time synchronization with other domains through the clock host in the domain. The time synchronization of each domain is achieved through the IEEE 802.1AS protocol. The SDN (Software Defined Network) controller at the control layer is responsible for coordinating the resource scheduling and traffic control of each domain.

[0057] Step 2: Establish a time synchronization mechanism.

[0058] Time-sensitive networks use the IEEE 802.1AS protocol for time synchronization. Devices in each TSN domain generate gPTP messages through the clock host in their domain and perform time synchronization error correction through the SDN controller to achieve accurate time synchronization across domains.

[0059] Based on the IEEE 802.1AS protocol, each TSN domain performs time synchronization through the clock host within the domain. The SDN controller captures the timestamp of each domain and calculates the time synchronization error, thereby ensuring accurate time synchronization across domains.

[0060] Time synchronization is the core issue to ensure traffic transmission in time-sensitive networks. By using the IEEE 802.1AS protocol, multiple TSN domains can be synchronized through precise clock synchronization messages (gPTP). The SDN controller captures the timestamp of each TSN domain and corrects the clock data based on propagation delay and time error, thereby ensuring that traffic between different domains can be transmitted with microsecond accuracy.

[0061] Step 201, use the IEEE 802.1AS standard to accurately synchronize the time between the components within the TSN network (time sensitive network). This standard uses the generic Precision Time Protocol (gPTP) to transmit synchronization frames from the Grand Master (GM) of the time sensitive network to each network node to achieve clock synchronization based on the measured link delay and the difference in clock frequency of different nodes. The GM clock is selected by the Best Master Clock Selection Algorithm (BMCA) based on the received announcement frame.

[0062] Step 202: The SDN controller captures the timestamp of each TSN domain and analyzes the time synchronization error between adjacent TSN network domains.

[0063] Specifically, the GM clock of each TSN synchronization domain generates a gPTP message with an embedded timestamp and sends it to the SDN controller. The SDN controller generates an entry timestamp based on the local clock and embeds it into the received gPTP message. The synchronization error between the TSN domain and the SDN control clock is calculated based on the timestamp message and the link propagation delay. Finally, the time synchronization error between adjacent TSN domains is calculated based on the synchronization error between each TSN domain and the SDN controller clock.

[0064] Assume that TSN domain 1 and TSN domain 2 are two adjacent TSN network domains, such as Figure 3 As shown, the steps for calculating the time synchronization error between TSN domain 1 and TSN domain 2 include:

[0065] (1) Calculate the synchronization error between TSN domain 1 and the SDN controller clock: ;in, is the timestamp generated by TSN domain 1, is the timestamp generated by the SDN controller, is the propagation delay from TSN domain 1 to the SDN controller.

[0066] (2) Calculate the synchronization error between TSN domain 2 and the SDN controller clock: ,in, is the timestamp generated by TSN domain 2, is the timestamp generated by the SDN controller, is the propagation delay from TSN domain 2 to the SDN controller.

[0067] (3) Calculate the time synchronization error between TSN domain 1 and TSN domain 2: .

[0068] Step 3: Develop a unified scheduling time slot mechanism for CQF.

[0069] The unified scheduling time slot mechanism based on circular queue forwarding (CQF) (referred to as CQF mechanism) is used to achieve orderly reception and forwarding of data packets by dividing time into multiple fixed-length time slots. The CQF mechanism ensures deterministic transmission of time-sensitive traffic by alternating the reception and forwarding of queues.

[0070] The CQF mechanism divides time into time slots of fixed length, and data is received and sent in each time slot to ensure orderly scheduling of cross-domain traffic. This mechanism minimizes data transmission delay and jitter by cyclically using queues.

[0071] The CQF mechanism is a method for implementing deterministic scheduling through a circular queue. It divides time into multiple equal time slots, and performs data reception and transmission operations in each time slot. This method ensures that traffic transmission across multiple domains can be completed within the predetermined time window, avoiding queue overflow and packet loss.

[0072] CQF uses a Ping-Pong queue-based model to cyclically switch the transmission status of two queues to schedule TT (time-triggered flow) traffic and ensure deterministic latency communication, using the queue with the highest priority. and queue Used to receive and send TT streams. Each queue deploys a receive gate (Rx) and a transmit gate (Tx) to control the queue to forward or receive TT streams. When the Rx gate or Tx gate is open (O), the relevant queue can receive or send TT streams. When the Rx gate or Tx gate is closed (C), the relevant queue is prohibited from receiving or forwarding TT streams. In CQF, time is divided into n equal time slots .like Figure 4 As shown, in the even time slots (such as ),queue To receive a TT stream, its Rx gate is open and its Tx gate is closed; conversely, the queue For the TT flow in the forwarding queue, its Tx gate is open and the Rx gate is closed; similarly, in odd time slots (such as ),queue Transmit TT stream, its Rx gate is closed, Tx gate is open; queue To receive the TT stream, its Rx gate is open and its Tx gate is closed.

[0073] In CQF, the maximum end-to-end delay experienced by a TT flow is related to the number of hops and the length of the time slot. Therefore, its maximum end-to-end delay can be expressed as: .

[0074] The minimum end-to-end delay experienced by a TT flow can be expressed as: ;in, is the hop count of the traffic, Uniformly schedule time slot length for the entire network.

[0075] In addition, the downstream bridge must receive the frame from the upstream bridge in the same time slot and forward the received frame to the network in the next time slot. Therefore, the length of a time slot is related to the transmission delay of the bridge, the propagation delay of the link, and the processing delay of the bridge. However, the premise of using CQF is that the clocks of all network devices need to be accurately synchronized, which is difficult to achieve accurate clock synchronization of all network devices in multiple TSN network domains. Considering that when data is transmitted across TSN domains, a data packet can only be transmitted from one bridge in the current TSN domain to one bridge in the adjacent TSN domain at a time, therefore, a unified scheduling time slot mechanism for the entire network is designed to use the CQF model.

[0076] Due to the particularity of cross-domain traffic scheduling, the synchronization error between multiple TSN domains needs to be considered when designing the scheduling time slot length. The proposed unified scheduling time slot length for the entire network is expressed as: .

[0077] in, is the propagation delay of the data packet in the optical fiber. is the delay of the switch processing the data packet, It is the maximum value of the time synchronization error between adjacent TSN network domains.

[0078] in, is the transmission delay of the data packets in the queue at the switch, .in, is the number of TT flows that a queue can accommodate, is the maximum transmission length of a TT flow, It is the network bandwidth, and MTU stands for the Ethernet Maximum Transmission Unit.

[0079] In addition, when designing a unified scheduling time slot for the entire network, the impact of clock synchronization errors within the TSN network domain needs to be further considered.

[0080] Step 4: Cross-domain scheduling and routing optimization algorithm.

[0081] This embodiment proposes a cross-domain traffic scheduling algorithm based on the branching dueling Munchausen deep Q network (BD-MDRS). The algorithm converts the cross-domain scheduling and routing problems into decision-making problems in a multi-dimensional action space through Markov decision process (MDP) modeling, and combines the branching dueling Q-network (BDQN) and the Munchausen mechanism to optimize the selection of time slots and routes, ensuring the real-time nature of traffic scheduling and the reliability of routing selection.

[0082] This embodiment models the cross-domain scheduling and routing problem as a Markov decision process (MDP), and optimizes the time slot selection and routing selection through a multi-dimensional action space. That is, the cross-domain scheduling and routing problem is modeled as a Markov decision process (MDP), and the time slot selection and routing selection are processed through a branch adversarial Q network to optimize cross-domain traffic scheduling; and the Munchausen mechanism is combined to adjust the reward function by scaling the logarithmic probability of the strategy to improve the exploration and stability of the algorithm; and the shared features of the input state are extracted through the shared network core, and the decisions of time slot selection and routing selection are processed separately to optimize the certainty and efficiency of traffic scheduling.

[0083] In this embodiment, the cross-domain traffic scheduling and routing problem is modeled as an MDP, including a state space, an action space, and a reward function:

[0084] The state space (S) consists of network topology and flow information; network information includes topology and link load, while flow information includes source node, destination node, period, size, and maximum end-to-end delay required by TT flow. Node refers to the node where traffic is transmitted.

[0085] The action space (A) is divided into two dimensions:

[0086] Time slot selection (A 1 ): responsible for the allocation of time slots;

[0087] Adjacent edge selection (A 2 ): Select edges for routing.

[0088] In the scheduling model of this embodiment, time slot allocation is performed only in the first step, and subsequent actions are selected by increasing the time slot.

[0089] The reward function (R) takes into account the scheduling success rate, link utilization, and time slot utilization at the same time. The goal is to optimize the scheduling of cross-domain traffic, which is expressed as: .in, Indicates whether the traffic scheduling is successful under the action of the current step (time step), 1 if the scheduling is successful, otherwise 0; and represents the weight parameter; Represents adjacent edges The average link utilization is expressed as: .in, Is an adjacent edge In time slot The available queue capacity, is the total capacity of the link, Indicates the total number of time slots; the scheduling time is divided into n equal network-wide unified scheduling time slots t1, t2, …, t n ;, is the time slot utilization of the adjacent edge, that is, the ratio of the number of available time slots of the adjacent edge to the total number of time slots, expressed as: ,in, Representation Flow The adjacent edge The number of time slots available.

[0090] In this embodiment, the Branching Dueling Q-Network (BDQN) structure is used to handle the selection problem in a multi-dimensional action space. The network structure consists of three main parts:

[0091] Shared network core: extracts a shared representation of the input state s for multiple branches to estimate the Q values ​​of different sub-action dimensions;

[0092] Branch duel module: Each branch learns the value and advantage function of a specific sub-action dimension (such as time slot allocation or adjacent edge selection);

[0093] Output layer: Combines the value and advantage streams to compute the Q-value for each branch action.

[0094] The Branching Dueling Q-Network (BDQN) models time slot selection and route selection as two independent branches, extracts a shared representation of the input state through a shared network core, and optimizes these two action dimensions separately; each branch learns the Q value of the time slot and route selection: .in, represents the action in the action branch b estimated by the network The Q value, represents the value function of action branch b, Represents the action in action branch b The advantage function of Indicates the number of actions in action branch b, b is 1 or 2.

[0095] In this embodiment, in order to improve the algorithm's exploration ability, the BD-MDRS algorithm introduces the Munchausen mechanism, which encourages the agent to conduct effective exploration and find the optimal scheduling and routing strategy in a dynamic network environment by adjusting the strategy log probability in the reward function.

[0096] The Munchausen term is expressed as: ,in, is the temperature parameter that controls the degree of exploration, is the action under the softmax strategy The logarithmic probability of , derived from the target Q function, is expressed as: ,in, Represents the action in the target network under state s a Q value.

[0097] To derive the update rule of BD-MDRS, we first define the time difference (TD) target of each branch b. The TD target of branch b is Defined as: .in, is the immediate reward calculated using the reward function (R), is a scaling factor that balances exploration and exploitation during learning, is the discount factor for future rewards, is the target network Q value for the next state-action pair, using the parameter The target network is estimated. A b Represents the action space of action branch b.

[0098] The loss function for training BD-MDRS is defined as the mean square error (MSE) between the predicted Q value of each branch and the target Q value: .

[0099] Wherein, B represents the number of action branches b, which is 2 in this embodiment. It represents the action selected on action branch b.

[0100] Due to the dynamic changes in the branch action space (depending on the number of adjacent edges of the node), this embodiment proposes a fixed-length vector representation method for standardizing the output scheduling strategy. This method ensures that the dimension of the action space is consistent. The steps for determining the action space are as follows:

[0101] (1) Determine the maximum size of the action space.

[0102] First, determine the maximum number of adjacent edges a single node in the network may have. This number will define the maximum length of the unified action vector, that is, the maximum number of actions that can be chosen. For example, if the number of adjacent edges of any node in the network is at most 16, the length of the action vector is set to 16.

[0103] (2) Design a mapping table.

[0104] For each node, a mapping table is created to record the mapping relationship between its adjacent edges and the positions in the unified action vector. In this way, no matter how the position or adjacent edges of the node change, the structure of the action vector remains the same.

[0105] (3) Use a mask vector.

[0106] Because the actual valid actions may be different at each node, a dynamic mask vector is needed to represent the valid actions of the current node. The mask vector has the same length as the action vector, with valid actions being 1 and invalid actions being 0.

[0107] (4) Generation of action vector.

[0108] At each decision step, an action vector is generated based on the mapping table of the current node. The positions corresponding to valid adjacent edges in the vector are set to 1, and the rest are set to 0. For example, if a node has three adjacent edges and the maximum degree is 6, its action vector may be represented as [1, 0, 1, 0, 0,1], where the non-zero position indicates the selection of the corresponding adjacent edge.

[0109] (5) Use masks for action selection.

[0110] When selecting an action, the policy network generates a candidate action probability vector, which has the same length as the unified action vector. It applies element-wise multiplication (Hadamard product) through the mask vector to set the probability of invalid actions to zero, so that only valid actions are considered.

[0111] Example: Hypothetical Node The set of adjacent edges of , its mapping table is: , the action vector might be [1, 0, 3, 0, 5, 6], indicating the selection of edge , , and As possible actions, while the edge and Not selectable (assuming it is temporarily unavailable for some reason); the corresponding mask vector is [1, 0, 1, 0, 1, 1].

[0112] In this embodiment, the input of a time-sensitive network cross-domain traffic scheduling algorithm includes: network topology, replay memory capacity M, discount factor , temperature parameters , exploration rate , scaling factor , learning rate , batch size N, target network update frequency C.

[0113] In this embodiment, if Figure 1 As shown, the steps of a time-sensitive network cross-domain traffic scheduling algorithm include:

[0114] Step 401: Initialize network parameters: Initialize main network parameters and target network parameters ,and ; Initialize replay memory D and set capacity M.

[0115] Among them, both the main network and the target network adopt the Branching Dueling Q-Network (BDQN) architecture.

[0116] Step 402: At the beginning of each round: initialize the state of the time-sensitive network , and let time step t = 0, and let counter w = 0. Among them, the state consists of network topology and flow information; network information includes topology and link load, and flow information includes source node, destination node, cycle, size and maximum end-to-end delay of the flow.

[0117] Step 403: For the time step , in the state Next, through - Greedy strategy to choose actions .

[0118] Specifically, in the state In this case, each branch in the main network calculates multiple actions Q value , Represents the i-th action in action branch b; for each branch adversarial Q network branch b, with probability Choose a random action, or Select the action with the largest Q value, and get the action .

[0119] Step 404: In the environment (state of the time-sensitive network ) to perform actions , and observe the reward and the next time step state (action Status after execution) Among them, rewards It is calculated through the reward function.

[0120] Step 405: Storage transfer To replay memory (buffer) D.

[0121] Step 406: For branch b, when there are at least N transfers in the replay memory, sample a small batch containing N transfers from D; for each sampled transfer ( s , a , r , s' ), using the target strategy calculate Munchausen term If the number of transfers in the replay memory does not reach N, it will not be adopted and training will continue to execute steps 404 and 405.

[0122] Step 407: For branch b, transfer the sampled Input the main network to get the Q value in the main network under state s, that is, the predicted Q value ; Transfer the sample s' Input the target network and obtain , get the action when the maximum Q value in the target network .

[0123] Step 408: Based on , , r , , calculated , by minimizing the loss function of all branches and transitions , update the main network parameters .

[0124] Step 408, let t=t+1, w=w+1; determine whether w is less than C, and t is less than the set value, if so, return to step 403; if w is greater than or equal to C, and t is less than the set value, update the target network parameters , that is, every C time steps, update the target network parameters ; If t is greater than or equal to the set value (i.e., the end condition is met), end this round of training and output the optimal scheduling strategy ,in, and They represent the edge connecting the sender to the switch and the edge connecting the switch to the receiver, respectively. represents the tth time slot, Represents the edges between switches.

[0125] This embodiment provides a time-sensitive network cross-domain traffic scheduling method, which realizes time synchronization and deterministic traffic scheduling in a complex multi-domain environment through an innovative multi-domain time-sensitive network (MD-TSN) architecture and its cross-domain scheduling mechanism. Its architecture includes a control layer, a network layer, and a field layer. All TSN domains achieve time synchronization through the IEEE 802.1AS protocol, and use a unified scheduling time slot mechanism based on CQF to ensure the real-time and low latency of cross-domain data transmission.

[0126] Embodiment 2

[0127] The purpose of the second embodiment is to provide a time-sensitive network cross-domain traffic scheduling system, including:

[0128] An initialization module, configured to: initialize the state of the time-sensitive network;

[0129] An action selection module is configured to: for the current time step, in the state, after selecting an action through the main network and the greedy strategy, calculate the reward after the action is executed and the state of the next time step;

[0130] A replay memory module is configured to: store the state, action, reward of the current time step and the state of the next time step as a transition in the replay memory, and when the number of transitions in the replay memory reaches a threshold, sample multiple transitions from the replay memory;

[0131] The main network update module is configured as follows: for each sampled transfer, the time difference target value of each branch is calculated through the target network and the Munchausen mechanism; the predicted Q value of each branch is calculated through the main network; based on the time difference target value and the predicted Q value of each branch, the mean square error is calculated, and the parameters of the main network are updated based on the mean square error;

[0132] A target network update module is configured to: assign the parameters of the main network to the target network at intervals of a certain number of time steps;

[0133] The scheduling module is configured to: determine whether the end condition is met, if so, output the scheduling strategy and perform traffic scheduling; otherwise, enter the next time step and return to the main network and greedy strategy for action selection.

[0134] It should be noted here that each module in this embodiment corresponds to each step in Example 1 one by one, and the specific implementation process is the same, which will not be repeated here.

[0135] Embodiment 3

[0136] This embodiment provides a computer-readable storage medium on which a computer program is stored. The program is executed by a processor. When the program is executed by the processor, the steps in a time-sensitive network cross-domain traffic scheduling method as described in the above-mentioned embodiment 1 are implemented.

[0137] Embodiment 4

[0138] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the program, the steps in the time-sensitive network cross-domain traffic scheduling method as described in the above-mentioned embodiment 1 are implemented.

[0139] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0140] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for scheduling cross-domain traffic in a time-sensitive network, characterized in that: include: Initialize the state of the time-sensitive network; For the current time step, under the state of the current time step, after selecting an action through the main network and the greedy strategy, the reward after the action is executed and the state of the next time step are calculated; The state, action, reward of the current time step and the state of the next time step are stored in the replay memory as a transition. When the number of transitions in the replay memory reaches a threshold, multiple transitions are sampled from the replay memory. For each sampled transfer, the time difference target value of each branch is calculated through the target network and the Munchausen mechanism; Through the main network, calculate the predicted Q value of each branch; Based on the time difference target value and predicted Q value of each branch, the mean square error is calculated, and the parameters of the main network are updated based on the mean square error; The time difference target value is defined as: ,in, is the immediate reward calculated using the reward function (R), is a scaling factor that balances exploration and exploitation during learning, represents the Munchausen term obtained by the Munchausen mechanism, is the discount factor for future rewards, is the action in action branch b estimated by the target network The Q value, represents the state of the next time step, represents the action at the next time step, represents the target network parameters, ( s , a , r , s' ) indicates sampling transfer, s , a , r and s' Sample the current time step state, action, reward and next time step state in the transition respectively, A b Represents the action space of action branch b; assigns the parameters of the main network to the target network every several time steps; Determine whether the end condition is met. If so, output the scheduling strategy and perform traffic scheduling; otherwise, enter the next time step and return to the main network and greedy strategy to select actions.

2. A time-sensitive network cross-domain traffic scheduling method as claimed in claim 1, characterized in that: The time-sensitive network adopts a multi-domain time-sensitive network architecture, including a control layer, a network layer and a field layer. The control layer manages resource allocation and scheduling strategies of multiple domains through centralized network configuration and user configuration modules, and the network layer is used to transmit cross-domain time-sensitive traffic.

3. A time-sensitive network cross-domain traffic scheduling method as claimed in claim 1, characterized in that: The time-sensitive network uses the IEEE 802.1AS protocol for time synchronization. Each domain performs time synchronization through a clock host within its domain, and performs time synchronization error correction through a software-defined network controller.

4. A time-sensitive network cross-domain traffic scheduling method as claimed in claim 1, characterized in that: The time-sensitive network uses a network-wide unified scheduling time slot based on circular queue forwarding, and realizes the reception and forwarding of data packets by dividing time into multiple time slots of fixed length.

5. A time-sensitive network cross-domain traffic scheduling method as claimed in claim 1, characterized in that: The reward is calculated using a reward function, which is expressed as: ,in, Indicates whether the traffic scheduling is successful under the action of the current time step. and represents the weight parameter, Represents adjacent edges The average link utilization rate is is the time slot utilization of the adjacent edge.

6. A time-sensitive network cross-domain traffic scheduling method as claimed in claim 1, characterized in that: The mean square error is defined as: , where B represents the number of action branches b, Indicates the time difference target value, is the action in action branch b estimated by the main network The Q value, represents the state of the current time step, Indicates the main network parameters.

7. A time-sensitive network cross-domain traffic scheduling system, characterized in that: include: An initialization module, configured to: initialize the state of the time-sensitive network; An action selection module is configured to: for a current time step, after selecting an action through the main network and the greedy strategy under the state of the current time step, calculate the reward after the action is executed and the state of the next time step; A replay memory module is configured to: store the state, action, reward of the current time step and the state of the next time step as a transition in the replay memory, and when the number of transitions in the replay memory reaches a threshold, sample multiple transitions from the replay memory; A main network update module is configured to: for each sampled transfer, calculate the time difference target value of each branch through the target network and the Munchausen mechanism; calculate the predicted Q value of each branch through the main network; Based on the time difference target value and predicted Q value of each branch, the mean square error is calculated, and the parameters of the main network are updated based on the mean square error; The time difference target value is defined as: ,in, is the immediate reward calculated using the reward function (R), is a scaling factor that balances exploration and exploitation during learning, represents the Munchausen term obtained by the Munchausen mechanism, is the discount factor for future rewards, is the action in action branch b estimated by the target network The Q value, represents the state of the next time step, represents the action at the next time step, represents the target network parameters, ( s , a , r , s' ) indicates sampling transfer, s , a , r and s' Sample the current time step state, action, reward and next time step state in the transition respectively, A b Represents the action space of action branch b; A target network update module is configured to: assign the parameters of the main network to the target network at intervals of a certain number of time steps; The scheduling module is configured to: determine whether the end condition is met, if so, output the scheduling strategy and perform traffic scheduling; otherwise, enter the next time step and return to the main network and greedy strategy for action selection.

8. A computer-readable storage medium having a computer program stored thereon, the program being executed by a processor, characterized in that: When the program is executed by a processor, the steps in a time-sensitive network cross-domain traffic scheduling method as described in any one of claims 1-6 are implemented.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the time-sensitive network cross-domain traffic scheduling method as described in any one of claims 1-6 are implemented.

Images