Online incremental flow scheduling method, device and equipment based on time-sensitive test launch control network

Abstract

The invention discloses an online incremental flow scheduling method, device and equipment based on a time-sensitive test launch control network, which are suitable for solving the problem of mixed service flow resource allocation in a ground test launch control system network and improving the scheduling capability of the network on time-sensitive service flows and measurement service flows. The method comprises the following steps: carrying out service flow category division and attribute representation; network topology and resource detection; constructing a mixed service flow incremental transmission model; based on a mixed service flow incremental transmission model, establishing a common flow scheduling optimization target and constraint conditions of different types of service flows; configuring a Markov decision process according to the optimization target; executing an online incremental scheduling decision based on a traffic scheduling algorithm of deep reinforcement learning; according to the method and the device, the scheduling success rate of the periodic time-sensitive service flow and the measurement type service flow can be improved while the end-to-end transmission delay of the non-periodic burst time-sensitive service flow is optimized.

Description

Technical Field

[0001] This invention relates to the field of industrial network technology, and in particular to an online incremental traffic scheduling method, apparatus, and equipment based on a time-sensitive measurement and control network. Background Technology

[0002] Ensuring end-to-end deterministic low-latency and high-reliability transmission of differentiated time-sensitive and measurement-related service flows in ground-based telemetry, tracking, and command (TT&C) networks is crucial for the normal operation of the system. However, existing TT&C networks use traditional best-effort Ethernet mechanisms, which struggle to accurately guarantee the deterministic nature of network transmission metrics for differentiated service flows. Considering that Time-Sensitive Networking (TSN), proposed by the IEEE 802.1TSN working group, has the advantage of guaranteeing deterministic transmission of time-sensitive service flows while remaining compatible with traditional Ethernet protocols, combining TSN with TT&C networks to extend the deterministic guarantee capability of TT&C networks for service flows is a quick and efficient approach. Existing time-sensitive networks (TSNs) primarily reserve bandwidth resources to ensure deterministic transmission of time-sensitive service flows for which information is known in advance. However, most transmission and control commands in the measurement, launch, and control (MNC) network are non-periodic and bursty, making it difficult to reserve bandwidth resources in advance. Furthermore, periodic commands such as synchronization signals and measurement data in the MNC network need to be transmitted on the same network as non-periodic bursty service flows, resulting in significant differences in service flow characteristics. Therefore, it is difficult to directly use existing TSN mechanisms to guarantee the differentiated deterministic requirements of mixed service flows in the MNC network. The underlying queue shaping mechanism and service flow scheduling strategy of time-sensitive MNCs require further research and application. Summary of the Invention

[0003] The purpose of this invention is to provide an online incremental traffic scheduling method, apparatus, and device based on a time-sensitive measurement, control, and control network. By constructing incremental transmission models for different types of service flows, it supports differentiated deterministic forwarding of multiple flows online. Based on this, a traffic scheduling algorithm is designed to improve the resource adaptation efficiency between service flows and the time-sensitive measurement, control, and control network, thereby optimizing the scheduling success rate of service flows and end-to-end transmission latency.

[0004] This invention provides an online incremental traffic scheduling method based on a time-sensitive measurement and control network, comprising: Based on the characteristics of service flows in the time-sensitive measurement, control and control network, service flows are classified and their attribute information is characterized. The topology of the measurement, control, and transmission network is detected through the link layer discovery protocol, and the physical network is abstracted into related logical network resource representations. Based on different types of service flows, different priorities and queue reshaping mechanisms are configured to construct a hybrid service flow incremental transmission model. Based on the incremental transmission model of mixed service flows, common traffic scheduling optimization objectives and constraints for different types of service flows are established. Configure a Markov decision process according to the optimization objective; A traffic scheduling algorithm based on deep reinforcement learning executes online incremental scheduling decisions; Run the trained traffic scheduling algorithm to solve the scheduling mapping relationship between service flows and network resources, and obtain the final traffic scheduling result.

[0005] Preferably, the classification of service flows based on the service flow characteristics in the time-sensitive measurement and control network and the characterization of attribute information for each type of service flow include: The service flows in the measurement, control, and deployment network are divided into periodic time-sensitive service flows, non-periodic burst-type time-sensitive service flows, and measurement-related service flows; among them, The attribute information of the periodic time-sensitive service flow includes source address, destination address, data frame size, transmission period, latency requirements, jitter requirements, and reliability requirements. The attribute information of the non-periodic burst-type time-sensitive service flow includes source address, destination address, data frame size, latency requirements, jitter requirements, and reliability requirements. The attribute information of the measurement-type service flow includes source address, destination multicast group address, multicast path, data frame size, and worst-case latency requirement.

[0006] Preferably, the network resource representation includes: link path, link bandwidth, and time slice data capacity.

[0007] As a preferred approach, configuring different priorities and queue reshaping mechanisms according to different types of service flows, and constructing a hybrid service flow incremental transmission model includes: Periodic Time-Sensitive Service Flow Guarantee Mechanism: On the transmitting interface of the time-sensitive measurement and control network switch at each hop, a highest priority queue is allocated to periodic time-sensitive service flows, and the opening and closing times of queue gating are planned through a time-aware shaper, reserving dedicated time slices; when periodic services are transmitted within the predetermined time slice, the gating of other low-priority queues is closed to avoid time slice conflicts; non-periodic burst-type time-sensitive service flows are mapped to the same highest priority queue as periodic services, and a frame preemption mechanism is adopted; when concurrent with low-priority measurement service flows, the transmission of measurement services is interrupted to prioritize the burst-type services; if a periodic service of the same priority is being transmitted... If the traffic is interrupted, it will be queued and sent only after the interrupted traffic is completed. Emergency mechanisms for non-periodic burst traffic flows include: a bandwidth control mechanism for measurement services: measurement services are queuing and shaping based on credit values; credit values ​​are consumed when forwarding measurement services, and if the credit value is negative, queue gating is disabled until the credit value returns to non-negative; when a measurement service is preempted by a burst traffic flow, its credit value consumption is frozen until the burst traffic flow ends; a network-wide timing synchronization mechanism: a master-slave clock synchronization mechanism is used, with synchronization accuracy better than microseconds, and the number of network hops does not exceed the preset number of hops; each hop switch maintains a gating list based on the synchronization clock and periodically executes the opening and closing actions of queue gating to activate the traffic shaping strategy.

[0008] Preferably, the common traffic scheduling optimization objective and constraints for different types of service flows based on the incremental transmission model for mixed service flows include: The optimization objectives are to minimize the end-to-end latency of non-periodic burst-type time-sensitive service flows and maximize the scheduling success rate of periodic time-sensitive service flows and measurement-type service flows. The formula is as follows: , in, It represents the transmission delay between any two hops for non-periodic burst-type time-sensitive traffic flows, where Ni is the total number of time-sensitive measurement and control network switches along the route, and F... T It is a set of periodic time-sensitive service flows sent from the source end, F C It is a collection of measurement-type service streams sent from the source end. It is a set of periodic time-sensitive service flows that have been successfully scheduled. It is a collection of measurement-type service flows that have been successfully scheduled. These are the weighting factors for each sub-objective; The constraints include scheduling cycle constraints, time slice size constraints, service flow forwarding constraints, bandwidth capacity constraints, and service flow demand constraints.

[0009] Preferably, configuring the Markov decision process according to the optimization objective includes: The state space S contains the time slice occupancy information of each time-sensitive measurement and control network switch interface within a scheduling cycle, described by the formula: ,in Instruction No. Is the time slice of the slot on the time-sensitive control network switch interface occupied? It contains the attributes and requirements information of periodic, time-sensitive business flows. It contains the attribute and requirement information of non-periodic, burst-type, time-sensitive business flows. It measures the attributes and requirements of business processes. Action space A includes the transmission paths and transmission time slices for periodic time-sensitive service flows, the transmission paths for non-periodic burst-type service flows, and the transmission time slices for measurement-type service flows, described by the formula: ,in, and These represent the transmission path and transmission time slice for each periodic time-sensitive service stream. This represents the transmission path of each non-periodic burst-type service flow. This represents the sending time slice for each measurement-type service flow; The reward setting is derived by mapping two sub-optimization objectives, and the formula is described as follows: , Where r(t) is the total reward at step t, and r1(t) and r2(t) are the two parts of the reward obtained at step t, respectively. It is a weighting factor, n t It represents the number of switches along the path of the non-periodic burst-type time-sensitive traffic flow arriving at step t. It is a discount factor for transmission delay. It is the total number of periodic time-sensitive service flows and measurement-type service flows scheduled in step t. It is the discount factor for business flow scheduling.

[0010] As a preferred approach, traffic scheduling algorithms based on deep reinforcement learning perform online incremental scheduling decisions, including: Q-value-based training process: The evaluation neural network updates its model parameters by minimizing the loss value. The loss function is set as the mean squared error loss, and the formula is described as follows: , in, It is the sample size, Q W It is the decision evaluation neural network function, w is the parameter value of the decision evaluation neural network function, and y is the parameter value of the decision evaluation neural network function. t It is the target value based on the Q value, and the calculation formula is: , in, It is a discount factor. It is used to evaluate the target neural network function. For the parameters of the target neural network function, It is a decision-making objective neural network function. The parameters of the neural network function are the decision objective; The policy gradient of the decision-making evaluation neural network is designed and updated by combining the Q-value of the evaluation evaluation neural network. The formula is described as follows: , The parameters of the decision-making target neural network and the evaluation target neural network adopt a soft update mechanism, and the formula is described as follows: , , Among them, parameters The traffic scheduling algorithm contains four neural networks: decision evaluation neural network, decision target neural network, evaluation evaluation neural network, and evaluation target neural network. Each neural network consists of an input layer, an intermediate layer, and an output layer. The decision evaluation neural network is used to output corresponding actions according to different states. The number of neurons in its input layer is equal to the number of elements in the state space, the number of neurons in the intermediate layer is the same as that in the input layer, and the number of neurons in the output layer is equal to the number of elements in the action space.

[0011] The present invention also provides an online incremental traffic scheduling device based on a time-sensitive measurement and control network, comprising: The service flow classification and attribute characterization module is used to classify service flows based on the characteristics of service flows in the time-sensitive measurement, control and control network and to characterize the attribute information of each type of service flow. The network topology and resource detection module is used to detect the network topology of the measurement, control and transmission network through the link layer discovery protocol, and to abstract the physical network into related logical network resource representations. The mixed service flow incremental transmission model construction module is used to configure different priorities and queue shaping mechanisms according to different types of service flows to construct a mixed service flow incremental transmission model. The traffic scheduling constraint optimization problem construction module is used to establish common traffic scheduling optimization objectives and constraints for different types of service flows based on the incremental transmission model of mixed service flows. A Markov decision process design module is used to configure a Markov decision process according to the optimization objective. The traffic scheduling algorithm design module is used to execute online incremental scheduling decisions based on deep reinforcement learning-based traffic scheduling algorithms. The solution module is used to run the trained traffic scheduling algorithm, solve the scheduling mapping relationship between service flows and network resources, and obtain the final traffic scheduling result.

[0012] The present invention also provides an electronic device, comprising: The memory is used to store the processing program; The processor, when executing the processing program, implements the online incremental traffic scheduling method based on time-sensitive measurement and control network as described in the embodiments of the present invention.

[0013] The present invention also provides a readable storage medium, characterized in that the readable storage medium stores a processing program, which, when executed by a processor, implements the online incremental traffic scheduling method based on a time-sensitive measurement and control network as described in the embodiments of the present invention.

[0014] Compared with the prior art, the present invention has the following beneficial effects: This invention provides an online incremental traffic scheduling method based on a time-sensitive measurement and control network. First, it establishes a hybrid service flow incremental transmission model. By classifying service flows and establishing a time-slice avoidance mechanism, it ensures that time-sensitive service flows are allocated sufficient time-slice resources while preventing measurement-related service flows from starving. Considering the uncertain transmission times of aperiodic burst-type time-sensitive service flows, a traffic scheduling algorithm based on deep reinforcement learning is designed to learn the network distribution of aperiodic burst-type time-sensitive service flows over a period of time and plan path resources, thereby improving the scheduling performance of hybrid service flows. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the online incremental traffic scheduling method based on a time-sensitive measurement and control network in an embodiment of the present invention. Figure 2 This is a flowchart of an online incremental traffic scheduling method based on a time-sensitive measurement and control network according to another embodiment of the present invention; Figure 3 This is a structural block diagram of an online incremental traffic scheduling device based on a time-sensitive measurement and control network according to an embodiment of the present invention; Figure 4 This is a structural block diagram of an online incremental traffic scheduling device based on a time-sensitive measurement and control network according to another embodiment of the present invention; Figure 5 This is a structural block diagram of an online incremental traffic scheduling device based on a time-sensitive measurement and control network according to another embodiment of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] The term "comprising" and its variations as used herein are open-ended inclusion, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0018] Figure 1 This paper describes an application scenario of an embodiment of the present invention. In this scenario, tools such as a user registration interface and link probing protocols are used to uniformly collect characteristic information of different service flows and network resource status information in a time-sensitive measurement and control network. Joint scheduling of service flows and network resources sent by the front-end and back-end systems is then performed to meet the differentiated deterministic requirements of mixed service flow transmission over a shared network. The service flows include three types: periodic time-sensitive service flows, aperiodic burst-type time-sensitive service flows, and measurement service flows. The packet transmission period for periodic time-sensitive service flows is 400µs, 800µs, and 1000µs, with data frame sizes between 1KB and 1.5KB. The data frame size for aperiodic burst-type time-sensitive service flows is the same as that for periodic time-sensitive service flows. Measurement service flows occupy 70% of the bandwidth. The link bandwidth of the time-sensitive measurement and control network is set to 10Gbps. Both terminals and switches support time-sensitive queue shaping mechanisms. The service flow scheduling strategy is uniformly calculated by the controller and distributed to the network side. The traffic scheduling algorithm uses the Adam optimizer to train the neural network with a learning rate of 1×10⁻⁴, an experience pool size of 1024, and samples 32 trajectory samples per training iteration. A discount factor of 0.95 is used to obtain longer-term reward values. This embodiment does not impose specific limitations on the network topology or neural network structure of the algorithm in the online incremental traffic scheduling scenario of time-sensitive measurement and control networks.

[0019] To support differentiated deterministic scheduling of different service flows in a time-sensitive measurement, launch, and control network, this invention provides an online incremental traffic scheduling method based on a time-sensitive measurement, launch, and control network, including: S101: Based on the characteristics of service flows in the time-sensitive measurement, launch, and control network, service flows are categorized and their attribute information is characterized. Specifically, service flow categories are defined according to the characteristics of service flows in the measurement, launch, and control network. Specifically, control signaling in each subsystem of the measurement, launch, and control network requires ultra-low end-to-end transmission latency and ultra-high reliability, but the transmission time is uncertain; therefore, it is classified as a non-periodic burst-type time-sensitive service flow. Synchronization commands and clock synchronization signals transmitted before rocket launch are periodically transmitted and have deterministic end-to-end transmission latency and high reliability requirements; therefore, they are classified as periodic time-sensitive service flows. Multicast measurement data transmitted by the front-end system has high bandwidth utilization and is classified as measurement-type service flows.

[0020] Characterize service flow attribute information. Periodic time-sensitive service flow attribute information includes source address, destination address, data frame size, transmission period, latency requirements, jitter requirements, and reliability requirements, etc.; non-periodic burst-type time-sensitive service flows do not have a transmission period, so their attribute information includes source address, destination address, data frame size, latency requirements, jitter requirements, and reliability requirements, etc.; measurement-type service flow attributes include source address, destination multicast group address, multicast path, data frame size, worst-case latency requirements, etc. Thus, for periodic time-sensitive service flows (such as synchronization commands): by clearly defining their transmission period, latency / jitter requirements, and reliability requirements, dedicated time slices and priority queues can be pre-planned to ensure deterministic transmission (such as TDM-like mechanisms) and meet millisecond or even microsecond latency requirements; for non-periodic bursty service flows (such as control signaling): although they do not have a fixed period, by marking them with high reliability tags and real-time requirements, dynamic preemption mechanisms (such as frame preemption) are triggered to respond quickly in bursty scenarios and avoid task failure due to queuing delays; for measurement service flows (such as multicast data): by identifying their high bandwidth characteristics and lenient latency tolerance, credit value rate control or statistical multiplexing strategies are adopted to make full use of idle link resources, avoid crowding out critical service bandwidth, and achieve precise matching of service requirements and network capabilities.

[0021] S102: The topology of the measurement, transmission, and control network is detected through the link layer discovery protocol, and the physical network is abstracted into a related logical network resource representation. Specifically, the topology of the measurement, transmission, and control network is detected through the link layer discovery protocol, and the physical network is abstracted into a related logical network resource representation, including link paths, link bandwidth, time slice data capacity, etc. By proactively probing device connectivity in the network through link-layer discovery protocols (such as LLDP and BDPU), the system automatically generates real-time topology maps, replacing traditional manual drawing methods and significantly reducing maintenance workload. It dynamically tracks link status changes (such as adding / deleting links, node failures), triggering real-time updates of logical resources to ensure the network view always reflects the latest status. In test, development, and control scenarios, when devices frequently connect / disconnect or temporarily adjust cabling, the system can automatically identify new links and include them in the resource pool without manual intervention. By abstracting physical links into logical resources (such as "Link A → 10Gbps bandwidth" and "Time slice X → 2KB capacity"), the upper-layer scheduling algorithm does not need to focus on underlying hardware details but can concentrate on policy formulation. Through this "physical → logical" transformation, the test, development, and control network is transformed from a black box into a transparent and controllable resource pool, providing a solid network foundation for complex business scenarios requiring high reliability and low latency.

[0022] S103: Configure different priorities and queue shaping mechanisms according to different types of service flows to construct a hybrid service flow incremental transmission model; specifically, for periodic time-sensitive service flows in the measurement, transmission, and control network, use a time-aware shaper to shape queue traffic. The construction of the hybrid service flow incremental transmission model includes: a periodic time-sensitive service flow guarantee mechanism: on the sending interface of each hop of the time-sensitive measurement, transmission, and control network switch, allocate the highest priority queue for periodic time-sensitive service flows, and plan the opening and closing times of queue gating through a time-aware shaper, reserving dedicated time slices; when periodic services are transmitted in the predetermined time slice, close the gating of other low-priority queues to avoid time slice conflicts; map non-periodic burst-type time-sensitive service flows to the same highest priority queue as periodic services, using a frame preemption mechanism; when concurrent with low-priority measurement service flows, interrupt the transmission of measurement services to prioritize burst-type services; if encountering the same priority... If periodic services are in transit, they are queued and sent only after completion. Emergency mechanisms for non-periodic burst traffic include: a bandwidth control mechanism for measurement services: measurement services are queuing and shaping based on credit values; credit values ​​are consumed when forwarding measurement services, and if the credit value is negative, queue gating is disabled until the credit value returns to non-negative; when a measurement service is preempted by a burst traffic service, its credit value consumption is frozen until the burst traffic service ends; a network-wide timing synchronization mechanism: a master-slave clock synchronization mechanism is used, with synchronization accuracy better than microseconds, and the number of network hops does not exceed a preset number of hops; each hop switch maintains a gating list based on the synchronization clock and periodically executes queue gating on / off actions to activate the traffic shaping strategy.The above scheme reserves dedicated time slices for periodic services through a time-aware shaper, combined with a gating mechanism, to completely eliminate interference from low-priority services on periodic services, ensuring the stability of end-to-end latency (such as synchronization commands, clock signals, etc.). Time slices are planned based on the greatest common divisor of packet transmission cycles, effectively avoiding resource contention between different periodic service flows and improving the concurrent processing capability of multi-period services. The design of a highest-priority queue + dedicated time slice enables periodic services to achieve near-dedicated line transmission quality, meeting the stringent requirements of ultra-low jitter and zero packet loss in measurement, transmission, and control scenarios. Bursting services are included in the highest-priority queue, allowing them to interrupt low-priority measurement services through frame preemption. It achieves millisecond-level emergency response (such as equipment alarms and emergency control commands); if sudden and periodic services arrive simultaneously, it proactively queues and waits for the periodic service to complete before transmitting, ensuring the priority of sudden services while avoiding disruption of the determinism of periodic services; through the design of a shared high-priority queue, it provides the ability to "preempt on demand" for sudden services, adapting to unpredictable sudden traffic surges; based on a credit value-based queue shaping mechanism, it limits the bandwidth usage of measurement-type services (such as images and big data) within a controllable range, preventing them from excessively squeezing time-sensitive service resources; when measurement-type services are preempted by sudden services, the credit value is suspended, avoiding invalid transmissions caused by frequent interruptions and improving bandwidth utilization.

[0023] Those skilled in the art will understand that, on the transmitting interface of each time-sensitive measurement and control network switch, periodic time-sensitive service flows are forwarded using the highest priority queue. By planning the opening and closing times of queue gating, time slices are reserved for periodic time-sensitive service flows. When the planned periodic time-sensitive service flow is sent through the switch interface in the corresponding time slice, the gating of other low-priority queues is closed to avoid time slice conflicts between low-priority service flows and periodic time-sensitive service flows. The combination of master-slave clock synchronization mechanism (accuracy <1μs) and gating list ensures strict synchronization of time slice allocation and queue actions across switches, eliminating the problem of clock deviation accumulation in multi-hop networks.

[0024] Non-periodic burst time-sensitive traffic flows have the same priority as periodic time-sensitive traffic flows. Therefore, on each hop of the time-sensitive measurement and control network switch, non-periodic burst time-sensitive traffic flows and periodic time-sensitive traffic flows use queues with the same priority for data forwarding. Since the transmission time of non-periodic burst traffic flows is uncertain, it is difficult to guarantee deterministic transmission by reserving time slices in advance. Therefore, a frame preemption mechanism is used. When a low-priority measurement traffic flow is transmitted simultaneously with a non-periodic burst time-sensitive traffic flow, the preemption mechanism will interrupt the transmission of the measurement traffic flow, prioritizing the non-periodic burst time-sensitive traffic flow to preempt a time slice. If periodic time-sensitive traffic flows of the same priority are transmitted simultaneously, it is necessary to wait for the periodic time-sensitive traffic flow to complete its transmission before continuing to send non-periodic burst time-sensitive traffic flows.

[0025] Measurement-type service flows have a lower priority than time-sensitive service flows, therefore a low-priority queue is used for data forwarding. Since image data and other data consume significant bandwidth during network transmission, a credit-based queue traffic shaping method is used. Initially, the credit value is set to 0. When a measurement-type service flow is forwarded, it consumes credit value. After forwarding, if the credit value is negative, the queue gating closes, preventing the forwarding of the next measurement-type service flow. Credit value needs to accumulate. When the accumulated credit value reaches a non-negative value, the queue gating reopens, allowing the continued transmission of measurement-type service flows. When a frame of a measurement-type service flow is preempted by an aperiodic burst service flow during transmission, its credit value is frozen. Once the aperiodic burst service flow has finished transmitting, the credit value resumes consumption at the current position.

[0026] The entire network clock is configured with a master-slave synchronization mechanism, with synchronization accuracy set below microseconds. The number of hops for the time-sensitive measurement and control network switches used for synchronization does not exceed 8. Each time-sensitive measurement and control network switch maintains its own gating list based on the clock. Queue gating performs switching actions periodically according to the planned gating list to ensure that the configured queue traffic shaping mechanism is effective.

[0027] S104: Based on the incremental transmission model for mixed service flows, establish common traffic scheduling optimization objectives and constraints for different types of service flows. Specifically, based on the principle of the incremental transmission model for mixed service flows, establish optimization objectives and constraints for the co-network scheduling of periodic time-sensitive service flows, aperiodic burst-type time-sensitive service flows, and measurement service flows in the measurement, transmission, and control network. Specifically, the optimization objectives include minimizing the end-to-end latency of aperiodic burst-type time-sensitive service flows and maximizing the scheduling success rate of periodic time-sensitive service flows and measurement service flows, described by the formula: , in, is the transmission delay of non-periodic burst-type time-sensitive traffic flows between any two hops, Ni is the total number of time-sensitive measurement and control network switches along the route, FT is the set of periodic time-sensitive traffic flows sent from the source, and FC is the set of measurement traffic flows sent from the source. It is a set of periodic time-sensitive service flows that have been successfully scheduled. It is a collection of measurement-type service flows that have been successfully scheduled. These are the weighting factors for each sub-objective; The constraints include scheduling cycle constraints, time slice size constraints, traffic flow forwarding constraints, bandwidth capacity constraints, and traffic flow demand constraints. Specifically, the scheduling cycle constraint requires that the traffic scheduling cycle of the time-sensitive measurement and control network be equal to the least common multiple of the cycle of periodic time-sensitive traffic flows, as described by the formula: , Where cycle is the size of the scheduling period. Indicates the cycle of the business flow. It is a function for calculating the least common multiple.

[0028] The time slice size constraint requires that the time slice size for periodic time-sensitive service flows cannot exceed the greatest common divisor of the service flow cycles to avoid interference between service flows of different cycles. The formula is described as follows: , Where slot represents the time slice size. It is a function that calculates the greatest common divisor.

[0029] Service flow forwarding constraints require that different periodic time-sensitive service flows, as well as periodic time-sensitive service flows and measurement-type service flows, cannot experience forwarding conflicts on the same time-sensitive measurement and control network switch interface. Therefore, the arrival time of the next periodic time-sensitive service flow at the interface cannot be less than the departure time of the previous service flow from that interface. The formula is described as follows: , in, Indicates periodic time-sensitive business flows The time when the k-th data frame is sent from the source node Represents a data frame The number of hops from the source node to the s-th node It is in the and the Transmission delay between nodes Represents a data frame In the s The transmission latency on each node interface, Data frames representing measurement-type service flows.

[0030] Bandwidth capacity constraints require that the size of periodic time-sensitive service stream data frames transmitted within the corresponding time slice does not exceed the maximum data capacity that the link bandwidth can provide, as described by the formula: , Business flow requirements dictate that time-sensitive business flows have bounded end-to-end latency, jitter, and zero packet loss rate, as described by the formula: , in, and These represent the maximum and minimum end-to-end latency for time-sensitive traffic flows, respectively. , These represent the end-to-end transmission latency and jitter requirements for time-sensitive service flows, respectively.

[0031] The above scheme achieves differentiated Quality of Service (QoS) guarantees for multiple service flows: Periodic time-sensitive service flows (such as synchronization instructions): by maximizing the scheduling success rate target, it ensures that their data frames exclusively occupy the high-priority queue within the predetermined time slice, meeting the requirements of deterministic latency and zero packet loss rate; Non-periodic bursty service flows (such as control signaling): by minimizing the end-to-end latency target, combined with the frame preemption mechanism, it enables them to quickly traverse the network in emergency situations, avoiding task failure due to queuing delays; Measurement service flows (such as multicast data): by sharing the scheduling success rate target, it utilizes remaining resources for efficient transmission without affecting time-sensitive services, balancing bandwidth usage and data integrity; breaking the limitations of single-indicator optimization, it achieves "on-demand allocation" - ensuring determinism for critical services, ensuring response speed for bursty services, and ensuring throughput for background services. Enhanced efficient utilization and conflict avoidance of network resources: Scheduling cycle constraints (equal to the least common multiple of the cycle of periodic services): Aligning service flows of different cycles to a unified time frame, eliminating cross-cycle resource contention, and improving time slice reuse efficiency; Time slice size constraints (not exceeding the greatest common divisor of the cycles): Preventing the time slices of large-cycle services from excessively squeezing small-cycle services, avoiding resource fragmentation; Service flow forwarding constraints (arrival time of the next service ≥ departure time of the previous service): Through strict timing constraints, preventing forwarding conflicts on the same interface, ensuring pipelined processing efficiency; By combining mathematical modeling with engineering practice, the traffic scheduling of the measurement, transmission, and control network is transformed from experience-driven to data-driven, providing a verifiable and scalable solution for complex service scenarios with high reliability and low latency.

[0032] S105: Configure a Markov decision process according to the optimization objective; specifically, design the state space, action space, and reward of the Markov decision process based on the traffic scheduling constraint optimization problem. Specifically, the state space S contains, on the one hand, the occupancy status of time slices on the interfaces of each time-sensitive measurement and control network switch within a scheduling cycle; if a time slice is occupied, it is marked as 1, otherwise as 0; on the other hand, it contains the attributes and demand information of each service flow, described by the formula: The state space S contains the occupancy status of time slices on the interfaces of each time-sensitive measurement and control network switch within a scheduling cycle, described by the formula: , in Instruction No. Is the time slice of the slot on the time-sensitive control network switch interface occupied? It contains the attributes and requirements information of periodic, time-sensitive business flows. It contains the attribute and requirement information of non-periodic, burst-type, time-sensitive business flows. It measures the attributes and requirements of business processes. Action space A includes the transmission paths and transmission time slices for periodic time-sensitive service flows, the transmission paths for non-periodic burst-type service flows, and the transmission time slices for measurement-type service flows, described by the formula: , in, and These represent the transmission path and transmission time slice for each periodic time-sensitive service stream. This represents the transmission path of each non-periodic burst-type service flow. This represents the sending time slice for each measurement-type service flow; The reward setting is derived from two sub-optimization objectives. The first reward, r1, is set as the difference in end-to-end transmission latency of non-periodic burst-type time-sensitive traffic flows between two consecutive training steps. The second reward, r2, is set as the difference in the total number of periodic time-sensitive traffic flows and measurement traffic flows scheduled between two consecutive training steps. The formula is described as follows: , Where r(t) is the total reward at step t, and r1(t) and r2(t) are the two parts of the reward obtained at step t, respectively. It is a weighting factor, n t It represents the number of switches along the path of the non-periodic burst-type time-sensitive traffic flow arriving at step t. It is a discount factor for transmission delay. It is the total number of periodic time-sensitive service flows and measurement-type service flows scheduled in step t. It is a discount factor for traffic flow scheduling. This solution transforms the hybrid traffic flow scheduling problem into a learnable reinforcement learning framework through the refined design of Markov Decision Process (MDP). That is, by transforming complex network constraints into the state-action space boundary of MDP and encoding business requirements into reward signals, the traffic scheduling problem of measurement, transmission and control network is successfully transformed into a learnable and scalable reinforcement learning problem. This provides a solution with both theoretical rigor and engineering practicality for complex industrial scenarios with high reliability and low latency.

[0033] S106: A traffic scheduling algorithm based on deep reinforcement learning executes online incremental scheduling decisions; specifically, the neural network structure of the deep reinforcement learning algorithm is designed. The algorithm includes four neural networks: a decision evaluation neural network, a decision target neural network, a judgment evaluation neural network, and a judgment target neural network. Each neural network consists of an input layer, an intermediate layer, and an output layer. The decision evaluation neural network outputs corresponding actions based on different states. The number of neurons in its input layer is equal to the number of elements in the state space, the number of neurons in the intermediate layer is the same as that in the input layer, and the number of neurons in the output layer is equal to the number of elements in the action space. In particular, the decision evaluation neural network divides the neurons in the output layer into four action groups: the first group of neurons represents the transmission path of periodic time-sensitive traffic flows, the second group of neurons represents the transmission time slice of periodic time-sensitive traffic flows, the third group of neurons represents the transmission path of non-periodic burst traffic flows, and the fourth group of neurons represents the transmission time slice of measurement traffic flows. The decision target neural network is used to stabilize the training process, and its neural network structure is the same as that of the decision evaluation neural network. The evaluation neural network is used to assess the quality of state-action pairs. Its input layer neurons equal the total number of elements in the state and action spaces, the intermediate layer neurons equal the number of input layer neurons, and the output layer neurons are set to 1. The evaluation target neural network is used to stabilize the training process, and its neural network structure is the same as that of the evaluation evaluation neural network. The action space is decomposed into four independent outputs: periodic business paths / time slices, non-periodic business paths, and measurement-type time slices. This allows the model to specifically optimize scheduling strategies for different businesses, breaking through the limitations of a single action space and achieving refined control of mixed business flows. The multi-level neural network architecture improves decision-making accuracy and training stability, solving the problems of large policy fluctuations and slow convergence in traditional DRLs, making it particularly suitable for the high reliability requirements of testing, development, and control scenarios.

[0034] Initialize the time-sensitive measurement and control network environment and algorithm state. At the start of algorithm iteration, it is necessary to initialize the occupancy rate of each switch interface in the time-sensitive measurement and control network to 0 in each time slice, indicating that it is not occupied by traffic. At the same time, initialize the parameters of each neural network in the algorithm, including the connection weights between neurons, training period, experience pool size, learning rate, discount factor, etc.

[0035] Design the state transition process for traffic scheduling. During the iteration process, the algorithm needs to perform state transitions to collect historical experience. First, the agent observes the current state... This refers to the occupancy status of each time slice on the interface of each time-sensitive network switch. The decision-making and estimation neural network outputs corresponding mixed service flow scheduling actions based on the current strategy. Rewards r from environmental feedback t The state transitions to The intelligent agent judges s t+1 If the constraints are not met, the stop flag `done=1` is set, and the current iteration ends. If all business flows have been incrementally scheduled, `done=1` is set, and the current iteration ends. If the constraints are met but not all business flows have been scheduled, `done=0` is set, and the current iteration continues. During the iteration process, the agent records the state transition trajectory obtained at each step. The samples are stored in the experience pool as trajectory samples for training the neural network model.

[0036] The design employs a Q-value-based training process. After each iteration, the agent trains the neural network model based on historical trajectories. Specifically, the evaluation neural network updates its model parameters by minimizing the loss value. The loss function is set as the mean squared error loss, described by the formula: , in, It is the sample size, Q W It is the decision evaluation neural network function, w is the parameter value of the decision evaluation neural network function, and y is the parameter value of the decision evaluation neural network function. t It is the target value based on the Q value, and the calculation formula is: , in, It is a discount factor. It is used to evaluate the target neural network function. For the parameters of the target neural network function, It is a decision-making objective neural network function. The parameters of the neural network function are the decision objective; The policy gradient of the decision-making evaluation neural network is designed and updated by combining the Q-value of the evaluation evaluation neural network. The formula is described as follows: , The parameters of the decision-making target neural network and the evaluation target neural network adopt a soft update mechanism, and the formula is described as follows: , , Among them, parameters This solution provides an efficient, reliable, and scalable traffic scheduling solution for time-sensitive telemetry, measurement, and control networks through the adaptive capabilities of deep reinforcement learning, the stability optimization of neural network architecture, the native embedding of constraints, and the engineering-based modular design. It is especially suitable for complex scenarios with extremely high requirements for determinism and real-time performance, such as aerospace and industrial automation.

[0037] S107: Run the trained traffic scheduling algorithm to solve the scheduling mapping relationship between service flows and network resources, and obtain the final traffic scheduling result.

[0038] Example 2 Based on the same concept, such as Figure 3 As shown, the present invention provides an online incremental traffic scheduling device / system based on a time-sensitive measurement and control network, comprising: The service flow classification and attribute characterization module is used to classify service flows based on the characteristics of service flows in the time-sensitive measurement, control and control network and to characterize the attribute information of each type of service flow. The network topology and resource detection module is used to detect the network topology of the measurement, control and transmission network through the link layer discovery protocol, and to abstract the physical network into related logical network resource representations. The mixed service flow incremental transmission model construction module is used to configure different priorities and queue shaping mechanisms according to different types of service flows to construct a mixed service flow incremental transmission model. The traffic scheduling constraint optimization problem construction module is used to establish common traffic scheduling optimization objectives and constraints for different types of service flows based on the incremental transmission model of mixed service flows. A Markov decision process design module is used to configure a Markov decision process according to the optimization objective. The traffic scheduling algorithm design module is used to execute online incremental scheduling decisions based on deep reinforcement learning-based traffic scheduling algorithms. The solution module is used to run the trained traffic scheduling algorithm, solve the scheduling mapping relationship between service flow and network resources, and obtain the final traffic scheduling result. The implementation principle of the above module has been described in the previous embodiments, so it will not be repeated here.

[0039] Example 3 like Figure 4-5 As shown, based on the same concept, an electronic device is also provided in some embodiments of this application. This electronic device includes a memory and a processor, wherein the memory stores a processing program, and the processor executes the processing program according to instructions. When the processor executes the processing program, the online incremental traffic scheduling method based on a time-sensitive measurement and control network described in the foregoing embodiments is realized.

[0040] In some embodiments of this application, a readable storage medium is also provided. This readable storage medium can be a non-volatile readable storage medium or a volatile readable storage medium. The readable storage medium stores instructions that, when executed on a computer, cause an electronic device containing this readable storage medium to perform the aforementioned online incremental traffic scheduling method based on a time-sensitive telemetry and control network.

[0041] It is understood that, for the aforementioned online incremental traffic scheduling method based on time-sensitive measurement and control networks, if all of them are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0042] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0043] The program code for executing the technical solutions disclosed in this application can be written in any combination of one or more programming languages. These programming languages ​​include object-oriented programming languages—such as Java and C++—and conventional procedural programming languages—such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0044] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An online incremental traffic scheduling method based on a time-sensitive measurement and control network, characterized in that, include: Based on the characteristics of service flows in the time-sensitive measurement, control and control network, service flows are classified and their attribute information is characterized. The topology of the measurement, control, and transmission network is detected through the link layer discovery protocol, and the physical network is abstracted into related logical network resource representations. Based on different types of service flows, different priorities and queue reshaping mechanisms are configured to construct a hybrid service flow incremental transmission model. Based on the incremental transmission model of mixed service flows, common traffic scheduling optimization objectives and constraints for different types of service flows are established. Configure a Markov decision process according to the optimization objective; A traffic scheduling algorithm based on deep reinforcement learning executes online incremental scheduling decisions; Run the trained traffic scheduling algorithm to solve the scheduling mapping relationship between service flows and network resources, and obtain the final traffic scheduling result.

2. The online incremental traffic scheduling method based on a time-sensitive measurement and control network according to claim 1, characterized in that, The classification of service flows based on the characteristics of service flows in the time-sensitive measurement, control, and deployment network, and the characterization of attribute information for each type of service flow, include: The service flows in the measurement, control, and deployment network are divided into periodic time-sensitive service flows, non-periodic burst-type time-sensitive service flows, and measurement-related service flows; among them, The attribute information of the periodic time-sensitive service flow includes source address, destination address, data frame size, transmission period, latency requirements, jitter requirements, and reliability requirements. The attribute information of the non-periodic burst-type time-sensitive service flow includes source address, destination address, data frame size, latency requirements, jitter requirements, and reliability requirements. The attribute information of the measurement-type service flow includes source address, destination multicast group address, multicast path, data frame size, and worst-case latency requirement.

3. The online incremental traffic scheduling method based on a time-sensitive measurement and control network according to claim 1, characterized in that, The network resource representation includes: link path, link bandwidth, and time slice data capacity.

4. The online incremental traffic scheduling method based on a time-sensitive measurement and control network according to claim 1, characterized in that, Based on different priorities and queue reshaping mechanisms for different types of service flows, a hybrid service flow incremental transmission model is constructed, including: Periodic Time-Sensitive Service Flow Guarantee Mechanism: On the transmitting interface of the time-sensitive measurement and control network switch at each hop, a highest priority queue is allocated to periodic time-sensitive service flows, and the opening and closing times of queue gating are planned through a time-aware shaper, reserving dedicated time slices; when periodic services are transmitted within a predetermined time slice, the gating of other low-priority queues is closed to avoid time slice conflicts; non-periodic burst-type time-sensitive service flows are mapped to the same highest priority queue as periodic services, and a frame preemption mechanism is adopted; when concurrent with low-priority measurement service flows, the transmission of measurement services is interrupted to prioritize the burst-type services; if a periodic service of the same priority is being transmitted, it is queued and sent only after it is completed; Non-periodic Burst-Type Service Flow Emergency Mechanism: Measurement Service Bandwidth Management Mechanism: Queue shaping based on credit values ​​is adopted for measurement services; credit values ​​are consumed when forwarding measurement services, and if the credit value is negative, the queue gating is closed until the credit value recovers to non-negative; when a measurement service is preempted by a burst-type service, its credit value consumption is frozen, and consumption continues after the burst-type service ends. Network-wide timing synchronization mechanism: A master-slave clock synchronization mechanism is adopted, with synchronization accuracy better than microsecond level, and the number of network hops does not exceed the preset number of hops; each hop switch maintains a gating list based on the synchronization clock and periodically executes the opening and closing actions of queue gating to implement traffic shaping strategies.

5. The online incremental traffic scheduling method based on a time-sensitive measurement and control network according to claim 1, characterized in that, The common traffic scheduling optimization objectives and constraints for different types of service flows established based on the incremental transmission model for hybrid service flows include: The optimization objectives are to minimize the end-to-end latency of non-periodic burst-type time-sensitive service flows and maximize the scheduling success rate of periodic time-sensitive service flows and measurement-type service flows. The formula is as follows: ， in, It represents the transmission delay between any two hops for non-periodic burst-type time-sensitive traffic flows, where Ni is the total number of time-sensitive measurement and control network switches along the route, and F... T It is a set of periodic time-sensitive service flows sent from the source end, F C It is a collection of measurement-type service streams sent from the source end. It is a set of periodic time-sensitive service flows that have been successfully scheduled. It is a collection of measurement-type service flows that have been successfully scheduled. These are the weighting factors for each sub-objective; The constraints include scheduling cycle constraints, time slice size constraints, service flow forwarding constraints, bandwidth capacity constraints, and service flow demand constraints.

6. The online incremental traffic scheduling method based on a time-sensitive measurement and control network according to claim 1, characterized in that, The configuration of the Markov decision process according to the optimization objective includes: The state space S contains the time slice occupancy information of each time-sensitive measurement and control network switch interface within a scheduling cycle, described by the formula: ,in Instruction No. Is the time slice of the slot on the time-sensitive control network switch interface occupied? It contains the attributes and requirements information of periodic, time-sensitive business flows. It contains the attribute and requirement information of non-periodic, burst-type, time-sensitive business flows. It measures the attributes and requirements of business processes. Action space A includes the transmission paths and transmission time slices for periodic time-sensitive service flows, the transmission paths for non-periodic burst-type service flows, and the transmission time slices for measurement-type service flows, described by the formula: ,in, and These represent the transmission path and transmission time slice for each periodic time-sensitive service stream. This represents the transmission path of each non-periodic burst-type service flow. This represents the sending time slice for each measurement-type service flow; The reward setting is derived by mapping two sub-optimization objectives, and the formula is described as follows: , Where r(t) is the total reward at step t, and r1(t) and r2(t) are the two parts of the reward obtained at step t, respectively. It is a weighting factor, n t It represents the number of switches along the path of the non-periodic burst-type time-sensitive traffic flow arriving at step t. It is a discount factor for transmission delay. It is the total number of periodic time-sensitive service flows and measurement-type service flows scheduled in step t. It is the discount factor for business flow scheduling.

7. The online incremental traffic scheduling method based on a time-sensitive measurement and control network according to claim 1, characterized in that, The traffic scheduling algorithm based on deep reinforcement learning performs online incremental scheduling decisions, including: Q-value-based training process: The evaluation neural network updates its model parameters by minimizing the loss value. The loss function is set as the mean squared error loss, and the formula is described as follows: , in, It is the sample size, Q W It is the decision evaluation neural network function, w is the parameter value of the decision evaluation neural network function, and y is the parameter value of the decision evaluation neural network function. t It is the target value based on the Q value, and the calculation formula is: ， in, It is a discount factor. It is used to evaluate the target neural network function. For the parameters of the target neural network function, It is a decision-making objective neural network function. The parameters of the neural network function are the decision objective; The policy gradient of the decision-making evaluation neural network is designed and updated by combining the Q-value of the evaluation evaluation neural network. The formula is described as follows: ， The parameters of the decision-making target neural network and the evaluation target neural network adopt a soft update mechanism, and the formula is described as follows: ， ， Among them, parameters The traffic scheduling algorithm contains four neural networks: decision evaluation neural network, decision target neural network, evaluation evaluation neural network, and evaluation target neural network. Each neural network consists of an input layer, an intermediate layer, and an output layer. The decision evaluation neural network is used to output corresponding actions according to different states. The number of neurons in its input layer is equal to the number of elements in the state space, the number of neurons in the intermediate layer is the same as that in the input layer, and the number of neurons in the output layer is equal to the number of elements in the action space.

8. An online incremental traffic scheduling device based on a time-sensitive measurement and control network, characterized in that, include: The service flow classification and attribute characterization module is used to classify service flows based on the characteristics of service flows in the time-sensitive measurement, control and control network and to characterize the attribute information of each type of service flow. The network topology and resource detection module is used to detect the network topology of the measurement, control and transmission network through the link layer discovery protocol, and to abstract the physical network into related logical network resource representations. The mixed service flow incremental transmission model construction module is used to configure different priorities and queue shaping mechanisms according to different types of service flows to construct a mixed service flow incremental transmission model. The traffic scheduling constraint optimization problem construction module is used to establish common traffic scheduling optimization objectives and constraints for different types of service flows based on the incremental transmission model of mixed service flows. A Markov decision process design module is used to configure a Markov decision process according to the optimization objective. The traffic scheduling algorithm design module is used to execute online incremental scheduling decisions based on deep reinforcement learning-based traffic scheduling algorithms. The solution module is used to run the trained traffic scheduling algorithm, solve the scheduling mapping relationship between service flows and network resources, and obtain the final traffic scheduling result.

9. An electronic device, characterized in that, include: The memory is used to store the processing program; A processor, which, when executing the processing program, implements the online incremental traffic scheduling method based on a time-sensitive measurement and control network as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that, The readable storage medium stores a processing program, which, when executed by a processor, implements the online incremental traffic scheduling method based on a time-sensitive measurement and control network as described in any one of claims 1 to 7.

Images