A multi-agent cooperative scheduling method for incremental mixed flow in an industrial time-sensitive network

Abstract

The application discloses a multi-agent cooperative scheduling method for an incremental mixed flow in an industrial time-sensitive network, which comprises the following steps: analyzing the transmission characteristics of AVB flows and TT flows in the industrial time-sensitive network, and based on the network calculus theory, fine modeling of dynamic interference caused by the incremental mixed flow; in combination with the dynamic interference factors introduced by the incremental mixed flow, key transmission management variables of the AVB flows and the TT flows are determined; a flow scheduling model based on multi-agent cooperation is designed, and when the incremental mixed flow is accessed, an end-to-end transmission scheme is generated based on multi-agent cooperative decision-making; the trained agent is encapsulated into a multi-agent cluster and deployed into the industrial time-sensitive network, and is used for generating a transmission scheme for the incremental mixed flow online. The method can guarantee the delay constraint of the existing AVB flow, meet the transmission requirements of the incremental mixed flow, optimize the transmission scheme to improve the service quality of the industrial time-sensitive network, and fully support the stable operation of advanced applications in the industrial Internet of Things.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of industrial Internet of Things, and relates to a multi-agent cooperative scheduling method for incremental mixed flow in an industrial time-sensitive network. BACKGROUND

[0002] In recent years, Time-Sensitive Networking (TSN) focusing on low latency and high reliability communication has been extended to the wireless domain, building an industrial Internet of Things system with flexible access capability and high reliability. In the industrial Internet of Things enabled by TSN, various advanced industrial applications such as robot collaboration, remote control, and intelligent human-computer interaction are booming, relying on the deterministic transmission service provided by TSN. These advanced industrial applications are often multifunctional, and the various differentiated industrial data required by them are transmitted through TSN traffic of different priorities, resulting in a large number of mixed flows in the network. In addition, the industrial Internet of Things supports the dynamic deployment of customized applications in the industrial field without interrupting the operation to extend the industrial functions in real time, which means that the mixed flow in the industrial Internet of Things enabled by TSN will continue to increase. However, in the TSN standard, the priority of the Audio-Video Bridging (AVB) flow is lower than that of the Time-Triggered (TT) flow, and the available bandwidth of the AVB flow will be squeezed due to the increase of the mixed flow, thereby increasing the end-to-end latency. To ensure the stable operation of industrial applications in the network, the increase of mixed flow should not cause the existing AVB flow in the network to fail to meet the latency constraint; on the other hand, whether the incremental flow can be successfully transmitted also depends on whether the transmission scheme provided for it meets its own transmission requirements. Therefore, there is an urgent need for a mixed flow transmission management strategy that takes into account the above two conditions to optimize the transmission scheme of the incremental mixed flow. SUMMARY

[0003] To solve the above technical problems, the purpose of the present application is to provide a multi-agent cooperative scheduling method for incremental mixed flow in an industrial time-sensitive network, which first provides perfect modeling of the dynamic interference introduced by the increase of mixed flow, and then designs a traffic scheduling mechanism based on multi-agent cooperation, so as to improve the guarantee capability of the industrial time-sensitive network for the transmission of incremental mixed flow and meet the data transmission requirements of heterogeneous industrial applications.

[0004] The present application provides a multi-agent cooperative scheduling method for incremental mixed flow in an industrial time-sensitive network, comprising:

[0005] Step 1: analyze the transmission characteristics of AVB flow and TT flow in the industrial time-sensitive network, and based on the network calculus theory, combine the bandwidth competition and latency fluctuation under the incremental mixed flow scenario to finely model the dynamic interference caused by the increase of mixed flow;

[0006] Step 2: Combine the dynamic interference factors introduced by the incremental mixed flow, and determine the key transmission management variables of the AVB flow and the TT flow, covering transmission route determination and scheduling time slot division;

[0007] Step 3: Design a traffic scheduling model based on multi-agent collaboration, and generate an end-to-end transmission scheme that meets the delay and bandwidth constraints based on multi-agent collaborative decision-making when the incremental mixed flow is accessed;

[0008] Step 4: Encapsulate the trained agent as a multi-agent cluster and deploy it to the industrial time-sensitive network for online generation of transmission schemes for incremental mixed flows.

[0009] The multi-agent collaborative scheduling method for incremental mixed flows in an industrial time-sensitive network of the present application first analyzes the transmission mode of mixed traffic in the industrial time-sensitive network, and on this basis, models the dynamic interference caused by the incremental mixed transmission and determines the mixed flow transmission management variables; then designs a traffic scheduling method based on multi-agent collaboration, and configures a traffic adaptive neural network for the agent. This method can improve the transmission guarantee capability of the industrial time-sensitive network for incremental mixed flows, meet the data transmission needs of heterogeneous industrial applications, and achieve efficient scheduling and management of incremental mixed flows. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 is a flowchart of a multi-agent collaborative scheduling method for incremental mixed flows in an industrial time-sensitive network of the present application. DETAILED DESCRIPTION

[0011] As Figure 1 shown, the multi-agent collaborative scheduling method for incremental mixed flows in an industrial time-sensitive network of the present application comprises:

[0012] Step 1: Analyze the transmission characteristics of AVB flows and TT flows in the industrial time-sensitive network, including the priority mechanism and bandwidth occupation mode of the two; based on network calculus theory, combined with bandwidth competition and delay fluctuation in the incremental mixed flow scenario, model the dynamic interference caused by the incremental mixed flow in detail.

[0013] In the industrial time-sensitive network, the determinism of the TT flow hop-by-hop transmission is based on the reservation of dedicated time slots for it on each link, while the AVB flow is provided with a lower level of determinism due to its lower priority and is not allocated a time slot. This means that once the TT flow is allocated a time slot, it will not be disturbed by the bandwidth occupation that may be caused by subsequent incremental mixed flows. However, this is not the case for AVB. Whether it is an incremental TT flow or an incremental AVB flow, it will compress the available bandwidth of the existing AVB flow, thereby affecting its end-to-end delay and transmission. This phenomenon is a direct manifestation of the dynamic interference introduced by the incremental mixed flow.

[0014] The topology of a TSN network is denoted as is a set of network nodes, is a set of directed links:

[0015]

[0016] wherein, denote the start node, end node and bandwidth of link is a set of ports in the network, denotes the source port corresponding to link For any port , the links with this port as source are denoted as

[0017] In the advanced intelligent manufacturing scenario, the on-demand deployment of industrial applications will incrementally introduce mixed flows containing AVB flows and TT flows into the TSN network, causing denotes the index sequence of the incremental mixed flows, and the th incremental mixed flow is denoted as , which is defined as follows:

[0018]

[0019] wherein, denotes the start point of the incremental mixed flow, denotes the end point of the incremental mixed flow, denotes the data size of the incremental mixed flow, denotes the transmission period of the incremental mixed flow, denotes the type identifier of the incremental mixed flow, denotes the upper limit of the tolerable end-to-end delay of the incremental mixed flow.

[0020] When AVB flows or TT flows are incrementally deployed in the network, the transmission process of the existing AVB flows in the network will be disturbed, and their transmission performance will also be affected. Specifically, if the new flow is an AVB flow, it will exacerbate the bandwidth contention among all network AVB flows; and if the new flow is a TT flow, its occupation of a specific time slot will compress the available bandwidth of the AVB flow. The addition of mixed flows will increase the single-hop delay of the existing AVB flows; and then it may cause the upper limit of the end-to-end delay to be broken, resulting in unacceptable transmission performance. Based on this analysis, by modeling the delay in the single-hop transmission process of AVB flows, the dynamic disturbance caused by the incremental mixed flows can be effectively described.

[0021] ​​​​​According to network calculus theory, the upper limit of the single-hop delay of an AVB stream transmitted through a port is the maximum difference in the horizontal direction between the service curve and the arrival curve of the AVB stream transmitted through a certain port.

[0022] port Service curves for AVB streaming for:

[0023]

[0024] in, Indicates link bandwidth; Indicates port Time offset relative to network startup time At this location, the cumulative active duration of the dedicated timeslot and guard band for the TT stream is displayed. This function reflects the unavailable time window of the AVB stream and can be accessed by resolving the port. The gate control list is obtained directly. During the period when the AVB stream is not transmissible, The function value increases linearly with time; however, it remains constant during the period when the AVB stream can be transmitted.

[0025] For the network index is AVB stream Assume that it is generated by the terminal and passes through the first hop port. Upon joining the network, it is in The arrival curve can be characterized by the bucket-and-slot model:

[0026]

[0027] in, express The average generation rate within its cycle, Indicates its maximum possible instantaneous transmission volume; for Any port traversed in subsequent hops , its in The arrival curve is the previous hop port. Output curve:

[0028]

[0029] in, For flow In the port and its corresponding links The upper limit of single-hop delay during transmission; this value is added to the time synchronization error between nodes. Afterwards, as bias embedding exist arrival curve To depict the port Flow Output to port The effect.

[0030] When the incremental flow According to the transmission scheme specified for it When deployed on a network, it will cause updates to the service curves or arrival curves of some ports in the network: if For AVB flows, incremental deployment updates the aggregated AVB flows and their arrival curves on some ports, thereby increasing the single-hop latency of existing AVB flows in the network; if For TT streams, incremental deployment may compress the AVB stream transmission window and update the port service curve, thereby changing the upper bound of the single-hop latency of existing AVB streams.

[0031] Step 2: Considering the dynamic interference factors introduced by incremental hybrid streams, identify the key transmission management variables for AVB and TT streams, covering transmission route determination and scheduling time slot allocation, specifically:

[0032] Step 2.1: The difference between AVB and TT streams in whether they are allocated dedicated time slots leads to different transmission schemes: AVB streams only need to specify the route, while TT streams must determine the joint route and time slot scheduling scheme hop-by-hop. (Note the hybrid stream increment.) The transmission scheme is Its definition is:

[0033]

[0034] in, For incremental mixed flow The number of transmission hops, for In other words, Indicates incremental mixed flow In the The links traversed during the jump, For link set The Kleene positive closure represents the set of all potential routes with a hop count of at least one; when When it is a TT stream, considering that it needs to be transmitted within one supercycle... Then, it must be allocated on each link it passes through. One time slot; Indicates in Up to be assigned to TT stream of The index sequence of time slots, family It contains all The possible sequences of time slot indices, and denotes with the Cartesian product of the Kleene positive closure of ; each item in this closure corresponds to a potential end-to-end transmission scheme of TT flows with a joint routing-scheduling combination.

[0035] Step 2.2: To ensure the link order connection in traffic routing and avoid forming a loop, the routing in the transmission scheme needs to satisfy the following constraints:

[0036]

[0037] wherein, denotes the start node of link , denotes the end node of link , the end node of link , denotes the start node of link .

[0038] When is a TT flow, the time slots allocated to it on each link need to further satisfy the following constraints:

[0039]

[0040] wherein, is the th index in , corresponding to the time slot scheduled for the flow for the th transmission within a super cycle; H denotes the super cycle configuration of the TSN network, denotes the duration of a time slot; the offset between each pair of adjacent time slots must strictly match the flow period to ensure the periodicity of TT flow transmission.

[0041] Step 3: Design a traffic scheduling model based on multi-agent collaboration. When incremental mixed flows are accessed, generate an end-to-end transmission scheme that satisfies the delay and bandwidth constraints based on multi-agent collaborative decision-making. Specifically:

[0042] Step 3.1: Construct a multi-agent Markov decision process.

[0043] Convert the solution of the incremental mixed flow transmission management problem into a multi-agent Markov decision process. In this multi-agent Markov decision process, let denote the state space. When making transmission decisions for a certain incremental mixed flow, This represents the initial state quantity, which includes flow attribute information and the overall state of the network. The action space is shared by all agents; the incremental hybrid flow is formed through hop-by-hop cooperation among agents. In constructing an end-to-end transmission scheme, the output action of each agent involves the routing selection of the current hop, thus specifying which adjacent agent will continue the decision for the next hop. If the agents participate in sequence and cooperate to complete the shared... The jump-by-jump decision-making process is represented as follows:

[0044]

[0045] in, Indicates the decision-making process. To characterize the overall agent action of the end-to-end transmission scheme, corresponding to incremental hybrid streams end-to-end transmission scheme , For the first A smart agent The output single-hop transmission action, for the agent In other words, Defined as from the previous The action chain formed by the sequential decision-making outputs of each agent guides and constrains its current single-hop transmission decision; the overall decision of multiple agents can be represented as an action sequence composed of the single-hop actions of each agent, which corresponds to the end-to-end transmission scheme.

[0046] Step 3.2: Design the state input for the multi-agent system.

[0047] For the An intelligent agent participating in decision-making Its state input is determined by the initial state. With action chain Composition; initial state Contains Stream Attribute information (e.g., flow type, period, latency constraints, bandwidth requirements, etc.), and overall network information organized based on topological connectivity. (For example, the current transmission load of each link, time slot utilization, etc.); for intelligent agents Action chain It is concretized into the local observation state in the local network segment it maps to. This local observation characterizes the cumulative effect of preceding decisions on adjacent links.

[0048] In local observations, if incremental mixing flow For AVB flows, the agent focuses on the one-hop delay upper bound of both AVB flows on the adjacent link, and evaluates the potential impact on the link load via the transmission on the link; if For TT flows, the agent refines the observation granularity to the available time slots on the adjacent link, analyzes the fluctuations in the link transmission load and time slot utilization under different time slot allocation choices.

[0049] Initial state Local state Complete input formed by combination As the state input of the agent in hop-by-hop cooperation.

[0050] Step 3.3: Define differentiated action output modes for the agent for different incremental flow types.

[0051] For AVB flows, the agent outputs actions under the state The action For AVB flows, the agent outputs actions under the state The action contains both the next-hop transmission link and the set of time slots allocated on the link to ensure that the routing scheme and time slot allocation scheme of TT flows are compatible and conflict-free under the joint decision framework of routing and time slot scheduling. Through the above design, the agent is provided with a differentiated action space description and definition for incremental AVB flows and TT flows.

[0052] Step 3.4: To enable multiple agents to jointly optimize the end-to-end transmission performance of incremental mixed flows in the cooperation process, construct a reward function based on task achievement.

[0053] When multiple agents successfully generate an end-to-end transmission scheme that meets the bandwidth and delay constraints for a certain incremental flow through cooperation, the transmission scheme is considered as a valid configuration, and a positive reward is given to all agents involved in the decision-making process in the hop-by-hop decision-making process, with the reward value set to +1; if any hop in the decision-making chain cannot find a feasible next-hop link or available time slot, resulting in the inability to form a complete end-to-end transmission scheme, the cooperation is considered a failure, and a negative reward is given to all participating agents, with the reward value set to -1.

[0054] Step 3.5: Configure a flow-aware neural network (FANet) for each agent to realize the policy mapping from state input to action output, specifically:

[0055] For state input initial state Flow characteristics and network characteristics The system consists of two parts: a shared traffic feature extraction layer and a network feature extraction layer to extract high-dimensional features related to flow attributes and the overall network state; and a heterogeneous feature extraction layer to adaptively handle the local states corresponding to different traffic types. This allows for handling different dimensions under different conditions, such as AVB streams and TT streams. Feature extraction is performed; after feature extraction, the hidden features output from each feature extraction layer are concatenated, and the concatenated features are input into the corresponding decision module according to the traffic type, outputting the final action distribution or action selection; the agent... The decision-making strategy is represented as ,in for The policy parameters of a neural network.

[0056] Step 3.6: Train a multi-agent policy based on the proximal policy optimization algorithm; integrate the policy parameters of all agents into... Based on this, the overall decision-making strategy of the intelligent agent cluster is defined as follows: In the multi-agent Markov decision-making process, the agents obtain cumulative rewards by generating transmission schemes for incremental hybrid streams. And by leveraging the advantage function To measure in the initial state Take action below Characterized by incremental reward income compared to the average level To improve training efficiency and stability, a proximal policy optimization algorithm based on the Actor-Critic architecture is used to update the policy parameters.

[0057] The policy parameters are updated using a near-end policy optimization algorithm based on the Actor-Critic architecture, specifically as follows:

[0058] Old strategy Collect multiple rounds of interaction samples and calculate the advantage estimate for each time step; construct a model that includes the importance sampling ratio. objective function The difference in action selection between the new and old strategies is characterized by the importance sampling ratio, and a constraint mechanism is introduced into the objective function to limit the difference between the new and old strategies.

[0059] When importance sampling ratio When the deviation from 1 is too large, a truncation mechanism is used. right Cut to obtain ,Will Restricting the value to within between the initial agent and the next-hop agent, so as to avoid too drastic parameter update.

[0060] Finally, the parameter set is iteratively updated by using the policy gradient method, so that the new policy improves the solution performance of the incremental mixed flow transmission management problem under the premise of ensuring the stability of training:

[0061] Through the above training process, the converged multi-agent decision-making strategy is obtained

[0062]

[0063] Step 4: The trained agent is encapsulated as a multi-agent cluster and deployed in an industrial time-sensitive network to generate transmission schemes for incremental mixed flows online, specifically:

[0064] Step 4.1: When there is an incremental AVB flow or TT flow that needs to configure a transmission scheme, the transmission requirements of the flow are handed over to the initial agent for processing. The initial agent completes the first-hop transmission decision according to the flow type and the current network state, and outputs the corresponding single-hop transmission action.

[0065] Step 4.2: Determine whether the first-hop decision has formed an end-to-end transmission scheme covering the source node and the destination node. If not, transfer the decision-making right to the corresponding next-hop agent according to the next-hop link specified by the first-hop decision. The next-hop agent continues to output single-hop transmission actions under the constraints of its local observation state and historical action chain, and repeats the above process until a complete end-to-end transmission scheme is generated or it is determined that the transmission constraints of the incremental flow cannot be met under the current network conditions.

[0066] The above only describes the preferred embodiments of the present application and does not limit the idea of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.​​​

Claims

1. A multi-agent cooperative scheduling method for incremental hybrid flow in an industrial time-sensitive network, characterized in that, include: Step 1: Analyze the transmission characteristics of AVB and TT streams in industrial time-sensitive networks. Based on network calculus theory, and combined with bandwidth competition and latency fluctuations in incremental hybrid stream scenarios, refine the dynamic interference caused by incremental hybrid streams. Step 2: Combining the dynamic interference factors introduced by incremental hybrid streams, identify the key transmission management variables of AVB streams and TT streams, covering transmission route determination and scheduling time slot allocation; Step 3: Design a traffic scheduling model based on multi-agent cooperation. When incremental mixed-flow access occurs, generate an end-to-end transmission scheme that satisfies latency and bandwidth constraints based on multi-agent collaborative decision-making. Specifically: Step 3.1: Construct a multi-agent Markov decision process; Solving the incremental hybrid stream transport management problem is transformed into a multi-agent Markov decision process, in which we let... Representing the state space, when making transmission decisions for a given incremental hybrid stream, This represents the initial state quantity, which includes flow attribute information and the overall state of the network. The action space is shared by all agents; the incremental hybrid flow is formed through hop-by-hop cooperation among agents. In constructing an end-to-end transmission scheme, the output action of each agent involves the routing selection of the current hop, thus specifying which adjacent agent will continue the decision for the next hop. If the agents participate in sequence and cooperate to complete the shared... The jump-by-jump decision-making process is represented as follows: in, Indicates the decision-making process. To characterize the overall agent action of the end-to-end transmission scheme, corresponding to incremental hybrid streams end-to-end transmission scheme , For the first A smart agent The output single-hop transmission action, for the agent In other words, Defined as from the previous The action chain formed by the sequential decision-making outputs of each agent guides and constrains its current single-hop transmission decision; the overall decision of multiple agents can be represented as an action sequence composed of the single-hop actions of each agent, which corresponds to the end-to-end transmission scheme. Step 3.2: Design the state input for the multi-agent system; For the An intelligent agent participating in decision-making Its state input is determined by the initial state. With action chain Composition; initial state Contains Stream Attribute information And the overall network information obtained based on topological connectivity. ; Targeting intelligent agents Action chain It is concretized into the local observation state in the local network segment it maps to. This local observation characterizes the cumulative effect of preceding decisions on adjacent links; In local observations, if incremental mixing flow For AVB streams, the agent focuses on the upper bound of the single-hop delay of existing AVB streams on adjacent links in the network and evaluates... The potential impact on link load caused by transmission via this link; if For TT streams, the agent refines the observation granularity to the available time slots on adjacent links and analyzes... The fluctuations introduced to link transmission load and time slot utilization under different time slot allocation options; initial state With local state The complete input formed by combination As the state input of the agent in hop-by-hop cooperation; Step 3.3: Define differentiated action output methods for agents for different incremental flow types; For AVB streams, the intelligent agent In state Down-output actions For this hop routing, only the next-hop transmission link needs to be specified; however, for TT streams, the agent... In state Down-output actions It also includes the next-hop transmission link and the set of time slots allocated on that link, so as to ensure that the routing scheme and time slot allocation scheme of TT flow are compatible and conflict-free under the joint decision-making framework of routing and time slot scheduling; Step 3.4: To enable multiple agents to jointly optimize the end-to-end transmission performance of incremental hybrid streams during collaboration, construct a reward function based on task achievement. When multiple agents successfully collaborate to generate an end-to-end transmission scheme that satisfies bandwidth and latency constraints for an incremental stream, the transmission scheme is considered a valid configuration, and all agents participating in the decision-making process during the hop-by-hop decision-making process are given a positive reward, with the reward value set to +1; if no feasible next-hop link or available time slot can be found at any hop in the decision chain, resulting in the inability to form a complete end-to-end transmission scheme, then the collaboration is deemed a failure, and all agents participating in the decision-making process are given a negative reward, with the reward value set to -1; Step 3.5: Configure a flow-aware neural network FANet for each agent to achieve policy mapping from state input to action output; Step 3.6: Train a multi-agent policy based on the proximal policy optimization algorithm; integrate the policy parameters of all agents into... Based on this, the overall decision-making strategy of the intelligent agent cluster is defined as follows: In the multi-agent Markov decision-making process, the agents obtain cumulative rewards by generating transmission schemes for incremental hybrid streams. And by leveraging the advantage function To measure in the initial state Take action below Characterized by incremental reward income compared to the average level To improve training efficiency and stability, a proximal policy optimization algorithm based on the Actor-Critic architecture is used to update the policy parameters. Step 4: Encapsulate the trained agents into a multi-agent cluster and deploy it into an industrial time-sensitive network for online generation of transmission schemes for incremental hybrid streams.

2. The multi-agent cooperative scheduling method for incremental hybrid flow in industrial time-sensitive networks according to claim 1, characterized in that, The topology of the Industrial Time-Sensitive Network (TSN) is represented as follows: , For a set of network nodes, For a set of directed links: in, , and Representing links respectively The starting node, ending node, and bandwidth; A collection of ports in a network. Indicates link The corresponding source port, for any port The link with this port as the source is denoted as ; On-demand deployment for industrial applications will incrementally introduce mixed flows, including AVB and TT flows, into the TSN network, enabling... Let the index sequence of the incremental mixed stream be denoted as the first. The incremental mixed flow entering the network is Its definition is as follows: in, Indicates the starting point of the incremental mixing flow. Indicates the endpoint of the incremental mixing flow. Indicates the data size of the incremental mixed stream, Indicates the transmission period of the incremental mixed stream, Indicates the type identifier of the incremental mixed stream. This represents the upper limit of the tolerable end-to-end delay for incremental hybrid streams; Incremental hybrid streams exacerbate bandwidth contention among AVB streams across the entire network or compress the available bandwidth of AVB streams, and increase the single-hop latency of existing AVB streams. By modeling the latency during the single-hop transmission of AVB streams, the dynamic interference caused by incremental hybrid streams can be effectively characterized. The upper limit of the single-hop latency of AVB streams transmitted by a port is the maximum difference in the horizontal direction between the service curve and the arrival curve of the AVB stream transmitted by a certain port. port Service curves for AVB streaming for: in, Indicates link bandwidth; Indicates port Time offset relative to network startup time At this point, the cumulative activation duration of the dedicated time slot and guard band for the TT flow; For the network index is AVB stream Assume that it is generated by the terminal and passes through the first hop port. Upon joining the network, it is in The arrival curve is: in, express The average generation rate within its cycle, Indicates its maximum possible instantaneous transmission volume; for Any port traversed in subsequent hops , its in The arrival curve is the previous hop port. Output curve: in, For flow In the port and its corresponding links The upper limit of single-hop delay during transmission; this value is added to the time synchronization error between nodes. Afterwards, as bias embedding exist arrival curve To depict the port Flow Output to port The effect.

3. The multi-agent cooperative scheduling method for incremental hybrid flow in industrial time-sensitive networks according to claim 2, characterized in that, Step 2 specifically involves: Step 2.1: The transmission scheme for AVB streams only requires specifying the route, while TT streams must determine the joint route and time slot scheduling scheme hop-by-hop. Record the hybrid stream increment. The transmission scheme is Its definition is: in, For incremental mixed flow The number of transmission hops, for In other words, Indicates incremental mixed flow In the The links traversed during the jump, For link set The Kleene positive closure represents the set of all potential routes with a hop count of at least one; when When it is a TT stream, considering that it needs to be transmitted within one supercycle... Then, it must be allocated on each link it passes through. One time slot; Indicates in Up to be assigned to TT stream of The index sequence of time slots, family It contains all The possible sequences of time slot indices, and express and The Kleene positive closure of the Cartesian product covers all combinations of "link-slot index" union terms that contain at least one hop; each term in this closure corresponds to a potential routing-scheduling union TT stream end-to-end transmission scheme. Step 2.2: To ensure sequential link connection in traffic routing and avoid loop formation, the transmission scheme... Routes in the network must meet the following constraints: in, Indicates link The starting node, Indicates link The termination node, link The termination node, Indicates link The starting node; when When it is a TT stream, the time slots allocated to it on each link must further satisfy the following constraints: in, for The first in Each index corresponds to a stream. Within the supercycle The time slot scheduled for this transmission; H represents the supercycle configuration of the TSN network. This indicates the duration of a time slot; the offset between each pair of adjacent time slots must be precisely matched to the flow cycle. This is to ensure the periodicity of TT stream transmission.

4. The multi-agent cooperative scheduling method for incremental hybrid flow in industrial time-sensitive networks according to claim 1, characterized in that, Step 3.5 specifically involves: For status input initial state Flow characteristics and network characteristics The system consists of two parts: a shared traffic feature extraction layer and a network feature extraction layer to extract high-dimensional features related to flow attributes and the overall network state; and a heterogeneous feature extraction layer to adaptively handle the local states corresponding to different traffic types. This allows for handling different dimensions under different conditions, such as AVB streams and TT streams. Perform feature extraction; in After feature extraction is complete, the hidden features output from each feature extraction layer are concatenated, and the concatenated features are input into the corresponding decision module according to the traffic type, outputting the final action distribution or action selection; the agent... The decision-making strategy is represented as ,in for The policy parameters of a neural network.

5. The multi-agent cooperative scheduling method for incremental hybrid flow in industrial time-sensitive networks according to claim 1, characterized in that, In step 3.6, a near-end policy optimization algorithm based on the Actor-Critic architecture is used to update the policy parameters, specifically as follows: Old strategy Collect multiple rounds of interaction samples and calculate the advantage estimate for each time step; construct a model that includes the importance sampling ratio. objective function The difference in action selection between the new and old strategies is characterized by the importance sampling ratio, and a constraint mechanism is introduced into the objective function to limit the difference between the new and old strategies. When importance sampling ratio When the deviation from 1 is too large, a truncation mechanism is used. right Cut to obtain ,Will Restricting the value to within and This allows for adjustments to avoid overly drastic parameter updates. Finally, the policy gradient method is used to apply the parameter set. Iterative updates are performed to improve the performance of the new strategy in solving the incremental hybrid stream transport management problem while ensuring training stability. Through the above training process, the converged multi-agent decision-making strategy is obtained. .

6. The multi-agent cooperative scheduling method for incremental hybrid flow in industrial time-sensitive networks according to claim 1, characterized in that, Step 4 specifically involves: Step 4.1: When there is an incremental AVB stream or TT stream that needs to be configured with a transmission scheme, the transmission requirements of the stream are handed over to the initial agent for processing. The initial agent completes the first-hop transmission decision based on the stream type and the current network status, and outputs the corresponding single-hop transmission action. Step 4.2: Determine whether the first-hop decision has formed an end-to-end transmission scheme covering the source node and the destination node. If not, transfer the decision-making power to the corresponding next-hop agent according to the next-hop link specified by the first-hop decision. Under the constraints of its local observation state and historical action chain, the next-hop agent continues to output single-hop transmission actions and repeats the above process until a complete end-to-end transmission scheme is generated or it is determined that the transmission constraints of the incremental flow cannot be met under the current network conditions.

Images