A flow reservation method in distributed network configuration model based on reinforcement learning

Abstract

The present invention discloses a flow reservation method in a distributed network configuration model based on reinforcement learning. Under the distributed time-sensitive network configuration model, the method first defines a network configuration framework based on the Resource Allocation Protocol (RAP) and asynchronous traffic shaping (ATS), and introduces a delay configuration engine in the configuration framework. The engine can monitor the reception rate of time-sensitive data flows in the network and use this as feedback to dynamically adjust the delay bandwidth configuration reserved for each hop in the network through reinforcement learning. To this end, the method establishes a per-hop delay model based on asynchronous traffic shaping, and designs a reinforcement learning agent state space, action space and reward function. During each iteration, the reinforcement learning agent monitors the network's reception rate of data flows under the current per-hop delay configuration, and dynamically adjusts the per-hop delay configuration. Through the interaction between the environment and the reinforcement learning agent, the optimal delay budget configuration is selected to maximize the reception rate of data flows in the network.

Description

Technical Field

[0001] The present invention belongs to the field of time-sensitive networks, and in particular relates to a flow reservation method in a distributed network configuration model based on reinforcement learning. Background Art

[0002] With the rapid development of intelligent manufacturing, the demand for deterministic and reliable transmission delays and uniform network architectures is becoming increasingly urgent in the industrial sector. Ethernet's best-effort mechanism limits its ability to provide deterministic transmission, while some industrial real-time Ethernet networks are incompatible with each other. To this end, the IEEE Time-Sensitive Networking (TSN) Working Group has developed a series of standards to support deterministic transmission on Ethernet. A key component of this work is the Stream Reservation Protocol (SRP), which operates on the control plane and is primarily used to reserve bandwidth resources for time-sensitive traffic and provide end-to-end deterministic latency. SRP has been improved upon with the Resource Allocation Protocol (RAP).

[0003] In practical applications, reservation algorithms such as RAP are often combined with various traffic shaping algorithms. In a distributed time-sensitive network model, RAP is used in conjunction with Asynchronous Traffic Shaping (ATS). However, the delay calculated by ATS accumulates with each hop. In this case, traditional reservation algorithms configure a relatively loose delay guarantee at each hop during the reservation process to prevent data flows from being rejected because the per-hop delay exceeds the configured delay guarantee. However, an overly loose delay guarantee will waste bandwidth resources, while an overly tight delay guarantee will result in traffic being rejected. Both of these situations will reduce the amount of traffic accepted by the network. Therefore, determining the optimal per-hop delay guarantee during the reservation process is a very important issue that deserves research. Summary of the Invention

[0004] Purpose of the invention: In response to the above-mentioned optimization problem, that is, how to determine an optimal network configuration so as to maximize the reception rate of data flows in the network, the present invention proposes a flow reservation method in a distributed network configuration model based on reinforcement learning, which can dynamically adjust the per-hop delay guarantee to improve the acceptability of the flow. The method first simulates and analyzes the reservation process of the RAP+ATS distributed network, and establishes the optimization problem of the network configuration during the reservation process. Then, the per-hop delay of the ATS is modeled on the data side of the bridge, and the forwarding operation of the ATS shaping is simulated. Finally, a reinforcement learning agent is introduced into the network, and according to the feedback of the data flow acceptance rate in the network, the configuration of the per-hop delay guarantee is dynamically adjusted on the control plane of the bridge, thereby realizing the automatic configuration of the network and maximizing the number of data flows that can be deployed to the network.

[0005] Technical solution: To achieve the purpose of the present invention, the technical solution adopted by the present invention is: a flow reservation method in a distributed network configuration model based on reinforcement learning, which specifically includes the following steps:

[0006] (1) Based on the fully distributed configuration model defined by the IEEE 802.1Qcr standard, a TSN bridging network framework based on RAP+ATS based on reinforcement learning is established, and the network configuration optimization objectives and constraints in the reservation process are defined;

[0007] (2) Establish an ATS per-hop delay model on the data plane of the bridge, simulate the forwarding operation of ATS on the data plane and define the delay calculation formula;

[0008] (3) The proxy configures a delay guarantee for the control plane of each hop bridge according to the per-hop delay requirement of the input flow, calculates the current delay through the data plane ATS delay model, and decides whether to accept the data flow by checking whether the constraint conditions are met.

[0009] (4) Define the state space, action space, and reward function of the reinforcement learning agent in the network. Dynamically adjust the delay guarantee configuration by continuously interacting with the environment, and ultimately determine the optimal configuration that maximizes the data flow reception rate in the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a flow chart of the stream reservation method based on reinforcement learning;

[0011] Figure 2 It is the distributed network configuration framework of RAP+ATS based on reinforcement learning proposed by the present invention; DETAILED DESCRIPTION

[0012] The technical solution of the present invention is further described in detail below with reference to the accompanying drawings and implementation examples.

[0013] The flow chart of the flow reservation method in the distributed network configuration model based on reinforcement learning proposed by the present invention is shown in the attached figure. Figure 1 The specific steps include:

[0014] (1) Based on the fully distributed configuration model defined by the IEEE 802.1Qcr standard, a TSN bridging network framework based on RAP+ATS based on reinforcement learning is established, and the network configuration optimization objectives and constraints in the reservation process are defined;

[0015] Specifically, the distributed bridging network configuration framework of RAP+ATS based on reinforcement learning, the optimization objectives of the network configuration problem, and the steps for establishing the constraints are as follows:

[0016] (1a), such as Figure 2 As shown in the figure, a fully distributed TSN bridging network consisting of terminals, bridges, and physical links is considered in accordance with the IEEE 802.1Qcc standard. Network / user configuration information is transmitted directly within the bridge via the User Network Interface (UNI). The bridge data plane uses the ATS mechanism defined in the IEEE 802.1Qcr standard to forward data flows. The control plane uses the RAP protocol defined in the IEEE 802.1Qat standard to verify whether there are sufficient network resources for transmission and to reserve the required resources, thereby performing admission control operations on data flows.

[0017] (1b), such as Figure 2 As shown, a delay configuration engine is introduced into the network to monitor the reception rate of data flows whose admission is controlled by the RAP protocol in the network and make corresponding adjustments to the delay configuration.

[0018] (1c) Determine the optimization objectives and constraints in the network configuration problem. The goal is to find a delay-guaranteed configuration that maximizes the number of acceptable TSN flows in the scheduled process under the constraints of RAP. It can be expressed as:

[0019]

[0020]

[0021] Where R represents the acceptance rate of TSN flow in the network, N deploy and N accept represents the number of flows deployed and received in the network, d max (k,f i ) represents the flow f i In the delay bound of the k-th hop, Represents the configured per-hop delay guarantee.

[0022] (2) Establish an ATS per-hop delay model on the data plane of the bridge, simulate the forwarding operation of ATS on the data plane and define the delay calculation formula;

[0023] Specifically, the steps to establish the ATS per-hop delay model and simulate the ATS forwarding operation on the data plane are as follows:

[0024] (2a), the delay requirement of a data flow is generally given as an end-to-end delay bound, which can be calculated as the accumulation of the per-hop delay bounds of each hop along the path of flow f:

[0025]

[0026] d max (k, f) represents the delay bound of flow f at the kth hop, which is determined by the buffer delay d BU，max (k, f), media delay d MD，max (k), arrival recognition delay d AT，max (k) and processing delay d PR，max (k) Composition:

[0027] d max (k, f) = d BU，max (k,f)+d MD，max (k)+d AT，max (k)+d PR，max (k) (4)

[0028] Only the delay related to the data flow is considered, so d max (k, f) mainly depends on d BU，max (k, f), mainly comes from the competition of other data frames in the queuing stage and the store-and-forward operations of the port. Under the above conditions, the upper bound of the worst-case delay of flow f at the kth hop can be expressed as:

[0029]

[0030] (2b) According to the shaping algorithm defined by ATS, a qualified time T is assigned to TSN class traffic f eligibility , and enters the scheduling pool to wait for transmission, if T eligibility Less than or equal to the current time T current , then the flow f will be based on the qualified time T eligibility The best-effort traffic enters the BE queue and is forwarded after all TSN traffic is transmitted.

[0031] (3) The proxy configures a delay guarantee for the control plane of each hop bridge according to the per-hop delay requirement of the input flow, calculates the current delay through the data plane ATS delay model, and decides whether to accept the data flow by checking whether the constraint conditions are met.

[0032] Specifically, the agent configures a delay guarantee for each bridge hop and checks whether the delay constraint is met as follows:

[0033] (3a), in order to meet the delay requirements of different data flows, the delay configuration G at time t t It is defined as a set of four delays of different sizes G = {G j |j=0, 1, 2, 3}, the proxy needs to meet the delay requirements of flow f Allocate the minimum per-hop delay guarantee G that can meet its reservation requirements j :

[0034]

[0035] (3b), for flow f i , which can be accepted by the network during the reservation process under the following constraints:

[0036]

[0037] (4) Define the state space, action space, and reward function of the reinforcement learning agent in the network. Dynamically adjust the delay guarantee configuration by continuously interacting with the environment, and ultimately determine the optimal configuration that maximizes the data flow reception rate in the network.

[0038] Specifically, the definition of the state space, action space, and reward function of the reinforcement learning agent and the steps of the interaction between the agent and the environment are as follows:

[0039] (4a), the state information provided to the agent should include the current configuration of each hop and the TSN flow reception rate of the network under this configuration. The state space is required to include the number of flows deployed to the network, N. deploy , the delay requirement of flow f Static information about data flow, current network delay configuration G t And the number of data streams actually received by the current network The state space of the network environment is represented as follows:

[0040] (4b), the goal is to optimize the delay configuration G through the interaction between the agent and the network environment t To maximize the data flow reception rate, the delay budget configuration needs to be adjusted in each iteration, so action a t Representatives of G tAdjustment, that is, for each G j Perform addition or subtraction operations. j , there are 8 actions in total, so the action space can be expressed as: A = {a0, a1, a2, a3, a4, a5, a6, a7}.

[0041] (4c) uses the change in reception rate to quantify the reward, and the reward function is defined as the percentage change in reception rate between the current delay configuration and the previous configuration:

[0042]

[0043] (4d), at each time t, the agent selects the action with the largest Q value from the action space A to t Make adjustments and get a new configuration G t+1 , the environment will also transfer to a new state S t+1 , calculate the reward and update the Q value table:

[0044] Q(s t , a t )=Q(s t , a t )+α[r t+1 +γmax a∈A Q(s t+1 ,a)-Q(s t , a t )] (9)

[0045] At time t+1, the algorithm will start a new round of iteration, and the control plane of the bridge will be based on the new delay configuration G t+1 The agent determines whether reservations can be made based on the environment and constraints. Through continuous interaction between the environment and the agent, the delay configuration agent can ultimately provide a relatively optimal delay budget configuration for the current network environment, maximizing the network's data flow reception rate while satisfying the per-hop delay constraints.

[0046] The contents not described in detail in the present application belong to the prior art known to professional and technical personnel in this field.

Claims

1. A method for stream reservation in a distributed network configuration model based on reinforcement learning, comprising the following steps: Step 1: Based on the fully distributed configuration framework defined by the IEEE 802.1Qcr standard, a reinforcement learning-based RAP+ATS TSN bridging network framework is established, and the network configuration optimization objectives and constraints during the reservation process are defined. Step 2: Establish an ATS per-hop delay model on the data plane of the bridge, simulate the ATS forwarding operation on the data plane, and define the delay calculation formula; Step 3: The proxy configures a delay guarantee for the control plane of each bridge hop based on the per-hop delay requirement of the input flow. It calculates the current delay using the data plane ATS delay model and decides whether to accept the input data flow by checking whether the constraints are met. Step 4: Define the state space, action space, and reward function of the reinforcement learning agent in the network; and dynamically adjust the delay guarantee configuration by continuously interacting with the environment, ultimately determining the optimal configuration that maximizes the data flow reception rate in the network.

2. The method for stream reservation in a distributed network configuration model based on reinforcement learning according to claim 1, characterized in that: The distributed bridging network configuration framework of RAP+ATS based on reinforcement learning, the optimization objectives of the network configuration problem, and the steps for establishing the constraints are as follows: Step 1-1: In accordance with the IEEE 802.1Qcc standard, a fully distributed TSN bridge network is considered, consisting of terminals, bridges, and physical links. Network / user configuration information is directly transmitted in the bridge through the User Network Interface (UNI). The bridge data plane uses the ATS mechanism defined in the IEEE 802.1Qcr standard to forward data streams. The control plane uses the RAP protocol defined in the IEEE 802.1Qat standard to verify whether there are sufficient resources in the network for transmission and to reserve the required resources, thereby performing admission control operations on the data stream. Step 1-2: Introduce a delay configuration engine into the network to monitor the reception rate of data flows whose admission is controlled by the RAP protocol and adjust the delay configuration accordingly. Steps 1-3: Determine the optimization objectives and constraints in the network configuration problem. The goal is to find a delay-guaranteed configuration that maximizes the number of acceptable TSN flows in the scheduled process under the constraints of RAP. This can be expressed as: Where R represents the acceptance rate of TSN flow in the network, N deploy and N accept represents the number of flows deployed and received in the network, d max (k,f i ) represents the flow f i In the delay bound of the k-th hop, Represents the configured per-hop delay guarantee.

3. The method for stream reservation in a distributed network configuration model based on reinforcement learning according to claim 1, characterized in that: The steps to establish the ATS per-hop delay model and simulate the ATS forwarding operation on the data plane are as follows: Step 2-1: The delay requirement of a data flow is given by an end-to-end delay bound, which can be calculated as the accumulation of the per-hop delay bounds for each hop along the path of flow f: d max (k, f) represents the delay bound of flow f at the kth hop, which is determined by the buffer delay d BU,max (k, f), media delay d MD,max (k), arrival recognition delay d AT,max (k) and processing delay d PR，max (k) composition: d max (k,f)=d BU,max (k,f)+d MD,max (k)+d AT，max (k)+d PR,max (k) (4) Only the delay related to the data flow is considered, so d max (k, f) depends on d BU,max (k, f), comes from the competition of other data frames in the queuing stage and the store-and-forward operation of the port. Under the above conditions, the upper bound of the worst-case delay of flow f at the kth hop can be expressed as: Step 2-2: Assign a qualified time T to TSN traffic f according to the shaping algorithm defined by ATS eligibility , and enters the scheduling pool to wait for transmission, if T eligibility Less than or equal to the current time T current , then the flow f will be based on the qualified time T eligibility The traffic is forwarded to the next hop in order from low to high; the best-effort traffic enters the BE queue and is forwarded after all TSN traffic is transmitted.

4. The method for stream reservation in a distributed network configuration model based on reinforcement learning according to claim 1, characterized in that: The steps for the agent to configure the delay guarantee for each bridge hop and check whether the delay constraint is met are as follows: Step 3-1: To meet the delay requirements of different data flows, configure the delay G at time t t It is defined as a set of four delays of different sizes G = {G j |j=0, 1, 2, 3}, the proxy needs to meet the delay requirements of flow f Allocate the minimum per-hop delay guarantee G that can meet its reservation requirements j : Step 3-2: For stream f i , which can be accepted by the network during the reservation process under the following constraints:

5. The method for stream reservation in a distributed network configuration model based on reinforcement learning according to claim 1, characterized in that: The definition of the state space, action space, and reward function of the reinforcement learning agent and the interaction between the agent and the environment are as follows: Step 4-1: The state information provided to the agent should include the current configuration of each hop and the TSN flow reception rate of the network under this configuration. The state space is required to include the number of flows deployed to the network, N. deploy , the delay requirement of flow f Static information about the data flow, the current delay configuration of the network G t And the number of data streams actually received by the current network Regarding the dynamic information of the network state, the state space of the network environment is represented as: Step 4-2: The goal is to optimize the delay profile G by interacting with the network environment through the agent t To maximize the data flow reception rate, the delay budget configuration needs to be adjusted in each iteration, so action a t Representatives of G t Adjustment, that is, for each G j Perform addition or subtraction operations; for four G j , there are 8 actions in total, so the action space can be expressed as: A = {a0, a1, a2, a3, a4, a5, a6, a7}; Step 4-3: Use the change in reception rate to quantify the reward. The reward function is defined as the percentage change in reception rate between the current delay configuration and the previous configuration: Step 4-4: At each time t, the agent selects the action with the largest Q value from the action space A to act on G t Make adjustments and get a new configuration G t+1 , the environment will also transfer to a new state S t+1 , calculate the reward and update the Q value table: Q(s t ,a t )=Q(s t ,a t )+α[r t+1 +γmax a∈A Q(s t+1 ,a)-Q(s t ,a t )] (9) At time t+1, the algorithm will start a new round of iteration, and the control plane of the bridge will be based on the new delay configuration G t+1 Determine whether reservation is possible; through continuous interaction between the environment and the agent, the delay configuration agent can eventually provide a relatively optimal delay budget configuration under the current network environment, maximizing the network's data flow reception rate while satisfying the per-hop delay constraint.

Images