Analyzing and modeling jitter and delay behaviors, in particular for hybrid industrial time-sensitive networks

Abstract

The invention relates to a method for operating a network, in which a plurality of network devices, each having a respective configuration, are connected to one another for data exchange and data are exchanged via these connections, in which a dynamic delay, i.e. Jitter, is taken into account when determining the data transmission time, the network being a time-sensitive network, and determining an actual time of data transmission across each network device from the initial network device to the target network device taking into account a dynamic delay in which time synchronization jitter and forwarding jitter are taken into account.

Description

Technical Field

[0001] The invention relates to a method for operating a network according to the features of the preamble of claim 1, in which a plurality of network devices, each with their own configuration, are connected to one another for data exchange and exchange data via these connections, dynamic delays (jitter) being taken into account when determining the data transmission time. Background Art

[0002] Such methods are already known. This will be discussed in further detail in the following description (especially Section 1). Although the consideration of dynamic delays (jitter) when determining the data transmission time has improved the performance of the network operation in the prior art, these methods are not applicable to all networks, especially all hybrid industrial networks, because not all network components, especially their network devices (such as switches, hubs, etc.) are constructed and suitable for this. Therefore, the resulting defects will also be discussed in the following description (especially in Section 2). Summary of the invention

[0003] The object of the present invention is therefore to improve the performance of this known method, in particular with regard to the speed of the transmission time, and also to achieve a wider applicability.

[0004] This object is achieved by the features of claim 1 .

[0005] According to the invention, it is provided that the network is a time-sensitive network and the actual time of data transmission from the initial network device to the target network device across the network devices is determined taking into account the dynamic delay, wherein the time synchronization jitter and the forwarding jitter are taken into account in the dynamic delay. As a result, the transmission characteristics of the network, in particular the data transmission time, can theoretically be analyzed in practice based on the modeling of the network characteristics and / or based on the measurement of the network characteristics of the network, and based on this can be determined more accurately than the methods possible in the prior art. For this purpose, reference is made in particular to Section 4.3 (the first paragraph there) in conjunction with Section 7.3.1 (the second paragraph there) in the further description below in a supplementary and accurate manner.

[0006] In a specific implementation of the invention, it is provided that each network device inserts a time stamp in the data frame at its ingress port and its egress port. To this end, reference is made in particular to Section 6.1 described below (paragraph 1 therein) in a supplementary and precise manner.

[0007] In a further improvement of the present invention, it is provided that: the theoretical time for data transmission from the initial network device to the target network device across each network device is determined. The theoretical time for data transmission can be determined by calculation during network planning without first setting up the network in practice through hardware components. Here, the configuration of the entire network, parts of the network and / or its individual components (especially its network devices) can be changed (modeled) and the resulting data transmission time, i.e., the change in data transmission time, can be determined. From the determined changes, conclusions can be drawn and measures can be taken as to how the overall configuration or individual configurations need to be changed in order to improve (especially accelerate) data transmission in the network. For this purpose, reference is made in particular to Section 7.3.1 (the second paragraph there) in the following description in a supplementary and accurate manner.

[0008] In a further development of the invention, provision is made for this purpose that the actually determined time is compared with the theoretically determined time. From the comparison it can be seen whether a change to the set configuration improves (or possibly also worsens) the data transmission time. Therefore, the configuration can be changed again and a further time determination can be made. This process can be carried out until a time is determined that is sufficient or satisfactory for data transmission. After this, the theoretical planning of the network is completed, the components of the network are configured accordingly, and a network using these configured components is established in practice.

[0009] For this purpose, a further development of the invention provides that, when the comparison result exceeds a predeterminable threshold value, the configuration of at least one network device is changed for the purpose of commissioning between the initial network device and the target network device. Thus, the network properties can be influenced by changing the corresponding configuration in order to improve (in the sense of shortening) the data transmission time. This is possible not only in the planning of a network that does not actually exist, but also in a real network, i.e., a network that has already been set up.

[0010] According to the invention, it is conceivable that the method according to the invention is implemented only between two network devices that are interconnected and that data is exchanged via this connection. It is also conceivable that the method according to the invention is implemented on more than two network devices present in the topology of the network devices. Thus, any part of the network or any part of an adjacent or subordinate network can be selected, its data transmission time can be theoretically determined and its data transmission time can be improved after modeling its configuration.

[0011] The same applies to the parts of the network that have already been established and exist in reality.

[0012] In a further development of the invention, it is provided that the configuration of the initial network device and / or the target network device is also changed. Thus, in this case, the properties of the entire network are taken into account and the time behavior of the network is improved.

[0013] Finally, in a further development of the invention, it is provided that the configuration of the at least one network device is changed until the comparison result no longer exceeds a predefinable threshold value. Thus, after a configuration change (commissioning) of a single network device, a group of network devices or all network devices of the network under consideration, the time behavior of the network is optimized.

[0014] The method according to the invention therefore has the significant advantage that the time characteristics of the network can be analyzed theoretically before the network is established, and additionally or alternatively can also be analyzed after the network is actually built and put into use, and can be optimized by changing the configuration (modeling). This is of great significance in hybrid industrial time-sensitive networks, such as TSN networks.

[0015] With regard to further specific steps of the method according to the invention, its implementation and its underlying technical considerations (see in this regard in particular Section 3), reference is made to the following description. DETAILED DESCRIPTION

[0016] New industrial technologies such as edge computing and cloud control are transforming factory automation networks from separate purpose-built real-time networks to large, highly interconnected, multi-purpose networks. At the heart of this evolution is the real-time capable factory backbone network that connects the different production lines and machines. IEEE Time-Sensitive Networking (TSN) and IETF Deterministic Networking (DetNet) provide a range of new mechanisms for networking to enable the factory backbone and edge. These technologies promise to create ubiquitous deterministic communications across entire industrial facilities.

[0017] However, in practice, different parts of a factory network employ different TSN mechanisms to meet their specific quality and service requirements (QoS). As a result, different TSN domains are formed based on different approaches for achieving real-time traffic forwarding. Since each domain can provide a different level of real-time QoS, the guarantees for data traffic crossing domain boundaries do not apply. As a result, it is difficult to predict the latency and jitter of TSN when forwarding data traffic between domains. Therefore, it is not easy to make a reliable assessment of the quality of service (QoS) that can be achieved in a larger factory network.

[0018] In this paper, we first defined a model for time-sensitive communication across multiple TSN domains. With this model, we are able to evaluate the QoS requirements of industrial applications that need deterministic communication across multiple TSN domains. Our evaluation and real-world measurements on industrial-grade TSN hardware show the feasibility and applicability of our model to predict realistic best- and worst-case scenarios for latency in such hybrid TSN networks. This enables factory-wide QoS guarantees and helps identify critical bottlenecks within these guarantees.

[0019] 1. Introduction

[0020] Concepts such as Industry 4.0, smart manufacturing, edge computing, and big data require communication networks that are highly interconnected, ubiquitous, and deterministic. Today, factory networks are often composed of independent subnetworks that are incompatible with each other due to manufacturer-specific communication technologies that require information exchange between gateways in these subnetworks. In addition, these factory subnetworks are installed once, statically configured, and not designed for dynamic reconfiguration. An important driver for manufacturer-specific and static network architectures is to ensure the real-time requirements of industry. Machine vendors are responsible for the continued and safe operation of their machines after extensive machine certification. Hence the shift to higher internal interconnectivity. Factory networks require a time-deterministic network that provides reliable and protected communication for both time-sensitive and non-sensitive communications without reconfiguration. Ultimately, industry is moving towards a unified communication standard with IEEE Time-Sensitive Networking (TSN), which extends IEEE Ethernet for real-time capabilities. For networks that span multiple layer 2 networks, the IETF is developing the “Deterministic Networking” (DetNet) specification for TSN specifications

[32] to enable routing of TSN traffic. Manufacturer-specific communication technologies are incompatible with each other due to specific and proprietary improvements. Therefore, TSN must replace manufacturer-specific communication technologies in machine networks and become a standard that enables seamless communication between all components.

[0021] IEEE TSN is a series of standards developed by the IEEE 802.1 group and extended with additional scheduling algorithms for IEEE Ethernet to achieve hard real-time guarantees. Two of these schedulers radically change the switch's selection of frames for transmission: First, the Time Aware Shaper (TAS) implements a class-based Time Division Multiple Access (TDMA), which is not work-saving due to blocking the transmission of traffic with low priority within a specific retention time window. With sufficient time synchronization in the network, TAS enables TDMA schemes even on the frame or flow level. Secondly, the frame preemption (so-called FP) mechanism allows to prioritize frames with high priority over frames with low priority when transmitting. The transmission of low priority frames is continued there when the transmission of frames with high priority is completed.

[0022] Several methods for computing time schedules for TDMA schemes such as TAS have been proposed in the literature, e.g. [13, 18, 20, 21, 31]. However, all of these methods assume a unified network requiring three elements:

[0023] (1) Common time synchronization across all forwarding nodes,

[0024] (2) Use the same set of TSN mechanisms on all switches, and

[0025] (3) Consistent configuration of TSN mechanisms, such as the same timing schedule on all switches.

[0026] However, typical industrial networks are not uniform because they are diverse. Various vendors provide parts of the network. For example, a factory contains machines from different manufacturers. Since machine vendors sell their machines in many different factories and the machine network itself is closed and uniform, consistency across the entire factory is very unusual. In addition, vendors optimize the components inside the machines for efficient operation. Therefore, it is not economical to support the maximum configuration in each machine to guarantee consistency in the factory. Therefore, multiple machines can have a large bandwidth of heterogeneous network configurations without synchronization using TAS (Time Aware Shaper) with high-precision time synchronization on FP. However, not all machines are completely independent. For example, robots in a production line may need common time synchronization to be able to work together on the same product.

[0027] Summary: Factory networks are usually very heterogeneous environments and the homogeneity assumption cannot be met. Therefore, the aforementioned methods can be used within a single machine or a synchronized group of machines, but cannot be used in the entire network due to the non-deterministic transitions between heterogeneous sub-networks.

[0028] To achieve real-time guarantees for networks in heterogeneous factories, we present a model that shapes the transitions between different subnets. This paper describes the impact of different TSN mechanisms and the lack of time synchronization based on the development of a model for worst-case guarantees. Therefore, Section 3 introduces the prerequisites for our work and briefly outlines TSN mechanisms. Our contributions are mainly threefold. First, we analyze the impact of different TSN mechanisms, heterogeneous TSN configurations (e.g., network cycle time) and changing connection speeds in Section 4. Second, we define a best-case and worst-case model in Section 5 that covers TSN and DetNet flows across multiple independent TSN network segments as a complement to the calculation of worst-case guarantees and end-to-end delays.

[0029] Our model can realize the following application scenarios:

[0030] 1) Identify expensive configurations in the network;

[0031] 2) Verify applications with communications across different TSN domains; and

[0032] 3) Network debugging and fault identification.

[0033] We make our model code publicly available to derive guarantees for arbitrary network configurations. We evaluate the model in Section 6 by performing measurements on commodity industrial machines with TSN hardware. Evaluating the TSN worst-case model on commodity hardware is novel. Our results show feasibility and applicability. Finally, Section 2 discusses related work and Section 8 concludes our work.

[0034] Solutions exist for time-sensitive traffic in Industry 4.0 and smart manufacturing scenarios. However, our gap analysis shows that technology alone is not enough because it is limited to a single machine. Each industrial scenario requires a solution to make these market trends possible. In reality, industrial scenarios include networks with 100s to 1000s of switches and routers. Therefore, there is a problem that we want to solve and that is complex enough to require tool-based analysis.

[0035] 2. Related Work

[0036] In this section, we present related research results and discuss how our approach extends the state of the art. There are different research areas in the context of real-time networks: First, the research community is generally very active in the computation of TAS schedules or TDMA schedules for networks. We can further divide this research into pure deadline planning computation methods and joint deadline planning and routing computation methods. However, we consider these topics to be orthogonal, as we focus our research on the transition between multiple predefined heterogeneous networks. Therefore, deadline planning computation is not the subject of this paper and we refer the reader to [11,13,14,18-22,29-31] for further details. Second, we discuss different approaches to verify whether the network configuration (e.g., the computed deadline plan) meets the promised real-time guarantees. We discuss these approaches in detail because of their importance and their relevance to our approach and show how we complement existing work. Third, we consider Network Calculus (NC) as a complementary category, as it can be used for configuration verification and for the design of deadline plans. Here, we focus only on the application of NC in the context of TSN, as they form the basis of our approach. NC is a mathematical framework based on addition and subtraction algebra for argumentation and we optimized the latency for scenarios with a single TSN mechanism or a combination of multiple TSN mechanisms.

[0037] 2.1 Configuration Verification

[0038] Verification of time planning and general scheduling is a complex task and is therefore often included in scenarios with strict certification requirements, such as the aerospace industry. To this end, there are numerous methods and optimizations for Aviation Full Duplex (AFDX) Ethernet networks and their configurations. In contrast to our work, AFDX networks are similar to single TSN machine networks in that they are closed and uniform. Therefore, modeling methods for AFDX networks assume uniform configuration and time synchronization [8-10]. Aircraft components are often specifically designed or adapted to be used in specific combinations and configurations. Therefore, in contrast to factory networks, which consist of multiple components of different types and suppliers (including standard products), these networks are uniform and we cannot assume uniformity. As a result, configuration verification of TSN networks has not yet been widely used. Gou et al.

[17] , Lv et al.

[25] , and Frimpong et al.

[16] proposed independent TSN configuration verification methods. However, all these methods require common time synchronization across all nodes in the network, which is not necessarily the case in factory networks.

[0039] 2.2 Network Calculus (NC)

[0040] Network calculus

[24] is a mathematical framework for computing delay and buffering upper bounds for a given network. Using addition and subtraction algebra, the NC model is able to provide narrow delay bounds. However, this accuracy comes at the cost of complexity. In contrast, our model takes a less complex approach, and we prioritize reducing complexity through simplification. Our evaluation (see Section 6) shows small differences between the computed worst-case delays and actual measurements. Therefore, we assume that in these scenarios, this simplification is acceptable.

[0041] Besides complexity, another drawback of NC is that the model is only valid under the specific conditions for which it was designed. Examples of such models can be found in [12, 27, 33, 34]. As a concrete example, Zhao et al.

[35] provide an NC model to be computed for the end-to-end delay for nodes that use a dedicated TAS gateway for the highest priority and a credit-based shaper (as described in IEEE 802.1Qav [1]) for other time-sensitive traffic. If the priority is changed, the model loses its applicability. This limitation of flexibility when combining TSN mechanisms and the restrictions on NC are well known, as described by Maile et al.

[26] . Since factory networks are very heterogeneous environments, the NC approach is not applicable. Regarding our approach, we trade flexibility in different scenarios for narrow bounds on the applicable NC.

[0042] 3 Background

[0043] The requirements for time-deterministic communication continue to grow in many industries and fields. However, to reduce complexity, we will limit the scope of application to industrial networks in the introduction. Together with our industry partners, we were able to verify the presented scenarios and assumptions. In this section, we will introduce the premises for discussing industrial control systems (ICS) based on next-generation TSN networks. After introducing an example architecture for an ICS network, we will continue with TSN and DetNet in this section. Finally, we standardize the types of traffic we use and discuss in this white paper in Section 3.2.

[0044] 3.1 Example ICS Network Architecture

[0045] Figure 1 shows an exemplary network architecture of the next generation ICS network based on the literature

[28] . Industrial networks have a hierarchical structure for two reasons:

[0046] A) Each network segment performs a dedicated task, and the above layers aggregate all networks with similar and related tasks;

[0047] B) For layered security.

[0048] The architecture introduces clear interfaces to apply the concepts of areas and lines

[23] . The machine network with the controller and all sensors and actuators is located at the lowest level. Each machine has its dedicated functionality and is designed and configured for the application scenario. Above this layer, there are multiple aggregation layers. In Figure 1, this is shown from the production line to the back end of production. Depending on the scale of supply, the network also includes additional layers, such as mobile communication networks.

[0049] Typically, the vendor of a network segment has full knowledge of all functions within that network segment, but little knowledge of the overall external traffic. This lack of knowledge will be further exacerbated in the future due to changes in dynamic manufacturing and production processes. Today, machine networks are isolated in order to compensate for these lack of knowledge and undesired impacts on critical applications. In the future, networks will be open to external data traffic in order to enable scenarios introduced by Industry 4.0 (e.g., virtual industrial control). Therefore, time-sensitive traffic inside the machine as well as time-sensitive traffic leaving and entering the machine requires QoS protection. The industrial market is focusing on TSN technology to achieve guaranteed QoS. We will discuss the details of this technology in Section 3.4.

[0050] 3.2 Traffic Classification

[0051] In industrial environments there are sometimes different types of traffic. Traffic is often event-based, while other traffic is periodic. Furthermore, industrial networks have: traffic with strict QoS requirements, traffic with less stringent QoS requirements, or even traffic with no specific QoS requirements. In general, any combination of traffic types may occur. For our model, we are not particularly interested in one class of traffic. We build our model on the assumption that we know all traffic in our network with all QoS requirements. This requires that all traffic with QoS requirements has a higher priority than data traffic without QoS requirements. Our model is not optimized for a specific configuration to achieve QoS requirements, and analyzes the latency that can be achieved for a fixed configuration. Therefore, the exact configuration between traffic types and priorities within the network (i.e., the Priority Code Point (PCP) value in the VLAN tag) is an input to the model and is not a limitation.

[0052] 3.3TSN Domain

[0053] As introduced in Section 3.1, vendors design each network segment in an industrial network for a specific application case. The vendors of different network segments create fixed configurations for specific purposes. A network segment with a consistent TSN configuration is called a TSN domain. At this stage, the machine vendor only knows the data traffic with QoS requirements that is relevant to the developed machine network. Therefore, the configuration of this TSN network includes not only resource reservations for all known traffic with QoS requirements, but also resource reservations for unknown traffic with QoS requirements. This results in resource reservations and protection for QoS requirements in a TSN configuration that uses the same TSN mechanisms on all nodes within the TSN domain and requires all nodes therein to be used for the ISN domain to be synchronized.

[0054] 3.4TSN Mechanism

[0055] Time-Sensitive Networking technology is specified by a set of standards developed in the TSN task group of IEEE 802.1. This set of standards implements standardized Ethernet networks. This provides time-deterministic transmission. Therefore, TSN is a key implementation for all Industry 4.0 and other converged network scenarios. In this section, we introduce three necessary TSN mechanisms to meet the QoS requirements for the described scenarios.

[0056] First, the choice was based on the current standardization status of IEC / IEEE 60802 [7] for industrial TSN configuration.

[0057] We first start with time synchronization, since it is the most important mechanism for supporting distributed and synchronous applications. Then, we describe two mechanisms to protect time-sensitive data traffic from data traffic with lower priority: time-aware shaper and frame preemption.

[0058] 3.4.1 Time Synchronization: IEEE 1588 (PTP) and IEEE 802.1AS

[0059] Synchronous applications require time synchronization between all end devices in the network. In the context of TSN, this is achieved by the Precision Time Protocol (PTP) defined by IEEE 1588 or its industrial configuration IEEE 802.1AS [6]. Both protocols use the concept of a master clock and distribute time with a constant measured delay between nodes to better compensate for errors. In PTP, the offset of time synchronization varies, but an accuracy of 1us can be achieved over a distance of 60 nodes [7]. In Section 6, we show the drift effect when the clocks are not synchronized with each other. We observed drifts of up to a few microseconds between two clocks over a period of several minutes. Obviously, a TSN domain can be synchronized directly with its neighboring domains or not. In addition, IEEE 802.1AS provides functionality for synchronization with external domains, so-called gPTP domains. Figure 2 shows three possible scenarios for time synchronization in use. In scenario A, two neighboring TSN domains are synchronized with each other, while in scenario B, they have different time perceptions. In scenario C, an intermediate domain forwards information about time synchronization between the upper and lower network segments. Therefore, both use the same time domain A to synchronize their applications without interfering with time domain B. In scenarios A and C, the virtualized SPS can take on the synchronization control task.

[0060] 3.4.2 Time-Aware Shaper (TAS): IEEE 802.1Qbv

[0061] Through Time Aware Shaper (TAS), the IEEE introduced a mechanism to provide a TDMA-like mechanism for Ethernet networks. Frames are classified and sorted in queues based on the Priority Code Point (PCP) value of their VLAN tags. Queues are opened and closed to transmit frames according to a configured schedule. This mechanism was originally introduced in IEEE 802.1Qbv[4] and is now integrated in IEEE 802.1Q[5].

[0062] Figure 3 visualizes the TAS mechanism. The egress port provides eight queues, one for each of the eight priority levels of the PCP field of the VLAN header. Thus, TSN supports up to eight traffic classes (TCs). Based on a periodic Gateway Control List (GCL), each egress port transmits frames of the TC in the time slots of the port open queue. We call this open port for transmission the TAS window. If multiple gateways are opened simultaneously, frames are selected by a strict priority mechanism that selects the highest priority frames first. The priority list for a particular TAS window is defined as priowindow. We use these variables to analyze whether a TC uses a window exclusively or shares it with other priority levels (i.e., exclusive gateways and shared gateways). For optimal operation, the TAS mechanism requires neighboring nodes that are synchronized with each other. Otherwise, frames arrive with unknown timing, which in the worst case leads to forwarding delays. We discuss the direct impact of synchronized and unsynchronized nodes in Section 4.2.6.

[0063] 3.4.3 Frame Preemption (FP): IEEE 802.1Qbu

[0064] The frame preemption mechanism allows frames of a certain priority (fast priority) to take over any other non-preemptive priority frames (i.e. frames not in the fast priority) by temporarily pausing transmission. This mechanism was originally introduced in IEEE 802.1Qbu[3] and has been incorporated into IEEE 802.1Q[5]. In addition, frame preemption requires modifications in the Ethernet bit transport layer, which were introduced in IEEE 802.3br[2].

[0065] Figure 4 shows two frames arriving at switch a with a time offset. The two flows have different layer 2 priorities in the priority code point (PCP) value in the VLAN tag. The network is configured so that PCP 7 has the only explicit priority. The preemption frame with PCP 6 arrives earlier than the fast frame with PCP 7 and arrives at the other port earlier. Therefore, the switch initiates the transmission of the PCP 6 frame first. Once the PCP 7 frame is ready for transmission, the switch preempts the transmission of the PCP 6 frame. This mechanism can only consider frames greater than or equal to 128 bytes.

[0066] 3.5 Deterministic Networking (DetNet)

[0067] Referring to Section 3.1, industrial networks consist of multiple independent machine networks (i.e., TSN domains). The combination of these TSN domains can form a large layer 2 network, or combine independent layer 2 networks into a layer 3 network. TSN targets layer 2 specifications and therefore all data traffic occurs within a layer 2 domain. Deterministic networking (DetNet) is a technology for enabling time-sensitive data traffic across multiple layer 2 networks. DetNet consists of a series of standards defined by the IETF

[15] that standardize the transition between two TSN domains at layer 3 forwarding nodes through TSN

[32] . In general, the specification requires deterministic forwarding latency and flows for transition so that the TSN flow uses the correct layer 2 addressing (e.g., correct VLAN tags, source addresses, and destination addresses) within the next TSN domain. We will explore the similarities in timing between layer 2 and layer 3 forwarding nodes in Section 4.2.

[0068] 4. System Model

[0069] In this section, we present our system model to analyze the best-case and worst-case delays of TSN and DetNet networks. Therefore, we introduce a generic forwarding model that is applicable to both TSN and DetNet nodes. Next, we introduce delays that are independent of the transmission selection algorithm (TSA). We then discuss the delay that is relevant to TSA, namely the queue delay, and evaluate its three different sub-latencies. Finally, we discuss the sources of jitter and the impact of jitter on network characteristics.

[0070] 4.1 Network Model

[0071] To derive a formal representation of our delay model, we first use a simple network model. The network consists of an arbitrary number of nodes v connected by edges e. To represent a full-duplex Ethernet network, we use a directed graph with always two connections between two nodes in order to model bidirectional transmission and to achieve bidirectional transmission at the same time. In this model, we do not distinguish between layer 2 and layer 3 networks, and we do not cover the TSN domain. However, for our delay calculation, time synchronization is crucial. Therefore, each node v is assigned a specific time range. We do not introduce a formal representation here, so we can simply describe that two nodes are adjacent and synchronized or not synchronized with each other. Finally, we realize the complete TSN configuration (i.e. TAS and FP) within the factory network. There are an arbitrary number of flows with QoS requirements within the network. We call the set of all flows with QoS requirements streams and the flow of interest s. For each flow, we know its priority prios, its cycle time and frame size bs. Again, this is similar for each configuration. For verification, we know all the paths that a flow takes through the network. Therefore, streamse defines the set of flows on an edge. As a basic premise of IEEE802.1 Ethernet networks, frames are forwarded when the line is idle, which means that node v will not be on the path of flow s twice.

[0072] 4.2 Forwarding Delay Model

[0073] Figure 5 shows an abstract view of forwarding with all delays and times highlighted. The model considers TSN nodes forwarding layer 2 frames and DetNet nodes forwarding layer 3 packets. In the following sections, we will discuss each delay and explain their impact in the worst-case delay model. Each of these delays is defined and analyzed per node or edge for a flexible model that supports a mix of different network devices.

[0074] 4.2.1 Propagation delay dprop

[0075] The propagation delay dprop is defined as the time it takes for the first bit to traverse the entire line. Therefore, this value depends on the line length and is static on each edge e.

[0076] 4.2.2 Transmission delay dtrans

[0077] The transmission delay dtrans depends on the connection speed and the frame size. In this article, we only refer to the frame size of Ethernet (i.e. layer 2). However, when implementing the model, we also consider the additional 20 bytes introduced due to the interframe gap, preamble and start frame boundary. This is purely for simplification reasons, because the layer 2 frame size associated with TSN networks is usually more typical than the layer 1 or layer 3 sizes.

[0078] Table 1 shows the transmission delay for a selection of individual bit sizes at different link speeds. This table is intended to provide a rough overview as background on the dimension of transmission delay, since transmission delay is related to interference delay as well as blocking delay.

[0079] 4.2.3 Processing Delay dproz

[0080] The processing delay dproz defines the duration used to process a frame or packet inside a forwarding node. This type of delay exists in both layer 2 and layer 3 forwarding nodes. The value varies depending on the node implementation (hardware vs. software) and the chip and software stack used in each node. However, we assume that this value is static on each node v.

[0081] 4.2.4 Interference delay dInterferenz

[0082] Higher levels may lead to more interfering flows in multiple domains on the same bridge. Depending on the configured QoS mechanisms, we can derive the interference delay dInterferenz in the worst case (as explained in this section), which defines the interference with frames of the same or higher priority on the egress port of the bridge. The set of flows that may interfere with the flow of interest s is a subset of all flows e transmitted on the edge:

[0083]

[0084] The interference delay depends on the edge connection speed e and is defined by the sum of all possible interfering transmission delays dtrans, calculated as follows:

[0085]

[0086] Next, we derive the set interferences,e,based on the QoS mechanisms on edge,e,.

[0087] Strict priority: With the strict priority mechanism, frames from all flows with equal and higher priority that may cause interference are taken into account. We model a subset of conflicting flows as follows:

[0088] interference s，e :={c|c∈streams e ∧prio c ≥prio s} (3)

[0089] As briefly introduced in Section 3.4.3, there are two categories of frames in terms of frame preemption: 1) fast frames, and 2) preempted frames. Depending on the assignment of a flow to one of these two categories, the interference calculation differs. A flow can only interfere with other fast flows if the desired flow is part of the fast category prioexpress. Strict priority is the mechanism used to distinguish between different flows within the fast category. We model the subset of flows that have the potential to interfere as follows:

[0090] interference s，e :={c|c∈streams e ∧prio c ∈prio express

[0091] ∧prio c ≥prio s} (4)

[0092] If the stream of interest is not part of the fast priority prioexpress, then it is part of the preemptive priority priopreemptible. In this case, all fast streams may preempt the stream of interest and thus cause interference delays. Within the preemptive priority, the strict priority selects the next stream to be transmitted. Therefore, we model the subset of streams with potential interference as follows:

[0093] interference s，e :={c|c∈streams e ∧(prio c ∈prio express ∨

[0094] (prio c ∈prio preemptible ∧prio c ≥prio s ))} (5)

[0095] Time-Aware Designer: Under the TAS mechanism, only switches transmit a priority subset according to the TAS configuration. Therefore, we only consider a subset of all flows with the highest priority in the priowindow. Additional flows that may cause interference must have equal or higher priority:

[0096] interference s，e :={c||c∈streams e ∧prio c ∈prio window

[0097] ∧prio c ≥prios} (6)

[0098] 4.2.5 Blocking delay dblck

[0099] The blocking delay defines the duration for which a frame of flow s is blocked on edge e, even if it has a lower priority. This delay depends on the TSN mechanism configured on that node. With strict priority as the transmission selection algorithm, an Ethernet switch selects the highest priority frame for transmission. However, if the switch has already started transmitting a frame with a lower priority before this, even though a frame with a higher priority is ready to send, this may cause a delay for traffic with a higher priority, possibly in the form of a maximum frame size. Therefore, IEEE802.1 introduces two mechanisms, TAS and frame preemption (FP), to reduce the impact on critical traffic.

[0100]

[0101] Therefore, we express the maximum blocking time dblck as a lower priority frame relying on the mechanism used in Equation 7. In this formula, we assume a maximum frame size of 1522 bytes, since this is the maximum value in standard IEEE Ethernet networks. However, IT networks often use Jumbo frames (e.g. for video data) with a size of 9022 bytes or even Jumbogrammes with a size of 65597 bytes. Therefore, the 1522 byte size in Equation 7 is a placeholder for the maximum frame size on edge e.

[0102] 4.2.6 Gateway delay dGate

[0103] The TAS mechanism keeps frames in the output queue when the gateway is closed, even if no other traffic exists. According to the network model in Section 4.1, the GCL configuration is known for each node. The gateway delay dgate defines the duration for which the node no longer forwards a flow because the gateway for this flow is closed. If the TAS mechanism is not configured, the gateway delay is set to 0. For each priority level, a transmission window in the GCL is opened in edge e for a duration of dwindowprio,e, which is opened at tomprio,e and closed at tcloseprio,e. The cycle time CT defines the recurring pattern of gateway open and gateway close events:

[0104]

[0105] Equation 8 applies to the asynchronous case, where we do not know the exact arrival times of the streams. In the synchronous case, dgate depends on the time of a frame that is ready for transmission at tenq. A frame may arrive too early in a cycle and wait for topen. Alternatively, a frame may not make it through the cycle, especially if it is transmitted too late or there is too much interference in the remaining open window where transmission is possible.

[0106]

[0107] It is important to note that Equation 9 does not directly assume that a frame is forwarded after tenq. We consider that frames may be delayed at the egress port by data traffic of equal and higher priority (see dInterferenz in Equation 2). Also, frames with lower priority may block the flow of interest for the duration of dblck (see Equation 7).

[0108] 4.3 Jitter Model

[0109] So far, the models have included static delays. However, Ethernet networks are not completely static. The biggest differences in the characteristics of synchronized networks are based on interference and blocking delays, as shown in Table 1 for the individual transmission delays. In addition, gateway delays also have a large impact on asynchronous scenarios. These two effects will be discussed in Section 5.1. There are two other types of jitter in TSN networks that we briefly point out here: A) time synchronization jitter and B) forwarding jitter.

[0110] First, time synchronization between two nodes is not stable over time. Therefore, the TAS port will not open with a perfect period on the network, but with small deviations. This type of jitter varies depending on the time synchronization mechanism. Protocols like NTP are less accurate than protocols like IEEE 1588 (PTP) or IEEE 802.1AS. The requirements document for the industrial TSN profile IEC / IEEE 60802 specifies a maximum jitter of 1 microsecond across 64 nodes in the network. Therefore, the current version of the IEC / IEEE 60802 specification requires the use of IEEE 802.1AS.

[0111] All components, such as PHY or switch chips have best-case and worst-case characteristics. Depending on the quality of the hardware components and the quality of the software stack used for software forwarding, the forwarding jitter will vary. Therefore, we introduced jitter definitions for each type of delay: propagation jitter jprope, transmission jtranse, and processing jprocv.

[0112] In Section 6.3 we analyze jitter and discuss its distribution. All jitter values ​​in this section are negligible compared to the effects of interference delay and blocking delay, typically in the range of a few nanoseconds to a microsecond compared to the effects of a few microseconds up to a few milliseconds. However, our goal is to obtain a complete model, and therefore present our implementation, covering all these types of jitter.

[0113] 5. Best and Worst Case Delay Models

[0114] In this section, a delay model for a complete factory network is constructed. Given the complete topology, whose configuration and flows with QoS requirements are described in Section 4.1, the model generates best and worst case delays for each flow. The model does not restrict the TSN configuration and topology, but analyzes the individual delays of each node and edge. The goal of this model is to compute the expected arrival window for each flow at each node in the network. The arrival window darriv is computed as the difference between the best and worst case delays. Figure 6 visualizes the arrival window magnified in gray over the distance within the network. The topology is a simple linear topology with optimal delay characteristics, visualized here in green. The worst case delays increase for each transmission independently of each other and are shown in red. The network is in the gray area during this transmission. At each node in the network, the difference between the best and worst case is called the arrival window darriv for a particular flow. Later in Section 7, we will use this arrival window for analysis to identify potential congestion in the worst case scenario and identify inefficient TSN configurations.

[0115] We introduce the model starting with the basic model. We compute the best-case and worst-case properties in Section 5.1. We introduce the effect of connection speed variations into our model in Section 5.2. Finally, we add congestion control and identification to our model in Section 5.3.

[0116] 5.1 Basic Delay Model

[0117] The delay of flow s on forwarding node v and edge e is defined as the sum of all the delays shown in Figure 5:

[0118]

[0119] Propagation delay and processing delay are static values ​​of the connection or forwarding node. Transmission delay depends on the connection speed and frame size. To complete this basic model represented by Equation 10, we define queuing delay as follows:

[0120]

[0121] When the TAS mechanism is configured on the egress port and frame, it must wait until the gateway is open and cannot be blocked due to delays. Therefore, we can always calculate the maximum of the two values ​​and derive a correct model as shown in Equation 11. In the best case, a flow does not interfere with other flows or is blocked by other data traffic. Therefore, we set dinterferences,e and dblcks,e to 0. Similarly, if TAS is not configured, we set dgates,e to 0. Otherwise, we apply Equation 8 or Equation 9 depending on the synchronization state. In fact, in the asynchronous scenario, the gateway delay dgates,e is set to 0 in the best case.

[0122] Additionally, in the best case, we subtract the jitter of the delay component differences. With this correction, we assume that all components on the forwarding path have optimal characteristics. The sum of all forwarding jitter is defined as:

[0123]

[0124] When the TAS mechanism is configured, we also assume that the gateway delay dgate is reduced by jtimesyncv. To calculate the worst case characteristics, we have to apply equations 2 and 7. Similar to the best case, we need to calculate the port delay delay dgate, but this time using the worst values ​​dinterferenzen and dblck. In addition, we add the sum of all forwarding jitters. When the TAS mechanism is configured, we increase dgate by jtimesyncv. As a result, this model calculates the best case delay and the worst case delay. The arrival window darriv represents the difference between these two delays. Figure 6 shows a small example construction of this arrival window.

[0125] 5.2 Impact of connection speed

[0126] In our delay model, we keep track of the size of all flows, expressed in bytes as bs. In addition, the topology model shows the connection speed on each connection. Therefore, the model is able to handle any variation in connection speeds across the network for industrial scenarios. This is a very important requirement, since sensors and actuators often have only 100Mbit / s interfaces, while the backbone network may run at 10GB / s. As shown in Table 1, these different connection speeds arise from different send durations dtrans as well as specific interference and blocking delays. Figure 6 shows the variation in connection speed with different tilt angles in the envelope.

[0127] 5.3 Impact of cycle time

[0128] In a TSN network, there are two types of cycle times. On the one hand, there is an application cycle time, with which application periodic data is transmitted. And on the other hand, in the TAS mechanism, the GCL works with a given network cycle time. This time, combined with the frame size, defines the bandwidth required for the application. The network cycle time, combined with the window duration of the GCL, defines the time for the possible throughput. If the forwarding node does not use TAS, the throughput will be limited by the connection speed. Even if TAS is not configured and we do not have a network cycle time, we still always have an application cycle time.

[0129] Based on our assumption that each TSN domain has a static and individual configuration, we cannot assume that the application cycle time and the network cycle time are the same. This change in cycle time can lead to an increase or decrease in the number of frames for a particular flow within a cycle. Figure 7 shows an example of a change in the network cycle time from one node to the second node. The first node (on the left) has a cycle time of 1ms, while the second node has 5ms. Therefore, five frames of the same flow from the first node will always be present in the next cycle. In order to avoid congestion inside the network, the configuration on the second node must be able to transmit all five frames. The same applies to end devices with an application cycle time when sending in a network with a TAS configuration. Similarly, the arrival window darriv of a flow can also be larger than the cycle time on the next node, and this leads to a doubling of the frames used for this flow within a cycle.

[0130] To satisfy the bandwidth increase requirement for each flow, we calculate three factors f that are used to increase the reserved bandwidth. First, we take into account the increased arrival window darriv and calculate fAnkunft. We calculate the ratio between the arrival window and the new clock period CTv. Next, we round this result up to always allow a complete flow to be within one of the possible periods:

[0131]

[0132] Secondly, the change in cycle time is similar and defined by fCT. If the cycle time on the second TSN node increases (i.e. it works slower), multiple original cycles are completed before the cycle continues and the second node is finished. Therefore, the second TSN node needs time to process more frames of the same flow than between the two cycle times according to the ratio:

[0133]

[0134] In the scenario where the cycle time on the second TSN node is reduced (i.e. working faster), we have to receive the frames of the flow in every cycle, because we do not model a flow that it should exist, e.g. every other cycle. Therefore, this has no impact on bandwidth reservation.

[0135] Finally, we calculate a factor fapp based on the difference between the application cycle time and the network cycle time. Similar to the previous scenario, we only consider the case where the network is periodic, i.e. CTnet is slower than the application cycle time CTapp:

[0136]

[0137] With these three values, the model improves the expected bandwidth in the worst case as follows:

[0138] b s,u =b s,v-1 ·f arriv ·f CT

[0139] And along the flow path s at the first TAS configuration node, the model increases the bandwidth as follows:

[0140] b s,v =b s,v-1 ·f app

[0141] 6. Evaluation

[0142] In the evaluation, we analyze different scenarios in the network and discuss the observed characteristics. The goal of the evaluation is to highlight the applicability of our model so that it covers a diverse combination of TSN configurations. In addition, the evaluation shows the feasibility of the model with a simple implementation. We divide our evaluation into the following parts. First, we introduce our measurement setup and methodology. Second, we specifically evaluate the impact of drift between unsynchronized clocks. Third, we analyze the capabilities of our measurement setup, namely the jitter of real TSN devices. After that, we focus on specific elements in our model, such as synchronous vs. unsynchronized characteristics, as well as interference and blocking delays due to cross traffic. Next, we discuss the variation of network cycle times.

[0143] 6.1 Apparatus and Method

[0144] We evaluate all using measurements on commercial TSN hardware. For all measurements, we have a linear topology with four devices: a transmitter "node 0" and three forwarding nodes. Each of these devices is able to insert hardware timestamps in the frames at the ingress and egress ports of the node. Thus, we can track each frame through the network and subsequently derive analytical features based on the packets.

[0145] Figure 8 shows the topology and three different data traffic paths. For each measurement, we have a measured traffic with 500-byte frames from "node 0" to "node 3". In addition, we can implement cross traffic on paths two and three. This cross traffic is in the form of Internet Mix (IMIX) and consists of three flows with frame sizes of 64 bytes, 570 bytes, and 1522 bytes. All graphs only show the measured traffic that exists, not the cross traffic. The duration of each measurement is between 25 minutes and five hours to show the stability of the results in terms of clock drift and jitter.

[0146] In general, we have two visualizations. First, we present the measurement curves over time. This visualization shows the trend over time, including the effects of drifting clocks (i.e., the constant rise and fall of values), jitter (when there are spikes), or stability (horizontal lines). Second, we show the measurements on a per-port basis on a per-node basis. In this visualization, we create a visualization that is compatible with arrival windows.

[0147] An important note for all measurements in this section is that the timestamps are based on each node's internal time. The nodes are synchronized via IEEE 802.1AS. Therefore, the results have an error of a few nanoseconds due to this synchronization.

[0148] 6.2 Time Synchronization

[0149] Figure 9 visualizes the impact of unsynchronized nodes in the network. The y-axis represents the offset relative to the sender in microseconds and the x-axis is the duration of the measurement. The four rows represent the measurement points for each device in the topology. In Figure 9a, all nodes within the network are synchronized using IEEE 802.1AS and we observe stable behavior. The accuracy of the time synchronization is within a few nanoseconds. Therefore, we observed no deviations during the measurement and the measured differences can be related to the forwarding time of the frame. In particular, the time to transmit a 500-byte frame over a 1 Gbit / s connection lasts about 4 us, while the processing delay of the switch is between 1 and 2 us. In contrast, in Figure 9b, only "Node 1" is synchronized with the sender, while "Node 2" is synchronized with "Node 3". We observe that the time base of the last two devices gradually deviates from the first two. However, the time characteristics between the two are consistent and each "Node 0" and "Node 1" and "Node 2" and "Node 3" remain stable. This scenario is characteristic of two unsynchronized TSN domains.

[0150] 6.3 Jitter

[0151] Low jitter is critical for correct network behavior in applications where time-sensitive conditions exist. High jitter can lead to different characteristics between cycles, such as interference with other traffic or interference in calculations on controllers and devices. Therefore, our model includes a jitter component within the network and calculates best-case and worst-case characteristics based on this. Figure 10 visualizes the measurements, including one measurement at "node 0" and two measurements at three other nodes. Figure 10a visualizes each timestamp inserted everywhere as a measurement run with the timestamp "node 0" as a reference. In fact, Figure 10a uses the same measurements as Figure 9a, with the added view of ingress and egress times. This figure clearly shows stable operation without interference with other network traffic. However, it can be seen that the upper line is thicker than the lower line, indicating the presence of jitter. To analyze the jitter within the forwarding process, Figures 10b to 10d show more detailed views. First, Figure 10b shows the jitter of the entire end-to-end delay. We subtract the timestamp of "node 0" from the output timestamp of each node and calculate the jitter in the resulting series of delays. We observe that the jitter increases with the distance (i.e. “Node 1” is closer to “Node 0” than “Node 3”). Furthermore, we also observe that the jitter distribution is wider at higher distances in the network. Secondly, Figure 10c visualizes the jitter in the processing time. To calculate the dataset, we subtracted the input timestamp of each node from the output timestamp of the same node. Since the same hardware is involved, the distribution is the same on each node. Furthermore, the jitter is related to the processing time and is in the range of only 10ns. Thirdly, Figure 10d shows the difference between the input and output of two adjacent nodes. This figure represents the jitter in the transmission delay. Again, their distribution is similar on all paths and is in the range of 30ns.

[0152] 6.4 Connection Speed

[0153] An obvious component within the model is the transit time of each flow on the edge. Figure 11 shows the delay between four nodes in a linear topology, in this scenario, the connection speed between "Node 0" and "Node 1", and between "Node 2" and "Node 3" is 1Gbit / s, while the connection speed between "Node 1" and "Node 2" is only 100Mbit / s. The delay is very stable over the 25 minute measurement interval as shown by Figure 11a. Figure 11b shows the exact measurement in a hop-by-hop view. This figure clearly shows that the connection speed between "Node 1" and "Node 2" is reduced and the edge becomes steeper, and thus the increased delay. The visualized timestamps are referenced to the output of each node. Therefore, each delay up to the node includes the propagation delay of the line (which applies equally to each connection), the transmission delay of the 500 byte frame (which depends on the connection speed) and the processing delay (which is a constant value and a static value of 1us for all nodes). Based on these components, the model presented in Section 5 is derived and in the best case the delay is 4.1us for a 1Gbit / s connection and 41us for a 100Mbit / s connection. There is no other traffic in this configuration except the measurement of data traffic. The worst case delay derived by the model is simply to add up all the jitter components as shown in Section 4.3 and evaluated in Section 6.3.

[0154] 6.5 Time-aware Shaper

[0155] In this section, we present the characterization evaluation of TAS. Figure 12 shows the results used for the discussion in this section. We use the same TAS configuration for both synchronous and asynchronous measurements. The “node 0” node sends a frame at the beginning of the time interval, and each of the subsequent three nodes turns on its gateway for the measurement traffic within 40us. In each node, we delay the time of turning on the gateway by 30us. The size of the measurement flow is only 500 bytes, which requires 4us at the beginning and about (30-5)us at the next node (including other delays such as processing delay). For an exhaustive analysis, all figures show the receiving time (i.e., rx) and transmitting time (i.e., tx) of each forwarding node.

[0156] In the synchronous scenario (see the two left graphs in Figure 12), all four nodes in the network are synchronized with each other. Figure 12a shows this static behavior, with a latency (difference between the rx value and the tx value of a node) of 25us over the entire course of the measurement duration. Figure 12c visualizes the same measurements on a per-node basis, and again taking into account two timestamps for each forwarding node. This figure shows the consistency of the latency behavior over time, since all flows use the same time period for transmission through the network. In the asynchronous scenario, "Node 0" and "Node 1" are synchronized with each other, while "Node 2" and "Node 3" are also synchronized with each other, but "Node 1" is not synchronized with "Node 2". Figure 12b uses a different scale than the envelope as a synchronization graph, since the transmission increases beyond the cycle time. The red line ("Node 2-rx") visualizes the drift between the two time domains. At the beginning of the measurement, the two time domains have similar time characteristics, but gradually diverge. The three upper rows ("node 2-tx", "node 3-rx" and "node 3-tx") show an increase in queuing after about 60 minutes in the measurement. Figure 12d shows this queuing effect. Only a small part of the flow in the first period can be transmitted. Because as the time difference increases, the frames between "node 2-rx" and "node 2-tx" must be queued.

[0157] 6.6 Interference and blocking delay

[0158] In this section, we analyze the impact of interference and blocking caused by other flows. In addition, we compare the measured results with the arrival windows calculated by the worst-case model. Figure 13 shows the three different settings used for this analysis. For each of the three scenarios, we model the cross traffic using IMIX. We inject cross traffic only on nodes 2 and 3 to have a stable reference characteristic on node 1. First, we discuss interference with flows of the same or higher priority. Next, we analyze the blocking delay on strict priority or frame priority.

[0159] 6.6.1 Interference delay

[0160] Figure 13a shows the input and output times on all nodes in the network for all frames. The thick green and blue bars show the forwarding deviation per second towards the third node. Figure 13b shows the same measurement process in light gray for a hop-by-hop view. In addition, the figure visualizes the transmission envelope calculated with our model in green and red. Since we inject all cross-traffic in an out-of-order and unsynchronized manner, the measured flows exploit the full transmission envelope. Using the knowledge of all flows, the model makes correct predictions in the worst case (see red line). In Appendix A.2, we show measurements with incorrect knowledge of flows with QoS requirements.

[0161] 6.6.2 Blocking Delay

[0162] To analyze the blocking delays (i.e., conflicts with traffic with lower priority), we start with the strict priority scenario in Figures 13c and 13b. We observe a regular delay of up to 12us per node, which indicates the blocking caused by the largest Ethernet frame. The evaluation shows that the worst-case model correctly predicts the blocking time in our scenario. Figures 13e and 13f show the same setup, but configured for frame preemption. The measured flows belong to the fast category, while the cross traffic is preemptive. We observe only small delays of up to 1us per node in Figure 13e. As shown in Table 1, this behavior is equivalent to the worst-case blocking caused during frame preemption. Therefore, the envelope in Figure 2 is thinner than in Figure 13f, and the jitter on the arrival of the frame preemption flow is also reduced.

[0163] 6.7 Changes in Network Cycle Time

[0164] In Section 5.3, we defined the increased bandwidth requirements for cycle time variation between two nodes. In addition, we derived the worst-case delay for flows with the TAS mechanism in Section 4.2.6. Therefore, in this section we evaluate two elements of our model. Figure 14 shows the measurement results in a topology and configuration scenario similar to that in Figure 7, where the cycle time of "Node 0" and "Node 1" is 1ms, while the cycle time of "Node 2" and "Node 3" is 5ms. Figure 14a visualizes the results using the send time as a reference and visualizes the delay of each flow for each next hop. Each value is the transmission time on each node and is therefore after the TAS gateway. Therefore, there is no significant delay on "Node 0" and "Node 1", only the delay that can be expected in the best case. The division into five different delay clusters on "Node 2" represents different buffered delay clusters. Only when "Node 2" forwards the frames of the measured flow every 5ms, it will receive every 1ms, while some frames will be forwarded directly (with some interference) and others will be buffered for 4ms. Referring to Equation 9 (measured based on synchronous topology), we can calculate the gateway delay for each of the five possibilities. As the worst case gateway delay we derive as follows:

[0165]

[0166] 7. Application scenario model

[0167] In this section, we present the application of the model defined in Section 5 to the entire network. As already described in Section 5.1, we do not consider the exact order of arrival times of flows at each node, since the model computes the amount of interference of flows. Therefore, we can compute the delay for each edge and each node separately. However, we add bandwidth to all flows whose arrival window is larger than the cycle time or have different cycle times between two nodes. Therefore, we have to recompute the delays of all nodes and edges associated with these bandwidths. Potentially, this recomputation leads to recursive recomputations. The upper bound on the number of recomputations is the number of ports in the network, since we have an acyclic directed graph. This upper bound is based on the assumption of acyclic and directed graphs, which are the basic principles of IEEE 802.1 Ethernet. Our model still has redundancy capabilities based on the ring protocol. In fact, even for every IEEE Ethernet redundancy, each flow will only pass through each port once.

[0168] We open-source our model implementation. The code in this repository implements the model introduced in this white paper and its application scenarios, which are described in the remainder of this section as example applications of the model. In the following sections, we present the four application scenarios of the model we specified in the introduction (i.e., end-to-end latency, overload detection, identifying inefficient transitions, and network debugging) with more details and examples. In Appendix A, we provide additional information on the specific uses of the open-source code.

[0169] 7.1 End-to-end delay / application requirements

[0170] The main motivation for the best-case and worst-case model is our first application case, namely computing the end-to-end delay and verifying application QoS requirements. To apply the model to this application, we run it on a given network configuration and a set of flows with QoS requirements. These flows may differ in terms of priority and QoS requirements. We compute the delay per hop for all flows in a first step. Next, we recompute all delays due to the increased bandwidth requirements of some flows (see Section 5.3). In the last step, we sum up all delays for each flow across all nodes and edges individually. The result is the best-case and worst-case end-to-end delay for each flow. These two values ​​are used to verify the application requirements (i.e., arrival time and maximum delay) in terms of arrival jitter. We provide an example of the calculations for this application case in Appendix A.1.

[0171] 7.2 Blockage / Bottleneck Analysis

[0172] In the second application scenario, we analyze bottlenecks in the topology and architecture. These bottlenecks can lead to link saturation and thus packet loss due to buffer overload. In Section 5.3, we introduced the application of increased bandwidth per flow for delay calculation. After calculating the best-case and worst-case delays, we compare the calculated bandwidth requirements with the available resources. For frame preemption and strict priority, the overall limit is the connection rate, while TAS artificially reduces the available bandwidth by priority. Finally, we calculate the worst-case scenario of link saturation in terms of cycle time and compare the required buffer size with the size available in the hardware. This results in an occupancy estimate for each edge in our topology. Appendix A.2 provides example calculations for this application scenario.

[0173] 7.3 Identifying Inefficient Domain Transitions

[0174] The model proposed in this paper is the basis for planning and optimization in TSN networks with multiple configurations and time synchronization. Our paper does not introduce algorithms for time planning or configuration optimization. However, the model identifies inefficient configurations and highlights the advantages of reconfiguration. The efficiency of the transition with the worst-case delay involves the potential introduced by this edge. In the first two application cases, we calculated all individual delays for each node and switch and all buffer requirements for each port. After that, we can evaluate each transition between two nodes based on efficiency. We provide example calculations on the ranking of efficient configurations in Appendix A.3.

[0175] 7.3.1 Network Debugging

[0176] Finally, we present the use case of network debugging. Finding unexpected delays in a large network is complex and time-consuming. Therefore, we consider network debugging as the main advantage of this model. However, when debugging, the quality depends on the devices used in the network, as these devices have different functionalities and functional elements for analysis. This section describes the debugging based on commercial industrial products of our industrial partners. Our research shows that other commercial devices also provide similar features in the industrial and IT fields.

[0177] In the first analysis, we compute all latency and connection requirements based on the topology, configuration, and flows used by our model. Next, we start debugging by comparing actual connection saturation with the computed saturation. Typically, network infrastructures aggregate saturation separately for each ingress port or even for each port per traffic class.

[0178] In the second analysis, we compare the calculated delay envelope with the actual characteristics in the network. Common network infrastructures have the ability to set up port mirroring. With this feature, we can copy all data traffic from one egress port of a forwarding node to a second port. Next, we obtain the traffic over several cycles and can analyze the characteristics of the periodic flows. Knowing the cycle time of each flow, we calculate the arrival time values ​​with the cycle time. This result allows us to generate a histogram with the deviation of each flow within the cycle. Finally, we compare the observed distribution with the calculated distribution. This analysis shows the accuracy of the verified flows.

[0179] We applied both debugging analysis methods to each egress port separately, which results in a linear increase in workload for each port added to the network. Since the first method is less complex and easier to automate, we used it as a preliminary check of performance in the network. Subsequently, we applied the acquisition method. On all ports we observed high saturation values ​​or flows with invalid QoS requirements. This simplified the identification of the cause of delays on these ports.

[0180] 8. Conclusion

[0181] IEEE TSN and IETF DetNet are a promising combination of mechanisms to create a common deterministic factory network. However, the use of different TSN mechanisms in TSN domains makes it difficult to determine QoS guarantees for TSN cross-domain traffic. Earlier work avoided this problem by assuming a homogeneous set of mechanisms for all domains and a unified time plan and time source. For example, all nodes would use a common time, configuration synchronization or a perfectly coordinated TAS time plan. However, in practice, this may not be possible for many cases because the combination of assembled machines and production lines is individually configured.

[0182] In this paper, we propose a model for computing the best-case and worst-case latency of heterogeneous industrial network architectures based on TSN. In particular, our model is able to handle the static TSN configuration of existing single machine networks and the lack of time synchronization between them. We also consider the jitter caused by the network infrastructure and is often ignored.

[0183] First, we analyze the impact of some TSN mechanisms on mixed time-critical and non-time-critical traffic in synchronous and asynchronous scenarios. Second, we use these individualized evaluations and build best-case and worst-case models. With these models, we consider the impact of different configurations across TSN network domains, such as changes in network cycle time and potential overload evaluation in worst-case scenarios. We highlight four specific application scenarios of the model in different network delivery and operation scenarios in typical TSN. Finally, we evaluate the applicability and accuracy of our model in a real test environment with industrial-grade real TSN hardware and TSN configurations, which is novel in the study. Our evaluation shows that the model is able to accurately predict the characteristics of real industrial TSN network devices. Therefore, with the help of this model, the achievable QoS guarantees between different TSN domains become computable and allow to determine whether they are sufficient for the operation of a specific industry, whether they are compliant, and whether they are reliable for the application.

[0184] AApplication scenarios

[0185] This section is an informal extension of the source model presented in this paper to illustrate the functionality of the various application scenarios we introduce to the public in Section 7.

[0186] In the examples in the appendix, we use the topology visualized in Figure 8. Unless otherwise stated, all devices in this topology are synchronized with each other.

[0187] A.1 End-to-end delay / application requirements

[0188] In this section, we present an application scenario for deriving the end-to-end delay and computing the arrival window at each node in the network.

[0189] We therefore compute all close arrival windows which are visualized in Section 6. Information about the best and worst case end-to-end delay enables the inspection of the QoS requirements of the application.

[0190] For our demonstration, we use the topology in Figure 8 and have set up the flows in Table 2.

[0191] The calculation of end-to-end delay starts with the following command:

[0192] python main.py a1 arrival_window__calculation

[0193]

[0194] A.2 Blockage / Bottleneck Analysis

[0195] In this section, we present an application scenario for identifying potential overloads inside a network. Generally speaking, congestion occurs when the traffic rate exceeds the connection speed. In addition, with the TAS mechanism, gateway entries artificially reduce the possible throughput on the link. For larger scenarios, it is still relatively simple to analyze the traffic paths and decide whether the frame size makes the setup work properly. However, as we introduced in Section 5.3, under different TSN configurations, multiple frames of the same flow need to be expected in the same cycle. Therefore, we created a scenario to analyze such an overload. For our demonstration, we use the topology as in Figure 8 and the flows set up in Table 2. However, we reduce the cycle time of flow 2 DC 7 until 330us. In our code base, this setup is presented after topology "a2". The analysis of possible congestion in the above scenario starts with the following command:

[0196] python main.py a2 congestion_identification

[0197]

[0198] To visualize the impact of congestion, we transferred this TSN configuration to the measurement artifact. Figure 15 shows the results.

[0199] A.3 Identifying Inefficient Transitions

[0200] In this section, we present an application scenario for identifying inefficient transitions between two adjacent TSN nodes based on their TSN configuration and the impact of other traffic with QoS requirements. For this purpose, during the calculation of the worst-case model, we track each worst-case delay delta between two nodes. Afterwards, we can sort these delay deltas in descending order and identify them as transitions with the greatest potential for improvement.

[0201] For our demonstration, we use the topology in Figure 8 and the flows set up in Table 2. Since this is a small example, we still keep it and follow the calculations manually. In the model, we have already noticed a possible high interference delay. In addition, we disabled the time synchronization between "node1" and "node2". Therefore, this analysis considers another significant impact on the worst-case latency. In our codebase, this setting is hidden under topology "a3".

[0202] Analysis of the inefficient transition in the above scenario begins with the following command:

[0203] python main.py a3 inefficient_transitions

[0204] The open source model output is as follows:

[0205]

[0206] The table shows a categorized view of the transitions based on their efficiency. The three "rx" latencies represent processing delays. In a large topology with different devices, these delays will not all be the same. The four "node 0" latencies visualize the delay at the output from the forwarding node, i.e. the queuing delay dWarteschlange. Obviously, the latency of "node 2-tx" is the highest, since the unsynchronized data traffic is forwarded in a TAS configuration. The worst-case latency of "node 3-tx" is also higher than that of "node 1-tx". This latency difference causes interference and therefore delays on "node 3", even though the two were previously synchronized on the node.

[0207] Graph label translation:

[0208] Figure 1: Example ICS network architecture

[0209] Figure 2: Possible settings for time synchronization: A) All synchronization, B) No synchronization between domains, and C) Communication across unsynchronized domains

[0210] Figure 3: Time-Aware Shaper (TAS): IEEE 802.1Qbv

[0211] Figure 4: Frame Preemption (FP): IEEE 802.1Qbu

[0212] Figure 5: Forwarding delay model

[0213] Table 1: Layer 2 transmission delay of the smallest IEEE Ethernet frame or smallest preemption frame fragment (64 bytes), the largest preemption frame fragment (127 bytes), the largest IEEE Ethernet frame (1522 bytes), the largest IP-Jumbo frame (9022 bytes), and the largest IP-Jumbogramm (65597 bytes) at different connection speeds

[0214] Figure 6: Visualization of the calculated arrival window of the proposed model for a virtual setting.

[0215] Figure 7: Cycle time changes from 1 ms on the left to 5 ms on the right

[0216] Figure 8: Evaluation setup; each node uses the TAS mechanism; the window from each node to other nodes is opened for 30us; then, the start of the TAS window in the path is shifted by 40us; TAS is open for priority levels 5, 6, and 7

[0217] Figure 9: Synchronous and asynchronous traces through the network, one view per packet; measurement duration: 60 minutes; 20 frames per second; 512 bytes per frame

[0218] (a) All nodes synchronize to the sender "Node 0"

[0219] (b) Only "Node 1" is synchronized to the sender; "Node 2" and "Node 3" are synchronized

[0220] Figure 10: Jitter analysis within a linear topology of four devices; each measurement is based on the device’s internal time and therefore includes jitter due to time synchronization

[0221] (a) Timestamps for all incoming data and outgoing ports

[0222] (b) End-to-end jitter: The jitter measured between “node 0” and all three nodes in the line

[0223] (c) Transaction jitter: The jitter measured between the ingress and egress of each of the three forwarders between nodes.

[0224] (d) Transmission jitter: The jitter measured between the output and input of two adjacent nodes.

[0225] Figure 11: Latency measurements in a linear topology with two different connection speeds; 100Mbit / s between “Node 1” and “Node 2”, and 1Gbit / s between “Node 0” and “Node 1” and between “Node 2” and “Node 3”

[0226] Figure 12: Synchronous and asynchronous traces through the network in a hop-by-hop view; GCL always opens the gateway 30 microseconds after the previous node; measurement duration: 300 minutes; 20 frames per second; 512 bytes per frame

[0227] (a) Synchronous tracking

[0228] (b) Asynchronous tracking

[0229] (c) Aggregate synchronization tracking based on network location

[0230] (d) Aggregate asynchronous tracking based on network location

[0231] Figure 13: Synchronous trace through a network with IMIX-based cross traffic; measurement duration: 30 minutes; 20 frames per second; 512 bytes per frame

[0232] (a) Single frame view with intersecting traffic of equal and higher priority.

[0233] (b) Hop-by-hop view of cross-traffic with equal and higher priority

[0234] (c) Single frame view of intersecting traffic with lower priority and strict priority

[0235] (d) Hop-by-hop view of cross-traffic with lower priority and strict priority

[0236] (e) Single frame view of interleaved traffic with lower priority and frame preemption

[0237] (f) Hop-by-hop view of cross-traffic with lower priority and frame preemption

[0238] Figure 14: In a synchronous topology, the TAS cycle time varies from 1 ms on “Node 0” and “Node 1” to 5 ms on “Node 2” and “Node 3”; measurement duration: 60 minutes; 20 frames per second; 512 bytes per frame

[0239] Table 2: Flow configuration for application scenario evaluation

[0240] Fig. 15: Synchronous trace through a network with IMIX-based cross-traffic, which causes congestion; measurement duration: 30 minutes.

Claims

1. A method for operating a network, wherein: A plurality of network devices, each with its own configuration, are connected to one another for data exchange and exchange data via these connections, wherein dynamic delays, i.e. jitters, are taken into account when determining the data transmission time, characterized in that the network is a time-sensitive network and the actual data transmission time from an initial network device to a target network device across the network devices is determined taking into account the dynamic delays, wherein time synchronization jitters and forwarding jitters are taken into account in the dynamic delays.

2. The method according to claim 1, characterized in that Each network device inserts a timestamp into the data frame at its ingress and egress ports.

3. The method according to claim 1 or 2, characterized in that Determine the theoretical time it takes for data to travel across each network device from the initial network device to the destination network device.

4. The method according to claim 3, characterized in that Compare the actual determined times with the theoretical determined times.

5. The method according to claim 4, characterized in that When the comparison result exceeds a predeterminable threshold value, the configuration of at least one network device is changed for the purpose of debugging between the initial network device and the target network device.

6. The method according to claim 5, characterized in that The configuration of the originating network device and / or the target network device is also changed.

7. The method according to claim 5 or 6, characterized in that The configuration of the at least one network device is changed until the comparison result no longer exceeds the predefinable threshold value.

Images