Analysing and modelling the jitter and delay behaviour of, in particular, mixed industrial time-sensitive networks

Abstract

The invention relates to a method for operating a network, wherein multiple network devices, each having their own configuration, are connected to one another for data exchange and exchange data via these connections, wherein dynamic delays (jitters) are taken into account in the determining of the time for the transmission of the data, wherein the network is a time-sensitive network and an actual time for the transmission of the data over network devices from a starting network device to a target network device is determined while taking the dynamic delays into account, wherein time synchronisation jitters and forwarding jitters are taken into account in the dynamic delays.

Description

[0001] Analyzing and modeling the jitter and delay behavior of mixed industrial time-sensitive networks

[0002] Description

[0003] The invention relates to a method for operating a network, wherein a plurality of network devices, each having its own configuration, are connected to one another for data exchange and exchange data via these connections, wherein dynamic delays (Jiter) are taken into account when determining the time for the transmission of the data, according to the features of the preamble of patent claim 1.

[0004] Such methods are already known. They will be discussed in more detail in the description below (particularly in section 1). Determining the time for data transmission while taking dynamic delays (jitter) into account has, in the prior art, already brought improvements in network performance. However, these methods cannot be applied to older, and in particular not to all mixed industrial networks, because not all network components, especially their network devices such as switches, hubs, or the like, are designed and suitable for this purpose. The resulting disadvantages will also be discussed in the description below (particularly in section 2).

[0005] The invention is therefore based on the object of improving such a known method with regard to its performance, especially with regard to the speed of the transmission time, and also to achieve a more general applicability.

[0006] This problem is solved by the features of patent claim 1.

[0007] According to the invention, the network is a time-sensitive network and an actual time for the transmission of data across the network devices from an originating network device to a destination network device is determined taking into account the dynamic delays, with time synchronization jitter and forwarding jitter being taken into account in the dynamic delays. As a result, the transmission behavior of a network, in particular the time for data transmission, can be analyzed theoretically after modeling the network behavior and / or after measuring the network behavior of a network in practice and, based on this, determined much more precisely than was possible in the prior art. For additional and more precise details, reference is made in particular to Section 4.3 (1st paragraph therein) in conjunction with Section 7.3.1 (2nd paragraph therein) of the description below.

[0008] In a specific development of the invention, each network device inserts a timestamp into a data frame on its ingress port and its egress port. For additional and more precise details, please refer in particular to Section 6.1, paragraph 1, of the description below.

[0009] In a further development of the invention, it is provided that a theoretical time for the transmission of data across the network devices from the source network device to the destination network device is determined. The theoretical time for data transmission can be determined mathematically when a network is planned without it first being constructed in practice using hardware components. The configuration of the entire network, parts of the network and / or its individual components (in particular its network devices) can be changed (modeled) and the resulting times, i.e. their change, for data transmission can be determined. Conclusions can be drawn from the determined change and measures can be derived as to how the overall configuration or individual configurations must be changed in order to improve (in particular accelerate) data transmission within the network.For additional and more precise information, please refer in particular to section 7.3.1 (2nd paragraph) of the description below.

[0010] In a further development of the invention, the actually determined time is compared with the theoretically determined time. From this comparison, it can be deduced whether a change made to the set configuration has led to an improvement (or possibly also to a deterioration) in the data transmission time. Accordingly, the configuration can be changed again and the time can be determined again. This can be done until a time has been determined that is sufficient or satisfactory for data transmission. After that, the theoretical planning of the network is complete, the network components are configured accordingly, and the network with these configured components is set up in practice.

[0011] For this purpose, a further development of the invention provides that if the comparison exceeds a predeterminable threshold, the configuration of at least one network device is changed for debugging purposes between the source network device and the target network device. Thus, the network behavior can be influenced by changing the respective configuration to improve (in the sense of shortening) the data transmission time. This is possible both when planning a network that does not yet exist in reality and for implemented, i.e., already established, networks.

[0012] According to the invention, it is conceivable that the method according to the invention is carried out only between two network devices that are connected to each other and exchange data via this connection. It is also conceivable that the method according to the invention is carried out across more than two network devices that are present in the topology of the network device. Thus, any parts of the network or a neighboring or subordinate network can be selected, whose data transmission time is determined theoretically and improved after modeling their configuration. The same applies to parts of a network that has already been set up and exists in reality.

[0013] In a further development of the invention, the configuration of the source network device and / or the destination network device is also changed. In this case, the behavior of an entire network is considered and its timing behavior is improved.

[0014] Finally, in a further development of the invention, the configuration of at least one network device is changed until the comparison no longer exceeds the predeterminable threshold value. Thus, after the configuration of a single network device, a group of network devices, or all network devices of the network under consideration has been changed (debugging), the timing of this network is optimized. The method according to the invention thus has the significant advantage that the timing of the network can be analyzed theoretically before its construction, but additionally or alternatively also after its construction in reality and commissioning, and can be optimized by changing configurations (modeling). This is a significant advantage, especially in mixed industrial time-sensitive networks, such as TSN networks.

[0015] With regard to further concrete steps of the method according to the invention, their implementation and the underlying technical approach (see in particular section 3 in this regard), reference is made to the following description.

[0016] SUMMARY

[0017] New industrial technologies such as edge computing and cloud-based control are transforming factory automation networks from isolated, purpose-built real-time networks to large, highly interconnected, multi-purpose networks. At the core of this development are real-time-capable factory backbone networks that connect different production lines and machines. IEEE Time-Sensitive Networking (TSN) and IETF Deterministic Networking (DetNet) offer a set of new network mechanisms for implementing both the factory backbone and the edge. These technologies promise to create ubiquitous, deterministic communication across an entire industrial plant.

[0018] In practice, however, different parts of the factory network are deployed, each with different TSN mechanisms to achieve their specific quality of service (QoS) requirements. This creates different TSN domains based on different approaches to achieving real-time traffic forwarding. Since each domain can provide a different level of real-time QoS, guarantees do not apply to traffic traversing a domain across its boundary. Therefore, it is difficult to predict delays and jitter for TSN when forwarding traffic between domains. Therefore, making reliable statements about the achievable QoS in larger factory networks is not exactly easy to assess.

[0019] In this article, we first define a model for time-critical communication across multiple TSN domains. Using this model, we are able to evaluate the QoS requirements of industrial applications that require deterministic communication across multiple TSN domains. Our evaluations with real-world measurements on industrial-grade TSN hardware demonstrate the feasibility and applicability of our model to predict the actual best-case and worst-case delays in such a mixed TSN network. This enables us to guarantee plant-wide QoS and helps identify bottlenecks that could compromise these guarantees.

[0020] 1. INTRODUCTION

[0021] Terms such as Industry 4.0, Smart Manufacturing, Edge Computing and Big Data require a highly interconnected, ubiquitous deterministic communication network.

[0022] However, factory networks often consist of isolated subnetworks that are more incompatible due to mutual manufacturer-specific communication technologies that require information exchange between gateways in these subnetworks. Furthermore, these factory subnets are installed once, statically configured, and not designed for dynamic reconfiguration. A key driver for this manufacturer-specific and static network architecture is to ensure the real-time requirements of the industry. Machine suppliers are liable for the continuous and secure operation of their machines after they have passed extensive certifications. Therefore, the transformation towards greater internal connectivity leads to this.

[0023] A factory network requires a time-deterministic network that enables reliable and secure communication for both time-critical and non-critical communication without reconfiguration. Finally, the industry is moving toward a unified communication standard with IEEE Time-Sensitive Networking, which is extensible with IEEE Ethernet for real-time capabilities. For networks spanning multiple Layer 2 networks, the IETF is working on the "Deterministic Networking" (DetNet) over TSN specification

[0032] to enable the routing of TSN traffic. Vendor-specific communication technologies are mutually incompatible due to specific and proprietary enhancements. Therefore, TSN must replace manufacturer-specific communication technologies within machine networks with a standard that enables seamless communication between all components.

[0024] IEEE TSN is a family of standards developed by the IEEE 802.1 group. It extends IEEE Ethernet with additional scheduling algorithms to enable hard real-time guarantees. Two of these schedulers fundamentally change how switches select frames for transmission: First, the Time-Aware Shaper (TAS) implements class-based time-division multiplexing (TDMA), which is not work-efficient due to blocking the transmission of low-priority traffic within certain reserved time slots. With sufficient time synchronization in the network, the TAS even enables a TDMA scheme at the frame or stream level. Second, the frame preemption, also called the (FP) mechanism, allows high-priority frames to be given priority over low-priority frames during transmission. The low-priority frames continue to be transmitted when the high-priority frames have completed their transmission.

[0025] Various approaches to calculating schedules for TDMA mechanisms, such as TAS, have been proposed in the literature, e.g., [13, 18, 20, 21, 31]. However, all these approaches assume that a unified network requires three elements:

[0026] (1) common time synchronization across all forwarding nodes,

[0027] (2) Using the same set of TSN mechanisms on all switches and

[0028] (3) consistent configuration of the TSN mechanisms, e.g., the same schedule on all switches

[0029] However, typical industrial networks are not uniform, as they are different, with vendors supplying parts of the network. For example, a factory includes machines from different manufacturers. Since machine vendors sell their machines in many different factories, but the

[0030] While a machine network itself is self-contained and uniform, uniformity across the entire factory is very unusual. Additionally, vendors optimize components within the machine for efficient operation. Therefore, supporting maximum configurations in each machine to guarantee uniformity across the factory is uneconomical. As a consequence, a large number of machines using TAS with high-precision time synchronization on FP without synchronization can have a wide range of heterogeneous network configurations. However, not all machines are completely independent. For example, robots in a

[0031] Production lines may require common time synchronization to enable collaborative work on a product.

[0032] In summary, factory networks are typically very heterogeneous environments and do not meet the homogeneity assumptions. Therefore, the above-mentioned approaches can be used within a single machine or groups of synchronized machines, but not across the entire network due to non-deterministic transitions between heterogeneous subnetworks. To enable real-time guarantees of a network in a heterogeneous factory, we provide a model that formalizes the transition between different subnetworks. This paper describes the influence of different TSN mechanisms and the lack of time synchronization by developing a model for worst-case guarantees. Therefore, Section 3 introduces the prerequisites for our work and gives a brief overview of TSN mechanisms. Our contribution is threefold. First, we will discuss the influence of different TSN mechanisms, heterogeneous TSN configurations (e.g.,We analyze network cycle times and vary link speeds in Section 4. Second, in Section 5, we define a best-case and worst-case model covering TSN and DetNet streams of multiple independent TSN network segments, in addition to calculating worst-case guarantees and end-to-end latency.

[0033] Our model enables the following use cases:

[0034] 1) Identification of costly configurations within the network,

[0035] 2) Validation of applications with communication across different TSN domains and

[0036] 3) Network debugging and error detection.

[0037] We open the code of our model to derive guarantees for arbitrary network configurations. We evaluate the model with measurements on commercially available industrial machines with TSN hardware in Section 6. The evaluation of a TSN worst-case model on commercially available hardware is novel. Our results demonstrate feasibility and applicability. Finally, Section 2 discusses related work, and Section 8 concludes our work.

[0038] Solutions for time-critical traffic exist in the Industry 4.0 and smart manufacturing scenarios. However, our gap analysis shows that technology alone is insufficient, as it is limited to a single machine. Every industrial scenario requires a solution to enable these market trends. In fact, industrial scenarios encompass networks with 100 to 1,000 switches and routers. Therefore, the problem we are addressing exists and is complex enough to require a tool-based analysis.

[0039] 2. RELATED WORK

[0040] In this section, we present related research results and discuss how our approach extends the state-of-the-art. In the context of real-time networking, there are several research areas: First, the research community on computing TAS schedules or TDMA schedules for networks in general is very active. We can further distinguish these into approaches based on pure scheduling and combined scheduling and routing. However, we consider these topics orthogonal because we focus our research on the transition between multiple heterogeneous networks that are already predefined. Therefore, schedule computation is beyond the scope of this paper, and for further details, we refer the reader to [11, 13, 14, 18-22, 29-31]. Second, we discuss various approaches to verify whether a network configuration, e.g., a computed schedule, meets the promised real-time guarantees.We discuss these approaches in detail, as they are highly important, and they relate to our approach, showing how we complement existing ones. Third, we consider Network Calculus (NC) as a complementary category, as it can be used for configuration verification and schedule design. We focus only on the use of NC in the context of TSN, as they form the basis for our approach. NC is a mathematical framework based on minus-plus algebra for proof, and we optimize it for delays for scenarios with a single TSN mechanism or multiple TSN mechanisms in combination.

[0041] 2.1 Configuration check

[0042] Schedule verification, and more generally, is a complex task and therefore usually found in scenarios with strict certification requirements, such as in the aviation industry. Numerous approaches and optimizations exist for Aviation Full-Duplex (AFDX) Ethernet networks and their configurations. Compared to our work, AFDX networks are comparable to individual TSN machine networks because they are self-contained and uniform. Therefore, modeling approaches for AFDX networks assume a uniform configuration and time synchronization [8-10]. The components of an aircraft are usually specifically built or adapted to be used in a certain combination and configuration. Thus, these networks are uniform, unlike factory networks, which consist of many components of different types and vendors, including standard products, so we cannot assume uniformity.Therefore, a.

[0043] Configuration verification for TSN networks has not yet been widely applied. Gou et al.

[0017] , Lv et al.

[0025] and Frimpong et al.

[0016] each present independent approaches for TSN-

[0044] Configuration verification. However, all of these approaches require common time synchronization across all nodes in the network, which is not necessarily the case in factory networks.

[0045] 2.2 Network Calculus (NC)

[0046] Network Calculus

[0024] is a mathematical framework for calculating upper delay and buffer bounds for a given network. Using minus-plus algebra, NC models can yield tight delay bounds. However, this precision comes at the expense of complexity. In contrast, our model uses a less sophisticated approach, and we prioritize reducing complexity over downsizing. Our evaluations (cf. Section 6) show a small discrepancy between the calculated worst-case delay and the actual measurements. Therefore, we assume that this reduction is acceptable in these scenarios.

[0047] In addition to complexity, another disadvantage of NC is that the models only work under the specific conditions for which they are designed. Examples of such models can be found in [12, 27, 33, 34]. As a concrete example, Zhao et al.

[0035] provides an NC model to calculate end-to-end delays for nodes that use exclusive TAS gates for the highest priority and the Credit-Based Shaper (as specified in IEEE 802.1Qav [1]) for other time-critical traffic. If the prioritization were changed, the model would lose its applicability. This limitation of flexibility when combining TSN

[0048] Mechanisms and the limitation of NC are known, as described by Maile et al. in

[0026] .

[0049] Since factory networks are very heterogeneous environments, the NC approach is inappropriate. Our approach combines flexibility with application in different scenarios, in contrast to the narrow limitations of NC. 3 BACKGROUND

[0050] The need for time-deterministic communication is constantly growing in many sectors and industries. However, to reduce complexity, we limit the scope of this presentation to industrial networks. Together with our industrial partner, we are able to validate the presented scenarios and assumptions. This section presents the prerequisites for discussing next-generation Industrial Control Systems (ICS) based on TSN networks. After introducing an example architecture of an ICS network, we continue this section with an introduction to TSN and DetNet. Finally, we specify the traffic types we use and discuss in this white paper in Section 3.2.

[0051] 3.1 Example ICS network architecture

[0052] Figure ! presents an example network architecture for next-generation ICS networks based on

[0028] . Industrial networks have a hierarchical structure for two reasons:

[0053] A) each network segment performs a dedicated task, and the above layers aggregate all networks with similar and related tasks,

[0054] B) for hierarchical security.

[0055] The structure introduces obvious interfaces to apply the concept of zones and lines [23J]. Machine networks with controllers and all sensors and actuators are located at the lowest level. Each machine has dedicated functionality and is designed and configured for the use case. Above this level are several aggregation levels. Figure 1 illustrates this starting from the production line and production backbone. Depending on the deployment size, the network includes additional layers such as a cellular network.

[0056] Typically, the provider of a network segment has full knowledge of all functions within that network segment, but knows little about all external traffic. Due to dynamic manufacturing and changes in the production process, this lack of knowledge will increase in the future. Today, machine networks are isolated to compensate for this lack of knowledge and unwanted influences from critical applications. In the future, the networks will be open to external traffic to enable scenarios introduced by Industry 4.0 (e.g., virtual industrial controllers). Therefore, time-critical traffic within the machine and time-critical traffic entering and leaving the machine require QoS protection. The industrial market is focusing on TSN technology to achieve guaranteed QoS. We cover the details of this technology in Section 3.4.

[0057] 3.2 Traffic classes

[0058] In an industrial environment, there are sometimes different types of traffic. Traffic is more event-based, other traffic is cyclical. In addition, industrial networks have traffic with strict QoS requirements, with less strict QoS requirements, or even without specific QoS requirements. In general, any combination of traffic types can occur. For our model, we do not focus specifically on one category of traffic. We build our model on the assumption that we know all traffic with all QoS requirements in our network. This requires that all traffic with QoS requirements has a higher priority than traffic without QoS requirements. Our model is not optimized for configurations for specific QoS requirements to be achieved and analyzes the achievable delays for a fixed configuration. Therefore, an accurate assignment of traffic types to priorities within the network (i.e.the Priority Code Point (PCP) value in the VLAN tag) is an input to the model and not limited to it.

[0059] 3.3 TSN domain

[0060] As introduced in Section 3.1, the vendor designs each network segment in an industrial network for a specific use case. The vendors of the various network segments create a fixed configuration for that specific purpose. Such a network segment with a consistent TSN configuration is referred to as a TSN domain. At this stage, the machine vendor only knows the traffic with QoS requirements related to the developed machine network. Therefore, the configuration of this TSN network includes not only resource reservations for all known traffic with QoS requirements, but also resource reservations for unknown traffic with QoS requirements.This results in resource reservations and protection of QoS requirements in a TSN configuration that uses and requires the same TSN mechanisms on all nodes within the TSN domain: all nodes therein for the ISN domain to be synchronized.

[0061] 3.4 TSN mechanisms

[0062] The technology, Time-Sensitive Networking, is specified by a set of standards developed within the TSN Task Group of IEEE 802.1. This set of standards enables standardized Ethernet networks. These provide time-deterministic transmission. Therefore, TSN is the critical implementation for all Industry 4.0 and other converged network scenarios. In this section, we introduce three required TSN mechanisms to meet the QoS requirements for the scenarios described.

[0063] Initially, this selection is based on the current standardization status for the industrial TSN profile IEC / IEEE 60802 {?].

[0064] We'll start with time synchronization, as it's the most important mechanism for supporting distributed and synchronized applications. We'll then describe two mechanisms for protecting time-critical traffic from the influence of lower-priority traffic: Time-Aware Shaper and Frame Preemption.

[0065] 3.4.1 Time synchronization: IEEE 1588 (PTP) and IEEE 802.1AS.

[0066] Synchronized applications require time synchronization between all end devices in the network. In the context of TSN, this is achieved using the Precision Time Protocol (PTP) defined in IEEE 1588 or its industry profile IEEE 802.1AS [6]. Both protocols use grandmaster concepts and distribute time with a constant delay between nodes to better compensate for errors. In PTP, the offset for time synchronization varies but can achieve an accuracy of 1ps over a distance of 60 nodes [7]. In Section 6, we discuss the effect of drifting clocks when they are not synchronized. We observe a drift between the two clocks of several microseconds within minutes. Obviously, TSN domains can be either directly synchronized with their neighboring domain or not. In addition, IEEE 802.1AS provides a time synchronization (TSN) feature.1AS provides the function for synchronizing with an external domain using so-called gPTP domains. Figure 2 shows these three possible scenarios for the use of time synchronization. In scenario A), two neighboring TSN domains are synchronized with each other, whereas in scenario B) they have a different understanding of time. In scenario C), the middle domain passes the time synchronization knowledge between the upper and lower network segments. Thus, both use the same time domain A to synchronize their applications with time domain B without interference. In scenarios A) and C), the virtualized PLC can take over synchronized control tasks.

[0067] 3.4.2 Time-Aware Shaper (TAS): IEEE 802. IQbv.

[0068] With the Time-Aware Shaper (TAS), the IEEE has introduced a mechanism to provide a TDMA-like mechanism for Ethernet networks. Frames are classified based on the priority code point (PCP) value of their VLAN tag and sorted into queues. The queues open and close according to a configured schedule for transmitting frames. The mechanism was originally introduced in IEEE 802.1Qbv [4] and is now integrated into IEEE 802.1Q (5).

[0069] Figure 3 visualizes the TAS mechanism. An egress port provides eight queues, one for each of the eight priorities of the POP field of the VLAN header. Thus, TSN supports up to eight traffic classes (TCs). Based on a repeating gate control list (GCL), each egress port transmits the TCs' frames in the time slots in which the port opens the queue. We refer to this open port for transmission as a TAS window. When multiple gates are open simultaneously, frames are selected by the strict priority mechanism, which selects the frame with the highest priority first. The priority list for a particular TAS window is defined as priowindow. We use this variable to analyze whether a TC uses a window exclusively or shares it with other priorities (i.e., exclusive gates and shared gates). For optimal operation, the TAS mechanism requires the neighboring nodes to be synchronized with each other.Otherwise, frames arrive with unknown timing, which in the worst case leads to forwarding delays. We discuss the direct impact of synchronized and unsynchronized nodes in Section 4.2.6.

[0070] 3.4.3 Frame Preemption (FP): IEEE 802.1Qbu.

[0071] The frame preemption mechanism allows frames of specific priorities (express priorities) to overtake any other frame outside the preemptive priorities (i.e., frames not in the express priorities) by temporarily pausing their transmission. This mechanism was originally introduced in IEEE 802.1Qbu [3] and is now integrated into IEEE 802.1Q [5]. Furthermore, frame preemption also requires a change in the physical layer of Ethernet, introduced by IEEE 802.3br [2].

[0072] Figure 4 shows two frames arriving at switch a with a time offset. Both streams have different Layer 2 priorities in the priority code point (PCP) value in the VLAN tag. The network is configured so that PCP 7 has the only express priority. The preemptive frame with PCP 6 arrives earlier and on a different port than the express frame with PCP 7.

[0073] Therefore, the switch starts transmitting the PCP 6 frame first. As soon as the PCP 7 frame is ready for transmission, the switch anticipates the PCP 6 frame. This mechanism can only accommodate frames with a size of 128 bytes and larger.

[0074] 3.5 Deterministic Networking (DetNet)

[0075] With reference to Section 3.1, industrial networks comprise multiple independent machine networks (i.e., TSN domains). These combinations of TSN domains can either form a large Layer 2 network or combine independent Layer 2 networks into a Layer 3 network. TSN is specified for Layer 2 and therefore handles all traffic within the Layer 2 domain. Deterministic Networking (DetNet) is the technology to enable time-critical traffic across multiple Layer 2 networks. DetNet comprises a set of standards defined by the IETF

[0015] , DetNet over TSN

[0032] specifies the transition between two TSN domains via a Layer 3 forwarding node. Most often, the specification requires deterministic

[0076] Forwarding delays and a stream for translation so that the TSN stream uses the correct Layer 2 addressing within the next TSN domain (e.g., correct VLAN tag, and source and destination addresses). We cover the similarities regarding the timing of Layer 2 and Layer 3 forwarding nodes in Section 4.2.

[0077] 4. SYSTEM MODEL

[0078] In this section, we present our system model to analyze the best-case and worst-case delays for TSN and DetNet networks. Therefore, we introduce a generic forwarding model that fits a TSN and DetNet node. Next, we introduce delays independent of the Transmission Selection Algorithm (TSA). After that, we discuss the TSA-dependent delay, i.e., the queueing delay, and evaluate its three different sub-delays. Finally, we discuss the sources and influences of delay on network behavior.

[0079] 4.1 Network Model To derive a formal representation for our delay model, we first use a simple network model. The network consists of any number of nodes v connected by edges e. To represent a full-duplex Ethernet network, we use a directed graph and always two connections between two nodes to model transmission in both directions and enable simultaneous transmission. Within this model, we do not distinguish between Layer 2 and Layer 3 networks, and we do not cover TSN domains. However, for our delay calculations, time synchronization is important. Therefore, each node v is assigned a specific time range. We do not use a formal representation here, so we can simply describe whether two nodes are adjacent and synchronized with each other or not.Finally, we are aware of the complete TSN configuration (i.e., TAS and FP) within the plant network. Within the network, there are any number of streams with QoS requirements. We denote the complete set of streams with QoS requirements as streams and the stream of interest as s. For each stream, we know its priority prios , its cycle time, and the frame size bs . Furthermore, this is similar for every configuration. For verification, we know all the paths the streams take through the network. Therefore, streamse defines the set of streams on an edge. A basic requirement for IEEE 802.1 Ethernet networks is to forward frames when the line is idle, which means that a node v should never be on the path of a stream s twice.

[0080] 4.2 Forwarding delay model

[0081] Figure 5 shows an abstract view of the forwarding, with all delays and times highlighted. This model considers TSN nodes forwarding Layer 2 frames and DetNet nodes forwarding Layer 3 packets. In the following sections, we discuss each of these delays and explain their effect within the worst-case delay model. For a flexible model that supports a mix of different network devices, each of these delays is defined and evaluated per node or edge.

[0082] 4.2.1 Propagation delay dprop

[0083] The propagation delay dprop defines the time it takes for the first bit to traverse the entire line. Therefore, this value depends on the line length and is static per edge e.

[0084] 4.2.2 0 transfer delay ngdtrans

[0085] The transfer time depends on the connection speed and the

[0086] Frame size. In this paper, we only refer to the frame size of Ethernet (i.e., Layer 2). However, when implementing the model, we also consider the additional 20 bytes introduced by the inter-frame gap, preamble, and initial frame delimiter. This is for simplicity only, since Layer 2 frame sizes are more typical in the context of TSN networks than Layer 1 or Layer 3 sizes. Table 1 shows a selection of transmission delays for a few bit sizes at different link speeds. This table is intended to provide a rough overview as a background to the dimensions of transmission delays, since the

[0087] Transmission delay is relevant for the interference delay as well as the blocking delay.

[0088] 4.2.3 Processing delay dproz

[0089] The processing delay dproc defines the time within the forwarding node to process the frame or packet. Both Layer 2 and Layer 3

[0090] Forwarding nodes have this type of delay. This value varies from node to node depending on its implementation, hardware vs. software, and the chips and software stacks used. However, we assume this value is static per node v.

[0091] 4.2.4 Interference delay dlnterference

[0092] A higher ranking in multiple domains on the same bridge potentially leads to more interfering streams. Depending on the configured QoS mechanisms, we can derive the worst-case interference delay dInterference as explained in this section. dInterference defines the interference with the same or higher priority frames on an egress port of a bridge. The set of streams that can interfere with the stream of interest s is a subset of all streams transmitted at the edge e: interferences^ c streamsg (1)

[0093] The interference delay depends on the connection speed e of the edge and is defined by the sum of all dtrans for the possible interferences, calculated in equation 1:

[0094] ^interference,., ~ ^rthScj, ® e^interference S:S

[0095] In the following, we derive the set inter fr rences.e based on QoS mechanisms on Edge e.

[0096] Strict Priority: With the strict priority mechanism, all frames from the stream of interest that have the same or higher priority and can potentially interfere are excluded. We model the subset of conflicting streams as follows: interferences,« — {c|ce streams« Aprio c > priority s} (3) Frame preemption

[0097] As briefly introduced in section 3.4.3, there are two categories of frames in Frame

[0098] Preemption: 1) Express frames, and 2) preemptive frames. Depending on the assignment of a stream to one of these two classes, the interference calculation is different. If the desired stream is part of the express category prloexpress, the stream can only interfere with other express streams. Strict priority is the mechanism used to decide between different streams in the express category. We model the subset of streams with possible interference as: interferences^ -= {c|ce streams,, A prio c e prio ex p ms

[0099] Aprioc > prio s ) (4)

[0100] If the stream of interest is not part of the explicit priorities prloexpress, it is part of the preemptive priorities priopreemptible. In this case, all express streams can preempt the stream of interest and therefore cause interference delay. Within the preemptive priorities, strict priority selects the next stream for transmission. Therefore, we model the subset of streams with possible interference as follows: interferences^ := (c|c € streams,. A fprio c e prio txp "ssV

[0101] (priOc &priOpreemptiblt ^prio c > priOs))} (5)

[0102] Time-aware designer. With the TAS mechanism, only the switch transmits a subset of priorities depending on the TAS configuration. Therefore, we only consider the subset of all streams with the highest priority in priowindow. Additional streams that can cause interference must be of the same or higher priority: interferences^ := {c|ce streams,. / \prio e £ prio vimiov

[0103] A price > priOs} W

[0104] 4.2.5 Lock delay dblck

[0105] The blocking delay defines the duration of a frame that blocks a stream s at the edge e, even though it has a lower priority. These delays depend on the TSN mechanisms configured on that node. Using strict priority as the transmission selection algorithm, Ethernet switches select the highest priority for the frame ready for transmission.

[0106] However, if the switch has just started transmitting a lower priority frame while the higher priority frame is ready to be sent, this will cause a delay for high priority traffic, possibly in the form of a maximum frame size. Therefore, IEEE

[0107] 802.1 introduced the two mechanisms TAS and Frame Preemption (FP) to reduce the impact on critical traffic.

[0108] 0 bytes TAS: exclusive gates

[0109] 1,522 tjrtes TAS: shared gates

[0110] 127 bytes EP: se express (?)

[0111] 1,522 bytes EP: s 6 preemptible

[0112] 1,522 bytes Strict Priority

[0113] Therefore, we represent the maximum blocking time dblck to denote a lower priority frame depending on the mechanism used in Equation 7. Within this formula, we assume a maximum frame size of 1,522 bytes, as this is the maximum in standard IEEE Ethernet networks. However, IT networks often use jumbo frames (e.g., for video data) with a size of 9,022 bytes or even jumbograms with a size of 65,597 bytes. Therefore, the size of 1,522 bytes in Equation 7 is a placeholder for the maximum frame size at the edge e.

[0114] 4.2.6 Gate delay dGate

[0115] The TAS mechanism keeps a frame in the output queue when the gate is closed, even if no other traffic is present. Based on the network model in Section 4.1, the GCL configuration is known on each node. The gate delay dgate defines the duration during which the node does not forward a stream because the gate for that stream is closed. If no TAS mechanism is configured, the gate delay is set to 0. For each priority, the transmission window in the GCL is open for the duration dwindowprio,e arn margin e; it opens at time topenprio,e and closes at tcloseprio,e . The cycle time CT defines the repeating patterns for the gate open and gate close events.

[0116] = CT ~ ^sdndawprto,, (®)

[0117] Equation 8 applies to the unsynchronized scenario, where we don't know the exact arrival times of streams. In the synchronized scenario, dgate depends on the time a frame is ready for transmission at tenq. A frame can either be too early in a cycle and wait for topping. Alternatively, a frame might not fit into the cycle, especially if it is too late for transmission or if there is too much interference within the remaining open window in which transmission is possible. en p™.« - (4nq M < fopen pr ", e ) (feose,., > (fenq pr(o e ,+ 4rans K> + <4ldt l f ,+ (9) ^rrterference E c ) ^CT) otherwise

[0118] It is important to note that Equation 9 does not directly assume forwarding of a frame to tenq. We mean that frames at the egress port can be delayed by traffic of equal and higher priority (cf. dlnterference in Equation 2). Increasingly, a lower-priority frame can block the stream of interest for the duration of dblck (cf. Equation 7).

[0119] 4.3 Jitter model

[0120] So far, the model includes static delays. However, Ethernet networks are not static at all. The biggest difference in behavior for synchronized networks is based on interference and blocking delays, as shown in Table 1 for the individual transmission delays. In addition, the gate delay also has a significant impact for unsynchronized scenarios. These two effects are discussed in Section 5.1. Within a TSN network, there are two other types of jitter, which we highlight here as follows: A) Time synchronization jitter and B) Forwarding jitter.

[0121] First, time synchronization between two nodes is not stable over time. Therefore, TAS ports across the network do not open in perfect cycles, but with small deviations. This type of jitter varies depending on the time synchronization mechanism. Protocols such as NTP have lower accuracy than those such as IEEE 1588 (PTP) or IEEE 802.1AS. The requirements document for the industrial ISN profile IEC / IEEE 60802 specifies a maximum jitter of 1 ps across 64 nodes in the network. Therefore, the current version of the IEC / IEEE 60802 specification requires the use of IEEE 802.1AS.

[0122] All components, such as the PHY or switching chip, have best-case and worst-case behavior. Depending on the quality of the hardware component and the software forwarding stack, forwarding jitter varies. Therefore, we introduce a jitter definition for each of these delays: propagation jitter: / prope , transmission jitter: ytranse , and processing jitter: jprocv .

[0123] In Section 6.3, we analyze jitter and discuss its distribution. Compared to the influence of noise delay and blocking delay, all jitter values ​​in this section are negligible, as are those in the range from a few nanoseconds to a microsecond, compared to a few microseconds to several milliseconds. However, we aim to obtain a complete model and thus present our implementation that covers all these types of jitter.

[0124] 5. BEST-AND-WORST-CASE DELAY MODEL This section builds the delay model for the complete factory network. Equipped with the complete topology, its configuration, and streams with QoS requirements in Section

[0125] As described in Section 4.1, the model generates a best-case and worst-case delay for each stream. This model does not restrict the TSN configuration and topology, but analyzes the individual delays per node and edge. The goal of this model is to calculate an expected arrival window for each stream on each node in the network. This arrival window is calculated as the difference between the best-case and worst-case delay. Figure 6 visualizes the increasing arrival window in gray over distance within the network. The topology is a simple line topology with best-case latency behavior, visualized here in green. The worst-case latency increased independently per traversal and is shown in red. The network is in the gray area within this traversal.At each node in the network, the difference between best-case and worst-case is determined, because a stream denotes the arrival window of that specific stream. Later, in Section 7, we use this arrival window for analysis to detect potential congestion in worst-case scenarios and identify inefficient TSN configurations.

[0126] We begin the presentation of the model with a basic model. We calculate the best-case and worst-case behavior in Section 5.1. We introduce the effects of changes in link speed into our model in Section 5.2. Finally, we add congestion control and detection to our model in Section 5.3.

[0127] 5.1 Basic delay model

[0128] The delay of a flow s on the forwarding node v and the edge e is defined as the sum of all delays shown in Figure 5: s>[) Vp fO pe + dtrans« + tfproc t , + dqueue s e (10)

[0129] The propagation and processing delays are static values ​​for the connecting or forwarding nodes. The transmission delay depends on the

[0130] connection speed and frame size.

[0131] To complete this basic model represented by Equation 10, we define the queueing delay as follows:

[0132] - möx f^a l e„- 41ck J ,,) + Vinierference i.t ( n )

[0133] If the TAS mechanism is configured on the egress port and the frame, it must wait for the gate to open; there can be no blockage due to delay. Therefore, we can always calculate the maximum of both values ​​and derive a correct model, as shown in Equation 11. In the best case, a stream does not interfere with another stream or get blocked by other traffic. Therefore, we set dinterferences,e and dblcks,e to 0. Likewise, if no TAS is configured, we set dgates,e to 0. Otherwise, we apply Equation 8 or Equation 9, depending on the synchronization state. Indeed, in the unsynchronized scenario, the gate delay dgates,e is set to 0 in the best case.

[0134] In addition, in the best case, we subtract the jitter for the difference between the delay components. This correction assumes an optimum behavior of all components on the forwarding path. The sum of all forwarding jitters is defined by:

[0135] Jprop e +jtrans, +jproc t

[0136] If the TAS mechanism is configured, we additionally assume that the gate delay dgate is reduced by / timesyncv. To calculate the worst-case behavior, we need to apply equations 2 and 7. Similar to the best case, we need to calculate the port delay dgate, but this time with the worst-case values ​​dlnterference and dblck. Additionally, we add the sum of the old forwarding jitter. If the TAS mechanism is configured, we increase dgate by / timesyncv. As a result, this model calculates the best-case delay and the worst-case delay. The arrival window dtriv denotes the difference between the two delays. Figure 6 shows a small example setup of this arrival window.

[0137] 5.2 Impact of connection speeds

[0138] In our delay model, we track the size of all streams in bytes bs . Furthermore, the topology model shows the link speed on each link. Therefore, the model is capable of handling any kind of changes in link speed across the entire network for industrial scenarios. This is a very important requirement, since sensors and actuators often only have 100 Mbps connections, but the backbone runs at, say, 10 GB / s. As shown in Table 1, these different link speeds result from a different transmission time dtrans and thus specific interference and blocking delays.

[0139] Figure 6 shows this change of connection speeds with the different inclination angles in the envelope.

[0140] 5.3 Influence of cycle times

[0141] Within a TSN network, there are two types of cycle times. On the one hand, there is the application cycle time, with which an application transmits cyclic data. On the other hand, in the TAS mechanism, the GCL operates with a given network cycle time. This time, in combination with the frame size, defines the bandwidth required by the application. The network cycle time, in combination with the GCL window duration, defines the time for possible throughput. If the forwarding node does not use TAS, the throughput is limited by the link speed. Even if no TAS is configured and we do not have a network cycle time, we still have an application cycle time.

[0142] Based on our assumption, each TSN domain has a static and individual configuration, so we cannot assume that the cycle time and the application's network cycle time are the same. Such changes in cycle time cause either more or fewer frames of a particular stream within a cycle. Figure 7 shows an example of the change in network cycle time from one node to the second. The first node (left) has a cycle time of 1 ms and the second node of 5 ms. Therefore, five frames of the same stream from the first node will always be present within the next cycle. To avoid congestion within the network, the configuration on the second node must be able to transmit all five frames. The same applies to the end device with an application cycle time when transmitting in a network with a TAS configuration.Similarly, the arrival window darriv of a stream may be larger than the cycle time on the next node, resulting in multiple frames for a stream within the cycle.

[0143] To meet the demand for increased bandwidth per stream, we calculate three factors / , which we use to increase the reserved bandwidth. First, we cover the increasing arrival window darriv and calculate / arrival. We calculate the ratio between the arrival window and the new clock time CTv . Next, we round this result to always allow the entire stream to be in one of the possible cycles:

[0144] Etriv = r^ärriv / ^Tol

[0145] Second, the change in cycle times is similar and is defined by / CT. If the cycle time on the second TSN node increases (i.e., it operates slower), several original cycles are completed before the cycle continues and the second node completes. Therefore, the second TSN node needs time to process more frames of the same stream than between the two cycle times, depending on the ratio:

[0146] In the scenario with decreasing cycle time on the second TSN node (i.e., operating faster), we must assume the stream's frames are present every cycle, since we are not modeling a stream where they are supposed to be present, for example, every other cycle. Therefore, this has no impact on bandwidth reservation.

[0147] Finally, we calculate the factor based on the difference between application cycle time and network cycle time / app. Similar to the previous scenario, we only cover the case where the network is cyclic, so time CTnet is slower than the application cycle time CTapp:

[0148] App ~ rCInrt / CTäppi

[0149] With these three values, the model increases the expected bandwidth in the worst case with: bs,i> — As.o— 1 • fmiv ' Jbt

[0150] And at the first TAS-configured node along the current path s, the model increases the

[0151] Bandwidth with:

[0152] ~ ' fapp

[0153] 6. EVALUATION

[0154] In the evaluation, we analyze various network scenarios and discuss the observed behavior. The goal of the evaluation is to highlight the applicability of our model, ensuring it covers a diverse mix of TSN configurations. Furthermore, the evaluation demonstrates the feasibility of the model with a simple implementation. We structure our evaluation as follows. First, we present our measurement setup and methodology. Second, we evaluate this, particularly the effect of drift between unsynchronized clocks. Third, we analyze the capabilities of our measurement setup, i.e., the jitter of real TSN devices. After that, we focus on specific elements in our model, such as synchronized vs. unsynchronized behavior and interference and locking delays due to cross-traffic. Next, we address changes in network cycle times.

[0155] 6.1 Setup and methodology

[0156] We evaluate everything with measurements on commercially available TSN hardware. For all measurements, we use a line topology with four devices: a transmitter ("Node 0") and three forwarding nodes. Each of these devices is capable of inserting a hardware timestamp into the frame on the node's ingress and egress ports. This allows us to track each frame across the network and analyze its behavior based on packet captures.

[0157] Figure 8 shows the topology and three different paths for the data traffic. For each measurement, we have measurement traffic with 500-byte frames from "Node 0" to "Node 3." Additionally, we can realize cross-traffic on paths two and three. This cross-traffic is implemented as an Internet Mix (IMIX) and consists of three streams with frame sizes of 64 bytes, 570 bytes, and 1522 bytes. All diagrams show only existing measurement traffic and no cross-traffic. The duration of each measurement ranges from 25 minutes to five hours to demonstrate the stability of the results with respect to clock drift and jitter.

[0158] Generally, we have two forms of visualization. First, we plot the measurement curve over time. This visualization represents the trend over time, which can be the effect of drifting clocks (continuously increasing and decreasing values), jitter (when there are spikes), or stability (horizontal lines). Second, we present the measurements at one port per node base. In this form of visualization, we create a visualization that is compatible with the arrival window. An important note for all measurements in this section is that the timestamps are based on the internal time of each node. The nodes are synchronized via IEEE 802.1AS. Therefore, the results contain an inaccuracy of a few nanoseconds due to this synchronization.

[0159] 6.2 Time synchronization

[0160] Figure 9 visualizes the impact of unsynchronized nodes in the network. The y-axis represents the offset to the transmitter in microseconds, and the x-axis represents the measurement duration. The four rows represent one measurement point per device in the topology. In Figure 9a, all nodes within the network are synchronized with IEEE 802.1AS, and we observe stable behavior. The time synchronization accuracy is within a few nanoseconds. Therefore, we observe no deviation during the measurement, and the measured difference can be related to the frame forwarding time. In particular, the transmission time of a 500-byte frame over a 1 Gbit / s link takes approximately 4 ps, and the switches have a processing delay between 1 and 2 ps. In comparison, in Figure 9b, only "Node 1" is synchronized with the transmitter, and "Node 2" is synchronized with "Node 3."We observe that the time base of the last two devices drifts away from the first two. However, the timing behavior between them is the same, and each "Node 0" and "Node 1" and "Node 2" and "Node 3" remains stable. This scenario is the behavior that two unsynchronized TSN domains will exhibit.

[0161] 6.3 Jitter

[0162] In applications where time-critical conditions exist, low jitter is crucial for correct network behavior. High jitter causes varying behavior from cycle to cycle, e.g., interference with other traffic or in the calculations on the controllers and devices. Therefore, our model includes the jitter components within the network and calculates the best-case and worst-case behavior. Figure 10 visualizes the measurements with one measurement at "Node 0" and two measurements at the other three nodes. Figure 10a visualizes each of the inserted timestamps everywhere as a measurement run with the "Node 0" timestamp as a reference. In fact, Figure 10a uses the same measurement as Figure 9a and adds the view of ingress and egress timings. This diagram clearly demonstrates stable operation without interfering with other network traffic.However, you can see that the lines at the top are thicker than those at the bottom, which indicates jitter. To analyze the jitter within the forwarding process, Figures 10b to 10d show detailed views.

[0163] First, Figure 10b shows the jitter for the total end-to-end latency. We subtract the timestamp "node 0" from the output timestamp of each node and calculate the jitter within it from the resulting series of delays. We see that the jitter increases with increasing distance (i.e., "node 1" is closer to "node 0" than to "node 3"). Furthermore, we observe a wider distribution of jitter at longer distances in the network. Second, Figure 10c visualizes the jitter of the processing time. To calculate the dataset, we subtract the input timestamp of each node from the output timestamp of the same nodes. The distribution is the same on each node because they use the same hardware. Furthermore, the jitter depends on the processing time and is in the range of only 10 ns. Third, Figure 10d shows the difference between the output and input of the two neighboring nodes.This diagram represents the jitter in the transmission delay. Again, its distribution is similar across all links and lies in the range of 30 ns.

[0164] 6.4 Connection speeds

[0165] A trivial component within this model is the transmission time of each stream per edge. Figure 11 shows the delays between four nodes in a line topology. In this scenario, the link speed between "Node 0" and "Node 1" and between "Node 2" and "Node 3" is 1 Gbit / s, and only 100 Mbps for the link between "Node 1" and "Node 2." The delays are very stable over the 25-minute measurement interval, as shown in Figure 11a. Figure 11b shows the precise measurement in a per-hop view. This figure clearly shows the reduced link speed between "Node 1" and "Node 2" with the steeper slope and thus the increased delay. The visualized timestamps have their reference point on the output side of each node.Therefore, each delay upstream of the node includes the propagation delay of the link (the same applies to each link), the transmission delay for a 500-byte frame (which depends on the link speed), and the processing delay (which is constant and a static value of 1 ps for all nodes). Based on these components, the model presented in Section 5 is obtained, and in the best case, the delay is 4.1 ps for the 1 Gbit / s links and 41 ps for the 100 Mbps link. There is no traffic other than the measurement of data traffic within this configuration. The worst-case delay derived from the model only requires the addition of all jitter components, as presented in Section 4.3 and evaluated in Section 6.3.

[0166] 6.5 Time-conscious shaper

[0167] In this section, we present the evaluation of the TAS's behavior. Figure 12 shows the results used for discussions in this section. We used the same TAS configuration for both the synchronized and unsynchronized measurements. The "Node 0" node sends a frame at the beginning of the interval, and each of the following three nodes opens its gate for measurement traffic for 40 ps. At each node, we shift the gate opening time by 30 ps. The measurement stream is only 500 bytes in size, the stream takes the first 4 ps, and at the next node, approximately (30-5) ps (including other delays such as processing delay). For a detailed analysis, all numbers show the reception time (i.e., rx) and transmission time (i.e., tx) of each forwarding node.

[0168] In the synchronized scenario (see the left two figures in Figure 12), all four nodes in the network are synchronized with each other. Figure 12a shows this static behavior with a latency of 25 ps (difference between a node's rx and tx values) throughout the entire measurement period. Figure 12c visualizes the same measurements on a per-node basis, again covering two timestamps per forwarding node. This figure demonstrates the consistency of latency behavior over time, as all streams use the same time period for transmission across the network. In the unsynchronized scenario, nodes "Node 0" and "Node 1" are synchronized with each other, and "Node 2" and "Node 3" are synchronized with each other, but "Node 1" is not synchronized with "Node 2." Figure 12b uses a different scale for the envelope than the synchronized figure, because the transmission increases by more than the cycle time.The red line ("node 2 - rx") visualizes the drift between the two time domains. At the beginning of the measurement, both time domains have similar timing behavior and drift apart from each other. The three upper time domains ("node 2 - tx", "node 3 - rx", and "node 3 - tx") indicate that increased queuing occurs after approximately 60 minutes into the measurement. This queuing effect is clearly visible in Figure 12d. Only a small subset of streams can be transmitted within the first cycle. This is because with the increased time difference, frames must be queued between "node 2 - rx" and "node 2 - tx".

[0169] 6.6 Interference and blocking delay

[0170] In this section, we analyze the effects of interference and blocking by other streams. Additionally, we compare the measured results with the arrival windows calculated by the worst-case model. Figure 13 shows three different settings for this analysis. For each of these three scenarios, we model the cross-traffic using the IM IX. We inject the cross-traffic only on nodes two and three to ensure stable reference behavior on the first node. First, we discuss interference with streams of equal and higher priority. Furthermore, we analyze the blocking delay for either strict priority or frame preference.

[0171] 6.6.1 Interference delay

[0172] Figure 13a shows the input and output times on all nodes in the network for all frames. The thick green and blue bars indicate the deviation in forwarding to the third node to the second. Figure 13b shows the same measurement sequence from a prehop view in light gray. Furthermore, this figure also visualizes the transmission envelope calculated with our model in green and red. Since we inject all cross-traffic unsorted and unsynchronized, the measurement stream uses the entire transmission envelope. However, with knowledge of all streams, the model correctly predicts the worst-case scenario (see red line). In Appendix A.2, we present measurements with incorrect knowledge about the streams with QoS requirements.

[0173] 6.6.2 Locking delay

[0174] To analyze the blocking delay (i.e., collision with lower-priority traffic), we start with the strict priority scenario in Figure 13c and Figure 13b. We observe regular delays with a maximum of 12 ps per node, indicating blocking by a maximum Ethernet frame. The evaluation shows that the worst-case model correctly predicts the blocking time in our scenario. Figures 13e and 13f illustrate exactly the same setup, but configured with frame preemption. The measurement stream is in the express category, and the cross-traffic is preemptive. We observe only small delays of up to 1 ps per node in Figure 13e. As shown in Table 1, this behavior corresponds to the blocking caused during frame preemption in the worst case. Therefore, the envelope in Figure 2 of Figure 13f is considerably thinner, and the stream arrival jitter is also reduced with frame preemption.

[0175] 6.7 Changes to the network cycle time

[0176] In Section 5.3, we define the increased bandwidth requirements for changes in cycle times between two nodes. Furthermore, in Section 4.2.6, we derive the worst-case delay for streams with the TAS mechanism. Therefore, we evaluate these two elements of our models in this section. Figure 14 shows the measurements within a topology and in a configuration scenario similar to the one shown in Figure 7. Nodes "Node 0" and "Node 1" have a cycle time of 1 ms, while nodes "Node 2" and "Node 3" have a cycle time of 5 ms. Figure 14a visualizes the result with the transmission time as a reference and visualizes the delay of each stream for each of the next hops. Each of the values ​​represents the transmission time at each node and therefore after the TAS gate. Thus, at nodes "Node 0" and "Node 1," no significant delay is visible, only the expected best-case delays.The division into five different delay clusters on "node 2" represents different clusters of buffer delays. However, only "node 2" forwards the frames for the measurement stream every 5 ms, receives them every 1 ms, some frames are forwarded directly (with some interference), and some frames are buffered for 4 ms. Referring to Equation 9 (the measurement is based on a synchronized topology), we can calculate the gate delay for each of the five possibilities. We derive the worst-case gate delay:

[0177] ~ = 0 ms + 5 ms - 1 ms = 4 ms?

[0178] 7. USE CASES FOR THE MODEL

[0179] In this section, we present the use of the model defined in Section 5 within a complete network. As already described in Section 5.1, we do not cover the exact order of streams per node from their arrival times, since the model calculates the amount of interference between the streams. Therefore, we can calculate the delays for each edge and node individually. However, we increase the bandwidth for all streams with an arrival window larger than the cycle time or different cycle times between two nodes. Therefore, we must recalculate the delays for all nodes and edges to which this bandwidth belongs. Potentially, this recalculation causes recursive recalculations. The upper limit for the number of recalculations is the number of ports in the network, since we have a directed graph without loops.This upper bound is based on the loop-free and directed graph assumptions, which are fundamental principles of IEEE 802.1 Ethernet. Our model is still redundancy-aware, based on loop protocols. In fact, even for any IEEE 802.1 Ethernet redundancy, each stream leaves each port only once. We open the implementation of our model. The code in this repository implements the model introduced in this white paper and the use cases described in the remainder of this section as exemplary usage of the model. In the following sections, we introduce the four use cases for the model we stated in the introduction (i.e., end-to-end latency, congestion detection, identification of inefficient transitions, and network debugging), with further details and examples. In Appendix A, we provide additional information on the specific usage of the open source code.

[0180] 7.1 End-to-end latency / application

[0181] Requirements

[0182] The main motivation for the best-case and worst-case models is our first use case, the calculation of end-to-end latencies and verification of application QoS requirements. To apply the model to this use case, we run it with a given network configuration and a set of streams with QoS requirements. These streams can differ in terms of priorities and QoS requirements. We calculate the per-hop latencies for old streams in the first step. Next, we recalculate all delays subject to increased bandwidth requirements for some streams (see Section 5.3). In the final step, we add all delays individually across all nodes and edges for each stream. The result is a best-case and a worst-case end-to-end delay per stream.

[0183] These two values ​​are used to validate the application's arrival jitter requirement (i.e., arrival time and maximum delay). We present an example of this calculation for this use case in Appendix A1.

[0184] 7.2 Analysis of traffic jams / bottlenecks

[0185] In the second use case, we analyze bottlenecks in the topology and design. Such bottlenecks can cause links to become saturated, resulting in packet loss due to overloaded buffers. In Section 5.3, we introduced the increased bandwidth per stream for use in delay calculations. After calculating the best-case and worst-case delays, we compare the calculated bandwidth requirements with the available resources. For frame preemption and strict priority, the overall limit is the link speed, and TAS artificially reduces the available bandwidth per priority. Finally, we calculate the worst-case saturation of a link based on the cycle time and compare the required buffer sizes with the sizes available in the hardware. This results in an occupancy evaluation per edge in our topology. Appendix A.2 presents an example calculation for this use case.

[0186] 7.3 Identification of inefficient domain transitions

[0187] The model presented in this paper is the basis for scheduling and optimization within TSN networks with diverse configurations and time synchronization. Our paper does not introduce an algorithm or configuration optimization for scheduling. However, the model identifies inefficient configurations and highlights the benefits of reconfiguration. The efficiency of a transition refers to the worst-case delay potential introduced on that edge. In the first two use cases, we calculate all individual delays per node and switch and all buffer requirements per port. After that, we can determine each transition between two nodes based on efficiency. We provide an example calculation for ranking the efficient configuration in Appendix A.3.

[0188] 7.3.1 Network debugging

[0189] Finally, we present the use case of network debugging. Searching for unexpected delays in large networks is complex and time-consuming. Therefore, we see network debugging as the main advantage of this model. However, the quality of debugging depends on the devices used in the network, as they have different functions and functional elements for analysis. This section describes debugging based on commercially available industrial products from our industrial partner. Our research shows that other commercially available devices in the industrial and IT sectors also offer similar features.

[0190] For an initial analysis, we calculate all delays and connection requirements based on the topology, configuration, and streams using our model. Next, we start debugging by comparing the actual connection saturation with the calculated saturation. Typically, the network infrastructure aggregates saturation either for each ingress port or even for each port per traffic class individually.

[0191] In a second analysis, we compare the calculated delay envelope with the actual behavior in the network. Common network infrastructure has the ability to set up port mirroring. This feature allows us to copy all traffic from one egress port of the forwarding nodes to a second port. Next, we capture traffic for a few cycles and can analyze the behavior of cyclic streams. Knowing the cycle time for each stream, we calculate the arrival time value using the cycle time. This result allows us to generate a histogram with the offset for each stream within the cycle. Finally, we compare the observed distribution with the calculated distribution. This analysis shows the degree of accuracy for the tested stream.

[0192] We apply both debugging analysis methods to each egress port individually, resulting in a linear increase in effort for each additional port in the network. Since the first method is less complex and easier to automate, we use it as the initial check of network performance. Later, we apply the capturing method. On all ports, we observe high saturation values ​​or streams with failed QoS requests. This simplifies the identification of causes of delays on these ports.

[0193] 8. CONCLUSION IEEE TSN and IETF DetNet represent a promising combination of mechanisms for creating a common deterministic factory network. However, the use of different TSN mechanisms in TSN domains makes determining QoS guarantees for TSN interdomain traffic difficult. Previous work has avoided this problem by assuming a homogeneous set of mechanisms and a uniform schedule and time source for all domains. For example, all nodes would use a common time, synchronization, or perfectly coordinated TSN schedules could be configured. In practice, however, this cannot be assumed for many cases because the combination of machines and lines being assembled is individually configured.In this paper, we present a model for calculating best-case and worst-case delays for heterogeneous industrial network architectures based on TSN. In particular, our model is capable of handling existing static TSN configurations of individual machine networks and the lack of time synchronization between them. We also consider the jitter caused by the network infrastructure, which is often neglected.

[0194] First, we analyze the impact of individual TSN mechanisms for mixed time-critical and non-critical traffic in synchronized and unsynchronized scenarios. Second, we use these individual evaluations to construct a best-case and a worst-case model. With these models, we cover the influences of different configurations across TSN network domains, such as changes in network cycle times and potential congestion assessments in worst-case scenarios. We highlight four specific use cases of this model with different network deployment and operation scenarios in typical TSN. Finally, we evaluate the applicability and accuracy of our model in a real-world testbed using actual industrial-grade TSN hardware and TSN configuration, which is unprecedented in research. Our evaluation demonstrates that the model can accurately predict the behavior of real industrial TSN network devices.The model makes it possible to calculate the achievable QoS guarantees across different TSN domains and to determine whether they are sufficient and acceptable for the operation of a particular industry and whether they are reliable for the applications.

[0195] A APPLICATION OF USE CASES

[0196] This section is an informal extension of the source model presented in this paper to explain the functionality of the various use cases that we presented to the public in Section 7.

[0197] In the examples in the appendix, we use the topology visualized in Figure 8. Unless otherwise specified, all devices in this topology are synchronized with each other.

[0198] Al end-to-end latency / application

[0199] Requirements

[0200] In this section, we introduce the use case for deriving the end-to-end delays and calculate the arrival window at each node in the network.

[0201] With this, we have calculated all the approximate arrival windows visualized in Section 6. The information about the best-case and worst-case end-to-end delays allows us to verify the application's QoS requirements.

[0202] For our demonstration, we use the topology in Figure 8 and have set up the streams in Table 2.

[0203] The calculation of end-to-end delays begins with the following

[0204] Command: python main , py al arrival^ window_calcul at ion

[0205] | port | best-case | worst-case |

[0206] [ I [ns ] | [ns ] |

[0207] | node 0-tx | 4135 | 17511 |

[0208] [ node 1 -rx | 5105 | 18541 |

[0209] | node 1 - tx | 44165 | 56511 |

[0210] | node 2- rx [ 45135 | 57541 |

[0211] | node 2- tx | 84165 | 114297 |

[0212] 1 node 3- rx | 85135 [ 115327 |

[0213] •| node 3- tx | 124165 | 1 54297 | A.2 Analyze traffic jams / bottlenecks

[0214] In this section, we present a use case to identify potential congestion within the network. Generally, congestion occurs when traffic rates exceed the link speed. Additionally, with the TAS mechanism, gate entries artificially reduce the potential throughput on a link. For larger scenarios, it is still easy to analyze the traffic paths and decide on the frame sizes to determine whether this setup can work. However, as we presented in Section 5.3, with different TSN configurations, multiple frames of the same stream can be expected in the same cycle. Therefore, we created a scenario to analyze this congestion. For our demonstration, we use the topology in Figure 8 and the stream configured in Table 2. However, we reduce the cycle times for streams 2 through 7 to 330 ps.In our repository, this setup is shown behind the topology "a2". The analysis of possible congestion in the scenario described above begins with the following command: python main . py a2 conge s tion _identification.

[0215] I Port | occupancy [%] |

[0216] ] node 0- tx | 1 |

[0217] | node 1 - tx | 1 |

[0218] | node 2 - tx | 233 |

[0219] ] node 3- tx | 233 ]

[0220] To visualize the effects of congestion, we transferred this TSN configuration to the test setup. Figure 15 shows the results.

[0221] A.3 Identification of inefficient transitions

[0222] In this section, we present the use case of identifying inefficient transitions between two neighboring TSN nodes based on their TSN configuration and the influence of other traffic with QoS requirements. To this end, we track each worst-case delay increment between two nodes during the worst-case model calculation. Afterwards, we can arrange these delay increments in descending order and identify them as the transitions with the greatest potential for improvement.

[0223] For our demonstration, we use the topology in Figure 8 and the current set in Table 2. Since this is a small example, we'll keep it for now and track the calculations manually. In the model, we already notice a potentially high interference delay. Additionally, we disable time synchronization between "Node 1" and "Node 2." Therefore, the analysis reveals another major influence on the worst-case delay. In our repository, this setup is hidden behind topology "a3."

[0224] The analysis of inefficient transitions for the scenario described above begins with the following command: python main . py a3 ine ffi ci en t_ tran siti ons

[0225] The open source model outputs the following:

[0226] | tran siti on | delay[ns] |

[0227] | node2-tx | 1004357 |

[0228] | node 3-tx | 91369 |

[0229] 1 node 1 - tx | 37970 |

[0230] 1 node 0- tx | 1751 1 |

[0231] | node 1 - rx | 1030 |

[0232] | node 2- rx | 1030 [

[0233] | node 3-rx | 1030 |

[0234] This table shows a sorted view of the transitions based on their efficiency. The three "rx" delays denote the processing delay. In a larger topology with different devices, these will not all be the same. The four "node O" delays visualize the delays at the output from a forwarding node, i.e.

[0235] Queue delay dQueue. Obviously, "Node 2-tx" has the highest delay because unsynchronized traffic is forwarded with a TAS configuration. "Node 3-tx" also has a higher worst-case delay compared to "Node 1-tx." This difference in delays leads to interference and thus delays on "Node 3," even though both are previously synchronized to the nodes. Translation of the labels in the figures:

[0236] Figure 1: Example ICS network architecture

[0237] Figure 2: Possible setups for time synchronization: A) old synchronized, B) no

[0238] Synchronization between domains and C) Communication across a non-synchronized domain

[0239] Figure 3: Time-Aware Shaper (TAS): IEEE 802.1Qbv

[0240] Figure 4: Frame Preemption (FP): IEEE 802.lQ.bu

[0241] Figure 5: Forwarding delay model

[0242] Table 1: Layer 2 transmission delay for a minimum lEEE Ethernet frame or minimum preemptive frame fragment (64 bytes), maximum preemptive frame fragment (127 bytes), a maximum lEEE Ethernet frame (1,522 bytes), a maximum IP jumbo frame (9,022 bytes) and a

[0243] Maximum IP jumbogram (65,597 bytes) at different connection speeds

[0244] Figure 6: Visualization of the calculated arrival window of the presented model for a fictitious setup

[0245] Figure 7: Change of cycle time from 1 ms left to 5 ms right

[0246] Figure 8: Evaluation setup; each node uses the TAS mechanism; the window from each node to the other is opened for 30 ps; next, the start of the TAS window in the path is shifted by 40 ps; the TAS is open for priorities 5, 6, and 7

[0247] Figure 9: Synchronized and non-synchronized traces through the network in a view per packet; measurement duration: 60 minutes; 20 frames per second; 512 bytes per frame

[0248] (a) All nodes are synchronized to the sender "Node 0"

[0249] (b) Only "Node 1" is synchronized to the sender; "Node 2" is synchronized with "Node 3" Figure 10: Jitter analysis within a line topology of four devices; each measurement is based on the internal time of the devices and therefore includes the jitter of the time synchronization yTime synchronization

[0250] (a) Timestamps for all incoming data and egress ports

[0251] (b) End-to-end jitter: Jitter measured between "node 0" and all three nodes in the line

[0252] (c) Processing jitter: Jitter measured between ingress and egress of each of the three forwarding routes between the nodes

[0253] (d) Transmission jitter: Jitter measured between output and ingress of two neighboring nodes

[0254] Figure 11: Delay measurements in a line topology with two different link speeds; 100 Mbps between Node 1 and Node 2, and 1 Gbps connections between Node 0 and Node 1 and between Node 2 and Node 3

[0255] Figure 12: Synchronized and non-synchronized traces through the network in a per-hop view; GCL always opens the gate 30 ps after the previous node; measurement duration: 300 minutes; 20 frames per second; 512 bytes per frame

[0256] (a) Synchronized tracing b) Non-synchronized tracing

[0257] (c) aggregated synchronized tracing based on network location

[0258] (c) aggregated non-synchronized trace based on network location

[0259] Figure 13: Synchronized traces through the network with cross-traffic based on IMIX; measurement duration: 30 minutes; 20 frames per second; 512 bytes per frame

[0260] (a) Single image view with cross traffic of equal and higher priority

[0261] (b) Per-hop view with cross traffic of equal and higher priority

[0262] (c) Single image view with lower priority cross traffic and strict priority

[0263] (d) Per-hop view with lower priority and strict priority cross traffic

[0264] (e) Single image view with lower priority cross traffic and frame preemption

[0265] (f) Per-hop view with lower priority cross-traffic and frame preemption Figure 14: Change of the TAS cycle time from 1 ms on "Node 0" and "Node 1" to 5 ms on "Node 2" and "Node 3" in synchronized topology; measurement duration: 60 minutes; 20 frames per second; 512 bytes per frame

[0266] Table 2: Stream configuration for use case evaluation

[0267] Figure 15: Synchronized tracing through the network with cross-traffic based on IMIX, resulting in congestion; measurement duration: 30 minutes

Claims

Patent claims 1. A method for operating a network, wherein a plurality of network devices, each having its own configuration, are connected to one another for data exchange and exchange data via these connections, wherein dynamic delays (jitter) are taken into account when determining the time for the transmission of the data, characterized in that the network is a time-sensitive network and that an actual time for the transmission of the data across the network devices from an originating network device to a destination network device is determined taking into account the dynamic delays, wherein time synchronization jitter and forwarding jitter are taken into account in the dynamic delays, 2. Method according to claim 1, characterized in that each network device inserts a timestamp into a data frame on its ingress port and on its egress port.

3. Method according to claim 1 or 2, characterized in that a theoretical time for the transmission of the data across the network devices from the source network device to the destination network device is determined.

4. Method according to claim 3, characterized in that the actually determined time is compared with the theoretically determined time.

5. The method according to claim 4, characterized in that if the comparison exceeds a predeterminable threshold value, the configuration of at least one network device is changed for the purpose of debugging between the source network device and the target network device.

6. Method according to claim 5, characterized in that the configuration of the source network device and / or the destination network device is also changed.

7. The method according to claim 5 or 6, characterized in that the configuration of the at least one network device is changed until the comparison no longer exceeds the predeterminable threshold value.

Images