A full-network telemetry method based on telemetry metadata reorganization

Abstract

The application discloses a full-network telemetry method based on telemetry metadata reorganization, and relates to the technical field of network communication.The full-network telemetry collects telemetry metadata of all network nodes in a specific range, and an INT collector constructs a data set by using INT telemetry reports of multiple continuous data packets; a telemetry metadata reorganization TMR algorithm is designed; telemetry metadata in the INT telemetry report is reorganized based on collected time and place, and TMR telemetry reports covering all network nodes are formed; a series of TMR telemetry reports obtained through the TMR algorithm processing are stored in a result set, and an accurate full-network view is formed.The application can more accurately reflect performance indexes of all nodes in the network, the performance is not affected by the length of a forwarding path, and is beneficial to network congestion identification and fault detection and the like.

Description

Technical Field

[0001] This invention relates to the field of network communication technology, specifically to a network-wide telemetry method based on telemetry metadata reassembly that reflects the performance and status of each node in the network. Background Technology

[0002] As networks continue to expand, traditional network measurement methods can no longer meet the needs of large-scale networks such as data centers. Emerging in-band network telemetry (INT) technology utilizes programmable switches to extract real-time device status and obtain fine-grained network performance data, such as hop-by-hop latency and queue occupancy. INT categorizes programmable switches (network nodes) into three roles: INT source nodes, INT transport nodes, and INT end nodes. As the first node on the telemetry path, the INT source node inserts telemetry commands into data packets and includes its own status as telemetry metadata, before forwarding the packet to the next-hop network node. The INT transport node is an intermediate network node on the path; it extracts its own network status based on telemetry commands to form telemetry metadata, also inserting it into data packets and forwarding them. The INT end node is the final node on the telemetry path, separating the original data packet from the INT telemetry header. The former is sent to the destination, while the latter generates an INT telemetry report, which is sent to the INT collector. Existing research on telemetry report analysis and processing is based on data packets; that is, all telemetry metadata collected from a single data packet forms an INT telemetry report. These fine-grained INT telemetry reports help the network quickly detect and locate faults, alleviating network congestion.

[0003] Traditional telemetry (INT) only defines how to acquire telemetry metadata, while the choice of which network node to collect data from is determined by network traffic and routing. Network nodes with high data traffic collect a large amount of INT data, while edge network nodes with low traffic collect insufficient INT data. However, network management often requires making optimal decisions based on a global view of the network situation. Within a specific network scope, a network-wide telemetry design covers the telemetry paths of all nodes to comprehensively collect INT data, while avoiding path overlap to reduce telemetry overhead, thus providing a complete network view.

[0004] Current research on network-wide telemetry generally focuses on how to orchestrate telemetry paths to obtain INT data from all network nodes. For example, the INT-path method (Pan T, Song E, Bian Z, et al. INT-path: Towards Optimal Path Planning for In-band Network-Wide Telemetry: Proc of IEEE Conference on Computer Communications, 2019[C].) uses the Euler algorithm to generate non-repeating telemetry paths covering all nodes in the network, which not only enables network-wide telemetry but also minimizes overhead.

[0005] Subsequent INT-react (Yuan Q, Li F, Pan T, et al. INT-react: An O(E)Path Planner for Resilient Network-Wide Telemetry Over Megascale Networks: Proc of the 30th IEEE International Conference on Network Protocols, 2022[C].2022), INT-probe (Pan T, Lin X, Song H, et al. INT-probe: Lightweight In-band Network-WideTelemetry with Stationary Probes.: Proc of IEEE 41st International Conference on Distributed Computing Systems, 2021[C].2021 / / ) and INT-Segment method (Yuan Q, LiF, Pan T, et al. INT-Segment: MTU-Adaptive Single-Path In-Band Network-WideTelemetry: Proc of IEEE 30th International Conference on Network Protocols, 2022 (C). 2022) both utilize graph theory methods to solve telemetry paths that can cover the entire network and have low overhead. The NetView method (Lin Y, Zhou Y, Liu Z, et al. NetView: Towards on-demand network-wide telemetry in the data center. [J]. Journal of Computer Networks, 2020.) can also formulate telemetry paths that cover all network nodes according to telemetry requirements.The DyPro method (Dallanora LM, Castro AG, Filho RI TD, et al. DyPro: Dynamic Probing Planning for In-Band Network Telemetry: Proc of 2022 IEEE Symposium on Computers and Communications, 2022 [C]. 2022) can also update the objective formula of the heuristic algorithm in a timely manner according to changes in telemetry requirements, solve for the optimal telemetry path, and achieve flexible network telemetry. Most of these methods only focus on how to collect telemetry metadata from all network nodes, and rarely pay attention to whether the collected telemetry metadata is accurate and timely. Only the INT-path and INT-react methods, when considering the orchestration of multiple paths, propose that the length of telemetry paths should be balanced as much as possible so that INT telemetry reports on different paths arrive at the INT collector as simultaneously as possible. However, they still do not consider the adverse impact of time differences in collecting telemetry metadata on the accuracy of performance analysis.

[0006] Existing network-wide telemetry methods collect in-band network telemetry (INT) data at each network node along the packet forwarding path and process the telemetry metadata of each packet to form an INT telemetry report, thus providing a network-wide view. However, factors such as network congestion and long paths can increase packet latency, leading to time lags in the collected telemetry metadata. For example, when network congestion occurs, the processing time of congested network nodes increases, resulting in a significant difference in telemetry metadata collection time before and after that node. When the telemetry path is long, the time difference between the collection of telemetry metadata at the source and end nodes is also significant, and some telemetry metadata becomes outdated by the time it reaches the INT collector. Processing and analyzing INT telemetry reports based on packets can easily lead to inaccurate performance analysis results because the mixed use of old and new telemetry metadata cannot reflect the actual network status within a specified time, affecting the accuracy and timeliness of network-wide telemetry. Summary of the Invention

[0007] The purpose of this invention is to address the aforementioned problems in the existing technology and, in order to improve the accuracy and timeliness of network-wide telemetry, provide a network-wide telemetry method based on Telemetry Metadata Recombination (TMR) that can more accurately reflect the performance and status of each node in the network and improve the accuracy of network-wide telemetry analysis. Its performance does not decrease with the increase of telemetry path length and can accurately and timely identify network congestion.

[0008] The present invention discloses a network-wide telemetry method based on telemetry metadata reassembly, comprising the following steps:

[0009] 1) The whole network telemetry collects telemetry metadata of all network nodes within a specific range. The INT collector builds a dataset from the INT telemetry reports of multiple consecutive data packets. Each report contains the telemetry metadata collected by the current data packet at each network node.

[0010] 2) Time synchronization: The time when each network node collects telemetry metadata in the INT telemetry report is synchronized with the source node's clock.

[0011] 3) Telemetry Metadata Reassembly (TMR) Algorithm: Based on the time and location (network node number) of telemetry metadata collected by each network node, a greedy algorithm is designed with the goal of minimizing the time variance to reassemble telemetry metadata collected at similar times, forming a TMR telemetry report covering all network nodes.

[0012] 4) A series of TMR telemetry reports obtained by the TMR algorithm are stored in the result set to form an accurate view of the entire network.

[0013] In step 1), the specific process of collecting telemetry metadata of all network nodes within a specific range using network-wide telemetry is as follows: On the telemetry path of N nodes, host A continuously sends M data packets, each data packet passing through the j-th network node R. j Collect its network status data in real time, and R j Telemetry metadata is inserted into the metadata stack, j = 1, ..., N; then, the INT end node generates an INT telemetry report and sends it to the INT collector. Each telemetry report contains R1 to R N The telemetry metadata is extracted; the original data packet is stripped and sent to host B to complete the network transmission task.

[0014] In step 2), the time for collecting telemetry metadata, i.e., the collection time of telemetry metadata, is defined as the time when the data packet leaves the network node. Although INT can directly obtain the time when telemetry metadata enters and leaves the network node, these nodes have their own clocks, and the overhead of synchronizing the clocks across the entire network is huge. Therefore, this invention ignores the data packet propagation delay (or estimates it as a constant), uses the clock of the source node as the standard y1 = EgressTime1, and expresses the time for the j-th network node to collect telemetry metadata as the collection time of the data packet in the previous network node plus the processing time of that node. After the INT collector receives the INT telemetry reports of M data packets, it calculates the collection time of each telemetry metadata according to formula (1) to form the INT telemetry report dataset W. This M×N dimensional matrix is ​​the input of the TMR algorithm, and the element W in the i-th row and j-th column is... ij This represents the telemetry metadata collected when the i-th data packet passes through the j-th network node, and the collection time.

[0015] yj =y j-1 +EgressTime j -IngressTime j (N≥j>1) (1)

[0016] In the formula, y j y represents the collection time of the telemetry metadata of the j-th network node. j-1 EgressTime represents the collection time of telemetry metadata for the (j-1)th network node. j IngressTime represents the time it takes for a data packet to leave the j-th network node. j This indicates the time when the data packet enters the j-th network node.

[0017] In step 3), the telemetry metadata reorganization (TMR) algorithm reorganizes the data based on the collection time of each element in the INT telemetry report dataset W. A greedy algorithm is designed with the goal of minimizing the time variance to form a TMR telemetry report dataset Z covering all network nodes. Specifically, this includes the following steps:

[0018] Step 1: Input and initialization;

[0019] Step 1.1: Input the INT telemetry report dataset W;

[0020] Step 1.2: Empty the TMR telemetry report dataset Z. The set S and candidate set B of TMR telemetry metadata are initialized to empty.

[0021] Step 1.3: Sort the elements in the first column of W in reverse order of collection time;

[0022] Step 2: Reorganize telemetry metadata and construct set S;

[0023] Step 2.1: Add the first element of column 1 in W to S, and then delete that element from W;

[0024] Step 2.2: Construct the candidate set B, which specifically includes the following steps:

[0025] Step 2.2.1: Calculate the average value of set S.

[0026] Step 2.2.2: Optimize the network node elements in set S that are not yet covered, and add them to set B. The relevant code is as follows:

[0027]

[0028] Step 2.2.3: Find the set with the closest collection time in set B. Add the elements to set S; at this point, the time variance increment of set S is minimized. The relevant code is as follows:

[0029]

[0030] S.add(temp2)

[0031] W.del(temp2)

[0032]

[0033] Repeat step 2.2 until S contains elements of all columns (i.e., traverse all network nodes); then save the result of set S (i.e., a TMR telemetry report) to result set Z, and then set S to empty;

[0034] Step 3: Repeat Step 2 until the K TMR telemetry reports are reassembled (K <M)。

[0035] This invention discloses a network-wide telemetry method based on Telemetry Metadata Recombination (TMR). It proposes to recombine telemetry metadata from different data packets based on the collection time of the telemetry metadata, grouping together telemetry metadata from similar times to form a more accurate TMR telemetry report, ensuring that each TMR telemetry report contains telemetry metadata from all network nodes.

[0036] Compared with the prior art, the present invention has the following outstanding advantages and technical effects:

[0037] This invention proposes a Telemetry Metadata Reassembly (TMR) method for network-wide telemetry. It uses a greedy algorithm to recombine telemetry metadata collected at similar times into a TMR telemetry report, thereby improving the accuracy of network-wide telemetry analysis. The reassembled TMR telemetry report reflects the actual network conditions and can accurately and promptly identify network congestion. Experiments demonstrate that, compared to traditional Intent (INT), the TMR method of this invention more accurately reflects the performance indicators of each network node, and its performance is unaffected by forwarding path length, which is beneficial for applications such as network congestion identification and fault detection. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the entire network telemetry process.

[0039] Figure 2 This is the experimental topology diagram. Among them, (a) is the basic environment and (b) is the congested environment.

[0040] Figure 3 The graph shows the time deviation curves for the INT and TMR methods. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the following embodiments will be used in conjunction with the accompanying drawings to further illustrate the invention. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. Rather, the invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of the invention as defined in the claims.

[0042] This invention collects telemetry metadata from all network nodes within a specific range. The INT collector constructs a dataset from the INT telemetry reports of multiple consecutive data packets. A telemetry metadata reassembly (TMR) algorithm is designed to reassemble the telemetry metadata in the INT telemetry reports based on the collection time and location, forming a TMR telemetry report covering all network nodes. A series of TMR telemetry reports obtained by the TMR algorithm are stored in the result set to form an accurate view of the entire network.

[0043] The whole-network telemetry method based on telemetry metadata reassembly described in this embodiment of the invention includes the following specific steps:

[0044] The whole network telemetry collects telemetry metadata from all network nodes within a specific range. Figure 1 This describes the network-wide telemetry process of collecting telemetry metadata along a telemetry path with N nodes. Host A continuously sends M data packets, each data packet passing through the j-th network node R. j Collect its network status data in real time, and R j Telemetry metadata is inserted into the metadata stack, j = 1, ..., N; then, the INT end node generates an INT telemetry report and sends it to the INT collector. Each telemetry report contains R1 to R... N The telemetry metadata is extracted; the original data packet is stripped and sent to host B to complete the network transmission task. The data reassembly (TMR) algorithm reassembles the telemetry metadata from multiple telemetry reports based on the collection time to form a TMR telemetry report.

[0045] Define the variables involved in the data recombination algorithm (as shown in Table 1) and describe the specific steps of the algorithm.

[0046] Table 1: List of Variables

[0047]

[0048] Among them, the collection time of telemetry metadata is defined as the time when the data packet leaves the network node; although INT can directly obtain the time when telemetry metadata enters and leaves the network node, these nodes have their own clocks, and the overhead of synchronizing the clocks of the whole network is huge; therefore, this invention ignores the data packet propagation delay (or estimates it as a constant), takes the clock of the source node as the standard y1=EgressTime1, and expresses the time when the j-th network node collects telemetry metadata as the collection time of the data packet in the previous network node plus the processing time of that node; after the INT collector receives the INT telemetry reports of M data packets, it calculates the collection time of each telemetry metadata according to formula (1) to form the INT telemetry report dataset W; this M×N dimensional matrix is ​​the input of the TMR algorithm, and the element W in the i-th row and j-th column is... ij This represents the telemetry metadata collected when the i-th data packet passes through the j-th network node, and the collection time.

[0049] y j =y j-1 +EgressTime j -IngressTime j (N≥j>1) (1)

[0050] In the formula, y j y represents the collection time of the telemetry metadata of the j-th network node. j-1 EgressTime represents the collection time of telemetry metadata for the (j-1)th network node. j IngressTime represents the time it takes for a data packet to leave the j-th network node. j This represents the time when the data packet enters the j-th network node. The specific algorithm steps are as follows:

[0051] Step 1: Input and initialization;

[0052] Step 1.1: Input the INT telemetry report dataset W;

[0053] Step 1.2: Empty the TMR telemetry report dataset Z. The set S and candidate set B of TMR telemetry metadata are initialized to empty.

[0054] Step 1.3: Sort the elements in the first column of W in reverse order of collection time;

[0055] Step 2: Reorganize telemetry metadata and construct set S;

[0056] Step 2.1: Add the first element of column 1 in W to S, and then delete that element from W;

[0057] Step 2.2: Construct the candidate set B, which specifically includes the following steps:

[0058] Step 2.2.1: Calculate the average value of set S.

[0059] Step 2.2.2: Optimize the network node elements in set S that are not yet covered, and add them to set B. The relevant code is as follows:

[0060]

[0061] Step 2.2.3: Find the set with the closest collection time in set B. Add the elements to set S; at this point, the time variance increment of set S is minimized. The relevant code is as follows:

[0062]

[0063] S.add(temp2)

[0064] W.del(temp2)

[0065]

[0066] Repeat step 2.2 until S contains elements of all columns (i.e., traverse all network nodes); then save the result of set S (i.e., a TMR telemetry report) to result set Z, and then set S to empty;

[0067] Step 3: Repeat Step 2 until the K TMR telemetry reports are reassembled (K <M)。

[0068] Experimental verification:

[0069] Figure 2 Figure (a) illustrates the basic experimental environment, where the INT protocol is deployed on four Barefoot Tofino switches, with each switch connected to the other via 100G bandwidth cables. Host C generates traffic and sends packets to R1; R1, as the INT source node, inserts telemetry commands, which are then forwarded and telemetry-monitored by other network nodes; and an INT telemetry report is generated at the INT end node R4. The collector INTBeat then processes and analyzes the INT telemetry report, and reassembles the telemetry metadata using the TMR algorithm to form a TMR telemetry report for further analysis.

[0070] Table 2 uses data packets 1# to 6# as examples to illustrate the composition of the INT telemetry report dataset: telemetry metadata collected when the 6 data packets pass through 6 network nodes: collection time and queue occupancy (measured in 80-byte units), with the average collection time of each network node used as the INT telemetry report time. Meanwhile, formula (2) lists the input matrix W of the TMR algorithm. To simplify the expression, W only lists the collection time, but each telemetry metadata also contains queue occupancy status data.

[0071] Table 2: INT Telemetry Report Dataset

[0072]

[0073]

[0074] After processing by the TMR algorithm, a series of TMR telemetry reports can be obtained and stored in the result matrix Z. Taking the first TMR telemetry report as an example, it consists of telemetry metadata collected by each network node at times (2315, 2369, 2340, 2335, 2343, 2348), as shown by the underlined elements in Table 2. The time of this TMR telemetry report is the average collection time of 2341 ns, and the time variance is 258, which is much smaller than the time variance of the INT telemetry report (as shown in Table 2). It can be seen that the TMR telemetry report can more accurately reflect the overall network view in real time.

[0075] Figure 2 Figure (b) shows the experimental topology of the congested environment. In this case, the cable bandwidth between R3 and R6 is reduced to 10G and filled with background traffic greater than 10G, thus creating congestion at R6. Queue occupancy is observed using the traditional INT method and the TMR method. In the previous basic environment experiments, due to abundant network resources, the queue occupancy of each network node was less than 3. Therefore, the observation that queue occupancy gradually increases indicates that network utilization is starting to increase and congestion is gradually forming.

[0076] Figure 3 The hop-by-hop telemetry metadata of 100 packets transmitted during network congestion was selected as the benchmark. Specifically, the collection time b of the 100 R6 telemetry metadata packets was used as the benchmark. i As a baseline (i = 1, 2, ..., 100): the i-th INT telemetry report contains data from b. i Time-collected R6 telemetry metadata; the i-th TMR telemetry report contains data from b. i R6 telemetry metadata was collected over time. Then, the accuracy of INT telemetry reports and TMR telemetry reports in detecting network congestion (e.g., ...) was examined. Figure 3As indicated by the indicator box, queue occupancy begins to increase continuously. The dashed line represents the time skew of the INT telemetry reports, where the i-th point represents the average collection time of the i-th INT telemetry report compared to b. i The difference between the two values. The solid line represents the time bias of the TMR telemetry reports, where the i-th point represents the difference between the average collection time in the i-th TMR telemetry report and b. i The difference is significant. It can be seen that after network congestion occurs, the deviation between the INT telemetry report and the baseline increases, because network congestion causes the telemetry metadata in the INT telemetry report to be a mixture of old and new data. After implementing Data Reassembly and Modulation (TMR), telemetry metadata from similar times is recombined, thus reducing the deviation to near zero. This also indicates that the TMR telemetry report more accurately reflects the actual network condition.

[0077] We also considered experimental scenarios where the forwarding path traversed different numbers of network nodes. Table 3 shows the average time variance of all INT telemetry reports and all TMR telemetry reports for N=4, 6, and 8. Experiments demonstrate that the performance of the TMR method does not deteriorate with the increase in the forwarding path, and its network node collection time variance fluctuates very little; while the collection time variance of the INT method increases rapidly with the increase in the path length.

[0078] Table 3: Mean Time Variance of INT and TMR Methods

[0079]

[0080] This invention proposes a novel Telemetry Metadata Reassembly (TMR) method. It aggregates and analyzes telemetry metadata collected from different data packets at similar times to form a TMR telemetry report covering all network nodes. An INT collector constructs a dataset from the INT telemetry reports of multiple consecutive data packets. Each report contains the telemetry metadata collected from each network node for the current data packet, and the collection time is synchronized. A greedy algorithm based on minimizing time variance is designed to reassemble the telemetry reports according to the time and location (network node number) of the telemetry metadata collection. Each TMR report contains the telemetry metadata collected from each network node at the same or similar time. Finally, the TMR telemetry report dataset is analyzed to obtain a more accurate view of the entire network. Compared with the traditional INT method, the TMR method can more accurately reflect the performance and status of each network node, and its performance does not decrease with the increase of telemetry path length.

[0081] The above embodiments are merely preferred embodiments of the present invention and should not be considered as limiting the scope of the present invention. All equivalent variations and improvements made within the scope of the present invention should still fall within the patent coverage of the present invention.

Claims

1. A network-wide telemetry method based on telemetry metadata reassembly, characterized in that... Includes the following steps: 1) The whole network telemetry collects telemetry metadata of all network nodes within the preset range. The INT collector builds a dataset from the INT telemetry reports of multiple consecutive data packets. Each report contains the telemetry metadata collected by the current data packet at each network node. 2) Time synchronization: The time when each network node collects telemetry metadata in the INT telemetry report is synchronized with the source node's clock. 3) Telemetry metadata reassembly (TMR) algorithm: Based on the time and location of telemetry metadata collection by each network node, a greedy algorithm is designed with the goal of minimizing the time variance to reassemble telemetry metadata collected at similar times to form a TMR telemetry report covering all network nodes. The telemetry metadata reassembly (TMR) algorithm is based on the INT telemetry report dataset. The collection times of each element are reorganized, and a greedy algorithm is designed with the goal of minimizing the time variance to form a TMR telemetry report dataset covering all network nodes. ; Specifically, the following steps are included: Step 1: Input and initialization; Step 1.1: Input the INT telemetry report dataset ; Step 1.2: Transfer the TMR telemetry report dataset Leave blank, A collection of TMR telemetry metadata and candidate set Initialize to empty, ; Step 1.3: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] The elements in the first column are arranged in reverse order of collection time; Step 2: Reorganize telemetry metadata and construct a set. ; Step 2.1: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] Add the first element of the first column. and in Delete the element; Step 2.2: Construct the candidate set Specifically, it includes the following steps: Step 2.2.1: Calculate the set average ; Step 2.2.2: For the set Optimize network node elements that are not yet covered and add them to the set. The relevant code is: for v=2:N do if It does not contain telemetry metadata for network node v. / / exist The element in column v is found using a binary search method to determine the element with the closest collection time. elements temp1= B.add(temp1) Step 2.2.3: In the set Find the collection time closest to Add the elements to the set At this time, the set The time variance increment is minimized; the relevant code is as follows: temp2= S.add(temp2) W.del(temp2) B = Repeat step 2.2 until... The collection contains elements from all columns, which means traversing all nodes in the network; The result is a TMR telemetry report saved to the results set. Then Leave blank; Step 3: Repeat step 2 until... The reorganization of the TMR telemetry report is complete. <M; 4) A series of TMR telemetry reports obtained by the TMR algorithm are stored in the result set to form an accurate view of the entire network.

2. The network-wide telemetry method based on telemetry metadata reassembly as described in claim 1, characterized in that... In step 1), the specific process of collecting telemetry metadata of all network nodes within a preset range using full-network telemetry is as follows: On the telemetry path of N nodes, host A continuously sends M data packets, each data packet passing through the j-th network node R. j Collect its network status data in real time, and R j Telemetry metadata is inserted into the metadata stack, j=1,…,N; then, the INT end node generates an INT telemetry report and sends it to the INT collector. Each telemetry report contains R1 to R... N The telemetry metadata is extracted; the original data packet is stripped and sent to host B to complete the network transmission task.

3. The network-wide telemetry method based on telemetry metadata reassembly as described in claim 1, characterized in that... In step 2), the time for collecting telemetry metadata is defined as the time when the data packet leaves the network node, using the source node's clock as the standard. The time for the j-th network node to collect telemetry metadata is represented as the time for the data packet to be collected at the previous network node plus the processing time of that node. After receiving M data packets of INT telemetry reports, the INT collector calculates the collection time of each telemetry metadata according to formula (1) to form the INT telemetry report dataset. ; This M×N dimensional matrix is ​​the input to the TMR algorithm, and the element in the i-th row and j-th column is... This represents the telemetry metadata collected when the i-th data packet passes through the j-th network node, and the collection time. In the formula, This indicates the time when the telemetry metadata of the j-th network node was collected. This indicates the collection time of telemetry metadata for the (j-1)th network node. This indicates the time when the data packet leaves the j-th network node. This indicates the time when the data packet enters the j-th network node.

Images