Clock-synchronous edge-based networking functionality

Abstract

Network traffic is monitored for coordinated control of data flow and anomaly detection. For a data flow transmitted between a sending host and a receiving host, a predetermined amount of sent or received network traffic of the data flow is recorded. The data flow is monitored for anomalies based on timestamps of data packets in the network traffic. In response to determining that an anomaly is not detected, the recorded sent and received network traffic is overwritten with new sent and new received network traffic, respectively. In response to determining that an anomaly is detected, the data flow is paused, which causes the sending host to store the recorded sent network traffic in a first buffer and the receiving host to store the recorded received network traffic in a second buffer.

Description

Technical Field

[0001]

[0002] The present disclosure generally relates to the coordinated control of network traffic in network transmission and data flow.

Background Art

[0002] Cross-reference to Related Applications

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 178,999, filed Apr. 23, 2021; U.S. Provisional Application No. 63 / 320,160, filed Mar. 15, 2022; U.S. Utility Application No. 17 / 725,408, filed Apr. 20, 2022; and U.S. Patent Application No. 17 / 520,836, filed Apr. 20, 2022, the entire disclosures of which are incorporated herein by reference, respectively.

[0003]

[0003] Modern Internet infrastructure typically includes large data centers that generate large amounts of network traffic. When demands are high, the output of the data center may be limited (e.g., by the capabilities of switches and gateways, etc.), and network traffic may have to be measured. Such transient congestion scenarios can cause bottlenecks and packet drops. To ensure the success of packet transmission when faced with such situations, systems have been developed that send acknowledgments from the receiving node to the transmitting node upon receiving a packet. However, these acknowledgments are not effective in that they are an additional cause of network traffic. Furthermore, these acknowledgments are limited to functioning in scenarios from a single transmitter to a single receiver. Still further, if an acknowledgment is not received, the packet may simply be retransmitted ad hoc, flow into the same congested switch, and fall into the same drop result, resulting in a scenario where the packet is constantly delayed or not received at all by the destination.

Prior Art Documents

Patent Documents

[0004] [Patent Document 1] U.S. Patent No. 10,623,173 [Brief explanation of the drawing]

[0005] [Figure 1]

[0004] Figure 1 is an exemplary system environment that performs a network camera and priority function according to an embodiment of the present disclosure. [Figure 2]

[0005] Figure 2 is a network traffic diagram showing a configuration in which multiple sending hosts send multiple data flows to a single receiving host, according to an embodiment of the present disclosure. [Figure 3]

[0006] Figure 3 is a network traffic diagram illustrating the timestamp operation on both the transmitting and receiving sides of data transmission according to an embodiment of this disclosure. [Figure 4]

[0007] Figure 4 is a data flow diagram showing the operation of the Netcam during the normal operating period and when an abnormality is detected during that period, according to an embodiment of this disclosure. [Figure 5]

[0008] Figure 5 is a network traffic diagram illustrating an embodiment of this disclosure in which a receiving host receives both high-priority and low-priority traffic from a transmitting host. [Figure 6]

[0009] Figure 6 is a data flow diagram showing that priorities are considered when determining the operation of the network camera according to an embodiment of this disclosure. [Figure 7]

[0010] Figure 7 is a flowchart illustrating an exemplary process for performing a network camera operation according to an embodiment of the present disclosure. [Figure 8]

[0011] Figure 8 is a flowchart illustrating an exemplary process for performing Netcam operation in multiple priority scenarios according to an embodiment of this disclosure. [Modes for carrying out the invention]

[0006]

[0012] The drawings and the following description relate to preferred embodiments for illustrative purposes only. From the following discussion, alternative embodiments of the structures and methods disclosed herein should be readily recognizable as viable alternatives that can be adopted without departing from the principles of the claims.

[0007]

[0013] This specification discloses a system and method for coordinating data flow control in the event of temporary congestion. “Netcam” monitors network traffic between clock-synchronized transmitting and receiving hosts, which is part of the data flow. The term “Netcam” as used herein is an abbreviation for “Network Camera,” and is a module that tracks network traffic and ensures remedial action is taken if the data flow traffic in a clock-synchronized system is delayed beyond acceptable limits. Netcam instructs transmitting and receiving hosts to buffer copies of the network traffic according to several parameters (e.g., buffering a certain number of packets, buffering packets according to a rolling window of time). Buffers may be overwritten on a rolling basis when the parameters are met (e.g., the oldest packet is overwritten when a new packet is transmitted or received and the buffer is full). If an anomaly is detected, Netcam may instruct all transmitting and receiving hosts to buffer data and to retransmit written packets to the transmitting host. Retransmissions may be subject to fluctuations (e.g., time delays between packet transmissions in a data flow) that cause delays or failures in packet transmission due to a given packet transmission sequence, but the fluctuations (jitter) nevertheless cause enough change for a retransmission attempt to succeed. The network may determine the need to write and retransmit packets depending on the priority of the data flow.

[0008]

[0014] Advantageously, the Netcam implementation disclosed herein enables improvements in both network transmission and forensic analysis. Improved network transmission is achieved by attempting to write to buffers across all machines for the most recent packet transmission in the data flow, allowing for the retransmission of accurate packet sets from many machines without relying on acknowledgment packets that may be lost or dropped across intricately intertwined machines. Furthermore, virtual machines may harbor bugs that are difficult to detect or isolate. Writing anomaly-associated packet sequences enables anomaly analysis, which in turn can identify faulty virtual machines. Further advantages and improvements become apparent from the following disclosures.

[0009]

[0015] Figure 1 shows an exemplary system environment that performs networkcam and priority functions according to an embodiment of the present disclosure. As shown in Figure 1, the networkcam environment 100 includes a transmitting host 110, a network 120, a receiving host 130, and Netcam This includes system 140. Only one transmitting host 110 and one receiving host 130 are shown, but this is for convenience and ease of illustration only; any number of transmitting and receiving hosts may be part of the network environment 100.

[0010]

[0016] The transmitting host 110 includes a buffer 111, a network interface card (NIC) 112, and a netcam module 113. The buffer 111 stores a copy of the output data transmission until one or more criteria are met for overwriting or discarding packets in the buffer. For example, the buffer may store data packets until it reaches its capacity, at which point the oldest buffered data packets may be discarded or overwritten. Other criteria may include the passage of time (e.g., discarding packets that have elapsed a predetermined amount of time since the transmission timestamp), the amount of buffered packets (e.g., after a predetermined amount of packets have been buffered, discarding or overwriting the oldest packets begins when a new packet is transmitted), etc.

[0011]

[0017] In an embodiment, buffer 111 stores information about a given output transmission rather than all packets. For example, a byte stamp may be stored rather than the packet itself, the byte stamp indicating the packet identifier and / or flow identifier and the timestamp when the packet (or aggregated data flow) was transmitted. In such an embodiment, the stored information does not need to be overwritten by the transmitting host 110 and / or Netcam The data may be stored in the persistent memory of system 140. This embodiment is not mutually exclusive with buffer 111, which stores a copy of the packet, and a combination of the two may be employed.

[0012]

[0018] NIC112 may be any type of network interface, such as a smart NIC. NIC112 interfaces with the transmitting host 110 and the network 120.

[0013]

[0019] The Netcam module 113 monitors certain conditions of the data flow and triggers functions based on the monitored data. For example, in response to the detection of network congestion, the Netcam module 113 may instruct all hosts that are part of the data flow to perform one or more of a variety of actions, such as pausing transmission, taking a snapshot of buffered data transmissions (i.e., writing buffered data to persistent memory), and performing other coordinated actions. As used herein, the term data flow may mean a collection of data transmissions between two or more interrelated hosts. Further details of the Netcam module 113 are described below in more detail in relation to Figures 2 to 8. The Netcam module 113 can be implemented in any component of the transmitting host 110. In one embodiment, the Netcam module 113 may be implemented in the NIC 112. In another embodiment, the Netcam module 113 may be implemented in the kernel of the transmitting host 110.

[0014]

[0020] Network 120 may be any network, such as a wide area network, a local area network, the Internet, or any other data transmission conduit between the transmitting host 110 and the receiving host. In some embodiments, network 120 may reside within a data center housing both the transmitting host 110 and the receiving host 130. In other embodiments, network 120 may facilitate cross-data center transmissions over any distance. The reference to a data center is illustrative, and the transmitting host 110 and the receiving host 130 may be implemented in any medium, including non-data center media.

[0015]

[0021] The receiving host 130 has a netcam buffer 131, a NIC 132, and a netcam module 133. The netcam buffer 131, the NIC 132, and the netcam module 133 operate in the same manner as the similar components described above for the transmitting host 110. The buffer 131 may be the same size or a different size from the buffer 111, and additionally or alternatively, may store the byte stamp of the received packet. Further distinctions between the components implemented in the transmitting host and the receiving host will be apparent based on the disclosure of FIGS. 2-8 below.

[0016]

[0022] The netcam system 140 has a clock synchronization system 141. The netcam system 140 may monitor data observed by netcam modules implemented in hosts, such as the netcam modules 113 and 133. The netcam system 140 may detect conditions that require an action by a netcam module and may instruct the affected netcam module to cooperate for a given data flow. The clock synchronization system 141 synchronizes one or more components of an individual host, such as a NIC, a kernel, or other components operating within the netcam module. Details of clock synchronization are described in Patent Document 1 related to joint ownership issued on April 14, 2020, the disclosure of which is hereby incorporated by reference in its entirety. The individual hosts are synchronized to the same reference clock at a very accurate level, enabling accurate timestamps across multiple hosts regardless of the location of the host, the host's bandwidth conditions, and jitter, etc. Further details of the netcam system 140 are disclosed below with reference to FIGS. 2-8. The netcam system 140 is an optional component in the netcam environment 100, and the netcam modules of the transmitting host and the receiving host can operate without relying on a centralized system, except relying on the reference clock for synchronization.

[0017] <​The Netcam environment 100 offers many advantages. Assuming that the Netcam module can run in the kernel or NIC (e.g., smart NIC) of a host (e.g., a physical host, a virtual machine, or any other form of host), the Netcam module is edge-based. In one embodiment, the Netcam function can run as an underlay, meaning it can run as a shim, for example, below the congestion control layer of an OSI system (e.g., Layer 3 of an OSI system). The Netcam module, and / or the Netcam system 140, may, upon detection of conditions, instruct a host to pause transmission of data flow across affected hosts, take snapshots (i.e., write some or all of the buffered data, such as the last N bytes transmitted and / or bytes transmitted in the last S seconds, where N or S are default values ​​or may be defined by the administrator), and perform any other actions disclosed herein. Further advantages and features are described below with reference to Figures 2 to 8.

[0018]

[0024] Figure 2 is a network traffic diagram illustrating an embodiment of the present disclosure in which multiple transmitting hosts transmit multiple data flows to a single receiving host. As shown in Figure 2, transmitting host 1 transmits data flow 211 to receiving host 200, transmitting host 220 transmits data flow 221 to receiving host 200, and any number of additional hosts may transmit their respective data flows (represented by data flow 231) to receiving host 200, as represented by transmitting host 230. As shown in Figure 2, the individual data flows transmitted by each host are different, but this is for convenience only, and two or more hosts may transmit data from the same data flow. Furthermore, a single transmitting host may transmit two or more different data flows to receiving host 200. Although only one receiving host is shown, a transmitting host may transmit data flows to any number of receiving hosts.

[0019]

[0025] To describe the operation of the network camera module in the transmitting host and the receiving host, refer to FIG. 3. FIG. 3 is a network traffic diagram showing the timestamp operation on both the transmitting side and the receiving side of data transmission according to an embodiment of the present disclosure. As shown in FIG. 3, when the transmitting host 310 transmits a packet to the receiving host 320, send the network camera module 113 of the host 310 records a transmission timestamp 311. Similarly, when the receiving host 320 receives a packet, the network camera module 133 of the receiving host 320 applies a reception timestamp. The timestamp reflects the time when the data packet is transmitted or received in the relevant components (e.g., NIC, kernel, etc.) introduced by the network camera module. The transmission timestamp may be stored in the buffers 111 and 131, attached to the packet, transmitted to the storage device in the network camera system 140, or any combination thereof.

[0020]

[0026] Since the transmitting host 310 is synchronized to the same reference clock as the receiving host 320, the elapsed time between the transmit timestamp 311 and the receive timestamp 321 reflects the one-way delay of a given packet. In one embodiment, upon receiving a given packet, the receiving host 320 sends an acknowledgment packet indicating the receive timestamp 321 to the transmitting host 310, thereby enabling the netcam module 113 to calculate the one-way delay by subtracting the transmit timestamp 311 from the receive timestamp 321. Other means of calculating the one-way delay are within the scope of this disclosure. For example, the transmit timestamp 311 may be attached to the data transmission, and the receiving host 320 may thereby calculate the one-way delay without the need for an acknowledgment packet. In yet another example, the netcam modules of the transmitting and receiving hosts may send timestamps to the netcam system 140 in batches or individually, and the netcam system 140 may calculate the one-way delay from them. For convenience and brevity, the following disclosure focuses on a scenario in which the sending host calculates the one-way delay based on the acknowledgment packet; however, those skilled in the art will recognize that all of these calculation methods are equally applicable.

[0021]

[0027] The network system then determines whether the one-way delay exceeds a threshold. For example, after calculating the one-way delay, the transmitting host 110 may compare the one-way delay to a threshold. The threshold may be predetermined or dynamically determined. A predetermined threshold may be set by default or set by an administrator. Different thresholds may be applied to different data flows depending on attributes such as one or more priorities of the data flows, as will be further explained below. The threshold may be dynamically determined depending on any number of factors, such as dynamically increasing as congestion decreases and decreasing as congestion increases (for example, because the delay is likely to indicate a problem of no congestion or mild congestion). In one embodiment, the threshold may be set on a per-host basis, as it may depend on the distance between the transmitting host and the receiving host. In such an embodiment, the threshold may be a predetermined multiplier of the minimum one-way delay between the transmitting host and the receiving host. That is, the minimum one-way delay is the minimum time required for a packet to propagate from the transmitting host to the receiving host. The magnification is typically at least 1.5x to 3x, but any magnification may be defined by the Netcam administrator. The threshold is equal to multiple multiplications of the minimum one-way delay. In response to a determination that the one-way delay exceeds the threshold, the Netcam module 113 may instruct the transmitting host 110 to take one or more actions.

[0022]

[0028] One or more of these actions may include pausing transmission from the transmitting host when the one-way delay is high, thereby reducing congestion and overall packet loss in network 120. The pause may be a predetermined time or may be determined dynamically in proportion to the magnitude of the one-way delay. In one embodiment, the pause may be equal to the one-way delay or may be determined by applying a multiplier defined by the administrator to the one-way delay. In one embodiment, Netcam may determine if a previous pause is in effect and, if so, reduce the pause time based on the pause time already elapsed since the previous acknowledgment packet. Furthermore, a given data flow is not limited to data flows that cause congestion, and therefore its pause period may be less than the one-way delay or a one-way delay threshold.

[0023]

[0029] Another possible action is to write some or all of the buffered data packets (e.g., from one or both of the sending and receiving hosts) to persistent memory in response to the one-way delay exceeding a threshold. Diagnostics can then be performed based on the buffered data packets (e.g., to identify network problems). Further actions are described in detail below with respect to Figures 4-8.

[0024]

[0030] In some embodiments, data flows may be associated with different priorities. The Netcam module may determine the priority of data flows based on an explicit identifier (e.g., a traffic layer identifier in the data packet header) or on an inference (e.g., a heuristic in which rules are applied to the packet header and / or payload to determine the priority type). As used herein, priority refers to a priority scheme for which types of data packets should be allowed to be transmitted and which should be suspended during periods of congestion. The priorities disclosed herein are considered in terms of avoiding insufficient link utilization or the need for explicit bandwidth allocation, and instead deciding which packets to transmit during periods of network congestion.

[0025]

[0031] To prioritize high-priority packets, a high one-way threshold may be assigned to high-priority traffic, and a relatively lower one-way threshold may be assigned to low-priority traffic. Thus, since a lower one-way delay must be detected for low-priority packets for the Netcam module to detect anomalies, anomalies will be detected more frequently for low-priority packets than for high-priority packets, while anomalies will only be detected for high-priority packets if the high one-way threshold is violated. Following the above description of determining the one-way threshold for a given host, different one-way thresholds may be applied in a priority-dependent manner to different data packets sent from or received by the same host. In embodiments of priority, the one-way threshold may be determined by the method described above (e.g., by applying a predetermined multiplier to the threshold), where the determination is further influenced by the application of a priority multiplier. The priority multiplier may be set by the administrator for any given type of priority, with higher multipliers for higher priorities and lower multipliers for lower priorities. Priorities do not have to be binary; any number of priority hierarchies may be defined, each corresponding to different or multiple types of data traffic and having different multipliers. Priorities and their associated multipliers may change over time for a given data flow (for example, priority may decrease at the start of transmission of different types of data packets that do not require high-latency transmission).

[0026]

[0032] In addition to using a one-way delay threshold based on the priority of a given packet, and alternatively to varying the one-way delay threshold, the Netcam module may operate so that the pause time of paused traffic during pause operation differs depending on the priority. Lower pause times may be assigned to high-priority traffic, and relatively higher pause times may be assigned to low-priority traffic. Low-priority traffic is paused while high-priority traffic is paused, ensuring that high-priority traffic can utilize more bandwidth by ensuring that low-priority traffic is paused more frequently than high-priority traffic during congestion periods. Pause times may be determined in the same manner as described above, but with the additional step of applying an additional pause multiplier to the pause time, a lower pause multiplier for high-priority traffic (e.g., a multiplier less than 1, such as 0.7x), and a higher pause multiplier for lower-priority traffic (e.g., a multiplier greater than 1).

[0027]

[0033] Priorities may be assigned in any number of ways. In one embodiment, one or more "carpool lanes" may be assigned that are available to data flows with eligible priorities. For example, a "carpool lane" may be an allocation of bandwidth that does not guarantee a minimum bandwidth for a given data communication, but is accessible only to data flows that meet the required parameters. Exemplary parameters may include one or more priorities that qualify the use of the reserved bandwidth of a given "carpool lane". For example, a carpool lane is required to have data flows with at least an intermediate priority, so in a three-priority system of low, medium, and high, both medium and high priorities would be eligible. As another example, there may be multiple "carpool lanes" (e.g., a carpool lane accessible by medium and high priority traffic, in addition to a carpool lane accessible only by high priority traffic).

[0028]

[0034] In one embodiment, guaranteed bandwidth may be allocated to a given priority. For example, high-priority data flows may be allocated a minimum bandwidth, such as 70 Mbps. In such an embodiment, any surplus unused bandwidth from the guaranteed bandwidth may be allocated to lower-priority data flows until bandwidth is required by data flows that qualify for the guarantee. The guaranteed bandwidth may be absolute or relative. A relative guarantee ensures that a given priority data flow receives at least a relative amount of bandwidth greater than that of a lower-priority data flow. For example, a high-priority data flow may be guaranteed three times the bandwidth of a low-priority data flow, and a medium-priority data flow may be guaranteed twice the bandwidth of a low-priority data flow.

[0029]

[0035] Referring to Figure 2, when two or more sending hosts send data from the same data flow, additional nodes alongside them, and for any receiving hosts that receive data from the data flow, may be referred to as a “cluster.” In one embodiment, a data flow can be identified by the collection of identifiers indicating that a data packet is part of a data flow, if all identifiers are found. For example, a Netcam module on any host may determine a flow identifier that identifies the data flow to which a packet belongs, based on a combination of the source address, destination address, source port number, destination port number, and protocol port number. Other combinations of identifiers may be used to identify the data flow that includes that packet. As previously mentioned, the hosts in a cluster are all synchronized to the same reference clock, so their form (e.g., servers, virtual machines, smart NICs, etc.) does not matter.

[0030]

[0036] In a scenario where data flows 211 and 221 are the same data flow, transmitting host 210, transmitting host 220, and receiving host 200 form a cluster. Following this example, buffering of data packets can occur across the hosts in the cluster at the flow level. That is, one or more Netcam modules and / or Netcam system 140 may record all packets transmitted or received within a parameter in the buffer of a host in the data flow, regardless of what parameter the buffer uses to record data and subsequently overwrite data (e.g., most recently transmitted packets, packets transmitted / received within a given time, etc.). In one embodiment, a representation of the time sequence relative to a reference clock is stored with the buffered data (e.g., transmit timestamp 311 and / or receive timestamp 321 are stored with the buffered data packets). Thus, transmitting hosts 210 and 220 may store data packets sharing a given flow ID in their buffer 111, and receiving host 200 may store received packets in buffer 131. Alternatively or additionally, transmitted and / or received packets may be sent to the Netcam system 140, which may buffer the received data.

[0031]

[0037] This advantage of buffering a certain amount of data on each host in the cluster enables different functions of the host netcam module in response to the detection of anomalies. Figure 4 is a data flow diagram showing the netcam operation during a normal operation period and when an anomaly is detected during that period, according to an embodiment of the present disclosure. The data flow 400 reflects the host operation and netcam operation (e.g., the netcam module or netcam system 140 of the transmitting / receiving host) during the normal operation period and the “anomaly function” (i.e., the operation taken when an anomaly is detected) period. The data flow 400 first shows the normal operation, in which the host transmits or receives a data flow 402, and then the netcam module or system (generally referred to as “netcam” in this figure) determines whether an anomaly has been detected 404 (e.g., based on one-way delay as described above). If no anomaly is detected, assuming that the storage of data packets has been full for some time, the host (e.g., of the cluster) overwrites those buffers 406 (e.g., meaning overwriting the oldest packets as described above, or following some other overwrite heuristic). Of course, if the buffer is not full, overwriting is unnecessary, and the buffer is saved to available memory. Normal operation repeats unless an anomaly is detected.

[0032]

[0038] An anomaly function occurs when an anomaly is detected. Different anomaly functions are disclosed herein, and data flow 400 focuses on illustrating a specific anomaly function for retransmission of buffered data. When a host (e.g., of a cluster) transmits / receives data flow information 408, the Netcam may detect an anomaly 410. As previously stated, an anomaly is detected when the one-way delay exceeds a threshold. In a cluster, it should be noted that the threshold may vary between hosts in the cluster, depending on the distance between the transmitting and receiving hosts. In response to the detection of an anomaly, the Netcam instructs all hosts in the cluster to save the buffered data 412. That is, even if an anomaly is detected on one host in the cluster, data from all nodes in the cluster is saved. This can be achieved by instructing a host to save the buffered data (or a portion of the data flow related to the buffered data) in persistent memory, or by maintaining the buffered data in the buffer and pausing data transmission, or by a combination of these using different instructions to different hosts. If pausing is used, the pausing time may vary across different nodes in the cluster, as previously stated. Regardless of how the data is stored, the Netcam may jitter the retransmission timing.414 It should be noted that the time sequences of packet transmission and reception are reflected in the stored data packets. The Netcam may jitter the retransmission timing by altering the time sequence (e.g., by creating a longer time gap (lag) between transmissions than the previous time gap, or by transmitting packets in a different order). The jitter may occur heuristically or randomly. The jitter may be applied if a previously attempted time sequence was the cause of failure (e.g., the previously attempted time sequence itself may cause excessive temporary congestion), and therefore, scenarios in which retransmission without jitter would fail may succeed with the application of jitter. The Netcam then retransmits the buffered data (or a portion thereof).416In some cases, it may be more efficient and computationally inefficient to retransmit the entire buffer containing data unrelated to the data flow or anomaly, rather than isolating the packets of the data flow associated with the anomaly. Normal operation then resumes until another anomaly is detected.

[0033]

[0039] Retransmission with fluctuation is just one example of normal functionality, and any number of functions can occur in response to the detection of anomalies. For example, in addition to or alternative to the anomaly function shown in data flow 400, buffered data may be written to persistent memory and stored for forensic analysis. In such a scenario, in response to the detection of anomaly, the Netcam may send an alert to the administrator and / or generate an event log indicating the anomaly. Any other previously described anomaly functions are equally applicable. As an example of forensic analysis, a known type of attack against systems such as data centers is a timing attack. A timing attack may have a "signature" and the packet intervals of the traffic can be learned (e.g., by training a machine learning model with timing patterns labeled by whether the timing pattern was a timing attack). Forensic analysis may be performed to determine whether the data was a timing attack. A timing attack can be blocked (e.g., by dropping data packets from the buffer in response to the Netcam module 113's determination that the buffered data represents a timing attack).

[0034]

[0040] As mentioned above, buffered data may include bytestamps (in contrast to, or in addition to, buffered packets). Bytestamps can be used for anomaly analysis (e.g., in forensic analysis, network debugging, security analysis, etc.). The advantage of using bytestamps is that they save memory space compared to using buffered data packets, and bytestamps require less computation to process. To identify the cause of an anomaly, bytestamps corresponding to the time period of the anomaly can be analyzed. The trade-off for using bytestamps over buffered packets is that buffered packet data is more robust and may provide further insight into anomalies.

[0035]

[0041] Figure 5 is a network traffic diagram illustrating an embodiment of the present disclosure in which a receiving host receives both high-priority and low-priority traffic from a transmitting host. As shown in Figure 5, transmitting host 510 sends a high-priority data flow 511 to receiving host 500, and transmitting host 530 sends a low-priority data flow 531 to receiving host 500. If network congestion occurs and an anomaly is detected, the transmitting hosts may process the high-priority and low-priority traffic differently. In one embodiment, since the low-priority data flow 531 is associated with a lower one-way delay threshold than the high-priority data flow 511, transmitting host 530 detects network congestion earlier than transmitting host 510. Therefore, transmitting host 530 takes corrective action, such as pausing network transmission of the low-priority data flow 531 for a limited time, while continuing transmission of the high-priority data flow 511 because its high one-way delay threshold has not yet been reached. If the high-priority data flow 511 actually reaches its high one-way delay threshold and a pause operation is taken responsively, the pause time may be shorter than the pause time for the low-priority data flow 531. This ensures that the high-priority data flow 511 resumes earlier and at a less congested time than it would face if the low-priority data flow 531 were not paused for extra time while the high-priority data flow 511 continues.

[0036]

[0042] Although shown as two separate sending hosts, sending hosts 510 and 530 may be the same host, in which case one sending host sends both high-priority and low-priority traffic to receiving host 500. Thus, the same sending host can continue to send high-priority data flows 511 as usual while taking corrective action (e.g., suspending) in response to the detection of anomalies in low-priority data flows 531. The sending host may have multiple buffers 111, each corresponding to a different priority.

[0037]

[0043] Figure 6 is a data flow diagram illustrating that priority is considered when determining Netcam operation according to an embodiment of the present disclosure. Data flow 600 is initiated when one or more transmitting hosts (e.g., transmitting host 110) transmit 602 the data flow and apply a transmit timestamp (e.g., transmit timestamp 311). A receiving host (e.g., receiving host 130) receives 604 the data flow and applies a receive timestamp (e.g., receive timestamp 321). Then Netcam operation occurs. As described above, Netcam operation can occur at the transmitting host (e.g., by receiving an ACK packet indicating a receive timestamp and using the Netcam module to calculate the one-way delay), at the receiving host (e.g., when a transmit timestamp is included in the data flow and the Netcam module calculates the one-way delay from it), at the Netcam system 140, or in some combination thereof.

[0038]

[0044] The Netcam determines the one-way delay of data packets in a data flow.606 As previously described, the calculation of the one-way delay may depend on the priority of the data flow, and therefore different data flows may have different one-way delay thresholds ("priority thresholds"). The Netcam compares the determined one-way delay to the individual priority thresholds.608 In response to the determination that the one-way delay is greater than the threshold for a given data flow, an abnormal function is initiated. As shown in Figure 6, some abnormal functions may include one or more of the following: pausing transmission of data flows associated with a given priority,612 and / or saving buffered data flows associated with a given priority (for example, for forensic analysis).614 As mentioned above, the pause time may vary depending on the priority level of the paused data flow.

[0039]

[0045] Figure 7 is a flowchart illustrating an exemplary process for performing a Netcam operation according to an embodiment of the present disclosure. The process 700 may be performed by one or more processors (for example, based on computer-readable instructions for performing the operation stored on a non-temporary computer-readable medium). For example, Netcam modules 113, 133, and / or the Netcam system 140 may perform the process 700 by executing some or all of the instructions. The process 700 is described for convenience in relation to Netcam module 113, but may be performed by any other Netcam module, and / or the system.

[0040]

[0046] Process 700 begins with the sending host (e.g., sending host 110) recording a first predefined amount of transmitted network traffic of the data flow in a first sequential criterion (e.g., recording it in buffer 111) by the sending host (e.g., receiving host 130), and the receiving host recording a second predefined amount of received network traffic of the data flow in a second sequential criterion (e.g., recording it in buffer 131) by the receiving host, where the sending host and the receiving host are clock-synchronized (e.g., using the reference clock of the clock synchronization system 141).

[0041]

[0047] The Netcam module 113 monitors for anomalies in the data flow 706 based on the timestamps of data packets in the network traffic (for example, by subtracting the transmission timestamp 311 from the reception timestamp 321 and comparing the result with a one-way delay threshold). The Netcam module 113 determines 708 whether an anomaly is detected during the monitoring period (for example, based on whether the comparison shows a one-way delay greater than the threshold). In response to the determination that no anomalies are detected during the monitoring period, the Netcam module 113The network camera module 113 may passively allow the recorded transmitted network traffic and recorded received network traffic to be overwritten with newly transmitted network traffic and newly received network traffic, respectively (for example, by recording the latest network traffic on top of the oldest recorded data packets and continuing the repetition of elements 702-708). In response to a decision that an anomaly has been detected during the monitoring period, the network camera module 113 pauses the data flow 712, instructs the transmitting host to save the recorded transmitted network traffic to a first buffer, and instructs the receiving host to save the recorded received network traffic to a second buffer.

[0042]

[0048] Figure 8 is a flowchart illustrating an exemplary process for performing Netcam operation in a multi-priority scenario according to an embodiment of the present disclosure. Process 800 may be performed by one or more processors (for example, based on computer-readable instructions for performing the operation stored on a non-temporary computer-readable medium). For example, Netcam modules 113, 133, and / or Netcam system 140 may perform process 800 by executing some or all of the instructions. Process 800 is described for convenience in relation to Netcam module 113, but may be performed by any other Netcam module, and / or system.

[0043]

[0049] Process 800 begins by identifying 802 a first data flow between a first transmitting host (e.g., transmitting host 110) and a receiving host (e.g., receiving host 130), the first data flow having high priority (e.g., high-priority data flow 511), and the transmitting and receiving hosts are synchronized using a common reference clock. Netcam module 113 (e.g., of a different transmitting host, or of the same transmitting host as transmitting host 110) identifies 804 a second data flow between a second transmitting host and a receiving host (e.g., low-priority data flow 531), the second data flow having low priority, where the second transmitting host may be the same as or different from the first transmitting host.

[0044]

[0050] The Netcam module 113 assigns a first delay threshold to the first data flow based on high priority 806 and a second delay threshold to the second data flow based on low priority 806. The first delay threshold exceeds the second delay threshold. The Netcam module 113 monitors a first one-way delay of the data packets of the first data flow in relation to the first delay threshold 808 and monitors a second one-way delay of the data packets of the second data flow in relation to the second delay threshold 810. In response to the determination that the first one-way delay of the data packets of the first data flow exceeds the first delay threshold, the Netcam module 113 pauses the transmission of the data packets of the first data flow from the first transmitting host to the receiving host for a first time 812. 2 In response to the determination that the second one-way delay of the data packets of the data flow exceeds the second delay threshold, the netcam module 113 pauses the transmission of the data packets of the second data flow from the second transmitting host to the receiving host for a second time exceeding the first time.

Claims

1. A method performed on a computer for monitoring anomalies associated with congestion in network traffic, wherein the method is: Regarding the data flow transmitted between the sending host and the receiving host, Based on a first sequential criterion, the transmitting host records a first predetermined amount of transmitted network traffic for the data flow. According to a second sequential criterion, the receiving host records a second predetermined amount of received network traffic of the data flow, wherein the transmitting host and the receiving host are clock-synchronized. The method involves monitoring the anomaly based on the timestamp of the data packet in the network traffic, wherein the anomaly is a state in which the data packet is delayed for a predetermined time or longer. To determine whether the abnormality is detected during the monitoring period, In response to the determination that no abnormality is detected during the monitoring period, The previously recorded transmitted network traffic and the previously recorded received network traffic are overwritten with newly transmitted network traffic and newly received network traffic, respectively. In response to the determination that the abnormality was detected during the monitoring period, The data flow is paused, the transmitting host is instructed to save the recorded transmitted network traffic to a first buffer, and the receiving host is instructed to save the recorded received network traffic to a second buffer, and The transmitting host is instructed to retransmit the transmitted network traffic associated with the anomaly from the first buffer. Includes, The retransmission of the transmitted network traffic is a computer-based method comprising the transmitting host fluctuating the timing of the transmission of two or more data packets to the receiving host, the fluctuating using one or more predetermined or random time intervals.

2. The computer-based method according to claim 1, wherein the first default amount and the second default amount are the same amount.

3. The method performed on a computer according to claim 2, wherein the same amount is measured based on either a predetermined number of the most recently transmitted or received packets, or a predetermined number of packets received in a time unit prior to the present.

4. The method performed by a computer according to claim 1, wherein the determination that the anomaly was detected during the monitoring period includes the determination that the one-way delay of a data packet transmitted from the transmitting host to the receiving host exceeds a delay threshold amount.

5. The method performed by the computer further includes using a transmit timestamp applied by the transmitting host to the data packet of the data flow as a comparison with a receive timestamp applied by the receiving host to the data packet, wherein the transmitting host and the receiving host are synchronized to the same reference clock, and the transmit timestamp and the receive timestamp are based on the same reference clock, the method performed by the computer according to claim 4.

6. The method executed on the computer according to claim 1, wherein the transmitting host is located in either the kernel or the transmitter's network interface card (NIC), and the receiving host is located in either the kernel or the receiver's NIC.

7. The method executed on a computer according to claim 6, wherein the transmitter is a physical machine or a virtual machine, and the receiver is a physical machine or a virtual machine.

8. The method performed on a computer according to claim 1, wherein the data flow includes data packets transmitted by a plurality of transmitting hosts, including the transmitting host, and each of the plurality of transmitting hosts records the data packets it transmitted as part of the data flow.

9. The method performed on a computer according to claim 8, wherein in a network camera system that monitors the data flow, in response to a decision that an anomaly has been detected during the monitoring period, the network camera system causes each of the plurality of transmitting hosts to store in its respective first buffer the data packets that have been transmitted as part of the data flow and recorded therein.

10. The method performed by a computer according to claim 9, wherein the operation by the Netcam system further includes, in response to a determination that the anomaly was detected during the monitoring period, causing the data packets associated with the anomaly to be retransmitted from the first buffer of each individual transmitting host, and restarting the data flow.

11. The method performed by a computer according to claim 1, wherein the retransmission of the data packets includes retransmitting all of the data packets from the first buffer of each individual transmitting host.

12. A non-temporary computer-readable medium including memory having encoded instructions for monitoring anomalies associated with congestion in network traffic, wherein the instructions, when executed by one or more processors, Regarding the data flow transmitted between the sending host and the receiving host, Based on a first sequential criterion, the transmitting host records a first predetermined amount of transmitted network traffic for the data flow, and According to a second sequential criterion, the receiving host records a second predetermined amount of received network traffic of the data flow, wherein the transmitting host and the receiving host are clock-synchronized. The method involves monitoring the anomaly based on the timestamp of the data packet in the network traffic, wherein the anomaly is a state in which the data packet is delayed for a predetermined time or longer. To determine whether the abnormality is detected during the monitoring period, In response to the determination that no abnormality is detected during the monitoring period, The previously recorded transmitted network traffic and the previously recorded received network traffic are overwritten with newly transmitted network traffic and newly received network traffic, respectively. In response to the determination that the anomaly was detected during the monitoring period, The data flow is paused, the transmitting host is instructed to save the recorded transmitted network traffic to a first buffer, and the receiving host is instructed to save the recorded received network traffic to a second buffer, and The transmitting host is instructed to retransmit the transmitted network traffic associated with the anomaly from the first buffer, the retransmission including the transmitting host fluctuating the timing of the transmission of two or more data packets to the receiving host, the fluctuating being done using one or more predetermined or random times. A non-temporary, computer-readable medium containing instructions for executing a command.

13. The non-temporary computer-readable medium according to claim 12, wherein the first default amount and the second default amount are the same amount.

14. The non-transient computer-readable medium according to claim 13, wherein the same quantity is measured based on either a predetermined number of the most recently transmitted or received packets, or packets received in a predetermined number of time units prior to the present.

15. The non-temporary computer-readable medium according to claim 12, wherein the determination that the anomaly was detected during the monitoring period includes the determination that the one-way delay of a data packet transmitted from the transmitting host to the receiving host exceeds a delay threshold amount.

16. The instruction further includes an instruction for determining a one-way delay by using a transmit timestamp applied by the transmitting host to the data packet of the data flow as a comparison with a receive timestamp applied by the receiving host to the data packet, wherein the transmitting host and the receiving host are synchronized to the same reference clock, and the transmit timestamp and the receive timestamp are based on the same reference clock, the non-temporary computer-readable medium according to claim 15.

17. The non-temporary computer-readable medium according to claim 12, wherein the transmitting host is located in either the kernel or the transmitter's network interface card (NIC), and the receiving host is located in either the kernel or the receiver's NIC.

18. The non-temporary computer-readable medium according to claim 17, wherein the transmitter is a physical machine or a virtual machine, and the receiver is a physical machine or a virtual machine.

19. The non-temporary computer-readable medium according to claim 12, wherein the data flow includes data packets transmitted by a plurality of transmitting hosts, including the transmitting host, and each of the plurality of transmitting hosts records the data packets it transmitted as part of the data flow.

20. The non-temporary computer-readable medium according to claim 12, wherein a byte stamp for the aforementioned data flow is recorded in persistent memory.

Images