Deploying Shadow Buffers in Clock-Synchronized Edge-Based Network Functions

Abstract

A regular buffer and a shadow buffer are maintained at the receiving host. In response to receiving a data flow from a sending host that is clocked with the receiving host using a common reference clock, a first representation of data of the data flow is stored in the regular buffer, the shadow buffer is transitioned from an idle state to an active state, and a counter in the shadow buffer indicating a unit of data traffic received is incremented.

Description

[Technical field]

[0001]

[0002] TECHNICAL FIELD This disclosure relates generally to network transmissions within data flows and coordinated control of network traffic. [Background technology]

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 320,160, filed May 13, 2022, the entire contents of which are incorporated by reference herein.

[0003]

[0003] Modern Internet infrastructures typically have large data centers that generate huge amounts of network traffic. When demand is high, the output of the data center may be constrained (e.g., by the capacity of switches, gateways, etc.) and network traffic may have to be metered. Such temporary congestion scenarios may cause bottlenecks and result in dropped packets. To ensure that packet transmissions are successful when faced with such conditions, systems have been developed that send acknowledgements from the receiving node to the sending node when a packet is received. However, these acknowledgements are inefficient in that they further contribute to additional network traffic. Furthermore, these acknowledgements are limited to working in single sender to single receiver scenarios. Furthermore, if an acknowledgement is not received, the packet will simply be retransmitted ad-hoc, potentially flowing into the same congested switch and leading to the same discard outcome, resulting in scenarios where the packet is delayed forever or is not even received by the destination. Additionally, these scenarios are rooted in congestion that has already occurred and are not sufficient to prevent congestion from occurring in the first place. [Brief description of the drawings]

[0004] [Figure 1]FIG. 1 illustrates an exemplary system environment for implementing a netcam and priority function according to one embodiment of the present disclosure. [Diagram 2]

[0005] FIG. 2 is a network traffic diagram illustrating multiple sending hosts sending multiple data flows to a single receiving host, according to one embodiment of the present disclosure. [Diagram 3]

[0006] FIG. 3 is a network traffic diagram illustrating time stamp operations at both the sending and receiving ends of a data transmission in accordance with one embodiment of the present disclosure. [Figure 4]

[0007] FIG. 4 is a data flow diagram illustrating netcam activity during normal operation and where anomalies are detected according to one embodiment of the present disclosure. [Diagram 5]

[0008] FIG. 5 is a network traffic diagram illustrating a receiving host receiving both high and low priority traffic from a sending host, according to one embodiment of the present disclosure. [Figure 6]

[0009] FIG. 6 is a data flow diagram illustrating netcam activity where priority is considered in determining the netcam activity according to one embodiment of the present disclosure. [Figure 7]

[0010] FIG. 7 is a flowchart illustrating an example process for performing netcam activities according to one embodiment of the disclosure. [Figure 8]

[0011] FIG. 8 is a flowchart illustrating an example process for performing netcam activities in a multiple priority scenario, according to one embodiment of the present disclosure. [Figure 9]

[0012] FIG. 9 is a data flow diagram illustrating netcam activity in which consideration of shadow buffers is shown, according to one embodiment of the present disclosure. [Figure 10]

[0013] FIG. 10 is a flowchart illustrating an exemplary process for performing netcam activity in coordination with shadow buffer considerations, according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005]

[0014] The figures and the following description relate to preferred embodiments by way of example only. From the following description, alternative embodiments of the structures and methods disclosed herein should be readily recognized as viable alternatives that may be employed without departing from the principles of what is claimed.

[0006]

[0015] A system and method for adjusting the control of data flows in the face of temporary congestion is disclosed herein. A "netcam" monitors network traffic between clock-synchronized sending and receiving hosts that are part of a data flow. The term "netcam" as used herein is a contraction of "network camera" and is a module that tracks network traffic and ensures that corrective action is taken if traffic of a data flow in a clock-synchronized system is delayed beyond acceptable limits. A netcam instructs sending and receiving hosts to buffer copies of network traffic according to some parameters (e.g., buffer a certain number of packets, buffer packets for a rolling window of time, etc.). The buffers may be overwritten on a rolling basis (e.g., overwriting the oldest packets when a new packet is sent or received and when the buffer becomes full) once the parameters are met. A netcam may cause all sending and receiving hosts to write buffered data where an anomaly is detected and may cause the sending host to retransmit the written packets. Retransmissions may be subject to jitter (e.g., time delay between packet transmissions of a data flow) such that if a transmission delay or failure occurs due to a given packet transmission sequence, the jitter causes enough variation that a retransmission attempt will nevertheless be successful. Netcam may determine the need to write and retransmit packets differently depending on the priority of the data flow. Netcam may instruct the receiving host's shadow buffer to monitor path utilization and capacity, and high utilization and / or low capacity may cause Netcam to predict impending anomalies and take corrective action similar to the case of a full buffer.

[0007]

[0016] Advantageously, the implementation of the netcam disclosed herein allows for both improved network transmission and forensic analysis. Improved network transmission occurs in that writing the most recent packet transmission and attempting to buffer across all machines in the data flow allows for retransmission of an exact set of packets from many machines without relying on acknowledgement packets that may be lost or dropped across a complex web of machines. Additionally, virtual machines may have bugs that are difficult to detect or isolate. Writing packet sequences associated with anomalies allows for failure analysis, which may allow identification of a faulty virtual machine. Additionally, using shadow buffers to predict anomalies may prevent scenarios where traffic is overly congested, allowing some capacity to remain on the path and corrective action to occur without pausing traffic. Additional advantages and improvements will become apparent from the disclosure below.

[0008]

[0017] 1 is a diagram illustrating an exemplary system environment for implementing the netcam and priority functions according to one embodiment of the present disclosure. As shown in FIG. 1, the netcam environment 100 includes a sending host 110, a network 120, a receiving host 130, and a clock synchronization system 140. Although only one of each of the sending host 110 and receiving host 130 is illustrated, this is merely for convenience and ease of illustration, and any number of sending hosts and receiving hosts may be part of the netcam environment 100.

[0009]

[0018] The transmitting host 110 includes a buffer 111, a network interface card (NIC) 112, and a netcam module 113. The buffer 111 stores copies of outbound data transmissions until one or more criteria for overwriting or discarding packets from the buffer are met. For example, the buffer may store data packets until capacity is reached, at which point the oldest buffered data packets may be discarded or overwritten. Other criteria may include time aging (e.g., discarding packets after a predetermined amount of time has passed since their transmission timestamp), amount of buffered packets (e.g., after a predetermined amount of packets have been buffered, begin to discard or overwrite the oldest packets as new packets are transmitted), etc.

[0010]

[0019] In one embodiment, buffer 111 stores information about a given outbound transmission, rather than the entire packet. For example, a byte stamp indicating an identifier for the packet and / or a flow identifier, as well as a timestamp when the packet (or aggregate data flow) was sent, may be stored instead of the packet itself. In such an embodiment, the stored information does not need to be overwritten and may be stored in persistent memory of transmitting host 110 and / or clock synchronization system 140. This embodiment is not mutually exclusive to buffer 111 storing copies of packets, and they may be employed in combination.

[0011]

[0020] The NIC 112 can be any type of network interface card, such as a smart NIC. The NIC 112 interfaces the sending host 110 and the network 120.

[0012]

[0021] The netcam module 113 monitors the data flow for certain conditions and triggers functions based on the monitored data. As an example, in response to detecting network congestion, the netcam module 113 may instruct all hosts that are part of the data flow to perform one or more of a variety of actions, such as pausing transmissions, taking a snapshot of buffered data transmissions (i.e., writing buffered data packets to persistent memory), and performing other coordinated activities. As used herein, the term data flow may refer to a collection of data transmissions between two or more hosts that are associated with each other. Further details of the netcam module 113 are described in more detail in connection with Figures 2-8 below. The netcam module 113 may be implemented in any component of the sending host 110. In one embodiment, the netcam module 113 may be implemented in the NIC 112. In another embodiment, the netcam module 113 may be implemented in the kernel of the sending host 110.

[0013]

[0022] Network 120 may be any network, such as a wide area network, a local area network, the Internet, or any other conduit for data transmission between sending host 110 and receiving host 130. In some embodiments, network 120 may be within a data center that houses both sending host 110 and receiving host 130. In other embodiments, network 120 may facilitate cross-data center transmission over any distance. Reference to a data center is merely exemplary, and sending host 110 and receiving host 130 may be implemented in any medium, including those that are not data centers.

[0014]

[0023] Receiving host 130 includes netcam buffer 131, NIC 132, netcam module 133, and shadow buffer 134. Netcam buffer 131, NIC 132, and netcam module 133 operate similarly to the analogous components described above with respect to sending host 110. Buffer 131 may be the same or different size as buffer 111, and may additionally or alternatively store byte stamps of received packets. Any further distinctions between these components implemented in a receiving host versus a sending host will be made clear based on the disclosure of Figures 2-8 below.

[0015]

[0024] Shadow buffer 134 may be used to track data traffic in a manner that allows for early warning of when congestion is imminent. For example, when data traffic is buffered, congestion may occur when the buffer is full, and the congestion prevents further data traffic from flowing until the congestion clears. Shadow buffers may increment a counter faster than regular buffers (e.g., may increment by 1.1 when one unit of data is received in the regular buffer) and / or may decrement a counter slower than regular buffers (e.g., may decrement by 0.9 or 0.95 when one unit of data is cleared in the regular buffer). As used herein, the term regular buffer may refer to the operation of buffer 111 and / or buffer 131 and / or other buffers disclosed herein having functionality similar to that of buffer 111 and / or buffer 131. Although only one shadow buffer 134 is shown in FIG. 1, multiple shadow buffers may be employed at the receiving host, with each of the shadow buffers being assigned to a different subset of data flows, such as individual data flows corresponding to the same application. The shadow buffers may be incremented / decremented at different rates (e.g., to indicate more congestion for low priority applications and less congestion for high priority applications). Alternatively, the shadow buffers may be incremented / decremented at the same rate, but different thresholding may be applied for different applications as to when a data flow should be considered to be experiencing congestion. Data buffered in the regular buffers includes data traffic (e.g., network packets) received by the receiver, and data is removed from the regular buffers once the data has been processed and / or routed to its next destination.The operations described herein of the netcam module 113 and / or the netcam system 140 that operate with respect to the conditions satisfied for the regular buffer may also be performed when the shadow buffer 134 indicates congestion.

[0016]

[0025] The netcam system 140 includes a clock synchronization system 141. The netcam system 140 may monitor data observed by netcam modules implemented in the hosts, such as netcam modules 131 and 133. The netcam system 140 may detect conditions that require action by the netcam modules and may send instructions to the affected netcam modules to take coordinated action for a given data flow. The clock synchronization system 141 synchronizes one or more components of each of the hosts, such as the NIC, kernel, or any other component on which the netcam modules operate. Details of clock synchronization are described in U.S. Patent No. 10,623,173, issued April 14, 2020 to Shared, which is incorporated herein by reference in its entirety. Individual hosts are synchronized to the same reference clock with very precise accuracy, allowing accurate time stamps between hosts regardless of host location, host bandwidth conditions, jitter, etc. Further details of the netcam system 140 are disclosed below with reference to Figures 2-8. The netcam system 140 is an optional component of the netcam environment 100, allowing the netcam modules in the sending host and / or receiving host to operate without relying on a centralized system other than relying on a synchronized reference clock.

[0017]

[0026] The netcam environment 100 has many advantages. The netcam module is edge-based, considering that it can run in the kernel or NIC (e.g., smart NIC) of a host (e.g., a physical host, a virtual machine, or any other form of host). In one embodiment, the netcam function can run as an underlay, meaning that it can run as a shim, on a layer of the OSI system below the congestion control layer (e.g., layer 3 of the OSI system). The netcam module and / or netcam system 140 can instruct the host to perform an activity if a condition is detected (e.g., congestion signals are detected using a shadow buffer), such as pausing transmission of data flows across the affected host, taking a snapshot (i.e., writing some or all of the buffered data, such as the last N bytes sent and / or the bytes sent in the last S seconds, where N or S may be default values ​​or may be defined by an administrator), and any other activity disclosed herein. Further advantages and features are described below with respect to Figures 2-8.

[0018]

[0027] 2 is a network traffic diagram illustrating multiple sending hosts sending multiple data flows to a single receiving host, according to one embodiment of the present disclosure. As shown in FIG. 2, sending host 1 sends data flow 211 to receiving host 200, sending host 220 sends data flow 221 to receiving host 200, and any number of additional hosts, represented by sending host 230, may send their respective data flows (represented by data flow 231) to receiving host 200. As shown in FIG. 2, the individual data flows sent by each of the sending hosts are different, but this is merely for convenience, as two or more sending hosts may send data from the same data flow. Furthermore, a single sending host may send two or more different data flows to receiving host 200. Although only one receiving host is illustrated, a sending host may send data flows to any number of receiving hosts.

[0019]

[0028] The operation of the netcam module at the sending host and the receiving host will now be described with reference to FIG. 3. FIG. 3 is a network traffic diagram illustrating the time stamp operation at both the sending and receiving sides of a data transmission according to one embodiment of the present disclosure. As shown in FIG. 3, when the sending host 310 sends a packet to the receiving host 320, the netcam module 113 of the receiving host 320 records a sending time stamp 311. Similarly, when the receiving host 320 receives the packet, the netcam module 133 of the receiving host 320 applies a receiving time stamp 321. The time stamp reflects the time when the data packet was sent or received by the associated component (e.g., NIC, kernel, etc.) in which the netcam module is installed. The sending time stamp may be stored in the buffers 111 and 131, may be attached to the packet, may be sent for storage in the netcam system 140, or any combination thereof.

[0020]

[0029] Since the sending host 310 is synchronized to the same reference clock as the receiving host 320, the time elapsed between the time of the send timestamp 311 and the receive timestamp 321 reflects the one-way delay of a given packet. In one embodiment, when a given packet is received, the receiving host 320 sends an acknowledgment packet to the sending host 310 indicating the receive timestamp 321, and the netcam module 113 can calculate the one-way delay by subtracting the send timestamp 311 from the receive timestamp 321. Other means of calculating the one-way delay are within the scope of this disclosure. For example, the send timestamp 311 may be appended to the data transmission, and the receiving host 320 may thereby calculate the one-way delay without the need for an acknowledgment packet. As yet another example, the netcam modules of the sending host and the receiving host may collectively or individually send timestamps to the netcam system 140, from which the netcam system 140 may calculate the one-way delay. For convenience and brevity, the scenario in which the sending host 110 calculates one-way delay based on acknowledgment packets will be the focus of the following disclosure, but those skilled in the art will recognize that either of these calculation means apply equally well.

[0021]

[0030] In one embodiment, the netcam system then determines whether the one-way delay exceeds a threshold. For example, after calculating the one-way delay, the sending host 110 may compare the one-way delay to a threshold. The threshold may be predetermined or dynamically determined. The predetermined threshold may be set by default or may be set by an administrator. As described further below, different thresholds may be applied to different data flows depending on one or more attributes of the data flows, such as the priority of the data flows. The threshold may be dynamically determined depending on any number of factors, such as dynamically increasing the threshold as congestion decreases and decreasing the threshold as congestion increases (e.g., because congestion is more likely to indicate a problem where congestion is not the cause of the delay or is a minor cause). In one embodiment, the threshold may be set on a per-host basis, as it may depend on the distance between the sending host and the receiving host. In such an embodiment, the threshold may be a predetermined multiple of the minimum one-way delay between the sending host and the receiving host. That is, the minimum time it takes for a packet to travel from the sending host to the receiving host would be the minimum one-way delay. The multiple is typically 1.5 to 3 times the minimum, but may be any multiple defined by the netcam administrator. The threshold is equal to the multiple of the minimum one-way delay. In response to determining that the one-way delay exceeds the threshold, the netcam module 113 may instruct the sending host 110 to take one or more actions.

[0022]

[0031] In additional or alternative embodiments, the determination of whether to take one or more actions may be performed using a separate measure of the state of the shadow buffer (e.g., shadow buffer 134). In short (discussed in further detail below), during a given data flow, and in parallel with buffering the data using the regular buffer, the netcam module 133 may direct the shadow buffer 134 to be incremented for each unit of data traffic received by the receiving host 320. The netcam module 133 may define a dynamic drain rate, which is the rate at which the netcam module 133 directs the shadow buffer 134 to be decremented. The dynamic drain rate may be determined by the netcam module 133 based on the number of data removed from the buffer 131 per unit time (e.g., multiplied by a factor that causes draining to occur slower in the shadow buffer 134 than in the buffer 131). The netcam module 133 may calculate the residence time as a function of the shadow buffer's 134 counter and the dynamic drain rate (e.g., the residence time may be calculated by dividing the value of the shadow buffer's counter by the dynamic drain rate). From this, the netcam module 133 may determine that the shadow buffer's one-way delay is the actual one-way delay (determined from the transmit and receive timestamps described above) aggregated with the residence time. The shadow buffer's one-way delay may be compared to a threshold (in addition to or instead of the regular buffer's one-way delay) and used to determine whether to take one or more actions.

[0023]

[0032] Whether driven by regular buffer or shadow buffer one-way delay, these one or more actions may include pausing transmissions from the sending host when the one-way delay is high, which reduces congestion and thereby generally reduces packet drops on the network 120. The pause may be a predetermined time or may be dynamically determined in proportion to the magnitude of the one-way delay. In one embodiment, the pause may be equal to the one-way delay or may be determined by applying an administrator-defined multiple to the one-way delay. In one embodiment, the netcam may determine whether a previous pause is being forced and, if so, reduce the pause time based on the amount of previous pause time that has already elapsed since the previously seen packet. Additionally, a given data flow may not be the only data flow contributing to congestion, and thus its pause time may be less than the one-way delay or one-way delay threshold.

[0024]

[0033] Another possible action is to write some or all of the buffered data packets (e.g., from either or both of the sending and receiving hosts) to persistent memory in response to a one-way delay exceeding a threshold. Diagnostics may then be performed on the buffered data packets (e.g., to identify network problems). Further actions are described with respect to Figures 4-8.

[0025]

[0034] In some embodiments, data flows may be associated with different priorities. The netcam module may determine the priority of a data flow based on an explicit identifier (e.g., a traffic layer identifier in a data packet header) or based on inference (e.g., based on heuristics where rules are applied to packet headers and / or payloads to determine priority type). As used herein, priority refers to a priority scheme where types of data packets should be allowed to be transmitted and should be paused during periods of congestion. The priorities disclosed herein are considered in the context of selecting packets to transmit during network congestion, avoiding the need to make explicit allocations of link underutilization or bandwidth, and instead.

[0026]

[0035] To prioritize high priority packets, a high one-way threshold may be assigned to high priority traffic and a low threshold relative to the high one-way threshold may be assigned to low priority traffic. These thresholds may be used to compare with either or both of the shadow buffer one-way delay and / or the regular buffer one-way delay. In this manner, low priority packets are detected as anomalies more frequently than high priority packets, since a lower one-way delay must be detected for anomalies to be detected by the Netcam module, whereas high priority packets are only detected as anomalies if they violate a higher one-way delay threshold. Following the above discussion of determining the one-way threshold for a given host, different one-way thresholds may be applied to different data packets sent by or received by the same host, depending on the priority. In a priority embodiment, the one-way threshold may be determined in the manner described above (e.g., by applying a predetermined multiple to the threshold), with the determination additionally affected by applying a priority multiple. The priority multiple may be set by an administrator for any given type of priority, but will be higher for higher priorities and lower for lower priorities. Priorities need not be binary, and any number of priority hierarchies may be established, each corresponding to a different type or types of data traffic, each having a different multiplier. Priorities and their associated multipliers may change over time for a given data flow (e.g., a priority may be reduced if the data flow begins transmitting a different type of data packet that does not require high latency transmission).

[0027]

[0036] In addition to or instead of using a priority multiple for the one-way delay threshold and varying the one-way delay threshold based on the priority of a given packet or data flow for which the packet is being transmitted, the netcam module may manipulate the pause time of traffic paused during a pause operation in a different manner depending on the priority. A low pause time may be assigned to higher priority traffic and a relatively high pause time may be assigned to lower priority traffic, ensuring that the lower priority traffic is paused more frequently than the higher priority traffic during periods of congestion, thereby ensuring that the higher priority traffic has more bandwidth available while the lower priority traffic is paused. The pause time may be determined in the same manner as above, but with an additional step of applying an additional pause multiple to the pause time, with a lower pause multiple (e.g., a multiple less than 1, such as 0.7x) for the higher priority traffic and a higher pause multiple (e.g., a multiple greater than 1) for the lower priority traffic.

[0028]

[0037] Priority may be assigned in any number of ways. In one embodiment, one or more "carpool lanes" may be assigned that may be used by data flows with eligible priorities. For example, a "carpool lane" may be a bandwidth allocation that does not guarantee a minimum bandwidth for a given data communication, but is only accessible by data flows that meet required parameters. Exemplary parameters may include one or more priorities that are eligible to use the reserved bandwidth of a given "carpool lane." As an example, a carpool lane may require that a data flow have at least a medium priority, and thus both medium and high priorities are eligible in a three priority system having low, medium, and high priorities. As another example, there may be multiple carpool lanes (e.g., a carpool lane that can be accessed by both medium and high priority traffic, plus a carpool lane that can only be accessed by high priority traffic).

[0029]

[0038] In one embodiment, guaranteed bandwidth may be assigned to a given priority. For example, a high priority data flow may be assigned a minimum bandwidth, such as 70 mbps. In such an embodiment, excess unused bandwidth from the guaranteed band may be assigned to a lower priority data flow until such time that bandwidth is required by the data flow covered by the guarantee. The guaranteed bandwidth may be absolute or relative. A relative guarantee ensures that a given priority data flow receives at least a certain relative amount more bandwidth than a low priority data flow. For example, a high priority data flow may be guaranteed three times the bandwidth of a low priority data flow, and a medium priority data flow may be guaranteed twice the bandwidth of a low priority data flow.

[0030]

[0039] Returning to FIG. 2, when two or more sending hosts send data from the same data flow, those nodes, in conjunction with any receiving hosts receiving data from the data flow, may be referred to as a "cluster." In one embodiment, the data flows may be identified by a set of identifiers that, when all are detected, represent that a data packet is part of the data flow. For example, the netcam module of any host may determine a flow identifier that identifies the data flow to which the packet belongs based on a combination of source address, destination address, source port number, destination port number, and protocol port number. Other combinations of identifiers may be used to identify the data flow of which the packet is a part. As previously mentioned, all hosts in a cluster are clocked to the same reference clock, regardless of their form (e.g., server, virtual machine, smart NIC, etc.).

[0031]

[0040] In a scenario where data flows 211 and 221 are the same data flow, sending host 210, sending host 220, and receiving host 200 form a cluster. Following this example, buffering of data packets (both across regular buffers and shadow buffers) may occur at a per-flow level across a cluster of hosts. That is, one or more netcam modules and / or netcam system 140 may record in a buffer of a host of a data flow all packets sent or received within any parameters that the buffer uses to record and then overwrite data (e.g., recently sent packets, packets sent / received within a given time, etc.). Furthermore, a receiving node that receives packets of a data flow from multiple sending hosts (e.g., receiving host 200 receiving packets from sending hosts 210 and 220) may maintain a single shadow buffer for the data flow or may maintain one separate shadow buffer for each of sending hosts 210 and 220. In one embodiment, an index of the time sequence relative to a reference clock is stored with the buffered data (e.g., transmit timestamp 311 and / or receive timestamp 321 are stored with the buffered data packets). Thus, transmitting host 210 and transmitting host 220 may store data packets 111 that share a given flow ID in a buffer, and receiving host 200 may store received packets in buffer 131. Alternatively or additionally, transmitted and / or received packets may be sent to netcam system 140, which may buffer the received data.

[0032]

[0041] From this advantage of buffering a certain amount of data at each host of the cluster, different functions of the host netcam module are possible in response to detection of an anomaly (e.g., the aforementioned conditions mentioned with respect to FIG. 2 above). FIG. 4 is a data flow diagram showing netcam activity during normal operation and where an anomaly is detected, according to one embodiment of the present disclosure. Data flow 400 reflects host activity and netcam activity (e.g., activities taken by a sending / receiving host or netcam module of the netcam system 140) during normal function and during "abnormal function" (i.e., actions taken if an anomaly is detected). Data flow 400 first shows normal function of the host sending or receiving 402 data flow, and the netcam module or system (generally referred to as "netcam" in this figure) determining 404 whether an anomaly is detected (e.g., based on one-way delay as described above). If no anomaly is detected, the hosts (e.g., of the cluster) overwrite 406 their buffers (meaning, for example, overwriting the oldest packets, or following some other overwrite heuristic, as described above), assuming that the buffers are full from previous storage of the data packet. Of course, if the buffers are not full, no overwriting is necessary, and the free memory of the buffer is stored. Normal functioning is repeated as long as no anomaly is detected.

[0033]

[0042] An anomaly function occurs when an anomaly is detected. Different anomaly functions are disclosed herein, and data flow 400 focuses on illustrating the specific anomaly function of retransmitting buffered data. In the case of sending / receiving 408 information of a data flow by a host (e.g., of a cluster), the netcam may detect 410 an anomaly. As described above, an anomaly is detected when the one-way delay (e.g., delay of the shadow buffer and / or regular buffer) exceeds a threshold. In the case of a cluster, the threshold may differ between hosts of the cluster depending on the distance between the sending host and the receiving host. In response to detecting an anomaly, the netcam instructs 412 to store the buffered data to all hosts of the cluster. That is, if an anomaly occurs even in one host of the cluster, data from all nodes of the cluster is stored. This may occur by instructing the host to store the buffered data (or parts thereof related to the data flow) in persistent memory, or by keeping the buffered data in the buffer and pausing data transmission, or by combining them with different instructions for different hosts. Note that if pausing is used, the pause time may vary across different nodes of the cluster, as described above. Regardless of how the data is stored, the netcam may jitter 414 the retransmission timing. The time sequence of packet transmission and reception is reflected in the stored data packets. The netcam may jitter 414 the retransmission timing by altering the time sequence (e.g., creating a longer delay during the previous time gap between transmissions, sending packets in a different order, etc.). Jitter may occur according to a heuristic or may be random. Jitter is applied in cases where a previously attempted time sequence was the cause of failure (e.g., because the previously attempted time sequence itself may have caused too much temporary congestion), and thus jitter may result in a jitter-free retransmission failing in such scenarios. The netcam then retransmits 416 the buffered data (or a portion thereof).Note that rather than quarantining packets of a data flow that are associated with the data flow or anomaly, it may be more expedient and computationally efficient to resend the entire buffer containing data that is not associated with the data flow or anomaly, after which normal functioning resumes until another anomaly is detected.

[0034]

[0043] Retransmission with jitter is just one example of an anomalous function, and any number of functions may occur in response to the detection of an anomaly. For example, in addition to or in the alternative to the anomalous functions shown in data flow 400, the buffered data may be written to persistent memory and stored for forensic analysis. In such a scenario, the netcam may send an alert to an administrator and / or generate an event log indicating the anomaly in response to the detection of the anomaly. Other previously mentioned anomalous functions are applicable as well. As an example of forensic analysis, a known type of attack against systems such as data centers is a timing attack. A timing attack may have a "signature" in that the intervals between packets of traffic can be learned (e.g., by training a machine learning model using timing patterns labeled with whether the timing pattern was a timing attack, by using pattern recognition, etc.). A forensic analysis may be performed to determine if the data was a timing attack. The timing attack may be blocked (e.g., by dropping data packets from the buffer if the netcam module 113 determines that the buffered data represents a timing attack).

[0035]

[0044] As mentioned above, the buffer data may include byte stamps (as opposed to, or in addition to, the buffer packets). The byte stamps may be used for analysis of anomalies (e.g., forensic analysis, network debugging, security analysis, etc.). The advantage of using byte stamps over buffered data packets is that storage space is saved and computational processing costs are reduced. The byte stamps for the period corresponding to the anomaly may be analyzed to determine the cause of the anomaly. The tradeoff in using byte stamps over buffered packets is that the buffered packet data may be more robust and provide further insight into the anomaly.

[0036]

[0045] FIG. 5 is a network traffic diagram illustrating a receiving host receiving both high priority traffic and low priority traffic from a sending host, according to one embodiment of the present disclosure. As shown in FIG. 5, a sending host 510 sends a high priority data flow 511 to a receiving host 500, and a sending host 530 sends a low priority data flow 531 to the receiving host 500. When network congestion occurs and an anomaly is detected, the sending host can treat the high priority traffic and the low priority traffic differently. In one embodiment, the sending host 530 detects the network congestion sooner than the sending host 510 because the low priority data flow 531 is associated with a lower one-way delay threshold than the high priority data flow 511. Thus, the sending host 530 can implement corrective actions such as pausing network transmission of the low priority data flow 531 for the pause time, while continuing transmission of the high priority data flow 511 because its higher one-way delay threshold has not yet been reached. When high priority data flow 511 reaches its high one-way delay threshold and a pause action is taken in response, the pause time may be shorter than the pause time of low priority data flow 531, thus ensuring that high priority data flow 511 resumes sooner and during a less congested time than would be encountered if low priority data flow 531 was not paused for the extra time while high priority data flow 511 continues.

[0037]

[0046] Similarly, with respect to the operation of shadow buffers, a high priority shadow buffer may be maintained separately by the receiving host 500 for the high priority data flow 511, and a low priority shadow buffer may be maintained separately by the receiving host 500 for the low priority data flow 531. The drain rates may be weighted differently on a priority basis. For example, a high priority shadow buffer may have a higher drain rate relative to the drain rate used for a low priority shadow buffer, and thus the high priority shadow buffer will be less likely to cause anomaly detection than a low priority shadow buffer.

[0038]

[0047] Although shown as two separate sending hosts, sending hosts 510 and 530 may be the same host where one sending host sends both high priority and low priority traffic to receiving host 500. Thus, the same sending host may take corrective action (e.g., quiesce) in response to detecting an anomaly in low priority data flow 531 while continuing to send high priority data flow 511 as normal. A sending host may have multiple buffers 111, with individual buffers corresponding to different priorities of data.

[0039]

[0048] 6 is a data flow diagram illustrating netcamming activity where priority is considered in determining netcamming activity, according to one embodiment of the present disclosure. Data flow 600 begins with one or more sending hosts (e.g., sending host 110) sending 602 a data flow and applying a send timestamp (e.g., send timestamp 311). A receiving host (e.g., receiving host 130) receiving 604 the data flow and applying a receive timestamp (e.g., receive timestamp 321). Netcamming activity then occurs. As described above, netcamming activity may occur at the sending host (e.g., by receiving an ACK packet indicating a receive timestamp and using a netcam module to calculate one-way delay), at the receiving host (e.g., if a send timestamp is included in the data flow and the netcam module calculates one-way delay therefrom), at the netcam system 140, or some combination thereof.

[0040]

[0049] The netcam determines 606 the one-way delay of the data packets in the data flow. As described above, the one-way delay calculation may depend on the priority of the data flow, and thus different data flows may have different one-way delay thresholds ("priority thresholds"). The one-way delay may be determined from the packets in general and / or aggregated with residence times to form a shadow buffer one-way delay. The netcam compares 608 the determined one-way delay (or delay, if shadow buffer one-way delay is used) to the respective priority thresholds. In response to a determination 610 that the one-way delay is greater than the threshold for a given priority data flow, an abnormal function is initiated. As shown in FIG. 6, some abnormal functions may include one or more of pausing 612 transmission of a data flow associated with a given priority and / or storing 614 (e.g., for forensic analysis) the buffered data flow associated with a given priority. As described above, the pause time may vary depending on the priority level of the paused data flow.

[0041]

[0050] 7 is a flow chart illustrating an example process for performing netcam activity, according to one embodiment of the disclosure. Process 700 may be executed by one or more processors (e.g., based on computer-readable instructions for performing operations stored in a non-transitory computer-readable memory). For example, netcam modules 113, 133, and / or netcam system 140 may execute some or all of the instructions to perform process 700. Process 700 is described with respect to netcam module 113 for convenience, but may be executed by any other netcam module and / or system.

[0042]

[0051] Process 700 begins, for a data flow transmitted between a sending host (e.g., sending host 110) and a receiving host (e.g., receiving host 130), with recording 702 (e.g., recording in buffer 111) by the sending host on a first circular basis, and recording 704 (e.g., recording in buffer 131) by the receiving host on a second circular basis, the sending host and receiving host being clock synchronized (e.g., using a reference clock of clock synchronization system 141).

[0043]

[0052] The netcam module 113 monitors 706 the data flow for anomalies based on timestamps of data packets in the network traffic (e.g., by subtracting the transmit timestamp 311 from the receive timestamp 321 and comparing the result to a one-way delay threshold). The netcam module 113 determines 708 whether an anomaly is detected during the monitoring (e.g., based on whether the comparison indicates that the one-way delay is greater than a threshold). In response to determining that no anomaly is detected during the monitoring, the netcam module 113 may passively allow the recorded outgoing network traffic and the recorded received network traffic to be overwritten 710 with newly sent network traffic and newly received network traffic, respectively (e.g., overwriting the oldest recorded data packets with the newest network traffic and continuing to repeat elements 702-708). In response to determining that an anomaly is detected during the monitoring, the netcam module 113 pauses 712 the data flow and causes the sending host to store the recorded outgoing network traffic in a first buffer and the receiving host to store the recorded received network traffic in a second buffer.

[0044]

[0053] 8 is a flow chart illustrating an example process for performing netcam activities in a multiple priority scenario, according to one embodiment of the disclosure. Process 800 may be performed by one or more processors (e.g., based on computer-readable instructions for performing operations stored in a non-transitory computer-readable memory). For example, netcam modules 113, 133, and / or netcam system 140 may execute some or all of the instructions to perform process 800. Process 800 is described with respect to netcam module 113 for convenience, but may be performed by any other netcam module and / or system.

[0045]

[0054] The process 800 begins with the netcam module 113 identifying 802 a first data flow between a first sending host (e.g., sending host 110) and a receiving host (e.g., receiving host 130), the first data flow having a high priority (e.g., high priority data flow 511), and the sending and receiving hosts being synchronized using a common reference clock. The netcam module 113 (e.g., of a different sending host or the same sending host as the sending host 110) identifies 804 a second data flow between a second sending host and the receiving host (e.g., low priority data flow 531), the second data flow having a low priority, and the second sending host may be the same or a different host as the first sending host.

[0046]

[0055] The netcam module 113 assigns 806 a first delay threshold to the first data flow based on a high priority and a second delay threshold to the second data flow based on a low priority, the first delay threshold exceeding the second delay threshold. The netcam module 113 monitors 808 a first one-way delay of data packets of the first data flow against the first delay threshold and monitors 810 a second one-way delay of data packets of the second data flow against the second delay threshold. In response to determining that the first one-way delay of data packets of the first data flow exceeds the first delay threshold, the netcam module 113 pauses 812 transmission of data packets of the first data flow from the first sending host to the receiving host for a first time. In response to determining that the second one-way delay of the data packets of the first data flow exceeds the second delay threshold, the netcam module 113 pauses 814 transmission of the data packets of the second data flow from the second sending host to the receiving host for a second time period that exceeds the first time period.

[0047]

[0056] FIG. 9 is a data flow diagram illustrating netcam activity in which consideration of shadow buffers is shown, according to one embodiment of the disclosure. Data flow 900 begins with a sending host sending 902 a data flow and applying a send timestamp, and a receiving host receiving 904 the data flow and applying a receive timestamp. These activities are performed in the manner described above with respect to elements 602 and 604 of FIG. 6. As mentioned with respect to FIG. 1, in one embodiment, a receiving host maintains both one or more regular buffers and one or more shadow buffers, where the regular buffers store data packets as they are received, and the shadow buffers maintain a counter that increments as data packets are received and drains according to a dynamic drain rate (i.e., decrements according to a dynamic drain rate per unit time). Different shadow buffers may be used for different data flows on the same receiving host, and different data flows may have different priorities.

[0048]

[0057] A shadow buffer may be in an idle state or an active state. In response to receiving traffic for a data flow, the netcam module 133 of the receiving host 130 may determine that a shadow buffer is in an active state (i.e., the shadow buffer for that data flow transitions from an idle state to an active state). In response to determining that traffic is no longer being received, the netcam module 133 may determine that a shadow buffer is in an idle state. For example, traffic may be considered no longer being received for a data flow for which at least a threshold time has elapsed since the last packet of the data flow was received. As another example, if traffic is consistently received packet-by-packet for a data flow over a unit of time, and a unit of time elapses in which no packets are received for the data flow, the netcam module 133 may determine that traffic is no longer being received. Thus, the netcam module 133 may continue to switch the state of a shadow buffer for a data flow from idle to active and back depending on whether traffic is received for the data flow. As described further below, the state of the shadow buffer is used by the netcam module 133 to determine other attributes associated with the shadow buffer, such as drainage rate.

[0049]

[0058] Assuming the shadow buffer was in an idle state at 904, the netcam module 133, in response to receiving the first packet of the data flow, transitions the shadow buffer from an idle state to an active state 905a and increments a counter of the shadow buffer indicating a unit of received data traffic 905b. If the shadow buffer is already in an active state, 905a is not performed, but 905b continues for each unit of traffic (e.g., packet) received. In one embodiment, the netcam module 133 increments the counter by multiplying the unit of received data traffic by a factor. For example, for every packet received, the counter may be incremented by multiplying the unit by a number greater than 1 (e.g., 1.01 or 1.1). As a specific example where multiple priorities exist, when a packet is received, the shadow buffer may be multiplied by 1.01 if a packet of a high priority data flow is received and by 1.1 if it is a low priority flow. The higher the factor, the more quickly the shadow buffer counter will have a number of threshold crossings that reflect an anomaly (eg, a pause in traffic and / or a scenario that merits the implementation of corrective action).

[0050]

[0059] Netcam (i.e., either Netcam System 140 or Netcam Module 133, or some distributed process) performs the Netcam activities shown in the rightmost column of Figure 9. For convenience, the activities are referred to as being performed by Netcam Module 133, although distributed or global processing by Netcam System 140 is equally possible.

[0051]

[0060] The netcam module 133 determines 906 a one-way delay of data packets for each data flow and determines 908 a dynamic drain rate for each shadow buffer corresponding to each data flow. Although 906 and 908 are shown sequentially in Figure 9, they may be performed in parallel with each other or in the reverse order from that shown. Element 906 may occur at any point during which it is shown in Figure 9 up until the occurrence of 914. Element 906 may be implemented in the same manner as described above with respect to 606 of Figure 6.

[0052]

[0061] The netcam module 133 may determine the dynamic drain rate based on the number of units of data removed from the regular buffer per unit time while the shadow buffer is in an active state. That is, if 3 bytes are removed from the regular buffer for transmission to the next node in the data flow per microsecond, the rate of 3 per microsecond is the basis on which the dynamic drain rate is determined, multiplied by a factor less than 1 (e.g., 0.9 or 0.95) so that draining from the shadow buffer occurs slower than draining from the regular buffer. The reason for decrementing the shadow buffer at a slower rate than the regular buffer is, again, so that if an anomaly occurs for the regular buffer, it is detected first using the shadow buffer. The netcam module 133 may select a factor by which to multiply the drain rate based on the priority of the data flow, where high priority data flows have a higher drain rate (e.g., 0.95-0.99) and medium and low priority data flows have lower drain rates (e.g., 0.9-0.94 for medium priority data flows and 0.85-0.89 for low priority data flows).

[0053]

[0062] The netcam module 133 may determine the dynamic drain rate at any cadence, such as every time a data packet is received by the receiving host 130, or at a slower cadence, such as every Nth data packet received in a given data flow. The netcam module 133 may limit the performance of determining 908 the dynamic drain rate to scenarios in which the shadow buffer is in an active state. If the shadow buffer is in an idle state, the netcam module 133 may take the last determined dynamic drain rate as the static drain rate used to decrement the shadow buffer until such time as the shadow buffer again enters an active state, after which the netcam module 133 may recalculate a new dynamic drain rate.

[0054]

[0063] The dynamic drain rate is used by the netcam module 133 for two purposes. First, the dynamic drain rate is used to decrement the shadow buffer counter over time. Second, the dynamic drain rate is used to calculate the "dwell time." As used herein, the term "dwell time" refers to a value that can be aggregated with the actual one-way delay of packets on a data flow as a congestion signal to determine if there is an anomaly in the data flow that requires corrective action to be taken.

[0055]

[0064] The netcam module 133 determines 910 the residence time as a function of the shadow buffer's counter (e.g., a proxy for the regular buffer's length with some additional length based on the incremental and drain multiplier factors) and the dynamic drain rate. In one embodiment, the netcam module 133 calculates the residence time by dividing the shadow buffer's counter value by the dynamic drain rate.

[0056]

[0065] The netcam module 133 determines 912 a congestion signal for the data flows based on the residence time. In one embodiment, the netcam module 133 determines the congestion signal by mathematically aggregating the one-way delay between the sending host and the receiving host along with the residence time. Similar to calculating a dynamic drain rate and incrementing a counter, the netcam module 133 may weight the residence time by a factor. For example, the residence time may be weighted according to the priority of the data flow, where a larger multiplier may be used for lower priority data flows and a smaller multiplier may be used for higher priority data flows (e.g., 1.01-1.05 for high priority data flows, 1.06-1.14 for medium priority data flows, and 1.15-1.30 for low priority data flows). This again causes higher priority data flows to be affected less frequently than lower priority data flows, causing their congestion signal to reach the threshold that triggers corrective action sooner.

[0057]

[0066] 6 for elements 610-614, the netcam module 133 may determine 914 that the congestion signal exceeds a threshold (e.g., a priority-specific threshold similar to the threshold used for the regular buffer) and may take corrective action. The corrective action may include storing 916 data or an indication of data for the associated data flow and / or pausing 918 transmission of the associated data flow.

[0058]

[0067] 10 is a flow chart illustrating an example process for performing netcam activity in coordination with consideration of a shadow buffer, according to one embodiment of the present disclosure. Process 1000 may be performed by one or more processors (e.g., based on computer-readable instructions for performing operations stored in a non-transitory computer-readable memory). For example, netcam modules 113, 133, and / or netcam system 140 may execute some or all of the instructions to perform process 1000. Process 1000 is conveniently described with respect to netcam module 133, but may be performed by any other netcam module and / or system.

[0059]

[0068] The process 1000 begins with the netcam module 133 maintaining 1002 a number of buffers at the receiving host, including a regular buffer and a shadow buffer (e.g., buffer 131 and shadow buffer 134). In response to receiving a data flow from a sending host clocked with the receiving host using a common reference clock, the netcam module 133 performs 1004: storing a first representation of data of the data flow in the regular buffer (e.g., storing data packets or metadata corresponding to the data packets in buffer 131), transitioning the shadow buffer from an idle state to an active state (e.g., if this is the start of traffic of the data flow since the last interruption of traffic), and incrementing a counter of the shadow buffer indicating a unit of data traffic received (e.g., a counter of the shadow buffer 134 corresponding to the data flow).

[0060]

[0069] The netcam module 133 determines 1006 a dynamic drain rate based on the number of units of data removed from the regular buffer per unit time while the shadow buffer is in an active state, and the shadow buffer returns to an idle state in response to an interruption of the receiving host receiving the data flow. The netcam module 133 calculates 1008 a residence time as a function of the shadow buffer's counter and the dynamic drain rate, and determines 1010 a congestion signal for the data flow based on the residence time (e.g., a congestion signal used to detect anomalies in the same manner as described with respect to 708 of FIG. 7).

Claims

1. 1. A computer-implemented method comprising: maintaining a plurality of buffers at the receiving host, the plurality of buffers including a regular buffer and a shadow buffer; in response to receiving a data flow from a transmitting host that is clock-synchronized with said receiving host using a common reference clock; storing a first representation of data of the data flow in the regular buffer; transitioning the shadow buffer from an idle state to an active state; incrementing a counter in said shadow buffer indicating a unit of data traffic received; determining a dynamic drain rate based on the number of units of data removed from the regular buffer per unit time while the shadow buffer is in the active state, the shadow buffer returning to an idle state in response to an interruption in reception of the data flow at the receiving host receiving the data flow; calculating a residence time of the shadow buffer as a function of the counter and the dynamic drainage rate; determining a congestion signal for the data flow based on the residence time; A computer-implemented method comprising:

2. 2. The computer-implemented method of claim 1, wherein the plurality of buffers comprises a plurality of shadow buffers, and the plurality of shadow buffers comprises the shadow buffer.

3. 3. The computer-implemented method of claim 2, wherein each of the plurality of shadow buffers serves a different subset of a plurality of data flows received by the receiving host.

4. 4. The computer-implemented method of claim 3, wherein each of the different subsets is determined to include data flows common to different given applications to which the data flows correspond, each different application having a different priority, and the emission rates are weighted based on the different priorities.

5. The computer-implemented method of claim 3 , wherein each of the different subsets is determined to include data flows that are common to a given set of senders.

6. 2. The computer-implemented method of claim 1, wherein incrementing the counter comprises multiplying the unit of received data traffic by a factor.

7. 2. The computer-implemented method of claim 1, further comprising decrementing the shadow buffer while the shadow buffer is in the active state at a slower rate than the data is removed from the regular buffer.

8. 8. The computer-implemented method of claim 7, wherein the slower rate is determined by multiplying the rate at which the data is removed from the regular buffer by a factor greater than or equal to 0.9 but less than 1.

9. 2. The computer-implemented method of claim 1, wherein the residence time is calculated by dividing the value of the counter for the shadow buffer by the dynamic drain rate.

10. 2. The computer-implemented method of claim 1, wherein the dynamic drainage rate is maintained in response to the shadow buffer transitioning to an idle state, and the dynamic drainage rate is reset in response to the shadow buffer returning to an active state.

11. 2. The computer-implemented method of claim 1, wherein determining the congestion signal comprises mathematically aggregating a one-way delay between the sending host and the receiving host with the residence time.

12. 12. The computer-implemented method of claim 11, wherein mathematically aggregating the one-way delays comprises weighting the dwell times by a factor.

13. A non-transitory computer-readable medium comprising instructions encoded on a memory that, when executed, cause one or more processors to perform operations, the instructions comprising: maintaining a plurality of buffers at the receiving host, the plurality of buffers including a regular buffer and a shadow buffer; in response to receiving a data flow from a transmitting host that is clock-synchronized with said receiving host using a common reference clock; storing a first representation of data of the data flow in the regular buffer; transitioning the shadow buffer from an idle state to an active state; incrementing a counter of said shadow buffer indicative of a unit of data traffic received; determining a dynamic drain rate based on the number of units of data removed from the regular buffer per unit time while the shadow buffer is in the active state, the shadow buffer returning to an idle state in response to an interruption in reception of the data flow at the receiving host receiving the data flow; calculating a residence time as a function of the counter and the dynamic drainage rate of the shadow buffer; determining a congestion signal for the data flow based on the residence time; A non-transitory computer readable medium having instructions for:

14. 14. The non-transitory computer-readable medium of claim 13, wherein the plurality of buffers comprises a plurality of shadow buffers, the plurality of shadow buffers comprising the shadow buffer.

15. 15. The non-transitory computer-readable medium of claim 14, wherein each of the plurality of shadow buffers serves a different subset of a plurality of data flows received by the receiving host.

16. 16. The non-transitory computer-readable medium of claim 15, wherein each of the different subsets is determined to include a data flow common to a different given application to which the data flow corresponds, each different application having a different priority, and the emission rate is weighted based on the different priority.

17. 20. The non-transitory computer-readable medium of claim 15, wherein each of the different subsets is determined to include data flows that are common to a given set of senders.

18. 14. The non-transitory computer-readable medium of claim 13, wherein the instructions to increment the counter include instructions to multiply the unit of received data traffic by a factor.

19. 14. The non-transitory computer-readable medium of claim 13, wherein the instructions further comprise instructions for decrementing the shadow buffer at a slower rate than the data is removed from the regular buffer while the shadow buffer is in the active state.

20. 20. The non-transitory computer-readable medium of claim 19, wherein the slower rate is determined by multiplying the rate at which the data is removed from the regular buffer by a factor greater than or equal to 0.9 but less than 1.

Images