Deployment of Shadow Buffers in a Clock-Synchronized Edge-Based Network Function

Abstract

A regular buffer and a shadow buffer are maintained at the receiving host. In response to receiving a data flow from a sending host that is clocked with the receiving host using a common reference clock, a first representation of data of the data flow is stored in the regular buffer, the shadow buffer is transitioned from an idle state to an active state, and a counter in the shadow buffer indicating a unit of data traffic received is incremented.

Description

Technical Field

[0001]

[0002] This disclosure generally relates to regulated control of network transmissions and network traffic within a data flow.

Background Art

[0002] Cross-reference to Related Applications

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 320,160, filed on May 13, 2022, which is hereby incorporated by reference in its entirety.

[0003]

[0003] Modern Internet infrastructure typically has large data centers that generate vast amounts of network traffic. When demand is high, the output of the data center can be constrained (e.g., by the capacity of switches, gateways, etc.), and it may be necessary to measure network traffic. Such temporary congestion scenarios can cause bottlenecks and packet drops. To ensure successful packet transmission when faced with such situations, systems have been developed that send an acknowledgment response from the receiving node to the sending node when a packet is received. However, these acknowledgment responses are inefficient in that they further contribute to additional network traffic. Additionally, these acknowledgment responses are limited to functioning in scenarios from a single sender to a single receiver. Further, if an acknowledgment response is not received, the packet is simply retransmitted ad hoc, potentially flowing into the same congested switch and resulting in the same discard outcome, leading to scenarios where the packet is permanently delayed or not even received by the destination. Additionally, these scenarios are rooted in the congestion that has already occurred and are not sufficient to prevent congestion from occurring in the first place.

Brief Description of the Drawings

[0004]

Figure 1

[0004] Figure 1 is a diagram showing an exemplary system environment for implementing a network camera and priority functions according to an embodiment of the present disclosure.

Figure 2

[0005] Figure 2 is a network traffic diagram showing that a plurality of transmission hosts transmit a plurality of data flows to a single reception host according to an embodiment of the present disclosure.

Figure 3

[0006] Figure 3 is a network traffic diagram showing timestamp operations on both the transmission side and the reception side of data transmission according to an embodiment of the present disclosure.

Figure 4

[0007] Figure 4 is a data flow diagram showing network camera activities during normal operation and locations where anomalies are detected according to an embodiment of the present disclosure.

Figure 5

[0008] Figure 5 is a network traffic diagram showing that a reception host receives both high-priority traffic and low-priority traffic from a transmission host according to an embodiment of the present disclosure.

Figure 6

[0009] Figure 6 is a data flow diagram showing network camera activities in which priorities are considered when determining network camera activities according to an embodiment of the present disclosure.

Figure 7

[0010] Figure 7 is a flowchart showing an exemplary process for executing network camera activities according to an embodiment of the present disclosure.

Figure 8

[0011] Figure 8 is a flowchart showing an exemplary process for executing network camera activities in scenarios with multiple priorities according to an embodiment of the present disclosure.

Figure 9

[0012] Figure 9 is a data flow diagram showing network camera activities in which consideration of a shadow buffer is shown according to an embodiment of the present disclosure.

Figure 10

[0013] FIG. 10 is a flowchart showing an exemplary process for performing webcam activity in consideration of and in coordination with a shadow buffer, according to one embodiment of the present disclosure.

DETAILED DESCRIPTION OF THE INVENTION

[0005]

[0014] The figures and the following description relate only to preferred embodiments for the purpose of illustration. From the following description, alternative embodiments of the structures and methods disclosed herein should be readily recognized as viable alternatives that may be employed without departing from the principles of what is claimed.

[0006]

[0015] Systems and methods are disclosed herein for adjusting control of a data flow in the face of temporary congestion. A “netcam” monitors network traffic between a clock-synchronized transmitter host and a receiver host that is part of the data flow. As used herein, the term “netcam” is a shortened term for “network camera” and is a module that tracks network traffic and ensures that corrective action is taken if traffic in the clock synchronization system delays beyond an acceptable limit. The netcam instructs the transmitter host and the receiver host to buffer a copy of the network traffic according to several parameters (e.g., buffer a specific number of packets, buffer packets for the time of a rolling window, etc.). The buffer can be overwritten on a rolling basis when the parameters are achieved (e.g., overwrite the oldest packet when a new packet is transmitted or received and when the buffer is full). The netcam may cause all transmitter hosts and receiver hosts to write buffer data in which an anomaly is detected, and may cause the transmitter host to retransmit the written packets. Retransmission may be subject to jitter (e.g., the time delay between packet transmissions in the data flow), such that if a transmission delay or failure occurs due to a given packet transmission sequence, the jitter nonetheless causes sufficient variation to succeed in the retransmission attempt. The netcam may determine the need to write and retransmit packets in different ways depending on the priority of the data flow. The netcam may instruct the shadow buffer of the receiver host to monitor path utilization and capacity, and high utilization and / or low capacity may cause the netcam to predict an impending anomaly and take corrective action similar to when the buffer is full.

[0007]

[0016] Advantageously, the webcam implementations disclosed herein enable both improved network transmission and forensic analysis. The improved network transmission occurs in that it enables retransmission of an accurate set of packets from many machines without relying on acknowledgment packets that may be lost or dropped across a complex web of machines by writing the latest packet transmissions and attempting buffering across all machines within the data flow. Further, virtual machines may have bugs that are difficult to detect or isolate. Writing packet sequences associated with anomalies enables fault analysis, which may enable identification of a malfunctioning virtual machine. Further, using a shadow buffer to predict anomalies may prevent scenarios where traffic becomes overly congested, leaving some capacity in the path and enabling corrective action to occur without pausing traffic. Additional advantages and improvements will become apparent from the following disclosure.

[0008]

[0017] FIG. 1 is a diagram showing an exemplary system environment for implementing a webcam and priority functions, according to one embodiment of the present disclosure. As shown in FIG. 1, the webcam environment 100 includes a transmitting host 110, a network 120, a receiving host 130, and a clock synchronization system 140. Only one of each of the transmitting host 110 and the receiving host 130 is shown, merely for purposes of illustration and ease, and any number of transmitting and receiving hosts may be part of the webcam environment 100.

[0009]

[0018] The transmitting host 110 includes a buffer 111, a network interface card (NIC) 112, and a network camera module 113. The buffer 111 stores a copy of the outbound data transmission until one or more criteria for overwriting or discarding a packet from the buffer are met. For example, the buffer may store data packets until it reaches its capacity, at which point the oldest buffered data packet may be discarded or overwritten. Other criteria may include the passage of time (e.g., discarding a packet after a predetermined time has elapsed since its transmission timestamp), the amount of buffered packets (e.g., after a predetermined amount of packets are buffered, the oldest packet is discarded or overwritten as new packets are transmitted), and the like.

[0010]

[0019] In one embodiment, the buffer 111 stores information regarding a given outbound transmission rather than the entire packet. For example, a byte stamp may indicate an identifier of the packet and / or flow identifier, as well as the timestamp at which the packet (or aggregate data flow) was transmitted, and may be stored instead of the packet itself. In such an embodiment, the stored information need not be overwritten and may be stored in the persistent memory of the transmitting host 110 and / or the clock synchronization system 140. This embodiment is not mutually exclusive with the buffer 111 that stores a copy of the packet, and they may be employed in combination.

[0011]

[0020] The NIC 112 can be any type of network interface card, such as a smart NIC. The NIC 112 interfaces the transmitting host 110 and the network 120.

[0012]

[0021] The network camera module 113 monitors the data flow for specific conditions and triggers functions based on the monitored data. As an example, in response to detecting network congestion, the network camera module 113 may instruct one or more of various operations to all hosts that are part of the data flow, such as pausing transmission, taking a snapshot of buffered data transmission (i.e., writing buffered data packets to persistent memory), and performing other coordinated activities. As used herein, the term data flow may refer to a collection of data transmissions between two or more hosts that are associated with each other. Further details of the network camera module 113 are described in more detail in connection with FIGS. 2-8 below. The network camera module 113 may be implemented in any component of the transmitting host 110. In one embodiment, the network camera module 113 may be implemented within the NIC 112. In another embodiment, the network camera module 113 may be implemented within the kernel of the transmitting host 110.

[0013]

[0022] The network 120 can be any network, such as a wide area network, a local area network, the Internet, or any other conduit for data transmission between the transmitting host 110 and the receiving host 130. In some embodiments, the network 120 can be within a data center that houses both the transmitting host 110 and the receiving host 130. In other embodiments, the network 120 can facilitate cross-data center transmissions over any distance. The reference to a data center is merely illustrative, and the transmitting host 110 and the receiving host 130 can be implemented on any medium, including those not in a data center.

[0014]

[0023] The receiving host 130 includes a netcam buffer 131, a NIC 132, a netcam module 133, and a shadow buffer 134. The netcam buffer 131, the NIC 132, and the netcam module 133 operate in the same manner as the analog components described above with respect to the transmitting host 110. The buffer 131 may be the same size or a different size than the buffer 111 and, additionally or alternatively, may store byte stamps of received packets. Any further distinctions between these components implemented at the receiving host with respect to the transmitting host will become apparent based on the disclosure of FIGS. 2-8 below.

[0015]

[0024] The shadow buffer 134 can be used to track data traffic in a way that enables early warning of when congestion is likely to occur. For example, when data traffic is buffered, congestion can occur when the buffer is full, and congestion prevents further data traffic from flowing until the congestion is cleared. The shadow buffer may increment the counter earlier than the regular buffer (e.g., when 1 unit of data is received in the regular buffer, it may increment by 1.1 each time), and / or may decrement the counter later than the regular buffer (e.g., when 1 unit of data is cleared in the regular buffer, it may decrement by 0.9 or 0.95 each time). As used herein, the term regular buffer may refer to the operation of buffer 111 and / or buffer 131, and / or the operation of other buffers disclosed herein having a function similar to the function of buffer 111 and / or buffer 131. Only one shadow buffer 134 is shown in FIG. 1, but multiple shadow buffers may be employed at the receiving host, and each of the shadow buffers may be assigned to a different subset of the data flow, such as an individual data flow corresponding to the same application. The shadow buffers can be incremented / decremented at different rates (e.g., to indicate more congestion for lower-priority applications and less congestion for higher-priority applications). Alternatively, the shadow buffers may be incremented / decremented at the same rate, but different thresholding may be applied to different applications with respect to cases where the data flow is considered to be facing congestion. The data buffered in the regular buffer includes the data traffic (e.g., network packets) received by the receiver, and the data is removed from the regular buffer when the data is processed and / or routed to the next destination.The operations described herein of the network camera module 113 and / or the network camera system 140 that operate with respect to the conditions satisfied by the regular buffer may also be implemented when the shadow buffer 134 indicates convergence.

[0016]

[0025] The network camera system 140 includes a clock synchronization system 141. The network camera system 140 may monitor data observed by network camera modules implemented on the host, such as network camera modules 131 and 133. The network camera system 140 may detect a state that requires an action by a network camera module and may send an instruction to the affected network camera module to take a coordinated action for a given data flow. The clock synchronization system 141 synchronizes one or more components of each host, such as a NIC, a kernel, or any other component on which the network camera module operates. Details of clock synchronization are described in commonly assigned U.S. Patent No. 10,623,173, issued April 14, 2020, which is hereby incorporated by reference in its entirety. Individual hosts are synchronized to the same reference clock with very high accuracy, enabling accurate timestamps between hosts regardless of the location of the host, the bandwidth conditions of the host, jitter, etc. Further details of the network camera system 140 are disclosed below with reference to FIGS. 2-8. The network camera system 140 is an optional component of the network camera environment 100, and the network camera modules of the transmitting host and / or the receiving host can operate the network camera modules without depending on a centralized system, except depending on the reference clock to be synchronized.

[0017]

[0026] The network camera environment 100 has many advantages. Considering that the network camera module is executable in the kernel or NIC (e.g., smart NIC) of a host (e.g., a physical host, a virtual machine, or any other form of host), it is edge-based. In one embodiment, the network camera function can be executed as an underlay, for example, as a shim, meaning it can be executed on a layer of the OSI system below the congestion control layer (e.g., layer 3 of the OSI system). The network camera module and / or the network camera system 140 can pause the transmission of the data flow across the affected host, acquire a snapshot (i.e., write some or all of the buffered data such as the last N bytes transmitted and / or the bytes transmitted in the last S seconds, where N or S may be default values or defined by the administrator), and instruct the host to execute an activity when a condition (e.g., a congestion signal is detected using a shadow buffer) such as any other activity disclosed herein is detected. Further advantages and functions are described below with respect to FIGS. 2-8.

[0018]

[0027] FIG. 2 is a network traffic diagram showing multiple sending hosts transmitting multiple data flows to a single receiving host according to an embodiment of the present disclosure. As shown in FIG. 2, sending host 1 transmits data flow 211 to receiving host 200, sending host 220 transmits data flow 221 to receiving host 200, and any number of additional hosts represented by sending host 230 can each transmit their respective data flow (represented by data flow 231) to receiving host 200. As shown in FIG. 2, the individual data flows transmitted by each of the sending hosts are different, but this is for convenience only, and two or more sending hosts may transmit data from the same data flow. Further, a single sending host may transmit two or more different data flows to receiving host 200. Only one receiving host is shown, but the sending hosts can transmit data flows to any number of receiving hosts.

[0019]

[0028] Here, referring to FIG. 3, the operation of the network camera module at the transmitting host and the receiving host will be described. FIG. 3 is a network traffic diagram showing the timestamp operation on both the transmitting side and the receiving side of data transmission according to an embodiment of the present disclosure. As shown in FIG. 3, when the transmitting host 310 transmits a packet to the receiving host 320, the network camera module 113 of the receiving host 320 records the transmission timestamp 311. Similarly, when the receiving host 320 receives a packet, the network camera module 133 of the receiving host 320 applies the reception timestamp 321. The timestamp reflects the time at which the data packet was transmitted or received by the relevant components (e.g., NIC, kernel, etc.) in which the network camera module is installed. The transmission timestamp may be stored in the buffers 111 and 131, may be attached to the packet, may be transmitted for storage in the network camera system 140, or any combination thereof.

[0020]

[0029] Since the transmitting host 310 is synchronized with the same reference clock as the receiving host 320, the elapsed time between the time of the transmission timestamp 311 and the reception timestamp 321 reflects the one-way delay of a given packet. In one embodiment, when a given packet is received, the receiving host 320 transmits a response packet indicating the reception timestamp 321 to the transmitting host 310, and the network camera module 113 can calculate the one-way delay by subtracting the transmission timestamp 311 from the reception timestamp 321. Other means of calculating the one-way delay are within the scope of the present disclosure. For example, the transmission timestamp 311 may be attached to the data transmission, and the receiving host 320 may thereby calculate the one-way delay without requiring a response packet. As yet another example, the network camera modules of the transmitting host and the receiving host may transmit the timestamps collectively or individually to the network camera system 140, and the network camera system 140 may calculate the one-way delay therefrom. For convenience and brevity, the scenario where the transmitting host 110 calculates the one-way delay based on the response packet will be the focus of the following disclosure, but those skilled in the art will recognize that any of these calculation means may equally apply.

[0021]

[0030] In one embodiment, the network camera system then determines whether the one-way delay exceeds a threshold. For example, after calculating the one-way delay, the transmitting host 110 may compare the one-way delay with a threshold. The threshold may be predetermined or dynamically determined. The predetermined threshold may be set by default or by an administrator. As further described below, different thresholds may be applied to different data flows depending on one or more attributes of the data flow, such as the priority of the data flow. The threshold may be dynamically determined according to any number of factors, such as increasing the threshold dynamically as congestion decreases or decreasing the threshold as congestion increases (e.g., because congestion is not the cause of the delay or is likely to indicate a minor problem). In one embodiment, the threshold may depend on the distance between the transmitting host and the receiving host and may be set for each host. In such an embodiment, the threshold may be a predetermined multiple of the minimum one-way delay between the transmitting host and the receiving host. That is, the minimum time required for a packet to travel from the transmitting host to the receiving host would be the minimum one-way delay. The multiple is typically between 1.5 and 3 times the minimum value, but may be any multiple defined by the administrator of the network camera. The threshold is equal to the multiple of the minimum one-way delay. In response to a determination that the one-way delay exceeds the threshold, the network camera module 113 may instruct the transmitting host 110 to take one or more actions.

[0022]

[0031] In additional or alternative embodiments, determining whether to take one or more actions may be performed using an individual measure of the state of a shadow buffer (e.g., shadow buffer 134). In short (more details are described below), during a given data flow, and in parallel with buffering data using a regular buffer, the netcam module 133 may instruct the shadow buffer 134 to be incremented for individual units of data traffic received by the receiving host 320. The netcam module 133 may define a dynamic drain rate, which is the rate at which the netcam module 133 instructs the shadow buffer 134 to be decremented. The dynamic drain rate may be determined by the netcam module 133 based on the number of data removed from buffer 131 per unit time (e.g., multiplying a factor that causes the drain to occur more slowly in shadow buffer 134 than in buffer 131). The netcam module 133 may calculate the residence time as the function of the counter of the shadow buffer and the dynamic drain rate (e.g., the residence time may be calculated by dividing the value of the counter of the shadow buffer by the dynamic drain rate thing ). From this, the netcam module 133 may determine that the one-way delay of the shadow buffer is the actual one-way delay aggregated with the residence time (determined from the transmission and reception timestamps described above). The one-way delay of the shadow buffer may be compared to a threshold (in addition to or instead of the one-way delay of the regular buffer) and used to determine whether to take one or more actions.

[0023]

[0032] Regardless of whether these one or more actions are driven by a regular buffer or a shadow buffer one-way delay, these one or more actions may include pausing transmission from the transmitting host when the one-way delay is high, which reduces congestion and thereby generally reduces packet loss on the network 120. The pause may be for a predetermined time or may be dynamically determined in proportion to the magnitude of the one-way delay. In one embodiment, the pause may be equal to the one-way delay or may be determined by applying an administratively defined multiple to the one-way delay. In one embodiment, the network camera may determine whether a previous pause has been enforced and, if so, may shorten the pause time based on the amount of the previous pause time that has already elapsed from the previously acknowledged packet. Further, a given data flow may not be the only data flow contributing to congestion, and thus, its pause time may be shorter than the one-way delay or the one-way delay threshold.

[0024]

[0033] Another possible action is to write some or all of the buffered data packets (e.g., from either or both the transmitting host and the receiving host) to persistent memory in response to a one-way delay exceeding a threshold. Then, a diagnosis may be performed on the buffered data packets (e.g., to identify network problems). Further actions are described with respect to FIGS. 4 - 8.

[0025]

[0034] In some embodiments, data flows may be associated with different priorities. The network camera module may determine the priority of a data flow based on an explicit identifier (e.g., an identifier of a traffic layer within a data packet header) or based on an inference (e.g., based on a heuristic where rules are applied to the packet header and / or payload to determine a priority type). As used herein, priority refers to a priority scheme where the type of data packet should be permitted to be sent and should be put on hold during periods of congestion. The priorities disclosed herein avoid the need for link underutilization or explicit bandwidth allocation and are instead considered in the context of selecting packets to send during network congestion.

[0026]

[0035] To prioritize high-priority packets, a high one-way threshold can be assigned to high-priority traffic, and a threshold that is relatively low with respect to the high one-way threshold can be assigned to low-priority traffic. These thresholds can be used to compare against either or both of the shadow buffer one-way delay and / or the regular buffer one-way delay. Thus, in order for an anomaly to be detected by the network camera module, a lower one-way delay needs to be detected for low-priority packets, whereas an anomaly is only detected for high-priority packets if the higher one-way delay threshold is violated, so low-priority packets are more frequently detected as anomalies than high-priority packets. Following the above description of determining the one-way threshold for a given host, different one-way thresholds can be applied to different data packets transmitted by or received by the same host depending on the priority. In an embodiment of the priority, the one-way threshold can be determined in the above manner (e.g., by applying a predetermined multiple to the threshold), and the determination is additionally affected by applying a multiple of the priority. The multiple of the priority can be set by an administrator for any given type of priority, but is higher for higher priorities and lower for lower priorities. The priority does not have to be binary, and any number of priority levels can be established, each corresponding to a different type or types of data traffic and each having a different multiple. The priorities and the multiples associated with them can change over time for a given data flow (e.g., if the data flow starts transmitting different types of data packets that do not require high-latency transmission, the priority can be reduced).

[0027]

[0036] In addition to or instead of using a priority multiple for the one-way delay threshold and varying the one-way delay threshold based on the priority of a given packet or data flow being transmitted, the network camera module may operate the idle time of the traffic paused during the idle operation in different ways depending on the priority. A lower idle time may be allocated to higher-priority traffic, and a relatively higher idle time may be allocated to lower-priority traffic, ensuring that lower-priority traffic is paused more frequently than higher-priority traffic during the congestion period, thereby ensuring that higher-priority traffic has more available bandwidth while lower-priority traffic is paused. The idle time may be determined in the same way as above, but using an additional step of applying an additional idle multiple to the idle time, with a lower idle multiple (e.g., a multiple less than 1 such as 0.7x) for higher-priority traffic and a higher idle multiple (e.g., a multiple greater than 1) for lower-priority traffic.

[0028]

[0037] Priority can be assigned in any number of ways. In one embodiment, one or more “carpool lanes” that can be used by data flows having a qualifying priority can be assigned. For example, a “carpool lane” can be a bandwidth allocation that does not guarantee a minimum bandwidth for a given data communication but is only accessible by data flows that meet essential parameters. Exemplary parameters can include one or more priorities having the qualification to use the reserved bandwidth of a given “carpool lane”. As an example, a carpool lane may require that a data flow have at least medium priority, and thus both medium and high priorities are eligible in a three-priority system having low, medium, and high priorities. As another example, there may be multiple carpool lanes (e.g., in addition to a carpool lane accessible by both medium- and high-priority traffic, a carpool lane accessible only by high-priority traffic).

[0029]

[0038] In one embodiment, the guaranteed bandwidth can be allocated to a given priority. For example, a high-priority data flow can be allocated a minimum bandwidth such as 70 mbps. In such an embodiment, the surplus unused bandwidth from the guaranteed bandwidth can be allocated to lower-priority data flows until the time when the bandwidth is required by the data flow for which the bandwidth is guaranteed. The guaranteed bandwidth may be absolute or relative. A relative guarantee ensures that a given priority data flow receives at least a certain relative amount more bandwidth than a lower-priority data flow. For example, a high-priority data flow can be guaranteed three times the bandwidth of a low-priority data flow, and a medium-priority data flow can be guaranteed twice the bandwidth of a low-priority data flow.

[0030]

[0039] Returning to FIG. 2, when two or more transmitting hosts transmit data from the same data flow, those nodes can cooperate and, in addition to any receiving host receiving data from the data flow, can be referred to as a "cluster". In one embodiment, a data flow can be identified by a set of identifiers that indicate that a data packet is part of the data flow when all are detected. For example, the network camera module of any host can determine a flow identifier that identifies the data flow to which a packet belongs based on a combination of source address, destination address, source port number, destination port number, and protocol port number. Other combinations of identifiers can be used to identify the data flow of which the packet is a part. As described above, all hosts in the cluster are clock-synchronized to the same reference clock regardless of their form (e.g., server, virtual machine, smart NIC, etc.).

[0031]

[0040] In a scenario where data flows 211 and 221 are the same data flow, the transmitting host 210, the transmitting host 220, and the receiving host 200 form a cluster. Following this example, buffering of data packets (across both regular and shadow buffers) can occur at a per-flow level across the host cluster. That is, one or more network camera modules and / or network camera system 140 can record all packets transmitted or received within any parameters used by the buffer in the buffer of the host of the data flow for the buffer to record data and then overwrite (e.g., the most recently transmitted packet, packets transmitted / received within a given time, etc.). Further, a receiving node that receives data flow packets from multiple transmitting hosts (e.g., receiving host 200 that receives packets from transmitting hosts 210 and 220) may maintain a single shadow buffer for the data flow or may maintain one separate shadow buffer for each of transmitting host 210 and transmitting host 220. In one embodiment, an indicator of the time sequence with respect to a reference clock is stored with the buffer data (e.g., transmit timestamp 311 and / or receive timestamp 321 are stored with the buffer data packet). Thus, transmitting host 210 and transmitting host 220 can store data packet 111 sharing a given flow ID in the buffer, and receiving host 200 can store received packets in buffer 131. Alternatively or additionally, transmitted and / or received packets can be sent to network camera system 140, and network camera system 140 can buffer the received data.

[0032]

[0041] From this advantageous point of buffering a specific amount of data at each host of the cluster, different functions of the host network camera module are possible in response to the detection of an anomaly (e.g., the aforementioned conditions referred to with respect to FIG. 2 above). FIG. 4 is a data flow diagram showing the network camera activity during normal operation and where an anomaly is detected, according to one embodiment of the present disclosure. Data flow 400 reflects host activity and network camera activity (e.g., the activity taken by the network camera module of the transmit / receive host or network camera system 140) during normal functioning and during “anomaly functions” (i.e., actions taken when an anomaly is detected). Data flow 400 first shows the normal function where the host transmits or receives the data flow of 402, and the network camera module or system (generally referred to as the “network camera” in this figure) determines 404 whether an anomaly is detected (e.g., based on one-way latency as described above). If no anomaly is detected, assuming the buffer is full from the previous memory of the data packet, the host (e.g., of the cluster) overwrites 406 those buffers (e.g., meaning overwriting the oldest packet or following some other overwrite heuristic as described above). Of course, if the buffer is not full, overwriting is not necessary and the data is stored in the free memory of the buffer. As long as no anomaly is detected, the normal function is repeated.

[0033]

[0042] An abnormal function occurs when an abnormality is detected. Different abnormal functions are disclosed in this specification, and data flow 400 focuses on illustrating a specific abnormal function of resending buffer data. In the case of transmitting / receiving 408 data flow information by a host (e.g., of a cluster), the network camera may detect 410 an abnormality. As described above, an abnormality is detected when a one-way delay (e.g., delay of a shadow buffer and / or a regular buffer) exceeds a threshold. In the case of a cluster, depending on the distance between the transmitting host and the receiving host, the threshold may be different among the hosts of the cluster. In response to the detection of the abnormality, the network camera instructs 412 to store the buffer data in all the hosts of the cluster. That is, even if an abnormality occurs in one host of the cluster, data from all the nodes of the cluster are stored. This can occur by instructing the host to store the buffer data (or that part related to the data flow) in persistent memory, or by holding the buffer data in the buffer and pausing data transmission, or by combining them with different instructions for different hosts. Note that when pausing is used, the pause time can vary across different nodes of the cluster as described above. Regardless of how the data is stored, the network camera may jitter 414 the retransmission timing. The time sequence of packet transmission and reception is reflected in the stored data packets. The network camera may jitter 414 the retransmission timing by changing the time sequence (e.g., creating a longer delay during a previous time gap between transmissions, transmitting packets in a different order, etc.). The jitter may occur according to a heuristic or may be random. The jitter is applied when the previously tried time sequence was the cause of the failure (e.g., because the previously tried time sequence itself may cause too many temporary congestions), and thus the jitter may result in a failure of the retransmission without jitter in such a scenario. Then, the network camera resends 416 the buffer data (or a part thereof).Rather than isolating packets of a data flow or data flows related to an anomaly, it may be more expedient and computationally efficient to retransmit the entire buffer containing data not related to the data flow or anomaly. Thereafter, normal functionality resumes until another anomaly is detected.

[0034]

[0043] Retransmission with jitter is just one example of an anomaly function, and any number of functions can occur in response to the detection of an anomaly. For example, in addition to or as an alternative to the anomaly function shown in data flow 400, buffer data may be written to persistent memory and stored for forensic analysis. In such a scenario, the netcam may send an alert to an administrator and / or generate an event log indicating the anomaly in response to the detection of the anomaly. Other aforementioned anomaly functions are equally applicable. As an example of forensic analysis, a known type of attack on a system such as a data center is a timing attack. A timing attack may have a "signature" in that the spacing between packets of traffic can be learned (e.g., by training a machine learning model using timing patterns labeled according to whether the timing pattern was a timing attack, by using pattern recognition, etc.). Forensic analysis may be performed to determine whether the data was a timing attack. A timing attack may be blocked (e.g., by the netcam module 113 dropping data packets from the buffer if it determines that the buffer data represents a timing attack).

[0035]

[0044] As described above, the buffer data may include byte stamps (as opposed to, or in addition to, buffer packets). The byte stamps can be used for anomaly analysis (e.g., forensic analysis, network debugging, security analysis, etc.). The advantage of using byte stamps over buffered data packets is that memory space is saved and the computational processing cost is low. The byte stamps for the period corresponding to the anomaly can be analyzed to determine the cause of the anomaly. The trade-off when using byte stamps over buffered packets is that the buffered packet data is more robust and can provide further insights into anomalies.

[0036]

[0045] FIG. 5 is a network traffic diagram showing a receiving host that receives both high-priority traffic and low-priority traffic from a transmitting host according to an embodiment of the present disclosure. As shown in FIG. 5, a transmitting host 510 transmits a high-priority data flow 511 to a receiving host 500, and a transmitting host 530 transmits a low-priority data flow 531 to the receiving host 500. When network congestion occurs and an anomaly is detected, the transmitting host can handle high-priority traffic and low-priority traffic differently. In one embodiment, since the low-priority data flow 531 is associated with a lower one-way delay threshold than the high-priority data flow 511, the transmitting host 530 detects network congestion earlier than the transmitting host 510. Accordingly, the transmitting host 530 may implement a corrective measure such as pausing the network transmission of the low-priority data flow 531, while for the high-priority data flow 511, since its higher one-way delay threshold has not yet been reached, the transmission can continue. When the high-priority data flow 511 reaches its high one-way delay threshold and a pause action is taken in response thereto, the pause time may be shorter than the pause time of the low-priority data flow 531, and thus, while the high-priority data flow 511 continues, it is guaranteed that the high-priority data flow 511 resumes earlier and with less congestion than would be faced if the low-priority data flow 531 were not paused for an extra period of time.

[0037]

[0046] Similarly, regarding the operation of the shadow buffer, the high-priority shadow buffer can be individually held by the receiving host 500 for the high-priority data flow 511, and the low-priority shadow buffer can be individually held by the receiving host 500 for the low-priority data flow 531. The discharge rate can be weighted differently based on priority. For example, the high-priority shadow buffer can have a higher discharge rate than the discharge rate used for the low-priority shadow buffer, and thus, the high-priority shadow buffer is less likely to cause the detection of an abnormality than the low-priority shadow buffer.

[0038]

[0047] Although shown as two separate sending hosts, the sending hosts 510 and 530 may be the same host where one sending host transmits both high-priority traffic and low-priority traffic to the receiving host 500. Thus, the same sending host can continue to transmit the high-priority data flow 511 as normal while taking corrective measures (e.g., pause) in response to the detection of an abnormality in the low-priority data flow 531. The sending host can have a plurality of buffers 111, and the individual buffers can correspond to different priorities of data.

[0039]

[0048] Figure 6 is a data flow diagram showing network camera activities where priorities are considered when determining network camera activities according to an embodiment of the present disclosure. Data flow 600 begins as one or more transmitting hosts (e.g., transmitting host 110) transmit 602 the data flow and apply a transmission timestamp (e.g., transmission timestamp 311). A receiving host (e.g., receiving host 130) receives 604 the data flow and applies a reception timestamp (e.g., reception timestamp 321). Then, network camera activities occur. As described above, network camera activities can occur at the transmitting host (e.g., by receiving an ACK packet indicating the reception timestamp and by using the network camera module to calculate the one-way delay), at the receiving host (e.g., when the transmission timestamp is included in the data flow and the network camera module calculates the one-way delay therefrom), in the network camera system 140, or in some combination thereof.

[0040]

[0049] The network camera determines 606 the one-way delay of data packets within the data flow. As described above, the one-way delay calculation can depend on the priority of the data flow, and thus, different data flows can have different one-way delay thresholds (“priority thresholds”). The one-way delay may generally be determined from the packets and / or aggregated with the dwell time to form a shadow buffer one-way delay. The network camera compares 608 the determined one-way delay (or delay if a shadow buffer one-way delay is used) with the respective priority thresholds. In response to the determination 610 that the one-way delay is greater than the threshold for a given priority data flow, an anomaly function is initiated. As shown in FIG. 6, some anomaly functions can include one or more of pausing 612 the transmission of the data flow associated with a given priority and / or storing 614 the buffer data flow associated with a given priority (e.g., for forensic analysis). As described above, the pause time can vary depending on the priority level of the paused data flow.

[0041]

[0050] FIG. 7 is a flowchart showing an exemplary process for performing webcam activity according to an embodiment of the present disclosure. Process 700 can be executed by one or more processors (e.g., based on computer-readable instructions stored in a non-transitory computer-readable memory for performing operations). For example, webcam modules 113, 133, and / or webcam system 140 can execute some or all of the instructions for performing process 700. Process 700 is described for convenience with respect to webcam module 113, but can be executed by any other webcam module and / or system.

[0042]

[0051] Process 700 begins with the sending host (e.g., sending host 110) recording 702 a first predetermined amount of the transmitted network traffic of the data flow (e.g., recording to buffer 111) on a first round-robin basis and the receiving host recording 704 a second predetermined amount of the received network traffic of the data flow (e.g., recording to buffer 131) on a second round-robin basis for the data flow transmitted between the sending host and the receiving host (e.g., receiving host 130), and the sending host and the receiving host are clock synchronized (e.g., using the reference clock of clock synchronization system 141).

[0043]

[0052] The network camera module 113 monitors 706 for anomalies in the data flow based on the timestamps of data packets in the network traffic (e.g., by subtracting the transmission timestamp 311 from the reception timestamp 321 and comparing the result to a one-way delay threshold). The network camera module 113 determines 708 whether an anomaly is detected during the monitoring (e.g., based on whether the comparison indicates that the one-way delay is greater than the threshold). In response to a determination that no anomaly is detected during the monitoring, the network camera module 133 may passively allow the recorded transmitted network traffic and the recorded received network traffic to be overwritten 710, respectively, by the newly transmitted network traffic and the newly received network traffic (e.g., overwriting the oldest recorded data packet with the latest network traffic and continuing to repeat elements 702 - 708). In response to a determination that an anomaly is detected during the monitoring, the network camera module 113 halts 712 the data flow and causes the transmitting host to store the recorded transmitted network traffic in a first buffer and the receiving host to store the recorded received network traffic in a second buffer.

[0044]

[0053] FIG. 8 is a flowchart showing an exemplary process for performing network camera activity in scenarios of multiple priorities, according to an embodiment of the present disclosure. Process 800 may be executed by one or more processors (e.g., based on computer-readable instructions stored in a non-transitory computer-readable memory for performing operations). For example, network camera modules 113, 133, and / or network camera system 140 may execute some or all of the instructions for performing process 800. Process 800 is described for convenience with respect to network camera module 113, but may be executed by any other network camera module and / or system.

[0045]

[0054] Process 800 starts when the network camera module 113 identifies 802 a first data flow between a first sending host (e.g., sending host 110) and a receiving host (e.g., receiving host 130). The first data flow has a high priority (e.g., high-priority data flow 511), and the sending host and the receiving host are synchronized using a common reference clock. The network camera module 113 (e.g., a sending host different from or the same as sending host 110) identifies 804 a second data flow (e.g., low-priority data flow 531) between a second sending host and the receiving host. The second data flow has a low priority, and the second sending host can be the same as or different from the first sending host.

[0046]

[0055] The network camera module 113 assigns 806 a first delay threshold to the first data flow based on the high priority and a second delay threshold to the second data flow based on the low priority. The first delay threshold exceeds the second delay threshold. The network camera module 113 monitors 808 the first one-way delay of the data packets of the first data flow with respect to the first delay threshold and monitors 810 the second one-way delay of the data packets of the second data flow with respect to the second delay threshold. In response to the determination that the first one-way delay of the data packets of the first data flow exceeds the first delay threshold, the network camera module 113 pauses 812 the transmission of the data packets of the first data flow from the first sending host to the receiving host for a first period of time. In response to the determination that the second one-way delay of the data packets of the second data flow exceeds the second delay threshold, the network camera module 113 pauses 814 the transmission of the data packets of the second data flow from the second sending host to the receiving host for a second period of time that exceeds the first period of time.

[0047]

[0056] FIG. 9 is a data flow diagram showing netcam activity with consideration of a shadow buffer according to an embodiment of the present disclosure. Data flow 900 begins as the transmitting host transmits 902 the data flow and applies a transmission timestamp, and as the receiving host receives 904 the data flow and applies a reception timestamp. These activities are performed in the manner described above with respect to elements 602 and 604 of FIG. 6. As mentioned with respect to FIG. 1, in one embodiment, the receiving host maintains both one or more regular buffers and one or more shadow buffers, where the regular buffer stores data packets when they are received, and the shadow buffer maintains a counter that increments when a data packet is received and drains according to a dynamic drain rate (i.e., decrements according to the dynamic drain rate per unit time). Different shadow buffers can be used for different data flows on the same receiving host, and different data flows can have different priorities.

[0048]

[0057] The shadow buffer can be in an idle state or an active state. In response to receiving traffic of a data flow, the network camera module 133 of the receiving host 130 may determine that the shadow buffer is in the active state (i.e., the shadow buffer for that data flow transitions from the idle state to the active state). In response to determining that traffic is no longer being received, the network camera module 133 may determine that the shadow buffer is in the idle state. For example, traffic may be considered to no longer be received for a data flow for which at least a threshold time has elapsed since the last packet of the data flow was received. As another example, if traffic for a data flow is consistently received packet by packet over a unit time and a unit time elapses during which no packets are received for the data flow, the network camera module 133 may determine that traffic is no longer being received. Accordingly, the network camera module 133 may continue to switch the state of the shadow buffer for a data flow from idle to active and back, depending on whether traffic is received for the data flow. As further described below, the state of the shadow buffer is used by the network camera module 133 to determine other attributes related to the shadow buffer, such as the discharge rate.

[0049]

[0058] Assuming that the shadow buffer was in an idle state at 904, in response to receiving the first packet of the data flow, the network camera module 133 transitions 905a the shadow buffer from the idle state to the active state and increments 905b the counter of the shadow buffer indicating the unit of received data traffic. If the shadow buffer is already in the active state, 905a is not performed, but 905b continues each time a unit of traffic (e.g., a packet) is received. In one embodiment, the network camera module 133 increments the counter by multiplying the unit of received data traffic by a coefficient. For example, for all received packets, the counter may be incremented by multiplying the unit by a number greater than 1 (e.g., 1.01 or 1.1). As a specific example where there are multiple priorities, when a packet is received, the shadow buffer may be multiplied by 1.01 when a packet of the high-priority data flow is received and may be multiplied by 1.1 when it is a low-priority flow. The higher the coefficient, the more quickly the shadow buffer counter reaches a number exceeding the threshold reflecting an anomaly (e.g., a traffic pause and / or a scenario worthy of corrective action).

[0050]

[0059] The network camera (i.e., either the network camera system 140 or the network camera module 133, or any of several distributed processes) performs the network camera activities shown in the rightmost column of FIG. 9. For convenience, the activities are referred to as being performed by the network camera module 133, but distributed processing or overall processing by the network camera system 140 is also possible.

[0051]

[0060] The network camera module 133 determines 906 the one-way delay of data packets for individual data flows and determines 908 the dynamic drain rate for each of the shadow buffers corresponding to the individual data flows. 906 and 908 are shown sequentially in FIG. 9, but these may be executed in parallel with each other or in an order opposite to the order shown. Element 906 may occur at any point while it is shown in FIG. 9 until the occurrence of 914. Element 906 may be implemented in the same manner as the method described above with respect to 606 of FIG. 6.

[0052]

[0061] The network camera module 133 may determine the dynamic drain rate based on the number of units of data deleted from the regular buffer per unit time while the shadow buffer is in the active state. That is, if 3 bytes are deleted from the regular buffer for transmission to the next node within the data flow per microsecond, a rate of 3 per microsecond is the basis on which the dynamic drain rate is determined, and a factor less than 1 (e.g., 0.9 or 0.95) is multiplied so that the drain from the shadow buffer occurs later than the drain from the regular buffer. The reason for decrementing the shadow buffer at a slower rate than the regular buffer is, to repeat, to enable detection first using the shadow buffer in case an abnormality occurs with respect to the regular buffer. The network camera module 133 may select a factor by which to multiply the drain rate based on the priority of the data flow, where high-priority data flows have a higher drain rate (e.g., 0.95 to 0.99), and medium-priority and low-priority data flows have a lower drain rate (e.g., 0.9 to 0.94 for medium-priority data flows, 0.85 to 0.89 for low-priority data flows).

[0053]

[0062] The netcam module 133 may determine the dynamic drain rate at any cadence, such as every time a data packet is received by the receiving host 130, or at a slower cadence, such as every Nth data packet received in a given data flow. The netcam module 133 may limit the performance of determining 908 the dynamic drain rate to scenarios in which the shadow buffer is in an active state. If the shadow buffer is in an idle state, the netcam module 133 may take the last determined dynamic drain rate as the static drain rate used to decrement the shadow buffer until such time as the shadow buffer again enters an active state, after which the netcam module 133 may recalculate a new dynamic drain rate.

[0054]

[0063] The dynamic drain rate is used by the netcam module 133 for two purposes. First, the dynamic drain rate is used to decrement the shadow buffer counter over time. Second, the dynamic drain rate is used to calculate the "dwell time." As used herein, the term "dwell time" refers to a value that can be aggregated with the actual one-way delay of packets on a data flow as a congestion signal to determine if there is an anomaly in the data flow that requires corrective action to be taken.

[0055]

[0064] The netcam module 133 maintains a counter for the shadow buffer (e.g., a proxy for the length of the regular buffer with some additional length based on the incremental and drain multiplier factors) and a dynamic drain rate. function In one embodiment, the netcam module 133 calculates the residence time by dividing the shadow buffer counter value by the dynamic drain rate.

[0056]

[0065] The network camera module 133 determines 912 a congestion signal for the data flow based on the residence time. In one embodiment, the network camera module 133 determines the congestion signal by mathematically aggregating the one-way delay between the transmitting host and the receiving host together with the residence time. Similar to calculating the dynamic discharge rate and incrementing the counter, the network camera module 133 may weight the residence time by a coefficient. For example, the residence time may be weighted according to the priority of the data flow, where a larger multiplier may be used for lower-priority data flows and a smaller multiplier may be used for higher-priority data flows (e.g., 1.01 - 1.05 for high-priority data flows, 1.06 - 1.14 for medium-priority data flows, 1.15 - 1.30 for low-priority data flows). This, although repetitive, causes higher-priority data flows to be affected less frequently than lower-priority data flows, and their congestion signals to reach the threshold that triggers corrective measures earlier.

[0057]

[0066] In a manner similar to the description of FIG. 6 for elements 610 - 614, the network camera module 133 may determine 914 that the congestion signal exceeds a threshold (e.g., a priority-specific threshold similar to the threshold used for the regular buffer) and may take corrective measures. The corrective measures may include storing 916 the data or the display of the data for the associated data flow and / or pausing 918 the transmission of the associated data flow.

[0058]

[0067] Figure 10 is a flowchart showing an exemplary process for performing webcam activity in consideration of and in coordination with a shadow buffer, according to one embodiment of the present disclosure. Process 1000 may be executed by one or more processors (e.g., based on computer-readable instructions stored in a non-transitory computer-readable memory for performing operations). For example, webcam modules 113, 133, and / or webcam system 140 may execute some or all of the instructions for performing process 1000. Process 1000 is described for convenience with respect to webcam module 133, but may be implemented by any other webcam module and / or system.

[0059]

[0068] Process 1000 begins with webcam module 133 maintaining 1002 a plurality of buffers at the receiving host, the plurality of buffers including a regular buffer and a shadow buffer (e.g., buffer 131 and shadow buffer 134). In response to receiving a data flow from a transmitting host that is clock-synchronized with the receiving host using a common reference clock, webcam module 133 stores 1004 a first display of the data of the data flow in the regular buffer (e.g., storing a data packet or metadata corresponding to the data packet in buffer 131), transitions the shadow buffer from an idle state to an active state (e.g., if this is the start of data flow traffic since the last interruption of traffic), and increments a counter of the shadow buffer indicating a unit of received data traffic (e.g., the counter of shadow buffer 134 corresponding to the data flow).

[0060]

[0069] The network camera module 133 determines 1006 a dynamic discharge rate based on the number of units of data deleted from the regular buffer per unit time while the shadow buffer is in an active state, and the shadow buffer returns to an idle state in response to an interruption of the receiving host that receives the data flow. The network camera module 133 calculates 1008 the residence time as the shadow buffer counter and the dynamic discharge rate, and determines 1010 the convergence signal of the data flow based on the residence time (for example, the convergence signal used to detect anomalies in the same manner as described with respect to 708 of FIG. 7). function and determines 1010 the convergence signal of the data flow based on the residence time (for example, the convergence signal used to detect anomalies in the same manner as described with respect to 708 of FIG. 7).

Claims

1. A method implemented on a computer, comprising: maintaining a plurality of buffers at a receiving host, the plurality of buffers including a regular buffer and a shadow buffer; in response to receiving a data flow from a transmitting host that is clock-synchronized with the receiving host using a common reference clock, storing a first display of the data of the data flow in the regular buffer; transitioning the shadow buffer from an idle state to an active state; incrementing a counter of the shadow buffer that indicates a unit of received data traffic; while the shadow buffer is in the active state, determining a dynamic discharge rate based on the number of units of the data deleted from the regular buffer per unit time, wherein the shadow buffer returns to the idle state in response to an interruption of the reception of the data flow at the receiving host that receives the data flow; calculating a residence time as a function of the counter of the shadow buffer and the dynamic discharge rate; determining a congestion signal for the data flow based on the residence time; A method implemented on a computer, comprising the above steps.

2. The plurality of buffers includes a plurality of shadow buffers, and the plurality of shadow buffers includes the shadow buffer. The method implemented on a computer according to claim 1.

3. Each of the plurality of shadow buffers acts on a different subset of a plurality of data flows received by the receiving host. The method implemented on a computer according to claim 2.

4. Each of the different subsets is determined to include a data flow common to a given set of applications corresponding to the data flow, and the individual different applications have different priorities, and the discharge rate is weighted based on the different priorities. The method implemented on a computer according to claim 3.

5. Each of the different subsets is determined to include a data flow common to a given set of transmitters. The method implemented on a computer according to claim 3.

6. Incrementing the counter includes multiplying a coefficient by the unit of the received data traffic, the method implemented on a computer according to claim 1. **Claim 7** The method implemented on a computer according to claim 1, further comprising decrementing the shadow buffer at a rate slower than the data is deleted from the regular buffer while the shadow buffer is in the active state. **Claim 8** The method implemented on a computer according to claim 7, wherein the slow rate is determined by multiplying the rate at which the data is deleted from the regular buffer by a coefficient that is 0.9 or more but less than 1. **Claim 9** The method implemented on a computer according to claim 1, wherein the residence time is calculated by dividing the value of the counter of the shadow buffer by the dynamic discharge rate. **Claim 10** The method implemented on a computer according to claim 1, wherein the dynamic discharge rate is maintained in response to the shadow buffer transitioning to the idle state, and the dynamic discharge rate is re-determined in response to the shadow buffer returning to the active state. **Claim 11** Determining the congestion signal includes mathematically aggregating the one-way delay between the transmitting host and the receiving host with the residence time, the method implemented on a computer according to claim 1. **Claim 12** Mathematically aggregating the one-way delay includes weighting the residence time by a coefficient, the method implemented on a computer according to claim 11. **Claim 13** A non-transitory computer-readable medium comprising encoded instructions on a memory, which when executed cause one or more processors to perform operations, the instructions being maintaining a plurality of buffers at a receiving host, the plurality of buffers including a regular buffer and a shadow buffer, the maintaining and in response to receiving a data flow from a transmitting host clock synchronized with the receiving host using a common reference clock, storing a first representation of the data of the data flow in the regular buffer and transitioning the shadow buffer from an idle state to an active state Incrementing a counter of the shadow buffer that indicates a unit of received data traffic; Determining a dynamic drain rate based on a number of units of the data deleted from the regular buffer per unit time while the shadow buffer is in the active state, where the shadow buffer returns to an idle state in response to an interruption of reception of the data flow at the receiving host that receives the data flow; Calculating a residence time as a function of the counter and the dynamic drain rate of the shadow buffer; Determining a congestion signal of the data flow based on the residence time; A non-transitory computer-readable medium comprising instructions for performing the above.

14. The non-transitory computer-readable medium according to claim 13, wherein the plurality of buffers includes a plurality of shadow buffers, and the plurality of shadow buffers includes the shadow buffer.

15. The non-transitory computer-readable medium according to claim 14, wherein each of the plurality of shadow buffers acts on a different subset of a plurality of data flows received by the receiving host.

16. The non-transitory computer-readable medium according to claim 15, wherein each of the different subsets is determined to include a data flow common to a different given application to which the data flow corresponds, the individual different applications have different priorities, and the drain rate is weighted based on the different priorities.

17. The non-transitory computer-readable medium according to claim 15, wherein each of the different subsets is determined to include a data flow common to a given set of senders.

18. The non-transitory computer-readable medium according to claim 13, wherein the instruction to increment the counter includes an instruction to multiply a coefficient by the unit of received data traffic.

19. The non-transitory computer-readable medium according to claim 13, wherein the instruction further includes an instruction to decrement the shadow buffer at a rate slower than the data is deleted from the regular buffer while the shadow buffer is in the active state.

20. The non-transitory computer-readable medium according to claim 19, wherein the slow rate is determined by multiplying the rate at which the data is deleted from the regular buffer by a coefficient that is 0.9 or more but less than 1.

Images