Efficient time-based correlation of data streams

Abstract

A technique for efficient data association is provided. A first data segment is received, and a first hash table from a plurality of hash tables is selected based on a timestamp associated with the first data segment. In addition, a first hash bucket in the first hash table is identified based on the first data segment. It is determined that the first hash bucket includes a second data segment. When it is determined that the first hash bucket meets a predefined criterion, the second data segment is removed from the first hash bucket, and the first data segment and the second data segment are associated.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of co-pending U.S. Provisional Patent Application No. 62 / 694,403, filed on July 5, 2018. The entire contents of the aforementioned related patent application are incorporated herein by reference. Technical Field

[0003] Embodiments presented in this disclosure generally relate to time-based association of data. More specifically, embodiments disclosed herein relate to techniques for associating different data sources in an efficient and scalable manner. Background Art

[0004] Data streams can come from various sources, and data streams are increasingly used in various processes today. For example, telemetry data is usually streamed to provide constant (or near-constant) monitoring. Such telemetry data can include data for monitoring any number of data sources, such as network telemetry, vehicle or equipment telemetry, weather telemetry, etc. Data streams can include a large amount of data, and the nature of the stream usually means that the data is received relatively continuously, with limited interruptions (or no interruptions). Therefore, efficient technology is needed to process such data to ensure that the monitoring of the data stream continues without delay or interruption. Depending on the specific data stream, such delays or interruptions can cause serious problems. In addition, in many embodiments, the data stream is received from a variety of different sources, and the data must be associated or aligned to facilitate the analysis of the data. Unfortunately, existing solutions require a lot of resources and cannot operate quickly enough to achieve real-time monitoring of larger data streams. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] In order to understand the above-mentioned features of the present disclosure in detail, the present disclosure briefly summarized above may be described in more detail by reference to embodiments, some of which are shown in the accompanying drawings. However, it should be noted that the drawings only show typical embodiments of the present disclosure, and therefore the drawings should not be considered as limiting the scope of the present disclosure, as the present disclosure may allow other equally effective embodiments.

[0006] Figure 1 is a block diagram illustrating a system for correlating data streams according to one embodiment disclosed herein.

[0007] Figure 2 A data association engine according to one embodiment disclosed herein is shown.

[0008] Figure 3 A multi-level association system according to one embodiment disclosed herein is shown.

[0009] FIG. 4A to FIG. 4B A cluster of a multi-level association system according to one embodiment disclosed herein is shown.

[0010] Figure 5 is a block diagram of a cluster of correlation engines according to one embodiment disclosed herein.

[0011] Figure 6 is a flow chart showing a method of associating data according to one embodiment disclosed herein.

[0012] Figure 7 is a flow chart illustrating a method for determining data association according to one embodiment disclosed herein.

[0013] Figure 8 is a flow chart illustrating a method of associating data using a ring buffer according to one embodiment disclosed herein.

[0014] Fig. 9 is a flow chart showing a method of associating data partitions according to one embodiment disclosed herein.

[0015] Fig.10 is a flow chart showing a method for associating data segments according to one embodiment disclosed herein.

[0016] To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements disclosed in one embodiment may be beneficially utilized on other embodiments without specific recitation. DETAILED DESCRIPTION

[0017] Various aspects of the invention are set out in the independent claims, preferred features are set out in the dependent claims. Features of one aspect may be applicable to each aspect alone or in combination with the other aspects.

[0018] According to one embodiment presented in the present disclosure, a method is provided. The method includes: receiving a first data record among a plurality of data records in a data stream. The method also includes: selecting a first element in a circular buffer based on a timestamp of the first data record, wherein the circular buffer includes a plurality of elements, each element corresponding to a corresponding time window. In addition, the method includes: identifying a first hash table associated with the first element in the circular buffer. The method also includes: generating a first hash value based on the first data record, and determining that a second data record is associated with the first hash value in the first hash table. In addition, the method includes: removing the second data record from the first hash table, linking the first data record and the second data record, and transmitting the linked first data record and the second data record to a downstream operator.

[0019] According to a second embodiment presented in the present disclosure, a computer program product is provided. The computer program product includes a computer-readable storage medium having a computer-readable program code contained therein, and the computer-readable program code can be executed by one or more computer processors to perform an operation. The operation includes: receiving a first data segment, and selecting a first hash table in a plurality of hash tables based on a timestamp associated with the first data segment. In addition, the operation includes: identifying a first hash bucket in a first hash table based on the first data segment, and determining that the first hash bucket includes a second data segment. In addition, when it is determined that the first hash bucket meets a predefined criterion, the operation includes: removing the second data segment from the first hash bucket, and associating the first data segment with the second data segment.

[0020] According to a third embodiment presented in the present disclosure, a system is provided. The system includes one or more computer processors and a memory containing a program that performs an operation when executed by the one or more computer processors. The operation includes: receiving a first data segment, and selecting a first hash table in a plurality of hash tables based on a timestamp associated with the first data segment. In addition, the operation includes: identifying a first hash bucket in a first hash table based on the first data segment, and determining that the first hash bucket includes a second data segment. In addition, when it is determined that the first hash bucket meets a predefined criterion, the operation includes: removing the second data segment from the first hash bucket, and associating the first data segment with the second data segment.

[0021] Example Embodiments

[0022] The new generation of network devices is able to export various telemetry data through software (i.e., software telemetry) and hardware (i.e., hardware telemetry) sensors. In an embodiment, telemetry data from multiple sensors and switches (or other devices) must be correlated and aligned in real time to ensure meaningful insights. However, when it comes to hardware telemetry data (e.g., traffic telemetry, streaming statistics, etc.), the data rates can be very high (greater than one million flows per second). This requires efficient ingestion and processing techniques. In various parts of this disclosure, network telemetry is used as an illustrative example. However, as will be appreciated by those of ordinary skill in the art, embodiments of the present disclosure can be readily applied to any data flow embodiment (e.g., in various embedded devices), or any implementation requiring association of data sources.

[0023] Existing time-based flow correlation engines cannot provide sufficient efficiency, which means that they cannot maintain real-time correlation of data, especially when the data rate increases. This leads to delays, which can seriously affect processing and monitoring work. In addition, existing processing engines are computationally expensive, which limits the horizontal expansion of microservice architectures. Embodiments of the present disclosure implement efficient correlation of traffic telemetry (FT) data, including data collected from hardware, application-specific integrated circuits (ASICs), forwarding engines in network nodes, etc. In addition, embodiments of the present disclosure implement efficient correlation of streaming buffer statistics (SSX) data, which can be derived from hardware, ASICs, forwarding engines, etc.

[0024] Embodiments of the present disclosure provide efficient algorithms and techniques for real-time correlation of high-rate data stream data (e.g., network telemetry data). In terms of resource requirements, the embodiments disclosed herein are lightweight, which enables horizontal expansion and multi-level deployment models. In addition, embodiments of the present disclosure provide O(1) efficiency, making the implementation much less sensitive to changes in data rates. Advantages of the present disclosure include better algorithmic efficiency in terms of space complexity and time complexity, very low memory usage compared to existing systems, high horizontal and vertical scalability for large use cases, and applicability to real-time correlation of data from arrays of sensors or data sources that are highly sensitive to rate and time.

[0025] In one embodiment of the present disclosure, these advantages are achieved by associating data stream data using one or more circular buffers (also referred to as time wheels, circular buffers, etc.). In an embodiment, each piece of data is placed in an appropriate element on the circular buffer based on a timestamp associated with each piece of data (e.g., a timestamp of a corresponding event, or a timestamp at which the data is recorded or sensed). In addition, in one embodiment, data is stored and associated based on establishing association logic using corresponding metadata, as discussed in more detail below. Although the embodiments herein are discussed with respect to circular buffers, in various embodiments, any buffer structure (including queues, linear buffers, etc.) can be utilized. In addition, although circular buffers are discussed, in an embodiment, the buffer can occupy linear memory, and each end of the buffer can be marked or annotated so that the read pointer and the write pointer can return to the beginning of the buffer after reaching either end. In addition, in an embodiment, each element or data block in the circular buffer does not need to be stored in a continuous block of memory, and can be stored between or in the parts of the memory associated with the circular buffer together with unrelated data.

[0026] In an embodiment of the present disclosure, a ring buffer is created, wherein each slot in the ring buffer is associated with a corresponding hash table, and wherein each slot corresponds to a time window. Data is received and segmented. For each data segment, a slot in the ring buffer is selected based on a timestamp associated with the data segment. The hash table corresponding to the identified slot is searched to identify a match to the data segment. If a match is found, the data is removed from the table and the two segments are associated. If no match is found, the segment is stored in the hash table.

[0027] Figure 1 1 is a block diagram of a system 100 for correlating data streams according to an embodiment disclosed herein. As shown, the system 100 includes a stream correlation device 105, a plurality of data sources 103A-N, and a correlated data sink 155. In the illustrated embodiment, the data sources 103A-N are network devices (e.g., routers, switches, etc.), and the data streams to be correlated include network telemetry. Of course, in various embodiments, the stream correlation device 105 can be used to process and correlate any type of data in any type of data stream from any type of sensor or device. In addition, although a single correlated data sink 155 is shown, in embodiments, any number of sinks may be utilized. Similarly, although the correlated data sink 155 is shown as a database, in embodiments, the correlated data sink 155 may include one or more programs or applications, devices, etc., depending on the specific implementation.

[0028] In the illustrated embodiment, the stream association device 105 typically receives data (e.g., data in one or more data streams) and associates the data based on a timestamp of the data, metadata associated with the data, and / or content of the data. In an embodiment, the specific function used to associate the data may be different based on the type of data being associated and the desired association. In an embodiment, a user or administrator may define the association function based on a specific implementation and the desired result. Although a single device is shown, in an embodiment, the stream association device 105 may operate as a distributed system across one or more devices. Similarly, in an embodiment, the operation of the stream association device 105 may be performed in a virtual environment (e.g., in one or more logical partitions or virtual machines). In addition, in an embodiment, multiple stream association devices 105 may operate in a cluster or grouping to distribute incoming workloads or provide different associations, as discussed in more detail below.

[0029] In the illustrated embodiment, the stream association device 105 includes a processor 110, a memory 115, a storage device 120, and a network interface 125. In the illustrated embodiment, the processor 110 retrieves and executes programming instructions stored in the memory 115, and stores and retrieves application data residing in the storage device 120. The processor 110 represents a single CPU, multiple CPUs, a single CPU with multiple processing cores, etc. Generally, the memory 115 is included to represent random access memory. The storage device 120 can be a disk drive or a flash-based storage device, and it can include fixed and / or removable storage devices, such as fixed disk drives, removable memory cards, or optical storage devices, network attached storage devices (NAS), or storage area networks (SAN). Through the network interface 125, the stream association device 105 can be communicatively coupled with other devices (including data sources 103A-N, associated data receivers 155, etc.).

[0030] As shown, the memory 115 includes a correlation engine 130. Although a single correlation engine 130 is shown, in an embodiment, multiple correlation engines 130 can operate sequentially or in parallel to perform multi-level correlation operations. In addition, in some embodiments, the correlation engine 130 can operate as an engine cluster or engine group to distribute the processing of incoming data. As shown, the correlation engine 130 includes a pre-processing component 135, a buffer component 140, and a post-processing component 150. In addition, in the illustrated embodiment, the buffer component 140 includes multiple hash tables 145A-N. Although the pre-processing component 135, the buffer component 140, and the post-processing component 150 are shown as discrete components, in an embodiment, their operations can be combined or separated among one or more other components instead.

[0031] In one embodiment, the pre-processing component 135 receives a data stream (e.g., from another correlation engine 130, another stream correlation device 105, or from one or more data sources 103A-N) and performs pre-processing to facilitate correlation of the data. In one embodiment, the pre-processing component 135 delineates the received data stream into discrete records, packets, portions, segments, or logical structures. For example, in such an embodiment, if the data stream includes streaming statistics (SSX) data, the pre-processing component 135 identifies SSX packets defined in the data. In addition, in some embodiments, the pre-processing component 135 filters the incoming data stream (or the identified or defined portions, packets, or records in the data stream) based on predefined criteria. For example, in such an embodiment, an administrator may wish to exclude data collected at certain times, related to certain streams or data, etc. from a specified device or interface.

[0032] Once the data has been preprocessed, the buffer component 140 receives the preprocessed data (e.g., a demarcated portion of a data stream) and buffers the data so that it can be associated. In an embodiment, buffering is used to provide the necessary time-cushion for sensor delays and / or transmission delays from one or more data sources 103A-N. In one embodiment, the buffer component 140 generates a hash value for each received data portion and stores the data portion in one or more hash tables 145A-N based on the hash value. In one embodiment, the buffer component 140 generates a key for each data portion based on the content of the portion or metadata associated with each data portion. For example, in one embodiment, the data stream includes flow data associated with a network flow. In such an embodiment, the buffer component 140 can generate a hash key for each data portion (e.g., for each flow or packet identified in traffic telemetry) based on the associated 5-tuple (e.g., source IP address, destination IP address, source port, destination port, and protocol). For example, in embodiments involving correlation of flow telemetry, each portion or record of flow telemetry may include data about a particular flow or network event, such as the entry or exit of a particular packet. Depending on the particular correlation desired, the hash key for each telemetry record may correspond to one or more of the packet's source and destination IP addresses, source and destination ports, and / or transport protocols.

[0033] Similarly, in embodiments involving SSX telemetry, the buffer component 140 can generate a hash key for each portion of SSX data based on metadata such as a switch, interface, queue (e.g., the switch, interface, and / or queue to which the portion of data relates or describes). In addition, in embodiments, different types of data (e.g., traffic telemetry and SSX data) can be associated by selecting association criteria that make sense for both data flows, as discussed in more detail below. In embodiments, the specific function used to generate the hash key (e.g., the specific data or metadata used) can depend on the type of data being associated and the type of association desired. In embodiments, these keys are defined based on user or administrator configuration.

[0034] In one embodiment, when a conflict is detected in hash tables 145A-N, buffer component 140 determines that the received record matches the record stored in hash tables 145A-N, and therefore these records should be associated and related to each other. For example, it is assumed that the first record stored in hash tables 145A-N corresponds to the departure of a packet from a first switch or router, and the newly received record corresponds to the next switch or router in the network that the packet enters. In an embodiment, buffer component 140 can pop up the record stored in hash tables 145A-N and associate or link it with the newly received record. In an embodiment, the data of this association is then transmitted downstream (e.g., to an association data receiver 155, to another association engine 130, and / or to another stream association device 105).

[0035] In one embodiment, in order to provide time-based association and content-based association, the buffer component 140 can assign each hash table 145A-N to a specific time window. For example, in one embodiment, the buffer component 140 utilizes a circular buffer with N elements, wherein each element includes a specific hash table 145A-N (or a pointer to a specific hash table 145A-N). In one embodiment, the number of elements in the circular buffer can be defined by an administrator. In an embodiment, each element in the circular buffer corresponds to a specific time window (e.g., 100 milliseconds). When the rotation time of the circular buffer has passed, the oldest data can be deleted or overwritten to make room for newer data. For example, assuming that the circular buffer includes ten elements, the time span of each element is one hundred milliseconds. In such an embodiment, the rotation time of the buffer is one second. That is, the end pointer of the circular buffer can point to the time of about one second in the past, and data older than one second can be discarded, as described in more detail below.

[0036] In an embodiment, upon receiving a record or portion of data, the buffer component 140 may determine a timestamp for the record. In an embodiment, this may correspond to the time of an event corresponding to the data record (e.g., the entry or exit of a packet, the addition of data to a queue, the removal of data from a queue, the receipt of data to be processed, the completion of processing, the triggering of a sensor, the time when a sensor reading or data value is recorded, etc.). Based on the timestamp, the buffer component 140 may identify the corresponding element in the circular buffer, thereby determining which hash table 145A-N to search. In this way, data is associated based on its content within a predefined time window.

[0037] In an embodiment, once an association is identified, the post-processing component 150 receives the associated data and performs any further operations. For example, in one embodiment, the post-processing component 150 filters the associated records based on predefined filtering criteria. In another embodiment, the post-processing component 150 compacts or compresses the records, or otherwise prepares them for transmission, storage, and / or processing by downstream entities. The associated data is then transmitted to the associated data receiver 155.

[0038] In an embodiment, there are several configuration options that can be used to adjust the specific implementation of the correlation engine 130. For example, in an embodiment, an administrator can define the amount of time that data should be buffered. In addition, in an embodiment, the administrator can define the width of each element in the time wheel (e.g., the amount of time included in the window). In an embodiment, a larger time bucket size corresponds to a smaller association granularity. In addition, in an embodiment, the administrator can define the memory available for the buffer, as well as the rate and amount of data to be associated with it. In addition, in an embodiment, a user or administrator can define the association type based on specifying specific data to be used when associating data (e.g., specific data or metadata in each record to be used to align the data).

[0039] Figure 2 A data association engine 130 according to one embodiment disclosed herein is shown. In the illustrated embodiment, the association engine 130 includes a circular buffer 215, and several blocks 205, 210, and 225 for processing data. As shown, data is consumed from one or more data pipelines, and at block 205, the data is pre-processed to delimit it into discrete data records. In addition, at block 210, any additional filtering and input processing is performed. The data record then advances to the circular buffer 215. As shown, the circular buffer 215 includes separate data elements 220.1, 220.2, ..., 220.M-1, and 220.M. In addition, as indicated by the ellipsis, the circular buffer 215 may include any number of data elements 220, each corresponding to a corresponding time window. In an embodiment, the number of data elements 220 is defined by a user or administrator. In addition, in an embodiment, the width of each data element 220 (e.g., the amount of time included therein) is also defined by a user or administrator. Furthermore, although a single hash table 145 is shown for clarity, in an embodiment, each data element 220 is associated with a corresponding hash table 145, as described above. In an embodiment, each corresponding data element 220 includes a pointer to the corresponding hash table 145.

[0040] As described above, in one embodiment, each data element 220 covers a predefined time window. In the illustrated embodiment, when a data record is received, the correlation engine 130 identifies the timestamp of the record and selects the data element 220 that covers the time window including the timestamp. For example, the first data record may correspond to data element 220.1, while the subsequent data record corresponds to data element 220.2. Note that data records are assigned to data elements 220 not based on when they are received but based on their timestamps. Thus, after placing the record in data element 220.2, the next data record may belong to data element 220.1, data element 220.M-1, or any other element. In an embodiment, once the appropriate data element 220 is identified, the correlation engine 130 utilizes the hash value of the record (which may be generated based on predefined criteria, as described above) to search the hash table 145 corresponding to the identified data element 220. If no match is found, in an embodiment, the correlation engine 130 inserts the data record into the hash table 145 at the position specified by the hash value and proceeds to the next data record.

[0041] As shown, hash table 145 is delimited into several entries or buckets. In an embodiment, a specific entry or bucket to which a data record belongs is defined based on the hash value of the data record. For example, in the illustrated embodiment, each entry corresponds to a hash value from "hasch1", "hash2" to "hashN". In an embodiment, the number of entries or buckets in each hash table 145 is defined by a user or administrator. In an embodiment, identifying one or more matching data records for a newly received data record includes: hashing the data record (or a certain identifying portion thereof) to determine the appropriate hash bucket, and determining whether there are one or more data records in the entry. If so, these records are considered as matches to the currently received record.

[0042] In an embodiment, if a match is identified in the hash table 145, the association engine 130 removes the matching data record from the hash table 145, associates, relates or links the two data records (e.g., the record stored in the hash table 145 and the newly received and processed record), and transmits them to block 225 for post-processing. In some embodiments, the associated data records are included in a combined data structure that references or points to all data records belonging to a specific association (e.g., an association of a specific type and a specific time window). As shown, in block 225, the association engine 130 can also perform any output processing, filtering and compression of the desired associated data records. In one embodiment, when each new data record is received, the association engine 130 also analyzes the timestamp to determine whether to rotate the buffer. For example, if the timestamp is earlier than the time covered by the earliest data element 220 (e.g., the timestamp is less than the current time minus the total wheel rotation time), the record is immediately discarded. Similarly, if the timestamp is newer than the window included in the latest data element 220, the correlation engine 130 will rotate the wheel by the width of one data element 220, thereby erasing data older than the rotation time (found in the last data element 220) and clearing space for a new hash table 145 for the latest time window.

[0043] In an embodiment, the rotation of the circular buffer 215 is achieved by moving the start pointer and the end pointer (also referred to as the start pointer and the end pointer, or the head and tail pointers). In some embodiments, the start pointer points to the newest data element 220, while the tail data pointer indicates the oldest data element 220. For example, after determining that the circular buffer 215 should be rotated, the correlation engine 130 can move the two pointers to point to their adjacent data elements 220. If the circular buffer 215 is implemented as a linear sequence of data elements 220, the correlation engine 130 can check to see if the updated pointer position places it outside the end of the buffer. If so, the pointer is placed at the other end of the buffer. In this way, the (one or more) pointers move down the buffer, one data element 220 at a time, until the cycle returns to the beginning.

[0044] In some embodiments, the correlation engine 130 is used to correlate two pieces of data. For example, in one embodiment, data from two sources can be correlated. In such an embodiment, when a match is found, the data is immediately removed from the circular buffer 215 and forwarded downstream. However, in some embodiments, the correlation engine 130 is configured to correlate three or more pieces of data. For example, if data from three or more sources (e.g., three or more sensors, network devices, etc.) are to be correlated, the correlation engine 130 will not immediately pop the data from the circular buffer 215 after identifying a match.

[0045] In one embodiment, if a match is identified, the association engine 130 (e.g., buffer component 140) determines the number of data sources to be associated and identifies the number of data items associated with the current hash value (e.g., how many data sources have been identified and associated). If the number of data items in a particular bucket of the selected hash table 145 (e.g., the number of matches) meets a predefined value, all of these data portions are removed from the ring buffer 215 and associated with each other. However, if the number of data items is below the defined value, the new data portion is inserted into the identified location and the association engine 130 continues to search for the remaining data.

[0046] For example, in a spine-leaf network topology, flow telemetry data may be collected from three nodes (e.g., an ingress leaf node, an egress leaf node, and a spine node). In such an embodiment, the correlation engine 130 may avoid removing data from the ring buffer 215 until data has been received from all three nodes. In an embodiment, a counter (e.g., a particular hash value) for a particular bucket is incremented each time data is inserted into the hash table 145. When a target count of flow records has been received (e.g., one from each data source), the data in the identified bucket is popped out and sent to the next processing stage as a set of associated data records. In addition, in an embodiment, similar logic exists when associating flow data with SSX data from multiple switches, and when associating multiple types of data for the same switch (e.g., ingress queue occupancy, egress queue occupancy, ingress queue drops, egress queue drops, etc.). Although flow telemetry and SSX data are used as example associations herein, embodiments of the present disclosure may be utilized to associate any type of data from any number of data sources.

[0047] In some embodiments, any data still remaining in the oldest data element 220 is lost when the circular buffer 215 rotates. In other embodiments, before overwriting the data element 220, the correlation engine 130 extracts any remaining data, links it appropriately, and forwards it downstream. For example, assume that a particular implementation requires data records from three different data sources to be associated. If a particular hash bucket has only two such records (e.g., the correlation engine 130 has not yet received the third record), the correlation engine 130 can still link, associate, or otherwise connect the two records and transmit them for further processing. In some embodiments, these incomplete associations are converted to different data receivers, or are to be modified or subjected to different output processing.

[0048] Figure 3A multi-stage correlation system 300 according to one embodiment disclosed herein is shown. In the illustrated embodiment, the independent correlation engines 130 work together as a fully functional single-node correlation engine 300 in the correlation system. As shown, each correlation engine 130 serves a different kind of input sensor (e.g., different types of data streams). In the illustrated embodiment, the system 300 includes a pipeline stage 350, a correlation stage 351, and a data receiver stage 352. The pipeline stage 350 includes three pipelines, one for each data type. As shown, the pipeline stage 300 includes a streaming statistics pipeline 305 for ingesting SSX data from a network device. Similarly, the pipeline stage 300 includes a flow telemetry pipeline 310 for ingesting flow data from a network device. In addition, the pipeline stage 350 includes a software telemetry pipeline 315 for ingesting software telemetry (e.g., information about network topology, neighboring devices, etc.). The data pipelines included in the system 300 are described for illustrative purposes only and are not intended to be limiting. In an embodiment, the configuration of system 300 may be used to implement multi-level association of any data.

[0049] In the illustrated embodiment, the software telemetry pipeline 315 feeds into a correlation engine 325 designated as the "FT level." In an embodiment, the correlation engine 325 receives and correlates flow telemetry data, such as entry and exit events from various devices, as described above. In an embodiment, the correlation engine 325 correlates the flow data based on a 5-tuple associated with the packets involved, as described above. Thus, in the illustrated embodiment, flow telemetry data is received from multiple points (e.g., multiple network devices) in a flow path (e.g., in a network). In an embodiment, for each flow, a ring buffer 215 is used to correlate the data. For example, in a leaf-spine topology, data is received from an ingress leaf node switch, an egress leaf node switch, and one or more spine node switches. In one embodiment, the correlation engine 325 correlates FT data from multiple supporting switches.

[0050] Furthermore, as depicted in the illustrated embodiment, when flow data records are correlated, the first stage additionally includes a path computation 330 stage. In an embodiment, the path computation 330 receives correlated flow telemetry data from a plurality of switches or devices and augments it with link-level data (e.g., Link Layer Discovery Protocol (LLDP) data and / or Cisco Discovery Protocol (CDP) data) from switches (including any switches in the path that do not support FT or hardware telemetry) to form an end-to-end flow path for each data packet or data flow from source to destination. In one embodiment, the path corresponds to a packet flow within a data center. In the illustrated embodiment, the data is received via the software telemetry pipeline 315.

[0051] Additionally, in the illustrated embodiment, a second correlation engine 320, labeled "SSX-FT Level," receives SSX data from the streaming statistics pipeline 305 and correlates it with the associated flow telemetry data from the correlation engine 325. For example, in an embodiment, the correlation engine 320 correlates the SSX data and the flow data based on metadata (e.g., the switch, interface, and / or queue associated with the SSX data and the flow data). That is, in an embodiment, the hash key used by the correlation engine 320 to identify a match in the corresponding ring buffer is based on metadata such as a particular device, interface, queue, port, or other component associated with the data. For example, in one embodiment, after an ingress event has been correlated (by the correlation engine 325) with its corresponding egress event, the correlation engine 320 identifies the SSX data associated with each of these ingress and egress events (e.g., from the device where the ingress / egress occurred) and correlates that data. In this way, the correlation engine 320 generates correlations for flow telemetry and SSX data (e.g., streaming buffer statistics), which provides accurate status for forwarding queues and buffers in each hop of the flow. In one embodiment, the second level of correlation provided by the correlation engine 320 can be distributed and requests can be fed to the target correlation engine 320 based on routing logic including hashing and / or stateful load balancing techniques.

[0052] In the illustrated embodiment, the system 300 includes a third level with a correlation engine 335 labeled "SSX-FT Final Level" that enables multiple correlation engine levels to be combined to form a larger system that distributes the load and handles larger scale depending on the number of switches / routers or flows being correlated. For example, more than one instance of the correlation engine 320 may be involved in the correlation of a flow. In one embodiment, each correlation engine 320 is designated to correlate flow telemetry to SSX data for a defined set of switches or devices. In such an embodiment, the correlation engine 325 will distribute the SSX correlation request across multiple correlation engines 320. All of these correlation engine 320 correlation levels will correlate the flow telemetry data based on the request for the corresponding switch. Therefore, in the illustrated embodiment, the final correlation engine 335 receives multiple groups of FT to SSX correlation data (each group corresponding to one or more devices) from each correlation engine 320. The correlation engine 335 can then correlate or collate the individual associations to form the final correlation data. In an embodiment, the FT-SSX correlation data from the second level (e.g., correlation engine 320) needs to be mapped again to the path calculation FT data from the first level (e.g., correlation engine 325 and path calculation 330). Since the second level (provided by correlation engine 335) can be distributed, in an embodiment, this third level (correlation engine 335) is used to aggregate this data from the second level and correlate it to the corresponding path calculation FT data.

[0053] In an embodiment, the final association data is then transmitted to the association data receiver 340. Therefore, as shown in the figure, embodiments of the present disclosure can be used to perform multi-level association. In addition, in an embodiment, data streams can be associated based on different factors at each level, based on the desired association and the nature of the underlying data. Nevertheless, the association technology disclosed herein still maintains 0(1) efficiency, thereby producing efficient association. In this way, as understood by those of ordinary skill in the art, embodiments of the present disclosure enable various association functions and operations to be implemented to suit the needs of a specific implementation.

[0054] FIG. 4A to FIG. 4B A cluster 400 of a multi-level correlation system 300 according to one embodiment disclosed herein is shown. The illustrated embodiment includes two multi-level correlation engines 300A and 300B, as described above with reference to Figure 3As described. In the illustrated embodiment, separate systems 300A-B are configured to interoperate so that data is clustered across each system. For example, as shown by arrow 445, when stream correlation data is transmitted from correlation engine 425B to correlation engine 420B, the data is also transmitted to correlation engine 420A to be correlated with SSX data received by system 300A. Similarly, as shown by arrow 450, when stream correlation is transmitted from correlation engine 425A to correlation engine 420A, the data is also transmitted to correlation engine 420B to be correlated with SSX data received by system 300B. In this way, each system shares correlations to ensure accurate results across the cluster.

[0055] Figure 5 5 is a block diagram of a cluster 500 of correlation engines according to one embodiment disclosed herein. In the illustrated embodiment, any number of pipelines 505.1-N in pipeline stage 530 may be fed into correlation stage 531. Furthermore, as shown, pipeline stage 530 may include any number of different correlation stages 515.1-M. In the illustrated embodiment, data is provided from pipeline stage 530 to correlation stage 531 via bus 510. In one embodiment, data from a particular pipeline 505 is always provided to a particular identified correlation stage 515. In other embodiments, correlation stage 515 is dynamically selected, such as by a hashing algorithm or a load balancing algorithm.

[0056] In the illustrated embodiment, each correlation stage 515 includes one or more correlation engines 130, each including one or more ring buffers 215. In the illustrated embodiment, once each stage has completed its correlation, the data is forwarded to the data receiver 525 via the second bus 520. In some embodiments, the data may additionally or alternatively be provided to one or more other correlation stages 515, as defined by the needs of a particular implementation.

[0057] Figure 6 6 is a flow chart illustrating a method 600 for associating data according to one embodiment disclosed herein. The illustrated method 600 begins at block 605, where the correlation engine 130 receives streaming data. As described above, the data may be a telemetry stream, a sensor data stream, a logging stream, or any other data. The method 600 proceeds to block 610, where the pre-processing component 135 selects data segments from the data stream. In one embodiment, the partitioning is based on a logical structure of the data stream. In some embodiments, the partitioning is performed by a pre-processing stage, and the segments are received sequentially by the correlation engine 130.

[0058] At block 615, the buffer component 140 identifies the timestamp of the selected segment. Then, the method 600 continues to block 620, where the buffer component 140 selects an appropriate hash table to process the record based on the timestamp. At block 625, the buffer component 140 searches the identified hash table based on the data included in the data segment. For example, as described above, in one embodiment, the buffer component 140 uses metadata about the data segment or data about a particular content record as a key to search the table. In an embodiment, the formulation of the key is determined by a user configuration based on the desired association.

[0059] Then, the method 600 proceeds to block 630, where the buffer component 140 determines whether there is a match in the table. That is, as described above, the buffer component 140 searches the selected hash table based on the generated key to identify a conflict with an existing record. If there is such a conflict, the buffer component 140 determines that the current segment should be associated with the stored segment. If there is no match, the method 600 proceeds to block 635, where the buffer component 140 inserts the segment into the table, and the method 600 returns to block 605 to receive additional streaming data. That is, in an embodiment, the method 600 proceeds to process the next identified segment in the data stream.

[0060] If the buffer component 140 identifies the association in the table, the method 600 continues to block 640, where the buffer component 140 increments the match counter associated with the generated key / bucket. Of course, in some embodiments, the buffer component 140 can be configured to associate data between two sources or two types of data. Therefore, in some embodiments, a counter is not used to determine the number of identified data segments, and once a conflict is found, data is immediately removed from the table. In addition, in some embodiments, the buffer component 140 determines the number of data records currently stored in the identified hash bucket by other means (rather than incrementing the counter).

[0061] At block 645, the buffer component 140 determines whether the match counter meets a predefined criterion (e.g., a threshold number of matches). In one embodiment, a user or administrator defines the criterion based on the number of data records to be associated. As described above, in various embodiments, different data sources, different types of data, etc. may be associated. Criteria may be defined so that data is removed from the circular buffer 215 only when a predefined number of associations or matches have been identified. If the criterion is not met, the method 600 proceeds to block 635, where the data is inserted into the identified table at the identified location. The method 600 then returns to block 605 to continue receiving data.

[0062] However, if the buffer component 140 determines that the criteria are met, the method 600 proceeds to block 650, where the buffer component 140 removes the identified match(es) from the hash table and associates all data with the current segment. This association may be accomplished by linking the records, including an indication of the association, storing or transmitting the records in a single data structure, or any other suitable technique for identifying an association. The method 600 then proceeds to block 655, where the correlation engine 130 transmits the association to a downstream operator. As described above, in embodiments, this may include one or more additional correlation engines, databases, applications, etc.

[0063] Figure 7 700 is a flowchart illustrating a method 700 for determining data association according to an embodiment disclosed herein. In one embodiment, the method 700 provides for Figure 6 . Method 700 begins after the timestamps of the segments have been identified in block 615. At block 705, the buffer component 140 determines the desired type of association. As described above, this can be defined by an administrator to ensure that the correlation engine 130 serves its intended purpose. For example, in an embodiment, the desired association can be about a 5-tuple of a record, a switch, port, interface, and / or queue associated with the record, and the like. Further, in an embodiment, the association can be about a sensor or device transmitting the data, the type of data included in the data stream, and the like. The examples given herein are not intended to be limiting, and other data associations will be apparent to those skilled in the art.

[0064] Once the desired association has been determined, the method 700 proceeds to block 710, where the buffer component 140 generates a hash key for the segment based on the association. For example, in an embodiment utilizing a 5-tuple of data groupings, generating the hash key may include concatenating each value in the 5-tuple into a single value. In some embodiments, the 5-tuple itself may serve as a key. Generally, in an embodiment, generating a hash key includes: identifying data for matching or associating records; extracting the data from the data segment (or from metadata associated with the segment); and formatting it for a hash algorithm, as defined in a configuration associated with the association engine. At block 715, the buffer component 140 generates a corresponding hash value for the key. Any suitable hash algorithm may be used to generate the hash value. Then, the method 700 proceeds to block 720, where the buffer component 140 identifies an appropriate hash table based on the timestamp of the segment.

[0065] For example, as described above, in an embodiment, the buffer component 140 utilizes a ring buffer 215 of data elements 220, each ring element 215 including a pointer to a different hash table 145. Furthermore, in an embodiment, each data element 220 includes data for a defined time window. In such an embodiment, identifying the appropriate hash table includes determining which data element 220 corresponds to the timestamp of the record, and selecting the associated hash table 145. The method 700 then ends, and the selected table is searched using the hash value in block 625.

[0066] Figure 8 8 is a flow chart illustrating a method 800 for associating data using a circular buffer according to one embodiment disclosed herein. In one embodiment, the method 800 is performed together with the method 600. That is, in an embodiment, for each record or segment processed by the method 600, the method 800 is also performed. In the illustrated embodiment, the method 800 begins at block 805, where the buffer component 140 receives a data record. At block 810, the buffer component 140 identifies a timestamp of the record. The method 800 then continues to block 815, where the buffer component 140 determines whether the timestamp is earlier than the data included in the circular buffer 215.

[0067] In an embodiment, whether the timestamp is before the circular buffer 215 is determined based on the position of the buffer mark or pointer and the total wheel rotation time. For example, assume that the most recent data element 220M ends at t=50, and the circular buffer 215 includes ten entries, each spanning one second. In such an embodiment, the circular buffer 215 spans 10 seconds of data. Therefore, the total wheel rotation time is ten seconds, and the buffer only includes data newer than t=40. Therefore, if the timestamp of the current record is less than t=40 (e.g., t=35), the record will fall out of the buffer.

[0068] If the timestamp is before the width of the circular buffer 215, the method 800 continues to block 820, where the record is discarded. The method 800 then returns to block 805. If the timestamp is not before the end of the buffer, the method 800 continues to block 825, where the buffer component 140 determines whether the timestamp appears after the buffer. That is, the buffer component 140 determines whether the timestamp of the current record or segment is newer than the latest window in the circular buffer 215. Continuing with the above example, if the timestamp is greater than t=50, the record is newer than the records allowed by the circular buffer 215.

[0069] If the timestamp occurs after the window covered by the latest data element 220, the method 800 continues to block 830, where the buffer component 140 rotates the buffer by one data element 220. That is, the buffer component 140 moves the marker forward in time by one data element 220, clears the hash table 145 residing in that element, and creates a new hash table 145. The method 800 then continues to block 835. Additionally, if the buffer component 140 determines at block 825 that the timestamp is not before the buffer, the method continues to block 835. That is, continuing with the example above, if the timestamp falls between 40 and 50, the buffer component 140 determines that the record falls somewhere in the circular buffer 215.

[0070] At block 835, buffer component 140 associates selected segmentation or record. For example, in one embodiment, buffer component 140 performs method 600 to insert a record in hash table 145 or identify a match in hash table 145, as described above. In this way, ring buffer 215 is kept up-to-date, and older data that have not yet been associated are discarded. In certain embodiments, buffer component 140 does not discard the data in the oldest data element 220, but pops up any record in each hash bucket, links them together, and associates them as part of downstream transmission. In one embodiment, if only a single data record is included in a specific hash entry, this record is discarded.

[0071] Fig. 9 9 is a flow chart illustrating a method 900 for associating a data segment (partition) according to one embodiment disclosed herein. The method 900 begins at block 905, at which the correlation engine 130 receives a first data record from a plurality of data records in a data stream. At block 910, the correlation engine 130 selects a first element in a circular buffer based on a timestamp of the first data record, wherein the circular buffer includes a plurality of elements, each corresponding to a corresponding time window. Then, the method 900 continues to block 915, at which the correlation engine 130 identifies a first hash table associated with the first element in the circular buffer. At block 920, the correlation engine 130 generates a first hash value based on the first data record. Then, the method 900 proceeds to block 925, at which the correlation engine 130 determines that the second data record is associated with the first hash value in the first hash table. At block 930, the correlation engine 130 removes the second data record from the first hash table. In addition, at block 935, the correlation engine 130 links the first data record and the second data record. Finally, the method 900 proceeds to block 940 where the correlation engine 130 transmits the linked first data record and the second data record to a downstream operator.

[0072] Fig.10 1 is a flow chart illustrating a method 1000 for associating data segments according to one embodiment disclosed herein. The method 1000 begins at block 1005, where the correlation engine 130 receives a first data segment. Then, the method 1000 proceeds to block 1010, where the correlation engine 130 selects a first hash table from a plurality of hash tables based on a timestamp associated with the first data segment. At block 1015, the correlation engine 130 identifies a first hash bucket in the first hash table based on the first data segment. In addition, at block 1020, the correlation engine 130 determines that the first hash bucket includes a second data segment. Then, the method 1000 continues to block 1020, where, upon determining that the first hash bucket meets a predefined criterion, the correlation engine 130 removes the second data segment from the first hash bucket and associates the first data segment with the second data segment.

[0073] In the foregoing, reference is made to the embodiments presented in the present disclosure. However, the scope of the present disclosure is not limited to the specifically described embodiments. Rather, any combination of the described features and elements, whether or not related to different embodiments, is contemplated for implementing and practicing the contemplated embodiments. In addition, although the embodiments disclosed herein may achieve advantages over other possible solutions or over the prior art, whether a particular advantage is achieved by a given embodiment does not limit the scope of the present disclosure. Therefore, unless expressly stated in (one or more) claims, the foregoing aspects, features, embodiments and advantages are merely illustrative and are not considered to be elements or limitations of the appended claims.

[0074] As will be appreciated by those skilled in the art, the embodiments disclosed in the present disclosure may be embodied as systems, methods, or computer program products. Thus, various aspects may take the form of a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, which are generally referred to herein as "circuits," "modules," or "systems." Additionally, various aspects may take the form of a computer program product embodied in (one or more) computer-readable media containing computer-readable program code.

[0075] Any combination of (one or more) computer-readable media can be utilized. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer-readable media will include the following: electrical connections with one or more lines, portable computer floppy disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In the context of this article, a computer-readable storage medium is any tangible medium that can contain or store programs for use by or associated with an instruction execution system, device or equipment.

[0076] A computer readable signal medium may include a propagated data signal containing a computer readable program code (e.g., in baseband or as part of a carrier wave). Such a propagated signal may take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that may communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0077] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0078] The computer program code for performing the operations of various aspects of the present disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Java, Smalltalk, C++, etc.) and conventional procedural programming languages ​​(such as "C" programming language or similar programming languages). The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network (including a local area network (LAN) or a wide area network (WAN)), or the connection to an external computer can be made (e.g., by using the Internet of an Internet service provider).

[0079] Hereinafter, various aspects of the present disclosure are described with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to embodiments proposed in the present disclosure. It should be understood that each block in the flowchart and / or block diagram, as well as the combination of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer programming instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to generate a machine, so that instructions executed by a processor of a computer or other programmable data processing device create a module for implementing a function / action specified in one or more blocks of a flowchart and / or block diagram.

[0080] These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing apparatus, or other device to function in a specific manner so that the instructions stored in the computer-readable medium produce an article of manufacture including instructions for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0081] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operating steps are performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable apparatus provide a process for implementing the functions / actions detailed in one or more blocks of the flowchart and / or block diagram.

[0082] Embodiments of the present invention may be provided to end users via a cloud computing infrastructure. Cloud computing generally refers to the provision of scalable computing resources as a service over a network. More formally, cloud computing may be defined as a computing capability that provides an abstract relationship between computing resources and their underlying technology architecture (e.g., servers, storage devices, networks), thereby enabling convenient on-demand network access to a shared pool of configurable computing resources that can be quickly configured and released with minimal management effort or service provider interaction. Thus, cloud computing allows users to access virtual computing resources (e.g., storage devices, data, applications, and even complete virtualized computing systems) in a "cloud" without having to consider the underlying physical systems (or the location of those systems) used to provide the computing resources.

[0083] Typically, cloud computing resources are provided to users on a pay-per-use basis, and users are charged only for the computing resources actually used (e.g., the amount of storage space consumed by the user or the number of virtualized systems instantiated by the user). Users can access any resource residing in the cloud from anywhere at any time via the Internet. In the context of the present invention, users can access applications (e.g., correlation engine 130) or related data available in the cloud. For example, the correlation engine 130 can be executed on a computing system in the cloud and process the data in the cloud. In this case, the correlation engine 130 can associate streaming data in real time in the cloud and store such identified associations at a storage location in the cloud. Doing so allows users to access the information from any computing system attached to a network (e.g., the Internet) connected to the cloud.

[0084] In summary, a technique for efficient data association is provided. A first data segment is received, and a first hash table from a plurality of hash tables is selected based on a timestamp associated with the first data segment. In addition, a first hash bucket is identified in the first hash table based on the first data segment. It is determined that the first hash bucket includes a second data segment. Upon determining that the first hash bucket meets a predefined criterion, the second data segment is removed from the first hash bucket, and the first data segment and the second data segment are associated.

[0085] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functions and operations of possible implementations of the systems, methods and computer program products according to various embodiments. In this regard, each block in the flowchart or block diagram can represent a part of a module, a fragment or a code, which contains one or more executable instructions for implementing a specific logical function. It should also be noted that in some alternative implementations, the functions mentioned in the blocks can occur in an order different from that mentioned in the accompanying drawings. For example, depending on the functions involved, the two blocks shown in succession can actually be executed substantially simultaneously, or the blocks can sometimes be executed in reverse order. It should also be noted that each block in the flowchart and / or block diagram, as well as the combination of blocks in the flowchart and / or block diagram, can be implemented by a dedicated hardware-based system that performs the functions or actions described in detail, or by a combination of dedicated hardware and computer instructions.

[0086] In view of the foregoing, the scope of the present disclosure is determined by the following claims.

Claims

1. A method for associating data streams, comprising: receiving a first data record of a plurality of data records in a data stream; selecting a first element in a circular buffer based on a timestamp of the first data record, wherein the circular buffer comprises a plurality of elements, each element corresponding to a respective time window; identifying a first hash table associated with the first element in the ring buffer; generating a first hash value based on the first data record; and Upon determining that a second data record has been associated with the first hash value in the first hash table: remove the second data record from the first hash table; linking the first data record and the second data record; and transmitting the linked first data record and the second data record to a downstream operator, Wherein, removing the second data record from the first hash table, linking the first data record and the second data record, and transmitting the linked first data record and the second data record to the downstream operator are performed when it is determined that a predefined criterion is met.

2. The method according to claim 1, further comprising: receiving a third data record; as well as Upon determining that the timestamp associated with the third data record is newer than a predefined threshold: selecting a second element in the ring buffer, wherein the second element corresponds to the oldest time window among the corresponding time windows; identifying a second hash table associated with the second element; and The second hash table is discarded.

3. The method according to claim 1, wherein: Determining that the predefined criterion is satisfied includes determining that a number of data records associated with the first hash value in the first hash table exceeds a predefined threshold.

4. The method according to claim 1 or 2, further comprising: receiving a third data record; generating a second hash value based on the third data record; Based on the first hash value, determining that the number of data records in the first hash table associated with the second hash value does not exceed a predefined threshold; as well as The third data record is inserted into the first hash table.

5. The method according to claim 1 or 2, further comprising: receiving a third data record; as well as Upon determining that the timestamp associated with the third data record is older than a predefined threshold, the third data record is discarded.

6. A computer program product comprising: A computer readable storage medium having computer readable program code embodied therein, the computer readable program code being executable by one or more computer processors to perform operations comprising: receiving a first data segment; selecting a first hash table among a plurality of hash tables based on a timestamp associated with the first data segment; Based on the first data segment, identifying a first hash bucket in the first hash table; determining that the first hash bucket includes a second data segment; and Upon determining that the first hash bucket satisfies a predefined criterion based on determining that a threshold number of data segments are stored in the first hash bucket: removing the second data segment from the first hash bucket; and The first data segment and the second data segment are associated.

7. The computer program product of claim 6, the operations further comprising: The first data segment and the second data segment are transmitted to one or more data receivers.

8. The computer program product according to claim 6 or 7, wherein: Receiving a first data segment comprises: receiving a data stream, wherein the data stream comprises a plurality of logical data units; and Based on the multiple logical data units, the data stream is divided into multiple data segments.

9. The computer program product according to claim 6 or 7, wherein: Selecting the first one of the plurality of hash tables includes identifying a first element in a ring buffer containing a plurality of elements, wherein each element of the plurality of elements is associated with a corresponding one of the plurality of hash tables.

10. The computer program product according to claim 9, wherein: Each respective element of the plurality of elements is associated with a respective time window.

11. The computer program product of claim 10, the operations further comprising: receiving a third data segment; as well as Upon determining that the timestamp associated with the third data segment is newer than a predefined threshold: selecting a second hash table among the plurality of hash tables, wherein the second hash table is associated with a second data element, wherein the second data element is associated with an oldest time window among the corresponding time windows; and The second hash table is discarded.

12. The computer program product according to claim 6 or 7, wherein: Identifying the first hash bucket in the first hash table includes: generating a hash key based on the first data segment according to a predefined configuration; generating a hash value based on the hash key; and The first hash bucket is identified based on the generated hash value.

13. The computer program product of claim 6 or 7, the operations further comprising: receiving a third data segment; Based on the third data segment, identifying a second hash bucket in the first hash table; as well as When it is determined that the second hash bucket does not meet the predefined criterion, the third data segment is inserted into the second hash bucket.

14. The computer program product of claim 6 or 7, the operations further comprising: receiving a third data segment; as well as Upon determining that the timestamp associated with the third data segment is older than a predefined threshold, the third data segment is discarded.

15. A system for correlating data streams, comprising: one or more computer processors; as well as a memory containing a program that, when executed by the one or more computer processors, performs operations including: receiving a first data segment; selecting a first hash table among a plurality of hash tables based on a timestamp associated with the first data segment; Based on the first data segment, identifying a first hash bucket in the first hash table; and When it is determined that the first hash bucket does not meet the predefined criterion, the first data segment is inserted into the first hash bucket.

16. The system of claim 15, wherein: Selecting the first one of the multiple hash tables includes: identifying a first element in a circular buffer containing multiple elements, wherein each of the multiple elements is associated with a corresponding hash table in the multiple hash tables, and wherein each corresponding one of the multiple elements is associated with a corresponding time window.

17. The system of claim 16, the operations further comprising: receiving a second data segment; as well as Upon determining that the timestamp associated with the second data segment is newer than a predefined threshold: selecting a second hash table among the plurality of hash tables, wherein the second hash table is associated with a second data element, wherein the second data element is associated with an oldest time window among the corresponding time windows; and The second hash table is discarded.

18. The system according to any one of claims 15 to 17, wherein the operations further comprise: receiving a second data segment; Based on the second data segment, identifying a second hash bucket in the first hash table; Determining that the second hash bucket includes a third data segment; as well as When it is determined that the second hash bucket meets the predefined criterion: removing the third data segment from the second hash bucket; as well as The second data segment and the third data segment are associated.

19. An apparatus for associating data streams, comprising means for implementing the method according to any one of claims 1 to 5.

20. A computer program product or computer readable medium comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method according to any one of claims 1 to 5.

Images