Historical Netflow Analysis System and Method

Abstract

This disclosure relates to a system and method for performing a historical netflow lookup of an IP address, such as an IPv4 address, in a cost-effective, time-effective, and storage-effective manner using a blob index and a bitset. A method for finding a historical netflow having a specific IP address includes the steps of: generating a blob index file for at least two historical netflow blobs; generating a bitset for the historical netflow blob with the longest duration among the at least two historical netflow blobs; performing a lookup using the bitset and at least two blob index files to find a historical netflow having the requested IP address in the netflow during the searchable period; and displaying the found historical netflow having the requested IP address.

Description

Technical Field

[0001] [Priority Claim / Related Patent Application] This PCT application claims priority under the Paris Convention and Article 19 of the US Patent Law and is a continuation application of US Patent Application No. 8 / 243,967 filed on September 8, 2023, the entire content of which is incorporated herein by reference.

[0002] [Technical Field of the Invention] The present disclosure relates to systems and methods for analyzing historical network flows, and more particularly, to systems and methods for efficiently finding IP addresses requested in historical network flows.

Background Art

[0003] Known techniques for analyzing computer network traffic involve using known network flow data (netflow data). An example of netflow data is shown in FIG. 1. This example shows information regarding the data flow between two computer nodes (a source node with an IP address of xxx.23.23.23 and a destination node with an IP address of yyy.33.44.55). Netflow data can also include the start time, duration, source and destination port numbers, number of packets, and packet size in the data flow. Using this data, various analyses of data traffic on a computer network, such as malware detection and botnet detection, can be performed.

[0004] In addition to analyzing current netflow data, it is possible to analyze historical netflow data to perform the same various analyses of data traffic on a computer network. One significant problem is that historical netflow data can contain millions of netflows. Each netflow for a specific period can be stored in a data blob, and a single netflow data blob with a short time interval (less than 1 second) can contain approximately 7 million netflows. Therefore, if a user wants to analyze a month's worth of historical netflow data, the storage capacity required to store that month's worth of netflow data becomes enormous, and the computing power required to search that month's worth of historical netflow data is prohibitively expensive. While it is possible to search each historical netflow data blob, it is extremely time-consuming and costly, and due to the volume of netflow data, it is not practical for time ranges spanning several months.

[0005] To make the analysis of historical netflow data easier, less expensive, and faster, it is desirable to reduce the amount of data that must be searched within the historical netflow data. The disclosed historical netflow analysis system and method overcome these problems related to historical netflow analysis, and this disclosure is directed toward this purpose. [Brief explanation of the drawing]

[0006] [Figure 1] This is an example of Netflow data. [Figure 2] This shows an example of botnet detection that can be performed using historical netflow. [Figure 3] This is an example of a system architecture for a historical network flow analysis system. [Figure 4] Figure 3 shows further details of the historical network flow analysis system backend. [Figure 5] This document presents an exemplary implementation of a historical network flow analysis system backend. [Figure 6] This document describes a method for finding historical netflows. [Figure 7] An example of a blob index file is shown below. [Figure 8] This shows a method for generating an IPv4 bitset. [Figure 9] This document describes how to search historical network flows using IPv4 bit sets. [Figure 10] Figure 5 shows an alternative method for finding historical net flows using the implementation configuration. [Figure 11] Further details on how to search the history bitset are provided. [Figure 12] This shows the tree structure used to search historical network flows. [Modes for carrying out the invention]

[0007] This disclosure is particularly applicable to IPv4 address protocol-based historical netflow analysis systems and methods, and the disclosure is described in this context. However, it will be understood that the system and method can be used with netflow data for computer network addresses using other address protocols and therefore have greater utility.

[0008] The historical netflow analysis system can reduce the processing of unnecessary data when searching within historical netflows by generating and using a blob index file for each new netflow blob, and each blob index file can be stored in blob data storage. The blob index file corresponding to each blob (described in detail below) is a summary of the complete set of unique IP addresses in that blob. Furthermore, the historical netflow analysis system can reduce the processing of unnecessary data by generating and using bit sets in different time ranges (monthly, daily, hourly, etc.) that further subdivide the historical netflow data, thereby avoiding further unnecessary data processing when analyzing historical netflow data.

[0009] A historical netflow determination system and method may be used in conjunction with any system that uses well-known netflows to evaluate computer-related traffic. For example, a historical netflow determination system and method may be used for malware detection using netflows as disclosed in U.S. Patent No. 0,742,660, and / or for forming graphs from netflows as disclosed in U.S. Patent No. 1,252,185, which covers graph streaming, all of which are owned by the same assignee as this patent application and are incorporated herein by reference and form part thereof.

[0010] Figure 2 shows a botnet detection process 100 that may use the historical network flow analysis technique described later. In the example in Figure 2, one or more botnets are used against the well-known Trickbot malware. The botnet may consist of victim terminals 106A-N, command and controller servers (or C&C servers) 104A-N, and a bot master 102. Victim terminals 106A-N are computers infected with the Trickbot malware. For example, victim terminals may be computers used by bank employees and infected with malware attached to a disguised email message. The victim terminals, for example, steal customer account numbers and passwords and send them to criminals behind the botnet. C&C servers 104A-N are computers that control victim terminals by sending commands. The bot master 102 is a computer that controls C&C servers 104A-N. In computer security, it is desirable to be able to detect each, preferably all, of the computers (and their IP addresses) that are part of a botnet in order to eliminate or mitigate the threat of botnets such as Trickbot described above. The historical netflow data analysis system and method disclosed below enable the efficient detection of botnets using historical netflow data over a predetermined period (e.g., one day, one month, several months).

[0011] Figure 3 shows an example system architecture of a historical netflow analysis system 30 that enables efficient retrieval of historical netflows while reducing storage requirements and increasing processing speed. System 30 in Figure 3 may be internally used tools, services provided to users (as shown in the example in Figure 3), or third parties that interface with other systems that use netflow for various computer network analyses. In the example in Figure 3, system 30 may have one or more computing devices 32, each of which can request historical netflow retrieval services from system 30 and then consume the output of the requested services (directly or via an API). In some cases, each computing device 32 may be connected to and communicate with a backend system 36 via a communication path 34, which is also connected to a store 38. Each computing device may have a processor, memory, a display, and communication circuits (such as WiFi, cellular, Bluetooth®, Ethernet®, etc.). For example, each computing device 32 could be a personal computer 32A, a smartphone device 32B (such as an Apple® iPhone® or Android® operating system-based device), a laptop computer 32C, and / or a tablet computer 32N. Each computing device 32 may have a known browser application that runs on the computing device 32 to interact with the backend system 36.

[0012] The communication path 34 may be a wired or wireless network (or a combination of both) that uses data and communication protocols for connecting and communicating between each computing device 32 and the backend system 36. Each part (or the entire communication path) may be wired (Ethernet®, DSL, cable, etc.) or wireless (Digital Computer Network, Cellular Data Network, WiFi, LAN, WAN, etc.). The system 30 in Figure 3 can operate on any communication path.

[0013] The backend 36 may be implemented using one or more computing resources, such as a server computer, blade server, processor, memory, virtual machine, or other cloud computing resources, and the processor of the backend 36 executes multiple lines of instruction / computer code so that the processor is configured to perform various operations of the NetFlow Analyzer 36A. The NetFlow Analyzer may have a historical NetFlow Analyzer Engine 36A1 (also implemented by executing multiple lines of instruction / computer code on the processor of the backend 36) that processes historical NetFlows and generates blob index files and bitsets, facilitating the execution of historical NetFlow searches that are less cumbersome, require less storage, and can be performed more quickly using technical elements compared to known techniques for analyzing historical NetFlows. The backend 36 may be coupled to one or more stores 38 (hardware or software or a combination) that store multiple historical NetFlow blobs, blob indexes, IPv4 bitsets, and the results of searches requested by each user via a browser and / or API.

[0014] Figure 4 shows further details of the historical netflow analysis system backend 36A1 of Figure 3, in which a user can submit a new historical netflow request via a browser interface and / or the API of the backend 36. The historical netflow analysis system backend 36A1 may have an index file generator 42, a bitset generator 44, and a searcher 46, which are coupled together to produce an output that is one or more identified historical netflows based on netflow blobs stored in store 38, a system-generated index file and a generated bitset, and an IP address requested by the user. Each element of the historical netflow analysis system backend 36A1 may be implemented by multi-line instruction / computer code executed by the processor of the backend system 36. Details of the blob index file and bitset will be described later. Each of the blob index file and bitset is a technical feature of the system that reduces the amount of data that must be searched to perform a historical netflow lookup, thereby reducing storage requirements and the time it takes to perform a historical netflow lookup. Generally, one or more blob index files may be generated from one or more netflow blobs, and one or more bitsets may be generated based on blob files and blob index files.

[0015] Figure 5 shows an exemplary implementation of the historical netflow analysis system backend, where a user of the system creates a new job (such as a search for a specific IPv4 address, like 1.2.3.4, for a period such as January 1, 2023 to January 25, 2023, using a set of historical netflows that may have different time periods, as described later) that is submitted to the system via an application programming interface (API) or the like. The user can also consume the output of the job generated by the system (via the API or web interface), which may be historical netflows containing the searched IP addresses. In this implementation, the system may have temporary blob storage 38B and blob storage 38A. Temporary blob storage 38B may be used to store data while a job is being processed and may store netflow blob files used during a particular process, blob index files for each blob file, and bit sets of netflow blobs. In one implementation, each blob storage may be Azure blob storage. Netflows found with the requested IP address can be stored in blob storage 38A so that they can be output to the user who requested the historical netflow search.

[0016] The processing element 50 may poll for new jobs, download and process all netflows stored in the temporary blob store 38B, and upload the blob index file for each blob file to the temporary blob storage 38B. The processing 50 may also upload netflows to the blob storage 38A in batches. The bitset creator 44 may retrieve the blob index file and generate one or more bitsets, which are stored in the temporary blob storage 38B. The searcher 46 may poll for historical netflow searches, retrieve blob indexes and bitsets from the temporary storage 38B to perform historical netflow searches, and send the found netflows to the history processing unit 52. The history processor 52 may retrieve the found netflows, process them based on the bitsets and searcher output, and output netflows including the requested IP address.

[0017] Figure 6 shows method 60 for finding historical netflows. Method 60 can be performed using the systems shown in Figures 3–5, or other known or future-developed systems capable of performing the processes of the method shown in Figure 6. This method can process all netflow blobs uploaded to the blob (with possible different time intervals) (e.g., exemplified by *.csv.gz), and for each netflow blob, this method can generate and upload a new "blob index file" (e.g., *.ipv4.gz) containing all the unique IPv4 addresses of that blob (62). An example of a blob index file is shown in Figure 7, where each IPv4 address is on a separate line in the index file. Since the blob index file for each netflow blob is typically 5–10 MB after compression, further elimination of unnecessary data is needed to enable rapid searching of historical netflows without enormous storage demands. This reduction of unnecessary data is achieved by the bitset generated by method (64) for each blob with different time intervals of netflow data.

[0018] An IPv4 address is typically represented as a string in dotted decimal notation, but can be converted to a decimal number, for example, such that "5.5.5.5" (dotted decimal notation) becomes 84215045 (decimal notation). There are 256×256×256×256 (2 32 ) possible values between "0.0.0.0" and "255.255.255.255". 2 32 A bit set with 2 bits can be represented by 67,108,864 64-bit words (about 537 MB). As a result, a compact "lookup table" can be created using IPv4 decimal as a bit index, and the byte size of the lookup table is constant regardless of whether the bit is set or not. By storing the compressed bit set in blob storage, its storage size is reduced. As a result, when using gzip, about 85% size reduction is obtained for the bit set in units of time, and about 50% size reduction is obtained for the bit set in units of month.

[0019]

[0020] Figure 8 shows a method 800 for generating an IPv4 bit set, where an example of an IPv4 address 5.5.5.5 (decimal 84215045) is converted into bits within a bit set to generate a bit set that can be used to search for historical network flows. As shown in Figure 8, the decimal IPv4 address is converted into a word index of a set 802 of 64-bit words. In one implementation, the decimal IPv4 address is divided by 64 and then rounded down to the nearest integer (floor math function) to find the correct 64-bit word. In the example of Figure 8, the correct 64-bit word is 1315860. The correct 64-bit word can be mapped to a 64-bit word index 804 to find the correct bit within the 64-bit word 804 bit set. This is a "bit operation". This can set or retrieve the bits representing one IPv4 address.For example, to convert the IPv4 address "5.5.5.5" to decimal form, the IPv4 address is split into four segments: 5, 5, 5, and 5. Next, each segment is converted to an integer. In this example, since each segment is already represented as an integer, each segment remains the same. Bit shifting and addition can be performed. Here, the 5 in the first segment is left-shifted by 24 bits (5 << 24 = 83886080), the 5 in the second segment is left-shifted by 16 bits (5 << 16 = 327680), the 5 in the third segment is left-shifted by 8 bits (5 << 8 = 1280), and the 5 in the fourth segment remains unchanged. The results of the bit shifts are added together to obtain the final decimal number (in this example, 83886080 + 327680 + 1280 + 5 = 84215045), and thereby the IPv4 address "5.5.5.5" corresponds to the decimal number 84215045.

[0021] Since the system may be limited by a 64-bit CPU for bitwise operations, it is necessary to fetch a 64-bit word from the bitset. The index of the bitset can be calculated, and here, using the decimal form of 5.5.5.5, we calculate the word index floor (84215045 / 64) = 1315860. Then, the generation of the bitset can be done by performing bitwise operations on the 64-bit word extracted using the word index floor. For the bitset, it is desirable to know which bits to look at in the 64-bit word (there are 64 bits), which can be calculated as 84215045%64=5 in the above example. To check whether the bit representing "5.5.5.5" (84215045) exists in the bitset, this method retrieves the 64-bit word at position 1315860 in the bitset, performs the actual bitwise operation on this 64-bit word, and determines whether the bit at position 5 is set. For example, "64bitword >> 5 & 1 == 1" is true if true, and false if false. This operation can be represented in pseudocode as (words[floor(84215045 / 64)] >> (84215045 % 64)) & 1 == 1.

[0022] Once a bitset is generated, this method can generate various bitsets for NetFlow blobs. As shown in Figure 9, this method can generate bitsets for NetFlow blobs for various time periods. For example, a job that can run every hour will summarize all *.ipv4.gz blob index files into a bitset for that hour, a job that runs every day will summarize all hourly bitsets into a bitset for that day, a job that runs every month will summarize all daily bitsets into a bitset for that month, and so on for different time intervals.

[0023] Returning to Figure 6, once bit sets for different time intervals have been generated, the method can use the generated bit sets to search for historical netflows for a specific IPv4 address (66). This search process will be described in detail later with reference to Figures 10-11. The search results are returned to the user as found historical netflows (68).

[0024] For example, if a user wants to find a specific IPv4 netflow within a certain time range (in one example, by submitting a job), two processes for the search may be executed as part of the search process 110 shown in Figure 10. In the first process, the search process (which may be implemented by the searcher 46 in Figure 5) searches the bitset and index files within the time range to find the relevant netflow blob to be processed (112).

[0025] The details of this process are shown in Figure 11. In the simple example used for illustrative purposes, the user is searching for the IPv4 address 1.2.3.4 between January 1, 2023 and January 25, 2023. In this simple example, for the sake of simplicity, the IPv4 address 1.2.3.4 is found at 00:00 (midnight) on January 1, 2023, but those skilled in the art will understand that the process, using the process described below, would find the requested IPv4 address at 11:40 a.m. on January 4, 2023, for example.

[0026] The bitset and index files for that period can be thought of as a "tree" (see Figure 12) where the root is a monthly bitset and the leaves are index files. During this search, the process may perform one or more checks (four in this simple example) at different levels of time granularity based on different levels of the tree. For example, in this simple example, the first check is at a monthly granularity, the second at a daily granularity, the third at an hourly granularity, and the fourth at a minute-level time granularity. Alternatively, in this example, the monthly granularity may be the longest time interval bitset, the daily granularity at a short time interval, and the minute-level granularity at a short time interval. During the search, each branch of the tree can be quickly searched or discarded at a coarse granularity. For example, if the daily index for the second day of the month is searched and the requested address is not found, the rest of that branch does not need to be searched, thus reducing the time required to complete the search.

[0027] In a simple example, January 2023 is the longest period that needs to be searched efficiently using the blob index file and bitset. In this simple example, the first check might be to determine if IPv4 address 1.2.3.4 occurs in "year=2023 / month=01 / ipv4.bitset.gz", which is the Netflow blob for Netflow during January 2023. If IPv4 address 1.2.3.4 is found in that Netflow blob, the next check is performed. If IPv4 address 1.2.3.4 is not found in the monthly Netflow blob (based on the blob index file and bitset), in this example, the search is complete and no further searches are needed, as the requested time range of the search is only January 2023. This search process starting from the longest period means that the process is complete if the IPv4 address is not found in the Netflow blob for the longest period, compared to other processes.

[0028] If a check of the monthly network blob (using the blob index file and bitset) indicates that the desired IPv4 address is in the monthly netflow blob, the search process may perform a second check to see if address 1.2.3.4 exists in "year=2023 / month=01 / day=01 / ipv4.bitset.gz", which is the network blob covering netflow for January 1, 2023 (a period narrower than the entire month of January). If IPv4 address 1.2.3.4 is found in that netflow blob, the next check is performed. If IPv4 address 1.2.3.4 is not found in the daily netflow blob (based on the blob index file and bitset), the search must perform checks for January 2-25, 2023, as these days fall within the requested period. Similar to the check for day 1, the checks for days 2-25 are performed in a similar manner using the netflow blob index file and bitset for those days.

[0029] If the address is found by checking for at least one day in January (1st to 25th), a third check determines whether address 1.2.3.4 exists in "year=2023 / month=01 / day=01 / hour=00 / ipv4.bitset.gz", which is a network blob covering netflow during 00:00 on January 1, 2023 (a period narrower than the entire month of January or a single day in January). If IPv4 address 1.2.3.4 is found within that netflow blob, the following check is performed: If IPv4 address 1.2.3.4 is not found within the hourly netflow blob for 00:00 (based on the blob index file and bitset), the search must perform a check for each of the other hours (1 through 23, representing 1:00 or 11:00) during the same day in January 2023 in which the requested address was found. Similar to the check at midnight, checks from 1 AM to 11 PM are performed in a similar manner using the netflow blob index files and bitsets for those times on a given day.

[0030] If the address is found by checking for at least one hour on any day in January, a fourth check determines whether the address 1.2.3.4 exists in the network blob "year=2023 / month=01 / day=01 / hour=00 / minute=00 / nfacctd n8429 netflow 20220101 0000.ipv4.gz", which covers netflow for the period of 00:00 on January 1, 2023 (or a period shorter than the entire month of January, a day in January, or 00:00 on January 1). If IPv4 address 1.2.3.4 is found in that minute-by-minute netflow blob, the network blob is added to the queue. If IPv4 address 1.2.3.4 is not found in the minute-by-minute netflow blob (based on the blob index file and bitset), there are no more network blobs to process in this branch of the tree, but more blobs may be found in other "branches". Therefore, the search process traverses a tree of bitsets and index files (see Figure 12 for a simple example) to search historical net flows in a time- and storage-efficient manner.

[0031] When using a search, the "best" scenario is that no IPv4 address matches are found within a time range using a monthly bitset. That search would complete in less than a minute. Searching each netflow file for that month would take considerable time, expense, and processing power. The "worst" scenario is that all index files find IPv4 address matches within a time range, which is equivalent to searching without a bitset or index files. However, if this were always true (which is not), the advantage of the searcher would still be finding the right blob to process for a given time range.

[0032] For more complex searches (longer-term searches using more bitset and index files), the number of checks performed during the search increases to narrow down / filter the search from checking the longest possible index and bitset files where the desired IPv4 address might exist, through one or more increasingly granular checks, to checking the shortest possible index and bitset files where the IPv4 address might exist.

[0033] Returning to Figure 10, the method places each NetFlow blob containing the requested address into a queue with a “poison pill” at the end to mark that the search for that NetFlow blob is complete (114). The method then performs a history NetFlow determination (in one implementation, for example, using the history processor 52 in Figure 5), where the history processing unit processes the next blob in the queue by consuming NetFlow blob names from the queue, retrieving a specific NetFlow blob, reading the NetFlow blob, and finding the flow in which the target IPv4 resides (116). The history processing unit process can start as soon as the first NetFlow blob name is queued. The process of searching for NetFlow blobs typically completes much faster than the history NetFlow processing, which requires a significant portion of the processing power (depending on the number of blob files found). The method determines whether there are further blobs in the queue (118), and if there are more NetFlow blobs, it can loop back to the processing process 116. If there are no more Netflow blobs to process, the method generates a summary of Netflow blobs, including IPv4 addresses, and returns these results to the user (120).

[0034] The start of the search (by searcher 46 in the implementation shown in Figure 5) differs depending on the time range of the requested search for the IPv4 address. Specifically, if the time range is one month or longer, the search may start with a "monthly bit set" search, as done in the simple example above. However, if the requested time range is only a few hours, the search starts with a "hourly bit set" search, as the month and day containing the IPv4 address are not the subject of the search and are already known by the requester of the search. Technical benefits provided by the disclosed systems and methods

[0035] Using known techniques, netflows can be stored in databases such as common SQL or NoSQL databases. These known techniques mean that the method for finding historical netflows is not a "search" (a novel combination of blob index files and bitsets), but rather an immediate "lookup" of the flow (e.g., giving all flows within a given time range), which is storage-intensive and processor-computation-intensive. An alternative known technique is to store only the lookup details regarding the IP address and the netflow blob in which the IP address resides. For example, for the amount of netflow volume contained in historical netflows, a known technique would likely only store IPv4 address->[blobs] information to find only the target blob given the search parameters (IP + time range). While the disclosed systems and methods may still require the same amount of processing, the cost of storing and retrieving "index" information is a significant differentiating factor.

[0036] Databases have costs associated with their uptime (excluding storage costs), while cloud blob storage only incurs storage costs. Disclosed bitset and blob index data can be stored in different tiers (hot, cold, and archive) at substantially different costs depending on the cloud storage tier, although databases (used in the most well-known techniques) are typically always "hot" on disks such as SSDs. Furthermore, while disclosed bitset / index data can be stored compressed, it cannot be retrieved using known techniques.

[0037] The greatest advantages are storage cost and storage capacity. Typically, there are multiple (e.g., 5) NetFlow blobs per minute, with 6 to 8 million flows per NetFlow blob. A typical NetFlow blob can have approximately 3 million unique IP addresses. Storing this information in a database would require approximately 3 million new entries / associations (one per unique IP address) pointing to this single blob, or approximately 15 million new entries for 5 blobs per minute. Storing this for several days, let alone several months, would rapidly become excessively large in scale. Therefore, the disclosed technique provides a technical solution that enables efficient storage and retrieval of historical NetFlows, which would be impossible and / or expensive using known techniques.

[0038] The foregoing description has been made for illustrative purposes with reference to specific embodiments. However, the above exemplary description is not intended to be exhaustive or to limit the disclosure to the exact form disclosed. Many modifications and variations are possible in light of the above teachings. The embodiments have been selected and described to best illustrate the principles of the disclosure and their practical application, thereby enabling those skilled in the art to best utilize the disclosure and its various embodiments with various modifications suitable for the specific use intended.

[0039] The systems and methods disclosed herein may be implemented via one or more components, systems, servers, appliances, or other subordinate components, or distributed among such components. When implemented as a system, such a system may include and / or be accompanied by components such as software modules, general-purpose CPUs, and RAM, among others, as found in general-purpose computers. In the implementation of this innovation existing on a server, such a server may include, or be accompanied by, components such as CPUs and RAM, as found in general-purpose computers.

[0040] In addition, the systems and methods of this specification may be achieved through implementations using heterogeneous or entirely different software, hardware, and / or firmware components beyond those described above. With respect to such other components (e.g., software, processing components, etc.) and / or computer-readable media associated with or embodying the present invention, for example, aspects of the innovations of this specification may be implemented in conjunction with a number of general-purpose or dedicated computing systems or configurations. Various exemplary computing systems, environments, and / or configurations suitable for use with the innovations of this specification may include, but are not limited to, personal computers, servers or server computing devices such as routing / connectivity components, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, consumer electronics devices, network PCs, other existing computer platforms, and distributed computing environments including one or more of the above systems or devices, and may include software or other components embodied in or on them.

[0041] In some examples, the aspects of the system and method are achieved, or can be achieved, via logic and / or logic instructions, including program modules, which are executed in relation to such components or circuits. Generally, program modules may include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific instructions as herein. The present invention may also be practiced in the context of distributed software, computers, or circuit configurations, where circuits are connected via communication buses, circuits, or links. In a distributed configuration, control / instructions may originate from both local and remote computer storage media, including memory storage devices.

[0042] The software, circuits, and components described herein also include and / or may utilize one or more types of computer-readable media. Computer-readable media may be any available media residing on, associated with, or thereby accessible on such circuits and / or computing components. Computer-readable media may include, but are not limited to, computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any way or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other media that may be used to store desired information and are accessible by computing components. Communication media may include computer-readable instructions, data structures, program modules, and / or other components. Furthermore, communication media may include wired media such as wired networks or direct wired connections, but any such type of media as defined herein does not include temporary media. Any combination of the above is also included within the scope of computer-readable media.

[0043] In this description, terms such as components, modules, and devices may refer to any type of logic or functional software element, circuit, block, and / or process that can be implemented in various ways. For example, the functions of various circuits and / or blocks may be combined with any number of other modules. Each module may be implemented as a software program stored in tangible memory (e.g., random access memory, read-only memory, CD-ROM memory, hard disk drive, etc.) and read by a central processing unit to implement the functions of the innovations described herein. Alternatively, a module may include programming instructions transmitted via a transmit carrier to a general-purpose computer or processing / graphics hardware. A module may also be implemented as a hardware logic circuit that implements the functions encompassed by the innovations described herein. Finally, a module may be implemented using dedicated instructions (SIMD instructions), field-programmable logic arrays, or any combination thereof that provides a desired level of performance and cost.

[0044] As disclosed herein, features consistent with this disclosure may be implemented through computer hardware, software, and / or firmware. For example, the systems and methods disclosed herein may be embodied in various forms, including data processors such as computers, which may also include databases, digital electronic circuits, firmware, software, or combinations thereof. Furthermore, while some of the disclosed implementations describe specific hardware components, the systems and methods consistent with the innovations herein may be implemented in any combination of hardware, software, and / or firmware. In addition, the features and other aspects and principles of the innovations herein may be implemented in various environments. Such environments and associated applications may include general-purpose computers or computing platforms that are specifically constructed to perform various routines, processes, and / or operations according to the present invention, or that are selectively activated or reconfigured by code to provide the necessary functionality. The processes disclosed herein are essentially independent of any particular computer, network, architecture, environment, or other device and may be implemented by a suitable combination of hardware, software, and / or firmware. For example, various general-purpose machines may be used with programs written in accordance with the teachings of the present invention, or it may be more convenient to construct dedicated devices or systems to perform the necessary methods and techniques.

[0045] The embodiments of the methods and systems described herein, including logic, may also be implemented as programmed functions in any of the various circuits, including programmable logic devices ("PLDs") such as field-programmable gate arrays ("FPGAs"), programmable array logic ("PAL") devices, electrically programmable logic and memory devices, as well as standard cell-based devices, and application-specific integrated circuits. Several other possibilities for implementing the embodiments include memory devices, microcontrollers with memory (such as EEPROMs), embedded microprocessors, firmware, and software. Furthermore, the embodiments may be embodied in microprocessors having software-based circuit emulation, discrete logic (sequential and combinatorial logic), custom devices, fuzzy (neural) logic, quantum devices, and hybrids of any of the above device types. The underlying device technologies can be provided by various component types, such as metal-oxide-semiconductor field-effect transistor (MOSFET) technologies like complementary metal-oxide-semiconductor ("CMOS"), bipolar technologies like emitter-coupled logic ("ECL"), polymer technologies (e.g., silicon-conjugated polymers and metal-conjugated polymer metal structures), and hybrid analog and digital designs.

[0046] It should also be noted that the various logics and / or functions disclosed herein may be made effective with respect to their behavior, register transfers, logic components, and / or other characteristics, using any number of combinations of hardware and firmware, and / or as data and / or instructions embodied in various machine-readable or computer-readable media. Computer-readable media in which such formatted data and / or instructions may be embodied include, but are not limited to, various forms of non-volatile storage media (e.g., optical, magnetic, or semiconductor storage media), but not temporary media. Throughout this specification, unless the context clearly indicates otherwise, words such as “comprise” and “comprising” should be interpreted in a comprehensive sense, as opposed to an exclusive or exhaustive sense; that is, “includes, but not limited to.” Where a word is used singular or plural, it also includes the plural or singular forms, respectively. In addition, the words “herein,” “hereunder,” “above,” “below,” and similar words refer to the entire application and not to any particular part thereof. Where the word “or” is used in reference to a list of two or more items, the word encompasses all of the following interpretations of the word: any of the items in the list, all of the items in the list, and any combination of the items in the list.

[0047] While specific currently preferred implementations of the present invention have been described herein, it will be apparent to those skilled in the art that various modifications and alterations of the implementations illustrated and described herein can be made without departing from the spirit and scope of the invention. Therefore, the present invention is intended to be limited only to the extent required by applicable legal regulations.

[0048] While the above has been described with reference to specific embodiments of the present disclosure, those skilled in the art will understand that modifications of these embodiments can be made without departing from the principles and spirit of the present disclosure, and that the scope of the present disclosure is defined by the appended claims.

Claims

1. A method for finding historical network flows with a specific IP address, A step of a historical netflow computer system to retrieve a plurality of historical netflow blobs, each containing a plurality of netflows, wherein each netflow contains data relating to data traffic between the Internet Protocol (IP) address of a source host and the said IP address of a destination host over a past period, and each of the historical netflow blobs contains the plurality of netflows over different past periods. The steps include: receiving a request from the historical netflow computer system to find a requested IP address during a searchable period, wherein the requested IP address can be found in at least two of the historical netflow blobs having netflows over different past periods; A step of generating blob index files for at least two of the historical netflow blobs using the historical netflow computer system, wherein each blob index file includes each of the unique IP addresses in the historical netflow blobs for different past periods. The steps include: generating a bitset for the longest-period historical netflow blob among the at least two historical netflow blobs using the historical netflow computer system, wherein the bitset includes a set of bits in a word representing each of the unique IP addresses in the historical netflow blob having the longest period; The steps include: using the historical netflow computer system to perform a search using the bit set and the at least two blob index files to find a historical netflow having the requested IP address in the netflows during the searchable period; The steps include displaying the found historical network flow having the requested IP address on the display of the computing device, A method that includes this.

2. The method according to claim 1, further comprising the step of forming a search tree having a root which is the generated bitset and a plurality of leaves, each leaf being a blob index file of the generated bitset having different time periods, and the step of performing the search further comprising checking in each blob index file whether the requested IP address appears in the blob index file.

3. The method of claim 2, wherein the step of performing the search further includes the steps of identifying a specific historical netflow blob to be processed having the requested IP address, using the blob index file corresponding to the specific historical netflow blob and the generated bitset corresponding to the specific historical netflow blob, and searching the identified specific historical netflow blob to find a historical netflow having the requested IP address.

4. The method according to claim 3, further comprising the steps of queuing the identified specific historical netflow blobs and finding the requested IP address in each identified specific historical netflow blob in the queue.

5. The method according to claim 1, wherein the period is one of one hour, one month, and one day.

6. The method according to claim 1, wherein the generated bit set is a 64-bit word.

7. The method according to claim 1, further comprising the step of generating a plurality of bit sets, each generated bit set having a period that varies between a shortest time interval, a short time interval, and a longest time interval, and the step of performing the search of the historical netflow further comprising the step of performing the search using the longest time interval, the step of performing the search using the short time interval if the requested IP address is included in the bit set of the longest time interval, and the step of performing the search using the bit set of the shortest time interval if the requested IP address is included in the bit set of the short time interval to find the historical netflow having the requested IP address.

8. The method according to claim 1, wherein the requested IP address is an IPv4 IP address.

9. A system for finding historical network flows with a specific IP address, A computer system for discovering historical network flows having a processor and memory, A computing device that can be connected to the aforementioned historical network flow discovery computer system and Equipped with, The processor of the aforementioned historical network flow discovery computer system is Retrieving multiple historical netflow blobs, each containing multiple netflows, wherein each netflow contains data relating to data traffic between the Internet Protocol (IP) address of a source host and the said IP address of a destination host during a past period, and each of the historical netflow blobs contains the said multiple netflows over different past periods. Receiving a request to find a requested IP address during a searchable period, wherein the requested IP address can be found in at least two of the historical netflow blobs having netflows over different past periods. To generate blob index files for at least two of the aforementioned historical netflow blobs, each blob index file containing each of the unique IP addresses within the historical netflow blobs for different past periods, To generate a bitset for the longest-period historical netflow blob among the at least two historical netflow blobs, wherein the bitset includes a set of bits in a word representing each of the unique IP addresses within the historical netflow blob having the longest period. Using the bitset and the at least two blob index files, perform a search to find historical netflows having the requested IP address in the netflows during the searchable period. It is configured to do the following: The display of the computing device displays the found historical network flow having the requested IP address. system.

10. The system according to claim 9, wherein the processor is further configured to form a search tree having a root which is the generated bitset and a plurality of leaves, where each leaf is a blob index file of the generated bitset having different time periods, and to check in each blob index file whether the requested IP address appears in the blob index file.

11. The system according to claim 10, wherein the processor is further configured to identify a specific historical netflow blob to be processed having the requested IP address, using the blob index file corresponding to the specific historical netflow blob and the generated bit set corresponding to the specific historical netflow blob, and to search the identified specific historical netflow blob to find a historical netflow having the requested IP address.

12. The system according to claim 11, wherein the processor is further configured to queue the identified specific historical netflow blobs and to find the requested IP address in each identified specific historical netflow blob in the queue.

13. The system according to claim 9, wherein the period is one of one hour, one month, and one day.

14. The system according to claim 9, wherein the generated bit set is a 64-bit word.

15. The system according to claim 9, wherein the processor is further configured to generate a plurality of bit sets, each generated bit set having a period of time that varies between a shortest time interval, a short time interval, and a longest time interval; to perform the search for the requested IP address in the historical netflow blob using the longest time interval; if the requested IP address is included in the bit set of the longest time interval, to perform the search for the requested IP address in the historical netflow blob using the short time interval; and if the requested IP address is included in the bit set of the short time interval, to perform the search for the requested IP address in the historical netflow blob using the bit set of the shortest time interval to find the historical netflow having the requested IP address.

16. The system according to claim 9, wherein the requested IP address is an IPv4 IP address.

Images