Apparatus, method, and system for generating a highly extensible and efficient composite recording index

Abstract

The present disclosure describes a method and system for processing and indexing data from a continuously updated distributed database. The method and system use a partitioned database architecture to return results to a query at regular intervals and reduce the memory size of indexed records. A database partition includes a plurality of data records corresponding to an index group of a single row of a rowKey table. An index group includes data records that are indexed at a predetermined indexing interval. The partitioned database architecture enables querying of record fields from the index group.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims priority to U.S. Provisional Patent Application No. PCT / US2023 / 021736, filed on June 3, 2022, entitled DEVICES, METHODS, AND SYSTEMS FOR GENERATING A HIGHLY - SCALABLE, EFFICIENT COMPOSITE RECORD INDEX, the disclosure of which is hereby incorporated by reference in its entirety.

[0002] This disclosure generally relates to network security, and more specifically, to improved systems and methods for processing and indexing files from a continuously updated database. In conventional indexing schemes, the system is overwhelmed by the number of records and can negatively impact read speed, memory table size, and query time. Some of these problems can be overcome by a significant increase in processing resources and memory space, but for this petabyte of data, this solution can be prohibitively expensive.

Summary of the Invention

[0003] The following summary is provided to facilitate an understanding of some of the innovative features specific to the aspects disclosed herein and is not intended to be a complete description. A complete understanding of the various aspects can be obtained by taking the entire specification, claims, and abstract.

[0004] In one non - limiting aspect, the present disclosure describes a method for indexing records in an extensible split data table that enables certain time queries. The method includes receiving, by a processor, a flow of pDNS records from one or more data sources; aggregating, by the processor, the flow of pDNS records into a distributed database; executing, by the processor, a first structured streaming job on the distributed database, where the first structured streaming job processes a first micro - batch of pDNS records, reformats the records into an extensible format, groups the records according to the first byte of the requested IP address, writes the pDNS records to a first set of group files among a plurality of group files, and a plurality of subsequent structured streaming jobs are continuously triggered and started within a predetermined period from a previous structured streaming job; executing, by the processor, an indexing job on a plurality of group files, where the indexing job reads a first set of group files among the plurality of group files, generates an index for all queryable fields of the pDNS records, and new indexing jobs are executed according to a predetermined indexing job interval; writing, by the processor, an index group to the first row of a rowKey table, where the index group is grouped according to the event timestamp of the indexed pDNS records indexed during a predetermined indexing job interval; querying, by the processor, the rowKey table by a set of queryable parameters of the pDNS records, where the query result is returned in a certain time, and may include.

[0005] In another non-limiting aspect, the present disclosure describes a system for indexing records returned by a query to an extensible database over a period of time.The system includes at least one memory communicatively coupled to at least one processor, at least one memory communicatively coupled to at least one processor, an input / output interface configured to access data from one or more external sources, wherein each of the plurality of external sources is communicatively coupled to the at least one processor, an input / output interface, and a database present in at least one memory and configured to store data. The at least one memory is configured to receive, by the at least one processor, a flow of pDNS records from one or more data sources, aggregate the flow of pDNS records into a distributed database, and execute a first structured streaming job on the distributed database, wherein the first structured streaming job processes a first microbatch of pDNS records, reformats the records into an extensible format, groups the records according to the first byte of the requested IP address, writes the pDNS records to a first set of group files among a plurality of group files, and further, a plurality of subsequent structured streaming jobs are continuously triggered and started within a predetermined period from a previous structured streaming job; execute an indexing job on the plurality of group files, wherein the indexing job reads a first set of group files among the plurality of group files, generates an index for all queryable fields of the pDNS records, and further, a new indexing job is executed according to a predetermined indexing job interval; write the index group to the first row of the rowKey table, wherein the index group is grouped according to the event timestamp of the indexed pDNS records indexed during a predetermined indexing job interval; query the rowKey table by a set of queryable parameters of the pDNS records by the processor, wherein the query result is returned within a certain time.

Brief Description of the Drawings

[0006]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

[0007] Corresponding reference numerals indicate corresponding parts throughout the several views. The embodiments described herein illustrate various aspects of the present invention in one form, and such embodiments should not be construed as limiting the scope of the present invention in any way.

Modes for Carrying Out the Invention

[0008] The applicant of the present application owns the following U.S. provisional patent applications, the entire disclosures of each of which are incorporated herein by reference in their entirety. - International Patent Application No. PCT / US2022 / 072739, filed on June 3, 2022, entitled DEVICES, SYSTEMS, AND METHODS FOR ENHANCING SECURITY INFORMATION & EVENT MANAGEMENT UPDATES FOR MULTIPLE TENANTS BASED ON CORRELATED, AND SYNERGISTIC DEPLOYMENT NEEDS - International Patent Application No. PCT / US2022 / 072743, filed on June 3, 2022, entitled DEVICES, SYSTEMS, AND METHODS FOR STANDARDIZING & STREAMLINING THE DEPLOYMENT OF SECURITY INFORMATION & EVENT MANAGEMENT ARTIFACTS FOR MULTIPLE TENANTS - International Patent Application No. PCT / US2022 / 082167, filed on December 21, 2022, entitled DEVICES, SYSTEMS, AND METHODS FOR PROVISIONING AND UPDATING SECURITY INFORMATION & EVENT MANAGEMENT ARTIFACTS FOR MULTIPLE TENANTS - International Patent Application No. PCT / US2022 / 082173, filed on December 21, 2022, entitled DEVICES, SYSTEMS, AND METHODS FOR STREAMLINING AND STANDARDIZING THE INGEST OF SECURITY DATA ACROSS MULTIPLE TENANTS - International Patent Application No. PCT / US2023 / 061069, filed on January 23, 2023, entitled DEVICES, SYSTEMS, AND METHODS FOR REMOTELY MANAGING ANOTHER ORGANIZATION’S SECURITY ORCHESTRATION, AUTOMATION, AND RESPONSE - International Patent Application No. PCT / US2023 / 0612894, filed on February 20, 2023, entitled DEVICES, SYSTEMS, AND METHODS FOR IDENTIFYING CYBER ASSETS AND GENERATING CYBER RISK MITIGATION ACTION BASED ON DOMAIN REDIRECTS - International Patent Application No. PCT / US2023 / 021736, filed on May 10, 2023, entitled DEVICES, SYSTEMS, AND METHODS FOR SUMMARIZING ANALYTIC OBSERVATIONS - International Patent Application No. PCT / US2023 / 022858, filed on May 19, 2023, entitled DEVICES, SYSTEMS, AND METHODS FOR INGESTING & ENRICHING SECURITY INFORMATION TO AUTONOMOUSLY SECURE A PLURALITY OF TENANT NETWORKS, and - US Provisional Patent Application No. PCT / US2023 / 022535, filed on May 17, 2023, entitled DEVICES, SYSTEMS, AND METHODS FOR IDENTIFYING CYBER ASSETS AND GENERATING CYBER RISK MITIGATION ACTIONS BASED ON A DEMOCRATIC MATCHING ALGORITHM

[0009] Numerous specific details are described in this disclosure and are described to provide a complete understanding of the overall structure, function, manufacture, and use of the aspects illustrated in the accompanying drawings. Well-known operations, components, and elements are not described in detail so as not to obscure the aspects described herein. The reader will understand that the aspects described and illustrated herein are non-limiting aspects. Thus, it will be understood that the specific structural and functional details disclosed herein may be representative and exemplary. Modifications and variations can be made without departing from the scope of the claims.

[0010] Before detailing the various aspects and methods of the systems disclosed herein, it should be noted that the exemplary aspects are not limited to the application or use in the details disclosed in the accompanying drawings and description. Of course, the exemplary aspects may be implemented or incorporated in other aspects, variations, and modifications and may be practiced or carried out in various ways. Further, unless otherwise indicated, the terms and expressions used herein are chosen for the purpose of describing the exemplary aspects for the convenience of the reader and are not intended for purposes of limitation. For example, any reference herein to a particular manufacturer, software suite, application, or development platform is understood to be merely intended to illustrate some of the many aspects of this disclosure. This includes any reference to trademarks. Thus, it should be understood that the devices, systems, and methods disclosed herein can be implemented to enhance any software update according to any purpose of use and / or user preference.

[0011] As used herein, the term "server" refers to, or may include, one or more computing devices that are operated or facilitated by communication and processing for multiple parties in a network environment, such as the Internet or any public or private network. As used herein, a reference to a "server" or "processor" may refer to a previously enumerated server and / or processor that performs a previous step or function, a different server, and / or processor, and / or a combination of servers and / or processors that are listed as performing the combination.

[0012] As used herein, the term "constant" may refer to one or more security information and event management (SIEM) functions that do not change during the issuance of an alert. For example, constants can include, in particular, Azure Sentinel Log Analytics functions. According to some non-limiting aspects, constants can be specifically configured according to the preferences and / or requirements of individual clients. For example, as described herein, alert rules may be the same for all client deployments. However, the use of the devices, systems, and methods disclosed herein can "fine-tune" the alert management method for each specific client using client-specific constants. In other words, each constant may include a whitelist of specific protocols, accounts, etc., that the alert rule manages separately (e.g., skips) those constants.

[0013] As used herein, the term "network" refers to, or can include, an entire enterprise information technology (IT) system introduced by a tenant. For example, a network can include a group of two or more nodes (e.g., assets) connected by any physical and / or wireless connection and configured to communicate and share information with one or more other nodes. However, the term "network" is not limited to any particular node or any particular means of connecting those nodes. A network can connect to Ethernet, an intranet, and / or an extranet, and can be configured to communicate with each other via ad-hoc connections (e.g., Bluetooth (R), Near Field Communication (NFC), etc.), local area connections ("LAN"), wireless local area networks ("WLAN"), and / or virtual private networks ("VPN") regardless of the physical location of each device, and can include any combination of assets (e.g., devices, servers, desktop computers, laptop computers, personal digital assistants, mobile phones, wearables, smart home appliances, etc.). A network can further include any tools, applications, and / or services deployed by a device or otherwise utilized by enterprise IT systems such as firewalls, email clients, document management systems, office systems, etc. In some non-limiting aspects, "network" can include third-party devices, applications, and / or services that are owned and controlled by a third party but for which the tenant has been approved to access the enterprise IT system.

[0014] As used herein, the term "platform" may include software architecture, hardware architecture, and / or combinations thereof. A platform may include any of a stand-alone software product, a network architecture, and / or a software product configured to be integrated within a software architecture and / or a hardware architecture as necessary to provide its technical benefits to the software product. For example, a platform may include any combination of a chipset, a processor, a logic-based device, memory, storage, a graphical user interface, a graphics subsystem, an application, and / or a communication module (e.g., a transceiver). In other words, a platform can provide the resources necessary to enable the technical benefits provided by software. According to some non-limiting aspects, the technical benefits provided by software are provided to other software employed by the physical resources of the ecosystem, or the physical resources within the ecosystem (e.g., APIs, services, etc.). According to other non-limiting aspects, a platform may include a framework for several software applications intended and designed to function together.

[0015] As used herein, the term "security monitoring platform" refers to, or may include, software configured to aggregate and analyze activities from many different resources across an information technology infrastructure. For example, a security monitoring platform may include a SIEM platform, and / or other types of platforms used for data monitoring and / or analysis (e.g., Splunk Enterprise Security, Microsoft Sentinel, Datadog Security Monitoring, ELK, etc.). The various aspects of the devices, systems, and methods disclosed herein related to SIEM may similarly be applicable to any type of security monitoring platform.

[0016] Examples of commonly implemented SIEMs include Azure Sentinel and Splunk Cloud, Devo, LogRhythm, IBM’s QRadar, Securonix, McAfee Enterprise Security Manager, LogPoint, Elastic Stack, ArcSight Enterprise Security Manager, InsightIDR, and others. Introducing Azure Sentinel as a cloud-based tool is widely accepted among managed security service providers (“MSSPs”), and for this reason, Azure Sentinel is described as a non-limiting example. However, of course, other SIEMs are contemplated by this disclosure. As with most SIEMs, introducing Azure Sentinel requires advanced skills and is a time-consuming task that is prone to errors. Each organization that requires a security solution has specific needs regarding ingestion log sources, detection / alert rules, automation of responses, monitoring such as reports, and alerts. Microsoft (MSFT) is often used by MSSPs to manage multiple clients, but the complexity of the initial configuration, introduction, and ongoing maintenance of artifacts (e.g., resource groups, log analytics workspaces, alert rules, workbooks, playbooks, etc.) has increased significantly. This can result in high costs for both MSSPs, which must employ more expensive specialists, and clients, who often bear at least a portion of the increasing costs. However, in many cases, there is overlap among some of the implementation needs of various clients.

[0017] For example, many organizations may require similar firewall monitoring solutions. In such cases, asset reuse, and reintroduction (and updates) can lead to significant cost savings and operational simplification. Unfortunately, known SIEM tools are technically unable to utilize such synergistic effects. Therefore, from initial provisioning to incident response automation, MSSPs have limited opportunities for reuse to capture efficiency across multiple clients. Thus, improved devices, systems, and implementation methods, as well as the issuance of SIEM client updates, are needed. Such enhancements can improve the technical performance and cost efficiency of SIEM, including the introduction of detection rules, visualization, investigation workbooks, and continuous maintenance.

[0018] It can be beneficial to aggregate data (e.g., log data, event data, threat intelligence data, etc.) from multiple platforms and provide that data to a system such as a SIEM platform to process and capture malicious behavior or draw other meaningful conclusions. For example, it can be beneficial to collect records from network devices, servers, domain controllers, and memory. The collected records can be stored, normalized, aggregated, and analyzed to discover trends, detect threats, and enable an organization to investigate alerts. Known SIEM tools (also referred to herein as SIEM detection engines) can provide some functionality, including monitoring of events, collection of data, and issuance of security alerts across the network, but such tools are typically customized to the implementing organization, i.e., more specifically, to a particular network architecture that can often be complex.

[0019] Specifically, because it is related to important data aggregation necessary to effectively protect the network, traditional tools are insufficient to efficiently monitor and aggregate data on a large scale. For example, to monitor and aggregate data across multiple tenant (or client) networks, an MSSP needs to receive approximately 2 million records (e.g., pDNS records) per second, and traditional tools need to be able to efficiently store, retrieve, and analyze related records for specifically requesting ip addresses, answering ip addresses, queried domain names (e.g., Qnames), and subdomains queried over a time range of several months in a cost-effective manner. Therefore, traditional tools cannot monitor and aggregate the records necessary to identify malicious activities in the footprint of interest, and thus cannot effectively identify important security metrics, especially including traffic of interest for security appliances, software vendors, and / or specific use cases.

[0020] In other words, traditional tools cannot technically aggregate and / or manage high-throughput records because of the nature and volume of those records, which require a "write speed" that exceeds their rated performance. Similarly, traditional tools are technically unable to maintain an efficient query index, and the resulting amount of data is very large for the user to effectively and / or efficiently search for records of interest for a specific tenant network, especially when managing the security of multiple tenant networks.

[0021] Accordingly, there is a need for an apparatus, system, and method for generating and utilizing a highly extensible composite record index to enhance network security. Such apparatus, systems, and methods have a number of practical applications and provide a number of technical improvements over known tools, including efficient querying and processing of records (e.g., pDNS data) for a particular protocol or domain, including records of volumes of billions of records in just a few seconds while maintaining low cost and high write throughput. Thus, such devices, systems, and methods can be used to repeatedly scale cloud-based data aggregation consistently without degrading the quality of search results.

[0022] The present disclosure presents such devices, systems, and methods, all of which provide a number of technical benefits and enable users to introduce cloud-based SIEM implementations, such as implementations of Azure Sentinel, on a large scale, repeatedly, and consistently, in one non-limiting aspect. For example, the devices, systems, and methods disclosed herein can provide (1) a record (e.g., pDNS) file partitioning scheme, (2) a streaming clustering algorithm for rapidly accumulating and emitting files using this scheme, (3) an efficient query index for those files implemented in Google Bigtable, and (4) an efficient algorithm for updating the query index as partitioned files are written. The resulting composite index can include split files and separate indexes, enabling SIEMs or other users to write two million records per second with their associated index values and query the resulting data for specific assets of interest within seconds among billions of written pDNS records.

[0023] A composite index can include a streaming distributed database that accumulates records from various sources. For example, a structured streaming job (e.g., Apache Spark) can be executed on a cloud-based platform (e.g., Google Cloud Dataproc) to continuously read and process a record stream from the composite index in small batches called micro-batches. Records can be grouped into each micro-batch by the first byte of the requested protocol, which improves the performance later in the pipeline. Subsequently, the records can be written as files (e.g., Apache Avro) on a cloud-based storage platform (such as Google Cloud Storage). According to the present disclosure, the grouped, written, and stored records can function as the primary data storage layer of the pDNS database and can support a very high write throughput (e.g., 6 million records per second). Conventional MSSP devices, systems, and methods are not only technically impossible to automate, but it is also not very realistic, if not impossible, for an MSSP to manually continuously aggregate and manage millions of tenant networks in real time.

[0024] FIG. 1 shows a system 1000 configured for security information and event management (SIEM) implementation across multiple tenants, according to at least one non-limiting aspect of the present disclosure. System 1000 can include a SIEM provider server 1002 comprising a memory 1004 and a processor 1006. In various aspects, SIEM provider server 1002 can comprise computer systems 5000 and their various components (e.g., processor 1006 may be similar to processor 5004, and memory 1004 may be similar to main memory 5006), which is described further with reference to FIG. 5.

[0025] In various aspects, memory 1004 may be configured to store instructions that, when executed by processor 1006, generate requests for data from multiple data sources 1010, 1020. Provider server 1002 receives petabytes of raw data from clients or third parties. The data may include global Internet traffic, and the network security computing system may be interested only in the fact of the overall data set. Upon receiving the raw data, the network provider server or computing system aggregates, processes, and indexes the data to create a queryable index table that indicates the original records. The index table may be stored locally on provider server 1002 or on backend server 1030. Further, provider server 1002 may operate as a front end and obtain query results from backend server 1030.

[0026] As the data set continues to grow, querying for desired fields becomes difficult due to the size of the data set. The database architecture faces a conflict of interest between query response time and the cost of computing resources. Due to the cost of computing resources for processing and indexing the data and storage space of the index table, it can be prohibitively expensive to return query results within a reasonable time. This disclosure describes a data indexing scheme for continuously updated data sets that include petabytes of data and require daily updates of scalability. The data indexing scheme provides a database architecture that partitions the index table and the original data to return query results to a query in a fixed time.

[0027] Compared with the disclosed data indexing scheme, in order to directly write individual records or cyber data into a distributed key-value database, such as Google BigTable, it is necessary to replicate data between keys of different fields or write some kind of back pointer. This data replication requires significantly more table nodes (up to four times as many) to maintain the write speed in a security operations center (SOC) environment.

[0028] Figure 2 shows a high-level flow diagram of the data indexing scheme. The system receives data 2002 from one or more data sources and aggregates 2004 the data into a distributed database. The system reformats 2006 the data into a common extensible format and writes the data to a file within a distributed blob storage system (e.g., Google Cloud Storage, Amazon S3, etc.). The system reads records from the file and generates an index based on the strings of the fields of the records. The system writes 2010 the index to a queryable key-value database. The key-value database receives 2012 a query from a front-end computing system and retrieves data based on the index indicating the location of the original record.

[0029] In various aspects, the system obtains cyber event data from one or more third - party sources and aggregates the cyber event data into a single dataset. The data can be received or obtained daily according to the type and change rate of the cyber event data. In one aspect, the system correlates the cyber event data to dynamic assets and provides an accurate assessment of the cyber event data to the third - party source as soon as possible. In one aspect, the cyber event data may include asset behavior at a particular point in time that correlates with malicious behavior or exposes the security of the system to risk. In another aspect, the cyber event data may include asset information such as software version, firmware version, update history, etc. Due to the dynamic nature of the cyber event data, the data can become stale and out - of - date in a short period (e.g., a few days or weeks). Therefore, the dataset needs to be continuously updated so that the system can maintain a continuity chain for dynamic assets.

[0030] When the system creates a query - able index table, the dataset can be queried based on a timestamp or an asset of interest. In one aspect, an entity is defined by its footprint, which includes a plurality of assets owned and controlled by the entity. All information related to the entity's footprint can be queried according to an IP address or domain and obtained by querying the entity's assets for a certain period of time. The main advantage of the data indexing scheme is that there is little change in the query response time as the amount of data or records in the dataset increases.

[0031] In this aspect, the network security computing system ingests data from multiple data sources and aggregates the data into a single distributed dataset. Next, the data is reformatted into its original format such as JSON of CSV text and translated into an extensible format such as an extensible JSON file like Apache Avro or Apache Parquet. The file is written to a distributed blob storage system (in this case Google Cloud Storage, with Amazon S3 being another example).

[0032] DNS records include multiple Internet Protocol address (IP address) fields (such as packet source IP, packet destination IP, and DNS response IP, etc.) and multiple Internet domain fields (such as query name or "qname", canonical name of the record or "cname", etc.).

[0033] For each data source, a mapping is created that explicitly specifies the relationship between the fields of the reformatted input and the original data. For example, a data source that includes a banner scan of IP addresses may have columns called "scanned_ip" which indicates the scanned IP address, "source_ip" which indicates the IP address that performed the scan, and "scan_time" which indicates the time at which the scan occurred. In this example, the mapping includes scanned_ip and source_ip as a list of IP addresses associated with the scan server event, and scan_time as a single timestamp. The schema is required to map at least one IP address or domain from the original data and exactly one timestamp from the original data.

[0034] The reformatting scheme can be defined as an abstract Java class, explicitly specifying the mapping of common extensible fields. The Java class can be configured to pull data from the original data into the corresponding fields. Using Java object classes provides access to higher-level class types such as open-source IP address classes. Additionally, Java objects allow for greater customization for serialization and deserialization of data for different contexts. For example, when writing to a backend database table for translation simplification and human readability, the data is serialized as JSON, but when the data needs to be processed at a high throughput rate, it may be advantageous to serialize the data with an optimized msgpack library. Finally, a way to construct indexing can be defined using Java object classes.

[0035] When the data is reformatted, the system writes the data into multiple fields of the rows of the key table, and the data input is written in order according to a consecutive IP address range. Additionally, the IPV4, IPV6, and domain name rowKey entries are stored in separate tables. This allows all rowKey tables to be sorted in lexicographical order according to the IP address and domain range. If multiple assets are associated with the same cyber event or data, different rowKey inputs are created for each associated asset. In one example, if the associated cyber assets include IPV4, IPV6, and domain names, rowKey inputs are created for each row of the same cyber event or data. The system reads the rowKey strings of the rowKey table and generates an index for each cyber event or data with a Java object class.

[0036] Generating an index from the rowKey table The first step is to ingest the pDNS records into a distributed dataset, where all pDNS records are stored in a common file type using a common format. The pDNS records are accumulated from multiple data sources into a streaming distributed database, and the database and structured streaming jobs are periodically executed on the database. The structured streaming jobs continuously read and process the pDNS data stream from the database in small batches called micro-batches. Each micro-batch contains multiple records that are grouped according to the first byte of the requested IP address and written to a group file. In one aspect, the group file is stored in a data serialization format such as JSON, XML, Apache avro file, etc. The JSON-based file format may be advantageous due to its ability to easily transmit data objects in key-value pairs and user-friendly human-readable text. Additionally, the size of the group file may be set according to a predetermined size, range, or micro-batch processing time.

[0037] The pDNS records are grouped or sorted according to the first byte of the requested IP address to take advantage of the natural grouping of the requested data. For example, a single IP address tends to request similar resources over time, and thus grouping the pDNS records by the first byte of the requested IP address improves the grouping of other fields such as the qname and answer IP address that will be indexed later. Additionally, grouping the pDNS records according to the first byte of the requested IP address reduces the shuffle data cost and improves the downstream DataFlow indexer job. The grouping scheme also provides a coarse first index across the requested IPs since the file path structure can be structured as follows. service: / / <bucket> / <write-year> / <write-month> / <write-day> / <write-hour> / <first-requesting-ip-byte> / <avro files>

[0038] With this directory structure, the system can query all DNS traffic for a specific group of IP addresses within a specific time range by reading all the data in the corresponding subdirectory.

[0039] The structured streaming job is forced to start micro-batch processing 5 minutes after the previous start time by using a micro-batch trigger. This ensures that the processing is not completely captured up to the latest offset, which causes problems by writing groups of files that are too small.

[0040] The system uses a dataflow indexer (e.g., Google Cloud Dataflow) job to encode the asset identifier (domain or IP address), observation time, and record as index values in a backend database table such as Google Bigtable. This index takes the form of a set of rowKeys that are used by the backend database table to associate the records with queryable fields. In various ways, the rowKey is a string that indicates the exact position of the saved row (data element or record). Further, the rowKey can be used by the database table to sort the rows according to each rowKey. To enable a fixed-time query for a specific record (e.g., cyber asset, cyber event), several copies of the record are stored for each asset associated with the record, and each copy has a rowKey board with a single asset identifier.

[0041] Next, the file names on the distributed storage system are written to the rows of the rowKey board table, and each rowKey board includes an asset identifier field, an index data field (e.g., cyber event data, pDNS records), and an event timestamp field. The asset identifier includes one or more of a list of associated IP addresses and / or a list of domains associated with the data field.

[0042] The database table includes a plurality of rowKeys, and each rowKey includes a plurality of fields within the table. In one aspect, the rowKey string indicates the exact location of the DNS record within the distributed database. The rowKey format may include one of the following formats. <asset_identifier>#<observation_timestamp>#<unique_hash> <asset_identifier>#<observation_timestamp>#<pDNS record>. <asset_identifier>#<observation_timestamp>#< hash_value representative of cyberEvent>.

[0043] asset_identifier: Encoding of either an IP address or a domain. The IP address is encoded as a hexadecimal representation of the IP address bytes, 4 bytes for ipv4 and 16 bytes for ipv6. The domain is encoded as the fully qualified domain name, but in reverse. For example, www.google.com is encoded as com.google.www.

[0044] observation_timestamp: The timestamp of the most accurate timestamp associated with the occurrence of the cyber event, encoded as an ISO 8601 string.

[0045] unique_hash: In one aspect, the uniqueness of a cyber event can be a set of columns that most clearly define an index. A unique hash is generated from a hash algorithm that receives the parameters of the cyber event as input. If either of two recorded events has the same value for these columns, it can be determined that both records correspond to the same event. The unique hash value allows the system to deduplicate multiple occurrences of the same data within a data table. As a result, the system does not return multiple occurrences of the same data.

[0046] When the system writes pDNS data records to a group file, the system reads the group file and performs a single data flow job indexer operation at a predetermined indexing interval. In one aspect, the indexing interval may be performed every hour. The operation of the data flow job indexer indexes all pDNS fields in the group file and stores the index in a rowKey table such as Google BigTable. To read data from the group file, the data flow job indexer operation lists a predetermined number of subdirectories for the first requested IP byte in parallel. The list of subdirectories can include all files, such as 256 subdirectories for Apache Avro files. This results in a substantial increase in the read speed (about 256 times slower) compared to the conventional read scheme of listing all files on a single node. Further, indexing the group file in predetermined indexing interval batches provides a substantial improvement in the indexed record throughput compared to a continuously streaming indexing scheme. In comparison, a continuously streaming indexing scheme runs the risk of processing records at a slower rate than they are received, backing off or doubling on write jobs that distort a database system or a "key-value" store (e.g., BigTable).

[0047] Each data flow indexer job reads grouped file data for one hour and generates an index for each queryable field of the pDNS record, including the metadata about the record, that points to the file containing the pDNS record. Next, the index is grouped by the time of the event timestamp of the pDNS record, and each group of the index is written as a single row in the rowKey table. Grouping the pDNS index into time groups uses a much smaller rowKey table cluster (about 4 times smaller) than would otherwise be needed to store the index of the data.

[0048] In another aspect, a hash value can be generated that represents a particular type of cyber event and does not uniquely identify an instance of the cyber event. In this aspect, cyber events of the same type may correlate with a representative hash value, such as the representative hash values disclosed in U.S. Provisional Patent Application No. 63 / 341,264, entitled METHOD AND SYSTEM FOR SUMMARIZING ANALYTIC OBSERVATIONS, filed May 12, 2022, the disclosure of which is hereby incorporated by reference in its entirety.

[0049] In one aspect, a SIEM platform executed by an MSSP server, such as that disclosed in U.S. Provisional Patent Application No. 63 / 294,570, filed December 29, 2021, titled DEVICES, SYSTEMS, AND METHODS FOR PROVISIONING AND UPDATING SECURITY INFORMATION & EVENT MANAGEMENT ARTIFACTS FOR MULTIPLE TENANTS, the disclosure of which is incorporated herein by reference in its entirety, can implement database tables to enhance network security. For example, cyber event data may be matched against behavior profiles during an indexing process. Based on the behavior profiles, the SIEM platform can identify behavior observations that correlate with a suspicious file among the tenant's network or the tenant's assets. The behavior observation data may be stored as metadata with the cyber event data or may require immediate action. The SIEM platform can consider immediate action necessary if the suspicious file contains malicious content, such as malware, ransomware, or other content indicating a cyber attack. In one aspect, the SIEM platform has administrative privileges to remotely delete suspicious files from one or more storage locations on the tenant's network. In another aspect, the SIEM platform communicates directly with the tenant security administrator who locally manages the tenant's network. When a suspicious file is detected and evaluated, the SIEM platform automatically generates an alert (such as an email, text, phone call, etc.) to the security administrator. The alert may include the identification of malicious content on the suspicious file, the location where the suspicious file is stored, and the measures or interventions necessary to mitigate the threats associated with the suspicious file.

[0050] Processing and Indexing of pDNS Records In one aspect, the data indexing scheme improves the processing speed and table size for processing and indexing a stream of DNS records, as well as the query response time for indexed pDNS records. The system receives a data stream of approximately two million new pDNS records per second and processes approximately 172 billion new pDNS records per day. The pDNS records enable the system to store DNS resolution data that the system uses to reference past DNS record values and identify potential security incidents or malicious infrastructure. DNS records are dynamic, and when a DNS record changes, it becomes difficult to identify and associate the previous value with the domain. Thus, pDNS records can be very useful in providing references to new DNS values. The pDNS records enable system administrators to determine when the DNS records change, the previous DNS values, and the new DNS values. Without pDNS records, it can be difficult to identify previous DNS records of malicious websites and associate those values with current DNS values.

[0051] The pDNS data stream is also useful to a security operations center for identifying patterns and creating predictive analytics models to identify malicious actors or cyberattacks. In various aspects, the pDNS records may be used to identify potentially malicious activities in the footprint of interest, potential security appliances, software vendors used by the company of interest, and traffic of interest for specific use cases, etc.

[0052] Ingest pDNS records and group the pDNS records in a structured streaming job Figure 3 shows a flowchart for reformulating records from a distributed database into an extensible format suitable for indexing. The system reads and reformulates data from a distributed database by starting a first microbatch process at 3002 using a structured streaming job. The system first reads 3004 a predetermined portion of data from the distributed database within the first microbatch. The amount of data within each microbatch may be determined based on a predetermined trigger interval between the first microbatch and subsequent microbatches started thereafter. Based on the start of the first microbatch, the system determines 3006 whether to trigger subsequent microbatches. The system reformulates the original records into an extensible format and reformulates 3008 the plurality of reformulated records into groups 3010 according to the first byte of the requested IP address. The system generates a group file associated with the plurality of reformulated records. In one aspect, the trigger timing between each microbatch correlates with the size and number of records within the group file.

[0053] Operation of the Dataflow Job Indexer Figure 4 shows a flowchart for generating an index indicating the location of the original records. The system starts 4002 a first dataflow indexer job based on a predetermined indexing interval. The system reads 4004 one or more group files based on a predetermined indexing interval. The system lists 4006 a predetermined number of subdirectories of the group file so that the system can read the subdirectories in parallel. The system generates 4008 an index for all records associated with a predetermined indexing interval range based on the file's index timestamp. The group index is written to a single row of the rowKey table that enables the front-end computing resources to query the table and retrieve the original records from the back-end server in a fixed time.

[0054] Database Basing Index Table For performance purposes, the index table is a separate table within a partitioned database architecture and can be optimized according to specific pDNS data sets. In various embodiments, each row of the index table includes an index for a predetermined number of pDNS records, such as up to 1000 records. The index can be a map-like object that includes the following structure. { "Group Filename” -> { BlockNumber of pDNS record-> [record metadata]}}

[0055] The group file block number is stored in addition to the group file name, so that at read time, the query system can directly request the block containing the record of interest, thus improving read performance.

[0056] Furthermore, the group file is described such that only one pDNS record per block exists, which means that each BlockNumber of the index uniquely identifies a single pDNS record. The record metadata includes additional fields about the pDNS record, such as whether the record was a DNS request or response, and can be used to filter out unwanted records.

[0057] To query the database, the first step is to query the backend database server using a set of predetermined parameters such as the IP address or domain of interest and the time range. The database returns an index to identify the files and offsets to those files that contain records matching the IP address or domain. The system may obtain the records by reading the files from the database of grouped files and scanning them into the corresponding records. In various aspects, the pNDS records may be queried according to the requesting IP address, the responding IP address, the queried domain name (qname), and the queried subdomain, and the processed pDNS records are mapped to the domain over a predetermined period such as several months.

[0058] Indexing layer that graphs IP connections and domain connections in the context of pDNS records In one aspect, the indexing database utilizes only the connections between IP addresses and domains within the DNS, rather than the full content of the DNS records. Thus, this aspect may enable an improvement in read performance by creating another index on the data. The new index represents a graph of the connections between IP addresses and domains within the data. The rowKey of this scheme may be stored in the following format. <requesting IP address> # <event timestamp bin> # <requested qname> -> Metadata regarding the request

[0059] The IP graph index provides a faster query for all domains requested by an IP within a specific time range, or finds all instances where an IP address has requested a domain over a specific window. The rowKey representing the connection between the domain and the IP address may be stored as follows. <requested qname> # <event timestamp bin> # <requesting IP> -> Metadata regarding the request

[0060] Similarly, the domain graph index provides an efficient query for all IP addresses requested for a particular domain (or any particular subdomain) within a given period. This indexing scheme utilizes the data indexing scheme disclosed above, but involves the variation of storing data in different rowKey tables.

[0061] FIG. 5 shows a diagram of a computing system 5000 according to at least one non-limiting aspect of the present disclosure. The computing system 5000 and various components included therein can be used to implement and / or execute any of the various components of the systems and methods 2000, 3000, and 4000 described above in connection with FIGS. 2-4, as described below.

[0062] According to a non-limiting aspect of FIG. 5, the computer system 5000 may include a bus 5002 (i.e., an interconnect), one or more processors 5004, main memory 5006, read-only memory 5008, removable storage media 5010, mass storage device 5012, and one or more communication ports 5014. As should be understood, components such as removable storage media are optional and not required in all systems. The communication port 5014 can be connected to one or more networks through which the computer system 5000 can receive and / or transmit data.

[0063] As used herein, a processor can mean one or more microprocessors, central processing units (CPUs), computing devices, microcontrollers, digital signal processors, or similar devices, or any combination thereof, regardless of architecture. The device for implementing the process may include, for example, a processor and those devices such as input devices and output devices appropriate for implementing the process.

[0064] The processor 5004 can be any known processor, such as, but not limited to, a processor manufactured and / or sold by INTEL®, AMD®, MOTOROLA®, etc., which is generally well-known to those skilled in the art and clearly defined in the literature. The communication port 5014 can be any of, for example, an RS-232 port for use in a modem-based dial-up connection, a 10 / 100 Ethernet port, a gigabit port using copper or fiber, or a USB port. The communication port 5014 can be selected according to the network, such as a local area network (LAN), a wide area network (WAN), a CDN, or any network to which the computer system 5000 is connected. The computer system 5000 may communicate with peripheral devices (e.g., a display screen 5016, an input device 5018) via an input / output (I / O) port 5020.

[0065] The main memory 5006 can be a random access memory (RAM) or any other dynamic storage device commonly known in the art. The read-only memory 5008 can be any static storage device such as a programmable read-only memory (PROM) chip for storing static information such as the instructions of the processor 5004. The mass storage device 5012 can be used to store information and instructions. For example, a hard disk such as the Adaptec® family of small computer serial interface (SCSI) drives, an optical disk, an array of disks such as a redundant array of independent disks (RAID) such as the Adaptec® family of RAID drives, or any other mass storage device can be used.

[0066] The bus 5002 communicatively couples the processor 5004 to other memory, storage, and communication blocks. The bus 5002 can be, for example, a PCI / PCI-X, SCSI, or a Universal Serial Bus (USB)-based system bus (or others) depending on the storage devices used. The removable storage medium 5010 can be any kind of external hard drive, floppy drive, IOMEGA® Zip drive, compact disk read-only memory (CD-ROM), compact disk rewritable (CD-RW), digital versatile disk read-only memory (DVD-ROM), and the like.

[0067] Aspects described herein may be provided as one or more computer program products, which may include a machine-readable medium having instructions stored thereon, which may be used to program a computer (or other electronic device) to perform a process. As used herein, the term "machine-readable medium" refers to any medium, plural media, or combination of different media involved in providing data (e.g., instructions, data structures) that can be read by a computer, processor, or similar device. Such media can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical or magnetic disks and other persistent memories. Volatile media includes dynamic random access memory, which typically constitutes a computer's main memory. Transmission media includes coaxial cables, copper wire, and fiber optics, including the wires that make up a system bus coupled to a processor. Transmission media can include, or convey, acoustic waves, light waves, and electromagnetic radiation, such as those generated during radio frequency (RF) and infrared (IR) data communications.

[0068] Machine-readable media can include, but are not limited to, floppy disks, optical disks, CD-ROMs, magneto-optical disks, ROMs, RAMs, erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or other types of media / machine-readable media suitable for storing electronic instructions. Further, aspects described herein may also be downloaded as a computer program product, where the program may be transferred from a remote computer to a requesting computer by a data signal embodied in a carrier wave or other propagation medium via a communication link (e.g., a modem or network connection).

[0069] Various forms of computer-readable media may be involved in carrying data (e.g., a sequence of instructions) to a processor. For example, the data may (i) be delivered from RAM to the processor, (ii) be carried on a wireless transmission medium, (iii) be formatted and / or transmitted according to a number of formats, standards, or protocols, and / or (iv) be encrypted in any of a variety of ways well known in the art.

[0070] A computer-readable medium can store program elements (in any suitable format) appropriate for implementing a method.

[0071] As shown, main memory 5006 is encoded with an application 5022 that supports the functions discussed herein (the application 5022 may be an application that provides some or all of the functions of the CD service described herein, including a client application). The application 5022 (and / or other resources described herein) may be embodied as software code, such as data and / or logical instructions (e.g., code stored in memory or on another computer-readable medium such as a disk) that support processing functions in accordance with the different aspects described herein.

[0072] During operation of one aspect, the processor 5004 accesses the main memory 5006 via the use of the bus 5002 to initiate, execute, run, interpret, or otherwise execute the logical instructions of the application 5022. Execution of the application 5022 generates the processing functions of the services associated with the application. In other words, the process 5024 represents one or more portions of the application 5022 that are executed within or on the processor 5004 within the computer system 5000.

[0073] In addition to process 5024 that carries out the operations discussed herein, note that other aspects described herein include application 5022 itself (i.e., unexecuted or non-executing logical instructions and / or data). Application 5022 can be stored on a computer-readable medium such as a disk (e.g., a repository), or within an optical medium. According to other aspects, application 5022 can also be stored in a memory-type system such as firmware, read-only memory (ROM), or, as in this example, executable code within main memory 5006 (e.g., within random access memory or RAM). For example, application 5022 can also be stored on removable storage medium 5010, read-only memory 5008, and / or mass storage device 5012.

[0074] One of ordinary skill in the art will understand that computer system 5000 can include other processes and / or software and hardware components, such as an operating system that controls the allocation and use of hardware resources.

[0075] The various aspects of the subject matter described herein are set forth in the numbered clauses below.

[0076] Clause 1: A method for indexing pDNS records in an extensible split data table that enables queries for a certain period of time, comprising: receiving, by a processor, a flow of pDNS records from one or more data sources; aggregating, by the processor, the flow of pDNS records into a distributed database; executing, by the processor, a first structured streaming job on the distributed database, wherein the first structured streaming job processes a first micro-batch of pDNS records, reformats the records into an extensible format, groups the records according to the first byte of the requested IP address, writes the pDNS records to a first set of group files among a plurality of group files, and a plurality of subsequent structured streaming jobs are continuously triggered and started within a predetermined period from a previous structured streaming job; executing, by the processor, an indexing job on a plurality of group files, wherein the indexing job reads a first set of group files among the plurality of group files, generates an index for all queryable fields of the pDNS records, and a new indexing job is executed according to a predetermined indexing job interval; writing, by the processor, an index group to the first row of a rowKey table, wherein the index group is grouped according to the event timestamp of the indexed pDNS records indexed during a predetermined indexing job interval; querying, by the processor, the rowKey table by a set of queryable parameters of the pDNS records, wherein the query result is returned within a certain period of time.

[0077] Clause 2: The method according to Clause 1, wherein the group file includes an Apache Avro file.

[0078] Clause 3: The method according to either Clause 1 or 2, where a predetermined period for triggering the following structured streaming job is within 5 minutes from the previous structured streaming job.

[0079] Clause 4: The method according to any one of Clauses 1 to 3, where a predetermined indexing job interval is 1 hour.

[0080] Clause 5: The method according to any one of Clauses 1 to 4, where the read operation of the indexing job further includes listing all file directories in the group file sub - directory according to the first byte of the requested IP address.

[0081] Clause 6: The method according to any one of Clauses 1 to 6, where the indexed fields of the pDNS record include the fields of the IP address request, the response to the IP address, and the queried domain name (qname).

[0082] Clause 7: The method according to any one of Clauses 1 to 6, where a set of queryable parameters for the pDNS record includes an asset identifier and a timestamp, the asset identifier includes the requested IP address, the response IP address, the queried domain name (qname), and the queried sub - domain, and the timestamp includes a specific time or a timestamp range.

[0083] Clause 8: The method according to any one of Clauses 1 to 7, where each pDNS record corresponds to the block number and file name of the group file that identifies the location of the pDNS record within the distributed dataset.

[0084] Clause 9: The method according to any one of Clauses 1 to 8, where the matching query of the index group points to the file name and block number of the matching pDNS record.

[0085] Clause 10: The method according to any one of Clauses 1 to 9, wherein the first structured streaming job and the indexing job are defined as higher-order Java classes.

[0086] Clause 11: A system for indexing records returned by a query for a certain period of time for an extensible database, comprising at least one processor, at least one memory communicatively coupled to the at least one processor, and an input / output interface configured to access data from one or more external sources, wherein each of the plurality of external sources is communicatively coupled to the at least one processor, an input / output interface, and a database present in at least one memory and configured to store data. The at least one memory is configured to receive, by the at least one processor, a flow of pDNS records from one or more data sources, aggregate the flow of pDNS records into a distributed database, and execute a first structured streaming job on the distributed database. The first structured streaming job processes a first micro-batch of pDNS records, reformats the records into an extensible format, groups the records according to the first byte of the requested IP address, writes the pDNS records to a first set of group files among a plurality of group files, and a plurality of subsequent structured streaming jobs are continuously triggered and started within a predetermined period from a previous structured streaming job. Execute an indexing job on a plurality of group files, wherein the indexing job reads a set of group files of the first set among the plurality of group files, generates an index for all queryable fields of the pDNS records, and a new indexing job is executed according to a predetermined indexing job interval. Writing the index group to the first row of the rowKey table, wherein the index group is grouped according to the event timestamp of the indexed pDNS records indexed during a predetermined indexing job interval. Querying the rowKey table by a set of queryable parameters of the pDNS records by the processor, wherein the query result is returned within a certain time.A system configured to store instructions to execute.,

[0087] Clause 12: The system according to Clause 11, wherein the group file includes an Apache Avro file.,

[0088] Clause 13: The system according to Clause 11 or 12, wherein a predetermined period for triggering the following structured streaming job is within 5 minutes from the previous structured streaming job.,

[0089] Clause 14: The system according to any one of Clauses 11 to 13, wherein a predetermined indexing job interval is 1 hour.,

[0090] Clause 15: The system according to any one of Clauses 11 to 14, wherein the reading operation of the indexing job further includes listing all file directories in the group file sub - director according to the first byte of the required IP address.,

[0091] Clause 16: The system according to any one of Clauses 11 to 15, wherein the indexed fields of the pDNS record include fields of the IP address request, the response to the IP address, and the queried domain name (qname).,

[0092] Clause 17: The system according to any one of Clauses 11 to 16, wherein a set of queryable parameters for the pDNS record includes an asset identifier and a timestamp, the asset identifier includes the requested IP address, the response IP address, the queried domain name (qname), and the queried sub - domain, and the timestamp includes a specific time or a timestamp range.,

[0093] Clause 18: The system according to any one of Clauses 11 to 17, wherein each pDNS record corresponds to the block number and file name of the group file that identifies the location of the pDNS record in the distributed dataset.,

[0094] Clause 19: A system according to any of Clauses 11 - 18, wherein a matching query of an index group points to the file name and block number of the matching pDNS record.

[0095] Clause 20: A system according to any of Clauses 11 - 19, wherein a first structured streaming job and an indexing job are defined as higher-order Java classes.

[0096] All patents, patent applications, publications, or other disclosure materials described in this specification are hereby incorporated by reference in their entirety as if each individual reference were explicitly incorporated by reference. All references said to be incorporated by reference in this specification, and any materials, or portions thereof, are incorporated in this specification only to the extent that the incorporated material does not conflict with the existing definitions, descriptions, or other disclosure materials described in this disclosure. Therefore, and to the extent necessary, the disclosure set forth in this specification prevails over any conflicting materials incorporated by reference in this specification, and the disclosure is set forth explicitly within the context of this application.

[0097] Various illustrative and exemplary aspects are described. The aspects described herein are to be understood as providing illustrative features of various details of various aspects of the present disclosure, and thus, unless otherwise specified, without departing from the scope of the present disclosure, as a matter of course, one or more features, elements, components, ingredients, materials, structures, modules, and / or aspects of the aspects of the present disclosure may be combined, separated, exchanged, and / or rearranged with one or more other features, elements, components, ingredients, materials, structures, modules, and / or aspects of the aspects of the present disclosure. Accordingly, those skilled in the art will recognize that various substitutions, modifications, or combinations of any of the illustrative aspects can be made without departing from the claimed subject matter. Further, those skilled in the art can recognize or confirm many equivalents to the various aspects of the present disclosure by reexamining this specification and using only routine experimentation. Accordingly, the present disclosure is not limited by the description of the various aspects, but only by the claims.

[0098] Those skilled in the art will generally recognize that terms used herein, and particularly in the appended claims (e.g., the body of the appended claims), are generally intended to be terms without limitation (e.g., the term "including" should be construed as "including but not limited to", the term "having" should be construed as "having at least", the term "includes" should be construed as "including but not limited to", etc.). It will be further understood by those skilled in the art that where a specific number of introduced claim limitations is intended, such intent is explicitly recited in the claims, and where there is no such recitation, such intent does not exist. For example, by way of illustration, the following appended claims may include the use of the introductory phrases "at least one" and "one or more" to introduce claim limitations. However, the use of such phrases should not be construed as implying that the introduction of a claim limitation by the indefinite article "a" or "an" limits any particular claim that includes such introduced claim limitation to a claim scope that includes only one such limitation, and the same applies to the use of the definite article used to introduce a claim limitation even where the same claim includes an introductory phrase such as "one or more" or "at least one", and an indefinite article such as "a" or "an" (e.g., "a" and / or "an" should generally be construed as meaning "at least one" or "one or more").

[0099] Furthermore, even if a specific number of the recited introduced claims is explicitly recited, one of ordinary skill in the art will recognize that such a recitation should typically be interpreted to mean at least the recited number (e.g., a mere recitation of "two recitations" will typically be understood to mean at least two recitations or two or more recitations without other qualifying language). Further, in these instances where conventions similar to "at least one of A, B, and C, etc." are used, generally, such a construction is intended in the sense that one of ordinary skill in the art will understand the convention (e.g., "a system having at least one of A, B, and C" includes, but is not limited to, a system having A alone, B alone, C alone, a system having A and B together, a system having A and C together, a system having B and C together, and / or a system having A, B, and C together, etc.). In instances where conventions similar to "at least one of A, B, or C, etc." are used, generally, such a construction is intended in the sense that one of ordinary skill in the art will understand the convention (e.g., "a system having at least one of A, B, or C" includes, but is not limited to, a system having A alone, B alone, C alone, a system having A and B together, a system having A and C together, a system having B and C together, and / or a system having A, B, and C together, etc.). In any of the description, claims, or drawings, disjunctive and / or phrases presenting two or more alternative terms will generally be understood by those of skill in the art to contemplate the possibility of including one of the terms, any of the terms, or both terms, unless the context indicates otherwise. For example, the phrase "A or B" will typically be understood to include the possibility of "A" or "B" or "A and B".

[0100] Regarding the appended claims, those skilled in the art will understand that the operations listed therein may generally be performed in any order. Also, although the claims are presented in sequential order, it should be understood that the various operations may be performed in other orders than those described, or simultaneously. Examples of such alternative orders include, unless the context otherwise indicates, repetition, interleaving, interruption, reordering, incrementing, preparation, supplementation, simultaneous, reverse, or other variant orders. Further, unless the context otherwise indicates, terms such as "responding", "relating", or other past participles generally do not intend to exclude such variants.

[0101] It should be noted that any reference to "one aspect", "aspect", "exemplification", "an exemplification", and the like means that the particular features, structures, or characteristics described in relation to the aspect are included in at least one aspect. Thus, the appearances of the phrases "in one aspect", "in an aspect", "in an exemplification", and "in an exemplification" at various places throughout this specification do not necessarily all refer to the same aspect. Further, the particular features, structures, or characteristics may be combined in any suitable manner in one or more aspects.

[0102] As used herein, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" include plural references.

[0103] For example, without limitation, directional terms used herein such as up, down, left, right, below, above, front, back, and variations thereof relate to the orientation of the elements shown in the accompanying drawings and, unless otherwise explicitly stated, are not limiting with respect to the claims.

[0104] As used herein, the term "about" or "approximately" means, unless otherwise specified, an acceptable error for a particular value as determined by one of ordinary skill in the art, which depends in part on how the value is measured or determined. In certain embodiments, the term "about" or "approximately" means within 1, 2, 3, or 4 standard deviations. In certain embodiments, the term "about" or "approximately" means within 50%, 200%, 105%, 100%, 9%, 8%, 7%, 6%, 5%, 4%, 3%, 2%, 1%, 0.5%, or 0.5% of a given value or range.

[0105] In this specification, unless otherwise indicated, all numerical parameters are to be understood as being preceded by the term "about", which in all cases reflects the inherent variability of the underlying measurement techniques used to determine the numerical value of the parameter. At a minimum, and not as an attempt to limit the application of the doctrine of equivalents to the claims, each numerical parameter herein is to be construed in light of the reported significant digits and by applying ordinary rounding techniques.

[0106] Any numerical range recited herein includes all sub-ranges subsumed within the recited range. For example, a range of "1 to 100" includes all sub-ranges between (and including) the recited minimum value of 1 and the recited maximum value of 100, i.e., all sub-ranges having a minimum value of 1 or more and a maximum value of 100 or less. Also, all ranges recited herein include the endpoints of the recited range. For example, a range of "1 to 100" includes the endpoints 1 and 100. Any maximum numerical limitation recited herein is intended to include all lower numerical limitations subsumed therein, and any minimum numerical limitation recited herein is intended to include all higher numerical limitations subsumed therein. Accordingly, Applicants reserve the right to amend this specification, including the claims, to expressly recite sub-ranges subsumed within the explicitly recited ranges. All such ranges are inherently described herein.

[0107] Any patent application, patent, non-patent publication, or other disclosure material referred to herein and / or listed in any application data sheet is hereby incorporated by reference into this specification, provided that the incorporated material is not inconsistent with this specification. Accordingly, and to the extent necessary, the present disclosure as expressly set forth herein prevails over any conflicting material incorporated by reference into this specification. Although said to be incorporated by reference into this specification, any material or portion thereof that conflicts with an existing definition, statement, or other disclosure material set forth herein is incorporated only to the extent that no conflict arises between the incorporated material and the existing disclosure material.

[0108] The terms “comprise” (and any form of comprise such as “comprises” and “comprising”), “have” (and any form of have such as “has” and “having”), “include” (and any form of include such as “includes” and “including”), and “contain” (and any form of contain such as “contains” and “containing”) are open-ended conjunctive verbs. As a result, a system that “comprises,” “has,” “includes,” or “contains” one or more elements possesses those one or more elements, but is not limited to possessing only those one or more elements. Similarly, an element of a system, device, or apparatus that “comprises,” “has,” “includes,” or “contains” one or more features possesses those one or more features, but is not limited to possessing only those one or more features.

[0109] The foregoing detailed description has described various forms of devices and / or processes by use of block diagrams, flowcharts, and / or examples. Where such block diagrams, flowcharts, and / or examples include one or more functions and / or operations, those skilled in the art will understand that each function and / or operation within such block diagrams, flowcharts, and / or examples can be implemented individually and / or collectively by a wide variety of hardware, software, firmware, or substantially any combination thereof. Those skilled in the art will recognize that some aspects of the forms disclosed herein can be implemented as one or more computer programs operating on one or more computers (e.g., as one or more programs operating on one or more computer systems), as one or more programs operating on one or more processors (e.g., as one or more programs operating on one or more microprocessors), as firmware, or substantially any combination thereof, and that all or part thereof can be equivalently integrated into an integrated circuit, and that the design of the circuitry, and / or the description of the software code, and also the firmware, are within the skill of those in the art in light of the present disclosure. Further, those skilled in the art will understand that the mechanisms of the subject matter described herein can be distributed in various forms as one or more program products, and that the exemplary forms of the subject matter described herein apply regardless of the particular type of signal-carrying medium used to actually carry out the distribution.

[0110] The instructions used to program the logic to implement the various disclosed aspects may be stored in memory within the system, such as dynamic random access memory (DRAM), cache, flash memory, or other storage devices. Further, the instructions may be distributed via a network or via other computer-readable media. Thus, a machine-readable medium is any mechanism, but not limited to, for storing or transmitting information in a form readable by a machine (e.g., a computer), such as floppy disks, optical disks, compact disks, read-only memory (CD-ROM), and magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic or optical cards, flash memory, or tangible machine-readable storage devices used to transmit information on the Internet via electrical, optical, acoustic, or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.). Thus, a non-transitory computer-readable medium includes any type of tangible machine-readable medium suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).

[0111] When used in any aspect of this specification, the term "control circuit" can refer to, for example, a wired circuit, a programmable circuit (e.g., a computer processor having one or more individual instruction processing cores, a processing unit, a processor, a microcontroller, a microcontroller unit, a controller, a digital signal processor (DSP), a programmable logic device (PLD), a programmable logic array (PLA), or a field programmable gate array (FPGA)), a state machine circuit, firmware storing instructions executed by a programmable circuit, and any combination thereof. The control circuit can be embodied, collectively or individually, as part of a larger system, such as an integrated circuit (IC), an application specific integrated circuit (ASIC), a system on chip (SoC), a desktop computer, a laptop computer, a tablet computer, a server, a smartphone, etc. Thus, as used herein, "control circuit" includes, but is not limited to, an electrical circuit having at least one discrete electrical circuit, an electrical circuit having at least one integrated circuit, an electrical circuit having at least one application specific integrated circuit, an electrical circuit forming a general purpose computing device configured by a computer program (e.g., a general purpose computer configured by a computer program that at least partially executes a process, and / or a device described herein, or a microprocessor configured by a computer program that at least partially executes a process, and / or a device described herein), an electrical circuit forming a memory device (e.g., in the form of a random access memory), and / or an electrical circuit forming a communication device (e.g., a modem, a communication switch, or an optoelectronic device). One of ordinary skill in the art will recognize that the subject matter described herein can be implemented in analog or digital fashion or some combination thereof.

[0112] When used in any aspect of this specification, the term "logic" can refer to an application, software, firmware, and / or circuitry configured to perform any of the foregoing operations. Software can be embodied as a software package, code, instructions, instruction sets, and / or data recorded on a non-transitory computer-readable storage medium. Firmware can be embodied as code, instructions, or instruction sets, and / or data hard-coded (e.g., non-volatile) in a memory device.

[0113] When used in any aspect of this specification, terms such as "component", "system", "module", etc. can refer to a computer-related entity, hardware, a combination of hardware and software, software, or software in execution.

[0114] When used in any aspect of this specification, "algorithm" refers to a self-consistent order of steps that produces a desired result, and "step" refers to an operation on a physical quantity and / or a logical state that can, although not necessarily, take the form of an electrical or magnetic signal capable of being stored, moved, combined, compared, and otherwise manipulated. These signals are commonly referred to as bits, values, elements, symbols, characters, terms, numbers, etc. These and similar terms may be associated with appropriate physical quantities and are merely convenient labels applied to these quantities and / or states. < / write-hour> < / write-day> < / write-month> < / write-year> < / bucket>

Claims

1. A method for indexing protective Domain Name System (pDNS) records in an extensible partitioned data table that allows queries for a set period of time, The process involves a processor receiving a data stream from one or more data sources, wherein the data stream includes a pDNS record. The processor aggregates the pDNS records into a distributed database, The processor executes a first structured streaming job on the distributed database, The processor performs indexing jobs on multiple group files, wherein the indexing jobs read a first set of group files from the multiple group files, generate a first group index for all queryable fields of the pDNS record, and subsequent indexing jobs are performed according to a predetermined indexing job interval. The processor writes an index group to the first row of the rowKey table, wherein the index group is grouped according to the event timestamps of the indexed pDNS records indexed during the predetermined indexing job interval. The processor queries the rowKey table according to the set of queryable fields of the pDNS record, The aforementioned processor returns query results in a fixed amount of time, regardless of the total number of rows in the rowKey table. Includes, The first structured streaming job described above is The processor processes the first microbatch of the pDNS records, The processor reformats the pDNS record into an expandable format pDNS record, The processor groups the extensible format pDNS records according to the first byte of the requested IP address, The processor writes the expandable format pDNS record to the first set of group files among the plurality of group files, such that a plurality of subsequent structured streaming jobs are successively triggered and started within a predetermined period from the previous structured streaming job. Methods that include...

2. The method according to claim 1, wherein the plurality of group files include Apache Avro files.

3. The method according to claim 1, wherein the predetermined period for triggering the plurality of subsequent structured streaming jobs is within 5 minutes from the previous structured streaming job.

4. The method according to claim 1, wherein the predetermined indexing job interval is 1 hour.

5. The method according to claim 1, further comprising listing all file directories in a group file subdirector according to the first byte of the requested IP address.

6. The method according to claim 1, wherein the first group index of the pDNS record includes a requesting IP address, a response IP address, and a queried domain name (qname).

7. The method according to claim 1, wherein the set of queryable fields for the pDNS record includes an asset identifier and a timestamp, the asset identifier includes a requesting IP address, a response IP address, a queried domain name (qname), and a queried subdomain, and the timestamp includes a specific time or timestamp range.

8. The method according to claim 1, wherein each of the pDNS records corresponds to a block number and file name of the plurality of group files that identify the location of the pDNS record in the distributed database.

9. The method according to claim 8, wherein the query result includes a matching query for the index group pointing to the block number and file name of the pDNS record in the distributed database.

10. The method according to claim 1, wherein the first structured streaming job and the indexing job are defined as higher-order Java classes.

11. The method according to claim 10, wherein the first structured streaming job and the indexing job are defined as higher-order Java classes.

12. A system for indexing records returned by queries over a certain period of time in an extensible database, At least one processor, At least one memory connected to the at least one processor, An input / output interface configured to access data from one or more external sources, wherein each of the one or more external sources is communicably coupled to at least one processor; A database located in at least one memory and configured to store the data, Equipped with, The at least one memory is controlled by the at least one processor. Receiving a data stream from one or more external sources, wherein the data stream includes protective Domain Name System (pDNS) records. The aforementioned pDNS records are aggregated into a distributed database, The first structured streaming job is executed on the distributed database, wherein the first structured streaming job is configured to process a first microbatch of pDNS records, reformat the records into an extensible format, group the records according to the first byte of a requested IP address, and write the pDNS records to a first set of group files among a plurality of group files, and a plurality of subsequent structured streaming jobs are executed sequentially within a predetermined period from the previous structured streaming job. The process involves executing an indexing job on the aforementioned group files, wherein the indexing job reads the first set of group files among the aforementioned group files, generates a first group index for all queryable fields of the pDNS record, and subsequent indexing jobs are executed according to a predetermined indexing job interval. The process involves writing an index group to the first row of the rowKey table, wherein the index group is grouped according to the event timestamps of the indexed pDNS records indexed during the predetermined indexing job interval. Querying the rowKey table according to the set of queryable fields in the pDNS record, Regardless of the total number of rows in the rowKey table, the query results will be returned within a fixed time. A system further configured to store executable instructions.

13. The system according to claim 12, wherein the plurality of group files include Apache Avro files.

14. The system according to claim 12, wherein the predetermined period for triggering the plurality of subsequent structured streaming jobs is within 5 minutes from the previous structured streaming job.

15. The system according to claim 12, wherein the predetermined indexing job interval is 1 hour.

16. The system according to claim 12, further comprising the indexing job listing all file directories in a group file subdirector according to the first byte of the requested IP address.

17. The system according to claim 12, wherein the index group of the pDNS record includes a requesting IP address, a response IP address, and a queried domain name (qname).

18. The system according to claim 12, wherein the set of queryable fields for the pDNS record includes an asset identifier and a timestamp, the asset identifier includes a requesting IP address, a response IP address, a queried domain name (qname), and a queried subdomain, and the timestamp includes a specific time or timestamp range.

19. The system according to claim 12, wherein the query result includes a matching query for the index group that points to the block number and file name of the pDNS record in the distributed database.

20. The system according to claim 19, wherein the query result includes a matching query associated with the block number and file name of the pDNS record in the distributed database.

Images